April 2017 - mice-discuss - lists.micemn.net

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: > > On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >> I would suggest that perhaps we look into filtering BGP (tcp/179) >> with an ACL prior to maintenance start on those specific ports being >> moved. Many other IXs are doing this for maintenance as a way to >> gracefully take things down, and let bilateral and RS sessions time >> out without killing active traffic. As we've noticed, not all >> members being moved are bothering to shut down sessions prior, which >> causes impact to/from those members. (i.e.: >> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) > > Don't even need ACLs. > > Just take down the route servers for the 2 hour period. > > Bilateral are unaffected and they can arrange things anyway with their > peers. I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > Adding another step to the process creates more complications as well, > and another point of failure if you screw up along the way. Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > Clean shutdown of bird is easier, quicker, and will for sure make the > multilateral peering not be further affected by bouncing repeatedly. Yes, great for MLPA, but not for bilateral. Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole.

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

> Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. It is worth pointing out that while I am on site doing the physical work, I interact with Doug, Anthony, Jay, and Richard to make sure things go smoothly, since each of them has their area of expertise. Jeremy

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: > I would suggest that perhaps we look into filtering BGP (tcp/179) > with an ACL prior to maintenance start on those specific ports being > moved. Many other IXs are doing this for maintenance as a way to > gracefully take things down, and let bilateral and RS sessions time > out without killing active traffic. As we've noticed, not all > members being moved are bothering to shut down sessions prior, which > causes impact to/from those members. (i.e.: > https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) Don't even need ACLs. Just take down the route servers for the 2 hour period. Bilateral are unaffected and they can arrange things anyway with their peers. Adding another step to the process creates more complications as well, and another point of failure if you screw up along the way. Clean shutdown of bird is easier, quicker, and will for sure make the multilateral peering not be further affected by bouncing repeatedly. > As an aside, thanks Jeremy for doing all this move work, it's not a > small effort and definitely appreciated. +1 -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years > this is just crazy? Nobody does maintenance during the day? Which industry? Because the Internet industry isn't 35 years old. Many people do maintenance during the day. So many. It's a 24x7 world now and for many businesses the time of day for work doesn't necessarily follow the sun. > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? Are you direct connected or are you via remote switch? If you are via remote switch then did your switch operator notify you? > Is MICE issuing credits for outages?? Planned don't count. And no SLA means .. no 'credit'. If you are via remote switch then you should take it up with them and not the community as a whole. But beyond that snark: this is volunteer work and as such Jeremy is already taking time out of his business day to do things for this group. Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, I surely did not mean to cause any feelings of un-appreciation for what you are doing. Just the timing of it and now I do understand the reasoning. Thanks and appreciate your time and efforts. John From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dean Bahls Sent: Wednesday, April 26, 2017 8:39 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com<mailto:hoyosa@gmail.com>> wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 26, 2017

Re: Arista Round 3 Moves

by Dean Bahls

Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

April 26, 2017

Re: Arista Round 3 Moves

by Colin Baker

On Wed, Apr 26, 2017 at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years this > is just crazy? Nobody does maintenance during the day? > > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? > > Is MICE issuing credits for outages?? I'll add that most MICE members are eyeballs whose peak usage occurs at night. Daytime is probably preferable for these networks. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net

April 26, 2017

Re: Arista Round 3 Moves

by Mike Johnston

On 2017-04-26 08:28, Ben Wiechman wrote: > > On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com > <mailto:hoyosa@gmail.com>> wrote: > > As an aside, thanks Jeremy for doing all this move work, it's not > a small effort and definitely appreciated. > > > I concur. This is greatly appreciated by us as well. Thank you. > I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

I do agree that filtering BGP prior to the move is a great thing to do. I know we attempted it in the past, and we had some issues getting it to work properly. Now that we are on the Arista, I have tested it and it seems to work well. I plan on utilizing it on all future maintenance on the Arista switch. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Wednesday, April 26, 2017 8:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Ben Wiechman

On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > As an aside, thanks Jeremy for doing all this move work, it's not a small > effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com

April 26, 2017