- mice-discuss - lists.micemn.net

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: MICE Graphs

by Richard Laager

On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard

April 26, 2017

MICE Graphs

by David Farmer

According to the graphs on the web page there was a network holocaust on MICE yesterday. Could someone please fix the graphs. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 26, 2017

Re: Arista Round 3 Moves

by Richard Laager

Doug, Please do go ahead and expand passive BFD. Everyone on one route server followed by the second route server a few days later should be fine. -- Richard > On Apr 26, 2017, at 10:51, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I think there is a lot more bilateral BGP going on across MICE than you think. > While I probably go talk to anybody, I have 11 unilateral peers on MICE besides > the two route servers. > > I think BFD for BGP would do a lot more good than BGP filtering if people would > implement it. > > Speaking of that, I have passive BFD testing on route server #1 for quite some time. > I've heard of zero problems. > > I would like to go more widespread than the test people (ie. offer it > to every member connection on route server #1) > > Since it is passive, the member end drives it, if they do it, the > route server will do it, if they don't, nothing else happens. I'd also like > to roll out passive BFD as default for new members. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > Bilateral are unaffected and they can arrange things anyway with their > peers. I think there is a lot more bilateral BGP going on across MICE than you think. While I probably go talk to anybody, I have 11 unilateral peers on MICE besides the two route servers. I think BFD for BGP would do a lot more good than BGP filtering if people would implement it. Speaking of that, I have passive BFD testing on route server #1 for quite some time. I've heard of zero problems. I would like to go more widespread than the test people (ie. offer it to every member connection on route server #1) Since it is passive, the member end drives it, if they do it, the route server will do it, if they don't, nothing else happens. I'd also like to roll out passive BFD as default for new members. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Wed, Apr 26, 2017 at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. Sadly, I can't thumbs-up this post as I can on FB... -Tk

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

I’ll also add, this is currently a draft with the IETF grow WG: https://tools.ietf.org/html/draft-ietf-grow-bgp-session-culling-01 (and one which I fully support) -- Andrew Hoyos hoyosa(a)gmail.com > On Apr 26, 2017, at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: >> >> On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >>> I would suggest that perhaps we look into filtering BGP (tcp/179) >>> with an ACL prior to maintenance start on those specific ports being >>> moved. Many other IXs are doing this for maintenance as a way to >>> gracefully take things down, and let bilateral and RS sessions time >>> out without killing active traffic. As we've noticed, not all >>> members being moved are bothering to shut down sessions prior, which >>> causes impact to/from those members. (i.e.: >>> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) >> >> Don't even need ACLs. >> >> Just take down the route servers for the 2 hour period. >> >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. > > Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. > I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > >> Adding another step to the process creates more complications as well, >> and another point of failure if you screw up along the way. > > Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. > Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. > > >

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: > > On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >> I would suggest that perhaps we look into filtering BGP (tcp/179) >> with an ACL prior to maintenance start on those specific ports being >> moved. Many other IXs are doing this for maintenance as a way to >> gracefully take things down, and let bilateral and RS sessions time >> out without killing active traffic. As we've noticed, not all >> members being moved are bothering to shut down sessions prior, which >> causes impact to/from those members. (i.e.: >> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) > > Don't even need ACLs. > > Just take down the route servers for the 2 hour period. > > Bilateral are unaffected and they can arrange things anyway with their > peers. I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > Adding another step to the process creates more complications as well, > and another point of failure if you screw up along the way. Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > Clean shutdown of bird is easier, quicker, and will for sure make the > multilateral peering not be further affected by bouncing repeatedly. Yes, great for MLPA, but not for bilateral. Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole.

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

> Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. It is worth pointing out that while I am on site doing the physical work, I interact with Doug, Anthony, Jay, and Richard to make sure things go smoothly, since each of them has their area of expertise. Jeremy

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: > I would suggest that perhaps we look into filtering BGP (tcp/179) > with an ACL prior to maintenance start on those specific ports being > moved. Many other IXs are doing this for maintenance as a way to > gracefully take things down, and let bilateral and RS sessions time > out without killing active traffic. As we've noticed, not all > members being moved are bothering to shut down sessions prior, which > causes impact to/from those members. (i.e.: > https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) Don't even need ACLs. Just take down the route servers for the 2 hour period. Bilateral are unaffected and they can arrange things anyway with their peers. Adding another step to the process creates more complications as well, and another point of failure if you screw up along the way. Clean shutdown of bird is easier, quicker, and will for sure make the multilateral peering not be further affected by bouncing repeatedly. > As an aside, thanks Jeremy for doing all this move work, it's not a > small effort and definitely appreciated. +1 -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017