April 2017 - mice-discuss - lists.micemn.net

Re: MICE Graphs

by David Farmer

I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > On 04/26/2017 11:32 AM, David Farmer wrote: > > According to the graphs on the web page there was a network holocaust on > > MICE yesterday. Could someone please fix the graphs. > > Anthony is actively working on this now. The RRDs were copied last > night, so the data is preserved. > > -- > Richard > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 27, 2017

Re: Arista Round 3 Moves

by Justin Krejci

Forcing an issue is going to happen when the port goes down. The point is the ACL can make it more graceful for those directly affected and also those indirectly affected by the port going down by ensuring all BGP connections (MLPA and bilateral) are terminated with the directly affected members before the link goes down for the directly affected members. In this context: directly affected member = those whose path to the main fabric are going to be physically moved (includes downstream members on an extension switch that is being moved) indirectly affected member = everyone else who are not going to have their own path to the fabric go down I am not sure who would intentionally prefer a less graceful option of losing all/most MICE connectivity. The addition of very short lived ACL's on the exchange fabric ports for directly affected members does not preclude members from implementing their own proactive measures; people are certainly free to heed the announcement that they are going to be directly affected and pre-emptively shutdown all of their own BGP sessions at whatever time they want in advance and re-activate them anytime they choose afterwards be it minutes, hours or days later. The addition of the exchange enacted ACLs will help minimize the impact to the indirectly affected members who have bilateral peering with directly affected members. ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Mike Horwath [drechsau(a)GEEKS.ORG] Sent: Wednesday, April 26, 2017 6:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 10:51:34AM -0500, Doug McIntyre wrote: > On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > > Bilateral are unaffected and they can arrange things anyway with their > > peers. > > I think there is a lot more bilateral BGP going on across MICE than > you think. While I probably go talk to anybody, I have 11 > unilateral peers on MICE besides the two route servers. Yep, I am assuming a lot of bilateral BGP peers. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: MICE Graphs

by Richard Laager

On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard

April 26, 2017

MICE Graphs

by David Farmer

According to the graphs on the web page there was a network holocaust on MICE yesterday. Could someone please fix the graphs. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 26, 2017

Re: Arista Round 3 Moves

by Richard Laager

Doug, Please do go ahead and expand passive BFD. Everyone on one route server followed by the second route server a few days later should be fine. -- Richard > On Apr 26, 2017, at 10:51, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I think there is a lot more bilateral BGP going on across MICE than you think. > While I probably go talk to anybody, I have 11 unilateral peers on MICE besides > the two route servers. > > I think BFD for BGP would do a lot more good than BGP filtering if people would > implement it. > > Speaking of that, I have passive BFD testing on route server #1 for quite some time. > I've heard of zero problems. > > I would like to go more widespread than the test people (ie. offer it > to every member connection on route server #1) > > Since it is passive, the member end drives it, if they do it, the > route server will do it, if they don't, nothing else happens. I'd also like > to roll out passive BFD as default for new members. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > Bilateral are unaffected and they can arrange things anyway with their > peers. I think there is a lot more bilateral BGP going on across MICE than you think. While I probably go talk to anybody, I have 11 unilateral peers on MICE besides the two route servers. I think BFD for BGP would do a lot more good than BGP filtering if people would implement it. Speaking of that, I have passive BFD testing on route server #1 for quite some time. I've heard of zero problems. I would like to go more widespread than the test people (ie. offer it to every member connection on route server #1) Since it is passive, the member end drives it, if they do it, the route server will do it, if they don't, nothing else happens. I'd also like to roll out passive BFD as default for new members. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Wed, Apr 26, 2017 at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. Sadly, I can't thumbs-up this post as I can on FB... -Tk

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

I’ll also add, this is currently a draft with the IETF grow WG: https://tools.ietf.org/html/draft-ietf-grow-bgp-session-culling-01 (and one which I fully support) -- Andrew Hoyos hoyosa(a)gmail.com > On Apr 26, 2017, at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: >> >> On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >>> I would suggest that perhaps we look into filtering BGP (tcp/179) >>> with an ACL prior to maintenance start on those specific ports being >>> moved. Many other IXs are doing this for maintenance as a way to >>> gracefully take things down, and let bilateral and RS sessions time >>> out without killing active traffic. As we've noticed, not all >>> members being moved are bothering to shut down sessions prior, which >>> causes impact to/from those members. (i.e.: >>> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) >> >> Don't even need ACLs. >> >> Just take down the route servers for the 2 hour period. >> >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. > > Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. > I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > >> Adding another step to the process creates more complications as well, >> and another point of failure if you screw up along the way. > > Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. > Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. > > >

April 26, 2017