MICE connectivity issues / elevated multicast traffic
Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson@integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image854150.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking
We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: Ping to AS6939: <https://www.facebook.com/InfotecManitoba> <https://twitter.com/InfotecManitoba> <https://www.linkedin.com/in/sergip/> Sergi Infotec Manitoba <tel:(844)%20777-3744> (844) 777-3744 <mailto:sergi@infotec.ca> sergi@infotec.ca <https://infotec.ca/> https://infotec.ca From: Ryan Nelson via mice-discuss <mice-discuss@lists.micemn.net> Sent: April 30, 2025 15:02 To: mice-discuss@lists.micemn.net Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: <tel:(218)%20336-2220> (218) 336-2220 <mailto:ryan.nelson@integrisit.com> ryan.nelson@integrisit.com <https://integrisit.com/> <https://integrisit.com/resources/> Integris Engage: Check out upcoming Education, Training & Networking
Jay, We are running BFD. Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson@integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image933818.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking From: Jay Hanke <jayhanke@southfront.io> Date: Wednesday, April 30, 2025 at 3:52 PM To: Sergii Polishchuk <sergi@infotec.ca> Cc: Ryan Nelson <ryan.nelson@integrisit.com>, mice-discuss@lists.micemn.net <mice-discuss@lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic EXTERNAL MESSAGE Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi@infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: [cid:ii_1968878db2dbdbb399] Ping to AS6939: [cid:ii_1968878db2d1f3c04fa] [facebook]<https://www.facebook.com/InfotecManitoba> [twitter]<https://twitter.com/InfotecManitoba> [linkedin]<https://www.linkedin.com/in/sergip/> [cid:ii_1968878db2d855d355] Sergi Infotec Manitoba [cid:ii_1968878db2d9374b66] (844) 777-3744<tel:(844)%20777-3744> [cid:ii_1968878db2da18c377] sergi@infotec.ca<mailto:sergi@infotec.ca> [cid:ii_1968878db2dafa3b88] https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson@integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:ii_1968878db2c4cff311]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave@lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net>
We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=4027... -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=1859... Andrew Hoyos hoyosa@gmail.com
Andrew’s theory about uplinks is correct: Ridgeview 3/35 and Po1 are part of its link to 511-7804 And 511-7504 po11 and 6/36 are part of its link to 511-7804 I don’t have any theories but spent a little time looking around, my notes so far: The usual background of non-unicast packets jumped from ~100/sec to ~250/sec rom 14:10 to 15:01 They were egressing lots of ports but po4 (Charter) is the only ingress port I’ve found so far - but I don’t think it’s volume accounts for it all. A number of members had traffic volumes change but the only ones who stopped all together that I see are: Integris and Hoyos on your remote WiscNet on their remote Conversely Xcel Entergy had a spike in traffic at around that time. And then we had trouble graphing NOCIX remote ports during that time too, which is odd. From: Andrew Hoyos <hoyosa@gmail.com> Date: Wednesday, April 30, 2025 at 4:08 PM To: Jay Hanke <jayhanke@southfront.io> Cc: Sergii Polishchuk <sergi@infotec.ca>, Ryan Nelson <ryan.nelson@integrisit.com>, "mice-discuss@lists.micemn.net" <mice-discuss@lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=4027... -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=1859... Andrew Hoyos hoyosa@gmail.com On Apr 30, 2025, at 15:52, Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> wrote: Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi@infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: <image009.png> Ping to AS6939: <image010.png> <image002.png><https://www.facebook.com/InfotecManitoba> <image003.png><https://twitter.com/InfotecManitoba> <image004.png><https://www.linkedin.com/in/sergip/> <image005.png> Sergi Infotec Manitoba <image006.png> (844) 777-3744 <tel:(844)%20777-3744> <image007.png> sergi@infotec.ca<mailto:sergi@infotec.ca> <image008.png> https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson@integrisit.com<mailto:ryan.nelson@integrisit.com> <image001.png><https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave@lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
Oh man, we were definitely the cause of this. We were troubleshooting a transponder card and put it into loopback without thinking about MICE consequences. We will make sure that doesn’t happen in the future. Please accept our sincere apologies for causing havoc. Dean From: Richard Laager <rlaager@micemn.net> Sent: Wednesday, April 30, 2025 9:04 PM To: mice-discuss@lists.micemn.net Subject: Re: MICE connectivity issues / elevated multicast traffic It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
Can I ask why port security did not trip on your MICE extension switch? From: Dean Bahls <dean.bahls@fiberminnesota.com> Sent: Thursday, May 1, 2025 7:18 AM To: Richard Laager <rlaager@micemn.net>; mice-discuss@lists.micemn.net Cc: Andrew Lambertson <andrew.lambertson@fiberminnesota.com> Subject: RE: MICE connectivity issues / elevated multicast traffic Oh man, we were definitely the cause of this. We were troubleshooting a transponder card and put it into loopback without thinking about MICE consequences. We will make sure that doesn’t happen in the future. Please accept our sincere apologies for causing havoc. Dean From: Richard Laager <rlaager@micemn.net <mailto:rlaager@micemn.net> > Sent: Wednesday, April 30, 2025 9:04 PM To: mice-discuss@lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
participants (8)
-
Andrew Hoyos -
Anthony Anderberg -
Dean Bahls -
Jay Hanke -
Jeremy Lumby -
Richard Laager -
Ryan Nelson -
Sergii Polishchuk