April 2025 - mice-discuss - lists.micemn.net

Re: MICE connectivity issues / elevated multicast traffic

by Anthony Anderberg

Andrew’s theory about uplinks is correct: Ridgeview 3/35 and Po1 are part of its link to 511-7804 And 511-7504 po11 and 6/36 are part of its link to 511-7804 I don’t have any theories but spent a little time looking around, my notes so far: The usual background of non-unicast packets jumped from ~100/sec to ~250/sec rom 14:10 to 15:01 They were egressing lots of ports but po4 (Charter) is the only ingress port I’ve found so far - but I don’t think it’s volume accounts for it all. A number of members had traffic volumes change but the only ones who stopped all together that I see are: Integris and Hoyos on your remote WiscNet on their remote Conversely Xcel Entergy had a spike in traffic at around that time. And then we had trouble graphing NOCIX remote ports during that time too, which is odd. From: Andrew Hoyos <hoyosa(a)gmail.com> Date: Wednesday, April 30, 2025 at 4:08 PM To: Jay Hanke <jayhanke(a)southfront.io> Cc: Sergii Polishchuk <sergi(a)infotec.ca>, Ryan Nelson <ryan.nelson(a)integrisit.com>, "mice-discuss(a)lists.micemn.net" <mice-discuss(a)lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=402… -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=185… Andrew Hoyos hoyosa(a)gmail.com On Apr 30, 2025, at 15:52, Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> wrote: Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: <image009.png> Ping to AS6939: <image010.png> <image002.png><https://www.facebook.com/InfotecManitoba> <image003.png><https://twitter.com/InfotecManitoba> <image004.png><https://www.linkedin.com/in/sergip/> <image005.png> Sergi Infotec Manitoba <image006.png> (844) 777-3744 <tel:(844)%20777-3744> <image007.png> sergi(a)infotec.ca<mailto:sergi@infotec.ca> <image008.png> https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> <image001.png><https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

April 30, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Jeremy Lumby

I have not been able to dig into a too deeply yet but there does appear to be a large amount of ARP and neighbor Discovery traffic coming from a handful of Networks Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> Date: 4/30/25 3:02 PM (GMT-06:00) To: mice-discuss(a)lists.micemn.net Subject: MICE connectivity issues / elevated multicast traffic _____ From : Ryan Nelson via mice-discuss [mice-discuss(a)lists.micemn.net] To : mice-discuss(a)lists.micemn.net [mice-discuss(a)lists.micemn.net] Date : Wednesday, April 30 2025 15:01:33 Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220 <tel:(218)%20336-2220> ryan.nelson(a)integrisit.com <https://integrisit.com/> Integris Engage <https://integrisit.com/resources/> : Check out upcoming Education, Training & Networking

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Andrew Hoyos

We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=402… -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=185… Andrew Hoyos hoyosa(a)gmail.com > On Apr 30, 2025, at 15:52, Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> wrote: > > Is everyone with issues running BFD? > > On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca <mailto:sergi@infotec.ca>> wrote: >> We lost connection with both route-servers for a minute and direct peering with some members (but not all): >> >> >> >> Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) >> >> Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down >> >> Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down >> >> Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) >> >> Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up >> >> >> >> Ping to AS42: >> >> >> >> <image009.png> >> >> >> >> Ping to AS6939: >> >> >> >> <image010.png> >> >> >> >> <image002.png> <https://www.facebook.com/InfotecManitoba> >> <image003.png> <https://twitter.com/InfotecManitoba> >> <image004.png> <https://www.linkedin.com/in/sergip/> >> >> >> <image005.png> >> >> Sergi >> >> Infotec Manitoba >> >> <image006.png> >> >> (844) 777-3744 <tel:(844)%20777-3744> >> <image007.png> >> >> sergi(a)infotec.ca <mailto:sergi@infotec.ca> >> <image008.png> >> >> https://infotec.ca <https://infotec.ca/> >> >> >> >> >> >> >> >> >> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net>> >> Sent: April 30, 2025 15:02 >> To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> >> Subject: MICE connectivity issues / elevated multicast traffic >> >> >> >> Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? >> >> Ryan Nelson >> >> Architect, Platform Services >> >> Office: (218) 336-2220 <tel:(218)%20336-2220> >> ryan.nelson(a)integrisit.com <mailto:ryan.nelson@integrisit.com> >> <image001.png> <https://integrisit.com/> >> Integris Engage <https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking >> >> >> >> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net <mailto:mice-discuss-leave@lists.micemn.net> > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Ryan Nelson

Jay, We are running BFD. Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image933818.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking From: Jay Hanke <jayhanke(a)southfront.io> Date: Wednesday, April 30, 2025 at 3:52 PM To: Sergii Polishchuk <sergi(a)infotec.ca> Cc: Ryan Nelson <ryan.nelson(a)integrisit.com>, mice-discuss(a)lists.micemn.net <mice-discuss(a)lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic EXTERNAL MESSAGE Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: [cid:ii_1968878db2dbdbb399] Ping to AS6939: [cid:ii_1968878db2d1f3c04fa] [facebook]<https://www.facebook.com/InfotecManitoba> [twitter]<https://twitter.com/InfotecManitoba> [linkedin]<https://www.linkedin.com/in/sergip/> [cid:ii_1968878db2d855d355] Sergi Infotec Manitoba [cid:ii_1968878db2d9374b66] (844) 777-3744<tel:(844)%20777-3744> [cid:ii_1968878db2da18c377] sergi(a)infotec.ca<mailto:sergi@infotec.ca> [cid:ii_1968878db2dafa3b88] https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:ii_1968878db2c4cff311]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net>

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Jay Hanke

Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca> wrote: > We lost connection with both route-servers for a minute and direct peering > with some members (but not all): > > > > Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD > adjacency down) > > Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD > adjacency down > > Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 > Unicast topology base removed from session BFD adjacency down > > Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset > (Peer closed the session) > > Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up > > > > Ping to AS42: > > > > > > Ping to AS6939: > > > > > > [image: facebook] <https://www.facebook.com/InfotecManitoba> > > [image: twitter] <https://twitter.com/InfotecManitoba> > > [image: linkedin] <https://www.linkedin.com/in/sergip/> > > > > *Sergi * > > Infotec Manitoba > > > > (844) 777-3744 > > sergi(a)infotec.ca > > https://infotec.ca > > > > > > > > > > > > *From:* Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> > *Sent:* April 30, 2025 15:02 > *To:* mice-discuss(a)lists.micemn.net > *Subject:* MICE connectivity issues / elevated multicast traffic > > > > Anyone else seeing some connectivity issues over MICE coinciding with an > uptick in multicast traffic coming from the core starting at around 2:10pm > CDT? > > *Ryan Nelson* > > *Architect, Platform Services* > > *Office:* (218) 336-2220 <(218)%20336-2220> > > ryan.nelson(a)integrisit.com > > <https://integrisit.com/> > > Integris Engage <https://integrisit.com/resources/> > : Check out upcoming Education, Training & Networking > > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

April 30, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Sergii Polishchuk

We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: Ping to AS6939: <https://www.facebook.com/InfotecManitoba> <https://twitter.com/InfotecManitoba> <https://www.linkedin.com/in/sergip/> Sergi Infotec Manitoba <tel:(844)%20777-3744> (844) 777-3744 <mailto:sergi@infotec.ca> sergi(a)infotec.ca <https://infotec.ca/> https://infotec.ca From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: <tel:(218)%20336-2220> (218) 336-2220 <mailto:ryan.nelson@integrisit.com> ryan.nelson(a)integrisit.com <https://integrisit.com/> <https://integrisit.com/resources/> Integris Engage: Check out upcoming Education, Training & Networking

April 30, 2025

MICE connectivity issues / elevated multicast traffic

by Ryan Nelson

Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image854150.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking

April 30, 2025

Re: MICE at 2T

by Anthony Anderberg

The good news is that I’ve finally (!) had a block of time to work on this - and am confident that the Aggregate graph now reflects reality, both today and historically. The bad news is that I was correct that I’d been double counting a few hundred gig by including ports that Jeremy added to our backbone links during this spring’s changes, so we’re back under 2T - for now at least. Cheers, Anthony From: David Farmer <farmer(a)umn.edu> Date: Saturday, March 29, 2025 at 7:33 PM To: Anthony Anderberg <AnthonyAnderberg(a)nuvera.net> Cc: "mice-discuss(a)lists.micemn.net" <mice-discuss(a)lists.micemn.net> Subject: Re: MICE at 2T Have you been able to validate the graphs? Are we regularly gating to 2 Tb now? On Sun, Mar 23, 2025 at 10:39 PM Anthony Anderberg <AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net>> wrote: I think I might be double-counting the packets that transverse the 511 building 7504-7804 link… I’ll look at that tomorrow and verify if I excluded the members of that LAG properly. Cheers, Anthony Sent from my iPhone On Mar 22, 2025, at 7:35 PM, David Farmer via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> wrote: I think Mice broke 2T in the last few days. But I'm having a hard time getting to the cacti graphs to confirm. Thanks. =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 30, 2025

Re: Non-MICE/perhaps dumb question

by Mike Horwath

On Fri, Apr 25, 2025 at 08:53:28PM +0000, Frank Bulk wrote: > Is that just the reality of it, or is there a broader product (other > than direct peering with Amazon) that would improve AWS performance > for all our customers? Is your customer willing to cover costs for direct connect? In my experience, this is very performant (depending on link speed you choose). Are they pushing into S3, some API endpoint, custom app, over SSH or FTP or another protocol? From home (over the comcastic networkz) I can saturate my upload (~350Mbps) without issue, and lab environment (~500Mbps, Atomic Data) can also hit the peak. This is over SSH (into instances) or S3 endpoints. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2025

RE: Non-MICE/perhaps dumb question

by Jeremy Lumby

Correct, Waves are very transparent to errors and the carrier selling them cannot oversubscribe them, as well as the path is known and not easily changed. Most IXes are quite transparent, and require extension uplink upgrades at 66% as well as path diversity. You get a lot of stability/predictability since most of those things you do not get if you are just buying transit. From: Ben Wiechman via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Friday, April 25, 2025 4:51 PM To: Jeremy Lumby <jlumby(a)mnvoip.com> Cc: mice-discuss(a)lists.micemn.net Subject: Re: Non-MICE/perhaps dumb question So your solution is to provide transport to a location closer to the AWS region, and then the customer establishes a private connection to AWS from there? Megaport may provide a similar transport concept. They have transport to many different DC in the US, and also had a cloud router product designed to allow remote routing for public cloud traffic w/o installing hardware the remote DC. I haven't looked at their solution in a long time to give a better judgement call regarding its viability for this use case, or whether it would only be useful on a customer by customer basis where each customer would need to have their own instance of the cloud switching in relevant data centers. This may also only be useful if direct connectivity would provide a benefit or if the customer has a lot of E/W traffic that could route remotely between public cloud applications. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com <mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com <http://arvig.com> On Fri, Apr 25, 2025 at 4:23 PM Jeremy Lumby <jlumby(a)mnvoip.com <mailto:jlumby@mnvoip.com> > wrote: I do have customers that utilize my extension switches to other IXes so that they can connect to Amazon directly in other regions. Unfortunately I have a west coast option, and you are looking for an east coast option. From: Frank Bulk <fbulk(a)mypremieronline.com <mailto:fbulk@mypremieronline.com> > Sent: Friday, April 25, 2025 3:53 PM To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: Non-MICE/perhaps dumb question I’ve been reviewing some of AWS’s connectivity products, but it seems like none of them are designed to improve the AWS experience for all of an ISP’s customers across AWS’s various regions. The background here is that we have a business customer using a key application hosted in AWS’s East Coast region. The applications’ document upload performance has been less than ideal, and their external consultant is blaming their ISP — us. As you know, once traffic leaves our network and hits upstream providers, our influence on the experience is very limited. My initial thought was to explore whether we could bring AWS closer to all of our customers through some kind of direct peering or other arrangement with AWS. However, from what I’m seeing, AWS’s connectivity options are mostly targeted at connecting a single enterprise to AWS, not enhancing general public access to AWS services. Is that just the reality of it, or is there a broader product (other than direct peering with Amazon) that would improve AWS performance for all our customers? Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE | Sioux Center, IA 51250 | <https://www.facebook.com/PremierCommunicationsSiouxCenter> Find us on Facebook phone: 712.722.3451 web: mypremieronline.com <https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049 _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net <mailto:mice-discuss-leave@lists.micemn.net>

April 25, 2025