- mice-discuss - lists.micemn.net

Re: T-Mobile Traffic Changes

by Richard Laager

FYI: This was caught in moderation because of the message size. I bumped the message size limit to 1 MiB. I'm not sure how people feel about this. If you disagree with "larger" messages on this list, please let me know. -- Richard

Aug. 1, 2025

T-Mobile Traffic Changes

by David Farmer

Last Night, around midnight, our traffic going to T-Mobile stopped going through MiCE. [image: image.png] [image: image.png] [image: image.png] It looks like they migrated traffic to their new MICE port. But we are no longer receiving any IPv4 routes from T-Mobile via MICE, but are receiving 5 IPv6 routes. We have also had instability (packet loss) with our IPv6 traffic from them for several weeks. Anyone else seeing this? Do you have any good contacts to get it resolved? Unfortunately, our contact with their NOC has not been returned or acknowledged. Thanks -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 1, 2025

Please welcome Charter Communications to MICE!

by Jeremy Lumby

Name: Charter Communications Switch: MICE 511 7504R Ports: E 6/4/1 in portchannel 7 ASN: 20115 IPv4: All IP info stays the same. Upgrading from 2x10G to 1x100G IPv6: Peering Contact: peering(a)charter.com Route Servers: NO Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

July 29, 2025

Please welcome UnitedHealth Group to MICE!

by Jeremy Lumby

Name: UnitedHealth Group Switch: MICE 511 (both switches) Ports: Adding 3/2/1 in Portchannel 5 to 6/21/1 on the 7504 (100G Switch - 206.108.255.200) Adding 3/9/1 in Portchannel 20 to 4/9/1 on the 7804 (400G Switch - 206.108.255.198) ASN: 10879 IPv4: All IP info stays the same. Adding ports for switch diversity as well as capacity of 2x2x100G IPv6: Peering Contact: inet_ops(a)uhc.com Route Servers: YES Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

July 28, 2025

Please welcome Macarne to MICE!

by Jeremy Lumby

Name: Macarne Switch: NOCIX Ports: E 5/21 ASN: 64289 IPv4: 206.108.255.23/24 IPv6: 2001:504:27:0:0:FB21::1/64 Peering Contact: peering(a)macarne.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

July 20, 2025

WiscNet AS2381 Router Maintenance

by Joshua Gorton

Heads up that WiscNet AS2381 will be performing router maintenance tonight at 10pm that will impact our MICE BGP peering sessions. Cheers, Josh

July 16, 2025

Please welcome Nova86 to MICE!

by Jeremy Lumby

Name: Nova86 LLC Switch: NOCIX Ports: E 5/22 ASN: 401538 IPv4: 206.108.255.22/24 IPv6: 2001:504:27:0:6:2082::1/64 Peering Contact: peering(a)nova86.net Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

June 25, 2025

Please welcome ServerForge to MICE!

by Jeremy Lumby

Name: ServerForge LLC Switch: NOCIX Ports: E 5/21 ASN: 23428 IPv4: 206.108.255.21/24 IPv6: 2001:504:27:0:0:5b84::1/64 Peering Contact: noc(a)serverforge.org Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

June 16, 2025

Please welcome Flexnetworks to MICE!

by Jeremy Lumby

Name: Flexnetworks Switch: MICE Core 511 Arista 7504-100G Port: E 4/22 ASN: 19523 IPv4: IP address info stays the same this is a port addition to a total of 2x10G in Po2 IPv6: Peering Contact: jay.mcmaster(a)flexnetworks.ca Route Servers: YES Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

June 16, 2025

WiscNet router maintenance

by Chris Wopat

WiscNet is performing maintenance on our router that peers on MICE - we've disabled peering until this completes around 1pm. Cheers, --Chris

June 14, 2025

RE: Yahoo peering down

by Jeremy Lumby

I have not heard anything, but I can confirm that the MICE switch is not receiving light. From: Frank Bulk <fbulk(a)mypremieronline.com> Sent: Tuesday, June 10, 2025 3:03 PM To: mice-discuss(a)lists.micemn.net Subject: Yahoo peering down Our peering sessions with Yahoo went down around 4 am, which matches with the graph here: http://micelg.usinternet.com/cacti/graph_view.php?action=tree <http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea f_id=58> &tree_id=1&leaf_id=58 Anyone happen to know what's up? Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE | Sioux Center, IA 51250 | <https://www.facebook.com/PremierCommunicationsSiouxCenter> Find us on Facebook phone: 712.722.3451 web: mypremieronline.com <https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049

June 10, 2025

Yahoo peering down

by Frank Bulk

Our peering sessions with Yahoo went down around 4 am, which matches with the graph here: http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… Anyone happen to know what's up? Frank [cid:a1ff5d57-4afc-4ec0-be61-51708bf94a4e.png] Frank Bulk Chief Technology Officer / Chief Operating Officer [cid:premiercomm_logo_cmyk_1388ffcb-8ec0-4842-9e40-a0e598482e7d.png] 339 1st Ave NE | Sioux Center, IA 51250 | Find us on Facebook<https://www.facebook.com/PremierCommunicationsSiouxCenter> phone: 712.722.3451 web: mypremieronline.com<https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049

June 10, 2025

Please welcome JTN Communications to MICE!

by Jeremy Lumby

Name: JTN Communications Switch: MICE Core Ridgeview-100G 7504 Port: Ethernet 3/32/4 (1x10G total) ASN: 398200 IPv4: 206.108.255.16/24 IPv6: 2001:504:27:0:6:1378::1/64 Peering Contact: info(a)jtncommunications.com Route Servers: NO Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

June 9, 2025

RE: [EXTERNAL] - Re: IRR import and router update processes

by Miles McCredie

Things now look the way I expect. Thank you [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: Richard Laager via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Tuesday, June 3, 2025 9:09 PM To: mice-discuss(a)lists.micemn.net Subject: [EXTERNAL] - Re: IRR import and router update processes CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. On 2025-06-03 14:38, Miles McCredie wrote: Are the IRR import and router update processes running as expected? No. Thanks for letting me know! The IXP Manager scheduler, and more specifically bgp4, seems to have failed in a "stuck running" way, as opposed to exiting with a failure status. bgpq4 was blocked on a read(). I don't know more than that, as I didn't have debug symbols for it, and it was a custom build to backport in a feature we needed. https://github.com/bgp/bgpq4/pull/90 if you're curious. This has been broken for some time, so there were a LOT of changes. Asking because I'm seeing some announcements being IRRd filtered that I don't expect. Can you check now, please? If it's still a problem, please let me know (on or off list) more specifically which prefix(es) are involved. Also, could you confirm which sources are being used? There are also some announcements with valid objects in NTTCOM that are being filtered. ARIN,RIPE,LACNIC,APNIC,AFRINIC,RADB,ALTDB,NTTCOM,LEVEL3,BELL -- Richard

June 4, 2025

Re: IRR import and router update processes

by Richard Laager

On 2025-06-03 14:38, Miles McCredie wrote: > > Are the IRR import and router update processes running as expected? > No. Thanks for letting me know! The IXP Manager scheduler, and more specifically bgp4, seems to have failed in a "stuck running" way, as opposed to exiting with a failure status. bgpq4 was blocked on a read(). I don't know more than that, as I didn't have debug symbols for it, and it was a custom build to backport in a feature we needed. https://github.com/bgp/bgpq4/pull/90 if you're curious. This has been broken for some time, so there were a LOT of changes. > Asking because I’m seeing some announcements being IRRd filtered that > I don’t expect. > Can you check now, please? If it's still a problem, please let me know (on or off list) more specifically which prefix(es) are involved. > Also, could you confirm which sources are being used? There are also > some announcements with valid objects in NTTCOM that are being filtered. > ARIN,RIPE,LACNIC,APNIC,AFRINIC,RADB,ALTDB,NTTCOM,LEVEL3,BELL -- Richard

June 4, 2025

IRR import and router update processes

by Miles McCredie

Are the IRR import and router update processes running as expected? Asking because I'm seeing some announcements being IRRd filtered that I don't expect. Also, could you confirm which sources are being used? There are also some announcements with valid objects in NTTCOM that are being filtered. Thanks [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond.

June 3, 2025

Liberty Communications Disconnect

by Jay Hanke

Liberty Communications [AS26725] has disconnected from MICE as of 6/1/2025. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/ MWPS 2025 https://www.eventcreate.com/e/mwps2025

June 2, 2025

RE: Non-MICE/perhaps dumb question

by Frank Bulk

Thanks for everyone's input and feedback. I ended up having a call with Cologix to get a better understanding of AWS' product and how it works. I shared this with our customer and have not yet received a response. Frank Frank Bulk -----Original Message----- From: Mike Horwath <drechsau(a)geeks.org> Sent: Saturday, April 26, 2025 9:30 AM To: Frank Bulk <fbulk(a)mypremieronline.com> Cc: mice-discuss(a)lists.micemn.net Subject: Re: Non-MICE/perhaps dumb question On Fri, Apr 25, 2025 at 08:53:28PM +0000, Frank Bulk wrote: > Is that just the reality of it, or is there a broader product (other > than direct peering with Amazon) that would improve AWS performance > for all our customers? Is your customer willing to cover costs for direct connect? In my experience, this is very performant (depending on link speed you choose). Are they pushing into S3, some API endpoint, custom app, over SSH or FTP or another protocol? From home (over the comcastic networkz) I can saturate my upload (~350Mbps) without issue, and lab environment (~500Mbps, Atomic Data) can also hit the peak. This is over SSH (into instances) or S3 endpoints. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

May 29, 2025

Please welcome Microsoft to MICE!

by Jeremy Lumby

Name: Microsoft Location: MICE Core 511 Ports: adding E3/1/1 to portchannel 35 (Arista 7504R) and adding E 3/8/1 to portchannel 11 (Arista 7804 R3)for a total of 2x2x100G ASN: 8075 IPv4: All IP info remains the same, this is a 2x100G port addition IPv6: Peering Contact: peering(a)microsoft.com Route Servers: NO Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

May 15, 2025

Please welcome Accuris Technologies to MICE!

by Jeremy Lumby

Name: Accuris Technologies Ltd. Switch: NOCIX Port: Ethernet 5/19 (10G) ASN: 52210 IPv4: 206.108.255.12/24 IPv6: 2001:504:27:0:0:CBF2::1/64 Peering Contact: domainsupport(a)accur.is Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

May 9, 2025

Please welcome BIFIV Intrepid OpCo, LLC to MICE!

by Jeremy Lumby

Name: BIFIV Intrepid OpCo, LLC Switch: MN VoIP 100G Port: E 12/1 ASN: 394056 IPv4: This is a port upgrade to 100G all IP info stays the same. IPv6: Peering Contact: peering(a)intrepidfiber.com Route Servers: NO Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

May 9, 2025

Please welcome Fort Dodge Fiber to MICE!

by Jeremy Lumby

Name: City of Fort Dodge dba Fort Dodge Fiber Switch: SouthFront Networks 10G Ports: Adding E2 for a total of 2x10G ASN: 400405 IPv4: This is a port addition, all IP info stays the same IPv6: Peering Contact: noc(a)fortdodgefiber.net Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

May 9, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Jeremy Lumby

Can I ask why port security did not trip on your MICE extension switch? From: Dean Bahls <dean.bahls(a)fiberminnesota.com> Sent: Thursday, May 1, 2025 7:18 AM To: Richard Laager <rlaager(a)micemn.net>; mice-discuss(a)lists.micemn.net Cc: Andrew Lambertson <andrew.lambertson(a)fiberminnesota.com> Subject: RE: MICE connectivity issues / elevated multicast traffic Oh man, we were definitely the cause of this. We were troubleshooting a transponder card and put it into loopback without thinking about MICE consequences. We will make sure that doesn’t happen in the future. Please accept our sincere apologies for causing havoc. Dean From: Richard Laager <rlaager(a)micemn.net <mailto:rlaager@micemn.net> > Sent: Wednesday, April 30, 2025 9:04 PM To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

May 1, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Dean Bahls

Oh man, we were definitely the cause of this. We were troubleshooting a transponder card and put it into loopback without thinking about MICE consequences. We will make sure that doesn’t happen in the future. Please accept our sincere apologies for causing havoc. Dean From: Richard Laager <rlaager(a)micemn.net> Sent: Wednesday, April 30, 2025 9:04 PM To: mice-discuss(a)lists.micemn.net Subject: Re: MICE connectivity issues / elevated multicast traffic It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

May 1, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Richard Laager

It sounds like we had a loop of some kind on the Fiber Minnesota extension switch. It was almost certainly, in some way, related to work they were doing. They are going to investigate further in the morning to try to understand what, specifically, happened. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

May 1, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Anthony Anderberg

Andrew’s theory about uplinks is correct: Ridgeview 3/35 and Po1 are part of its link to 511-7804 And 511-7504 po11 and 6/36 are part of its link to 511-7804 I don’t have any theories but spent a little time looking around, my notes so far: The usual background of non-unicast packets jumped from ~100/sec to ~250/sec rom 14:10 to 15:01 They were egressing lots of ports but po4 (Charter) is the only ingress port I’ve found so far - but I don’t think it’s volume accounts for it all. A number of members had traffic volumes change but the only ones who stopped all together that I see are: Integris and Hoyos on your remote WiscNet on their remote Conversely Xcel Entergy had a spike in traffic at around that time. And then we had trouble graphing NOCIX remote ports during that time too, which is odd. From: Andrew Hoyos <hoyosa(a)gmail.com> Date: Wednesday, April 30, 2025 at 4:08 PM To: Jay Hanke <jayhanke(a)southfront.io> Cc: Sergii Polishchuk <sergi(a)infotec.ca>, Ryan Nelson <ryan.nelson(a)integrisit.com>, "mice-discuss(a)lists.micemn.net" <mice-discuss(a)lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=402… -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=185… Andrew Hoyos hoyosa(a)gmail.com On Apr 30, 2025, at 15:52, Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> wrote: Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: <image009.png> Ping to AS6939: <image010.png> <image002.png><https://www.facebook.com/InfotecManitoba> <image003.png><https://twitter.com/InfotecManitoba> <image004.png><https://www.linkedin.com/in/sergip/> <image005.png> Sergi Infotec Manitoba <image006.png> (844) 777-3744 <tel:(844)%20777-3744> <image007.png> sergi(a)infotec.ca<mailto:sergi@infotec.ca> <image008.png> https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> <image001.png><https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

April 30, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Jeremy Lumby

I have not been able to dig into a too deeply yet but there does appear to be a large amount of ARP and neighbor Discovery traffic coming from a handful of Networks Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> Date: 4/30/25 3:02 PM (GMT-06:00) To: mice-discuss(a)lists.micemn.net Subject: MICE connectivity issues / elevated multicast traffic _____ From : Ryan Nelson via mice-discuss [mice-discuss(a)lists.micemn.net] To : mice-discuss(a)lists.micemn.net [mice-discuss(a)lists.micemn.net] Date : Wednesday, April 30 2025 15:01:33 Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220 <tel:(218)%20336-2220> ryan.nelson(a)integrisit.com <https://integrisit.com/> Integris Engage <https://integrisit.com/resources/> : Check out upcoming Education, Training & Networking

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Andrew Hoyos

We are, but saw issues to all peers (non BFD peers as well) There was definitely some uptick in non-unicast traffic in a few spots. I’m sure some are switch to switch links... 7804 - 3/2/1 -> ridgeview http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=402… -> Ridgeview 3/35/1 + po1 7504 - po11 + 6/36/1 http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=185… Andrew Hoyos hoyosa(a)gmail.com > On Apr 30, 2025, at 15:52, Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> wrote: > > Is everyone with issues running BFD? > > On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca <mailto:sergi@infotec.ca>> wrote: >> We lost connection with both route-servers for a minute and direct peering with some members (but not all): >> >> >> >> Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) >> >> Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down >> >> Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down >> >> Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) >> >> Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up >> >> >> >> Ping to AS42: >> >> >> >> <image009.png> >> >> >> >> Ping to AS6939: >> >> >> >> <image010.png> >> >> >> >> <image002.png> <https://www.facebook.com/InfotecManitoba> >> <image003.png> <https://twitter.com/InfotecManitoba> >> <image004.png> <https://www.linkedin.com/in/sergip/> >> >> >> <image005.png> >> >> Sergi >> >> Infotec Manitoba >> >> <image006.png> >> >> (844) 777-3744 <tel:(844)%20777-3744> >> <image007.png> >> >> sergi(a)infotec.ca <mailto:sergi@infotec.ca> >> <image008.png> >> >> https://infotec.ca <https://infotec.ca/> >> >> >> >> >> >> >> >> >> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net>> >> Sent: April 30, 2025 15:02 >> To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> >> Subject: MICE connectivity issues / elevated multicast traffic >> >> >> >> Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? >> >> Ryan Nelson >> >> Architect, Platform Services >> >> Office: (218) 336-2220 <tel:(218)%20336-2220> >> ryan.nelson(a)integrisit.com <mailto:ryan.nelson@integrisit.com> >> <image001.png> <https://integrisit.com/> >> Integris Engage <https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking >> >> >> >> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net <mailto:mice-discuss-leave@lists.micemn.net> > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Ryan Nelson

Jay, We are running BFD. Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image933818.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking From: Jay Hanke <jayhanke(a)southfront.io> Date: Wednesday, April 30, 2025 at 3:52 PM To: Sergii Polishchuk <sergi(a)infotec.ca> Cc: Ryan Nelson <ryan.nelson(a)integrisit.com>, mice-discuss(a)lists.micemn.net <mice-discuss(a)lists.micemn.net> Subject: Re: MICE connectivity issues / elevated multicast traffic EXTERNAL MESSAGE Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca<mailto:sergi@infotec.ca>> wrote: We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: [cid:ii_1968878db2dbdbb399] Ping to AS6939: [cid:ii_1968878db2d1f3c04fa] [facebook]<https://www.facebook.com/InfotecManitoba> [twitter]<https://twitter.com/InfotecManitoba> [linkedin]<https://www.linkedin.com/in/sergip/> [cid:ii_1968878db2d855d355] Sergi Infotec Manitoba [cid:ii_1968878db2d9374b66] (844) 777-3744<tel:(844)%20777-3744> [cid:ii_1968878db2da18c377] sergi(a)infotec.ca<mailto:sergi@infotec.ca> [cid:ii_1968878db2dafa3b88] https://infotec.ca<https://infotec.ca/> From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:ii_1968878db2c4cff311]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net>

April 30, 2025

Re: MICE connectivity issues / elevated multicast traffic

by Jay Hanke

Is everyone with issues running BFD? On Wed, Apr 30, 2025 at 3:48 PM Sergii Polishchuk <sergi(a)infotec.ca> wrote: > We lost connection with both route-servers for a minute and direct peering > with some members (but not all): > > > > Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD > adjacency down) > > Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD > adjacency down > > Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 > Unicast topology base removed from session BFD adjacency down > > Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset > (Peer closed the session) > > Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up > > > > Ping to AS42: > > > > > > Ping to AS6939: > > > > > > [image: facebook] <https://www.facebook.com/InfotecManitoba> > > [image: twitter] <https://twitter.com/InfotecManitoba> > > [image: linkedin] <https://www.linkedin.com/in/sergip/> > > > > *Sergi * > > Infotec Manitoba > > > > (844) 777-3744 > > sergi(a)infotec.ca > > https://infotec.ca > > > > > > > > > > > > *From:* Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> > *Sent:* April 30, 2025 15:02 > *To:* mice-discuss(a)lists.micemn.net > *Subject:* MICE connectivity issues / elevated multicast traffic > > > > Anyone else seeing some connectivity issues over MICE coinciding with an > uptick in multicast traffic coming from the core starting at around 2:10pm > CDT? > > *Ryan Nelson* > > *Architect, Platform Services* > > *Office:* (218) 336-2220 <(218)%20336-2220> > > ryan.nelson(a)integrisit.com > > <https://integrisit.com/> > > Integris Engage <https://integrisit.com/resources/> > : Check out upcoming Education, Training & Networking > > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

April 30, 2025

RE: MICE connectivity issues / elevated multicast traffic

by Sergii Polishchuk

We lost connection with both route-servers for a minute and direct peering with some members (but not all): Apr 30 14:02:18: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 reset (BFD adjacency down) Apr 30 14:02:19: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Down BFD adjacency down Apr 30 14:02:19: %BGP_SESSION-5-ADJCHANGE: neighbor 206.108.255.2 IPv4 Unicast topology base removed from session BFD adjacency down Apr 30 14:02:35: %BGP-5-NBR_RESET: Neighbor 206.108.255.2 active reset (Peer closed the session) Apr 30 14:03:13: %BGP-5-ADJCHANGE: neighbor 206.108.255.2 Up Ping to AS42: Ping to AS6939: <https://www.facebook.com/InfotecManitoba> <https://twitter.com/InfotecManitoba> <https://www.linkedin.com/in/sergip/> Sergi Infotec Manitoba <tel:(844)%20777-3744> (844) 777-3744 <mailto:sergi@infotec.ca> sergi(a)infotec.ca <https://infotec.ca/> https://infotec.ca From: Ryan Nelson via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: April 30, 2025 15:02 To: mice-discuss(a)lists.micemn.net Subject: MICE connectivity issues / elevated multicast traffic Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: <tel:(218)%20336-2220> (218) 336-2220 <mailto:ryan.nelson@integrisit.com> ryan.nelson(a)integrisit.com <https://integrisit.com/> <https://integrisit.com/resources/> Integris Engage: Check out upcoming Education, Training & Networking

April 30, 2025

MICE connectivity issues / elevated multicast traffic

by Ryan Nelson

Anyone else seeing some connectivity issues over MICE coinciding with an uptick in multicast traffic coming from the core starting at around 2:10pm CDT? Ryan Nelson Architect, Platform Services Office: (218) 336-2220<tel:(218)%20336-2220> ryan.nelson(a)integrisit.com<mailto:ryan.nelson@integrisit.com> [cid:image854150.png]<https://integrisit.com/> Integris Engage<https://integrisit.com/resources/>: Check out upcoming Education, Training & Networking

April 30, 2025

Re: MICE at 2T

by Anthony Anderberg

The good news is that I’ve finally (!) had a block of time to work on this - and am confident that the Aggregate graph now reflects reality, both today and historically. The bad news is that I was correct that I’d been double counting a few hundred gig by including ports that Jeremy added to our backbone links during this spring’s changes, so we’re back under 2T - for now at least. Cheers, Anthony From: David Farmer <farmer(a)umn.edu> Date: Saturday, March 29, 2025 at 7:33 PM To: Anthony Anderberg <AnthonyAnderberg(a)nuvera.net> Cc: "mice-discuss(a)lists.micemn.net" <mice-discuss(a)lists.micemn.net> Subject: Re: MICE at 2T Have you been able to validate the graphs? Are we regularly gating to 2 Tb now? On Sun, Mar 23, 2025 at 10:39 PM Anthony Anderberg <AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net>> wrote: I think I might be double-counting the packets that transverse the 511 building 7504-7804 link… I’ll look at that tomorrow and verify if I excluded the members of that LAG properly. Cheers, Anthony Sent from my iPhone On Mar 22, 2025, at 7:35 PM, David Farmer via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> wrote: I think Mice broke 2T in the last few days. But I'm having a hard time getting to the cacti graphs to confirm. Thanks. =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 30, 2025

Re: Non-MICE/perhaps dumb question

by Mike Horwath

On Fri, Apr 25, 2025 at 08:53:28PM +0000, Frank Bulk wrote: > Is that just the reality of it, or is there a broader product (other > than direct peering with Amazon) that would improve AWS performance > for all our customers? Is your customer willing to cover costs for direct connect? In my experience, this is very performant (depending on link speed you choose). Are they pushing into S3, some API endpoint, custom app, over SSH or FTP or another protocol? From home (over the comcastic networkz) I can saturate my upload (~350Mbps) without issue, and lab environment (~500Mbps, Atomic Data) can also hit the peak. This is over SSH (into instances) or S3 endpoints. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2025

RE: Non-MICE/perhaps dumb question

by Jeremy Lumby

Correct, Waves are very transparent to errors and the carrier selling them cannot oversubscribe them, as well as the path is known and not easily changed. Most IXes are quite transparent, and require extension uplink upgrades at 66% as well as path diversity. You get a lot of stability/predictability since most of those things you do not get if you are just buying transit. From: Ben Wiechman via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Friday, April 25, 2025 4:51 PM To: Jeremy Lumby <jlumby(a)mnvoip.com> Cc: mice-discuss(a)lists.micemn.net Subject: Re: Non-MICE/perhaps dumb question So your solution is to provide transport to a location closer to the AWS region, and then the customer establishes a private connection to AWS from there? Megaport may provide a similar transport concept. They have transport to many different DC in the US, and also had a cloud router product designed to allow remote routing for public cloud traffic w/o installing hardware the remote DC. I haven't looked at their solution in a long time to give a better judgement call regarding its viability for this use case, or whether it would only be useful on a customer by customer basis where each customer would need to have their own instance of the cloud switching in relevant data centers. This may also only be useful if direct connectivity would provide a benefit or if the customer has a lot of E/W traffic that could route remotely between public cloud applications. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com <mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com <http://arvig.com> On Fri, Apr 25, 2025 at 4:23 PM Jeremy Lumby <jlumby(a)mnvoip.com <mailto:jlumby@mnvoip.com> > wrote: I do have customers that utilize my extension switches to other IXes so that they can connect to Amazon directly in other regions. Unfortunately I have a west coast option, and you are looking for an east coast option. From: Frank Bulk <fbulk(a)mypremieronline.com <mailto:fbulk@mypremieronline.com> > Sent: Friday, April 25, 2025 3:53 PM To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: Non-MICE/perhaps dumb question I’ve been reviewing some of AWS’s connectivity products, but it seems like none of them are designed to improve the AWS experience for all of an ISP’s customers across AWS’s various regions. The background here is that we have a business customer using a key application hosted in AWS’s East Coast region. The applications’ document upload performance has been less than ideal, and their external consultant is blaming their ISP — us. As you know, once traffic leaves our network and hits upstream providers, our influence on the experience is very limited. My initial thought was to explore whether we could bring AWS closer to all of our customers through some kind of direct peering or other arrangement with AWS. However, from what I’m seeing, AWS’s connectivity options are mostly targeted at connecting a single enterprise to AWS, not enhancing general public access to AWS services. Is that just the reality of it, or is there a broader product (other than direct peering with Amazon) that would improve AWS performance for all our customers? Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE | Sioux Center, IA 51250 | <https://www.facebook.com/PremierCommunicationsSiouxCenter> Find us on Facebook phone: 712.722.3451 web: mypremieronline.com <https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049 _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net <mailto:mice-discuss-leave@lists.micemn.net>

April 25, 2025

Re: Non-MICE/perhaps dumb question

by Ben Wiechman

So your solution is to provide transport to a location closer to the AWS region, and then the customer establishes a private connection to AWS from there? Megaport may provide a similar transport concept. They have transport to many different DC in the US, and also had a cloud router product designed to allow remote routing for public cloud traffic w/o installing hardware the remote DC. I haven't looked at their solution in a long time to give a better judgement call regarding its viability for this use case, or whether it would only be useful on a customer by customer basis where each customer would need to have their own instance of the cloud switching in relevant data centers. This may also only be useful if direct connectivity would provide a benefit or if the customer has a lot of E/W traffic that could route remotely between public cloud applications. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Fri, Apr 25, 2025 at 4:23 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I do have customers that utilize my extension switches to other IXes so > that they can connect to Amazon directly in other regions. Unfortunately > I have a west coast option, and you are looking for an east coast option. > > > > *From:* Frank Bulk <fbulk(a)mypremieronline.com> > *Sent:* Friday, April 25, 2025 3:53 PM > *To:* mice-discuss(a)lists.micemn.net > *Subject:* Non-MICE/perhaps dumb question > > > > I’ve been reviewing some of AWS’s connectivity products, but it seems like > none of them are designed to improve the AWS experience for all of an ISP’s > customers across AWS’s various regions. > > > > The background here is that we have a business customer using a key > application hosted in AWS’s East Coast region. The applications’ document > upload performance has been less than ideal, and their external consultant > is blaming their ISP — us. As you know, once traffic leaves our network and > hits upstream providers, our influence on the experience is very limited. > My initial thought was to explore whether we could bring AWS closer to all > of our customers through some kind of direct peering or other arrangement > with AWS. > > > > However, from what I’m seeing, AWS’s connectivity options are mostly > targeted at connecting a single enterprise to AWS, not enhancing general > public access to AWS services. > > > > Is that just the reality of it, or is there a broader product (other than > direct peering with Amazon) that would improve AWS performance for all our > customers? > > > > Frank > > *Frank Bulk* > > *Chief Technology Officer / Chief Operating Officer* > > [image: PremierComm_logo_CMYK.png] > > 339 1st Ave NE *|* Sioux Center, IA 51250 *|* Find us on Facebook > <https://www.facebook.com/PremierCommunicationsSiouxCenter> > > *phone:* 712.722.3451 *web:* mypremieronline.com > <https://www.mypremieronline.com/> > *direct: *(712) 722-7152 *cell:* (712) 441-6049 > > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

April 25, 2025

RE: Non-MICE/perhaps dumb question

by Jeremy Lumby

I do have customers that utilize my extension switches to other IXes so that they can connect to Amazon directly in other regions. Unfortunately I have a west coast option, and you are looking for an east coast option. From: Frank Bulk <fbulk(a)mypremieronline.com> Sent: Friday, April 25, 2025 3:53 PM To: mice-discuss(a)lists.micemn.net Subject: Non-MICE/perhaps dumb question I've been reviewing some of AWS's connectivity products, but it seems like none of them are designed to improve the AWS experience for all of an ISP's customers across AWS's various regions. The background here is that we have a business customer using a key application hosted in AWS's East Coast region. The applications' document upload performance has been less than ideal, and their external consultant is blaming their ISP - us. As you know, once traffic leaves our network and hits upstream providers, our influence on the experience is very limited. My initial thought was to explore whether we could bring AWS closer to all of our customers through some kind of direct peering or other arrangement with AWS. However, from what I'm seeing, AWS's connectivity options are mostly targeted at connecting a single enterprise to AWS, not enhancing general public access to AWS services. Is that just the reality of it, or is there a broader product (other than direct peering with Amazon) that would improve AWS performance for all our customers? Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE | Sioux Center, IA 51250 | <https://www.facebook.com/PremierCommunicationsSiouxCenter> Find us on Facebook phone: 712.722.3451 web: mypremieronline.com <https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049

April 25, 2025

Non-MICE/perhaps dumb question

by Frank Bulk

I've been reviewing some of AWS's connectivity products, but it seems like none of them are designed to improve the AWS experience for all of an ISP's customers across AWS's various regions. The background here is that we have a business customer using a key application hosted in AWS's East Coast region. The applications' document upload performance has been less than ideal, and their external consultant is blaming their ISP - us. As you know, once traffic leaves our network and hits upstream providers, our influence on the experience is very limited. My initial thought was to explore whether we could bring AWS closer to all of our customers through some kind of direct peering or other arrangement with AWS. However, from what I'm seeing, AWS's connectivity options are mostly targeted at connecting a single enterprise to AWS, not enhancing general public access to AWS services. Is that just the reality of it, or is there a broader product (other than direct peering with Amazon) that would improve AWS performance for all our customers? Frank [cid:a1ff5d57-4afc-4ec0-be61-51708bf94a4e.png] Frank Bulk Chief Technology Officer / Chief Operating Officer [cid:premiercomm_logo_cmyk_1388ffcb-8ec0-4842-9e40-a0e598482e7d.png] 339 1st Ave NE | Sioux Center, IA 51250 | Find us on Facebook<https://www.facebook.com/PremierCommunicationsSiouxCenter> phone: 712.722.3451 web: mypremieronline.com<https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049

April 25, 2025

Re: Qwilt question

by David Farmer

According to the MICE port graphing, Qwilt's traffic recovered somewhat in late February. However, since February, we haven't seen any local traffic for Qwilt. [image: image.png] Thanks On Fri, Feb 14, 2025 at 1:04 PM Richard Laager <rlaager(a)micemn.net> wrote: > On 2025-02-13 08:20, Steve Savoy wrote: > > Has anyone heard anymore information on Qwilt and when the peering might > be back up? > > > I think it is very unlikely they will be reconnecting soon. > > They haven't paid in a year and are being disconnected for non-payment. > > -- > Richard Laager > Chief Manager, Director > Midwest Internet Cooperative Exchange LLC > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 23, 2025

Please welcome Cato Networks to MICE!

by Jeremy Lumby

Name: Cato Networks Inc Switch: MICE Core 511-100G 7504 Port: Ethernet 4/21 (10G) ASN: 13150 IPv4: 206.108.255.9/24 IPv6: 2001:504:27::335e:0:1/64 Peering Contact: peering(a)catonetworks.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

March 30, 2025

Re: MICE at 2T

by David Farmer

Have you been able to validate the graphs? Are we regularly gating to 2 Tb now? On Sun, Mar 23, 2025 at 10:39 PM Anthony Anderberg < AnthonyAnderberg(a)nuvera.net> wrote: > I think I might be double-counting the packets that transverse the 511 > building 7504-7804 link… I’ll look at that tomorrow and verify if I > excluded the members of that LAG properly. > > Cheers, > Anthony > > Sent from my iPhone > > On Mar 22, 2025, at 7:35 PM, David Farmer via mice-discuss < > mice-discuss(a)lists.micemn.net> wrote: > > > I think Mice broke 2T in the last few days. > > But I'm having a hard time getting to the cacti graphs to confirm. > > Thanks. > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 30, 2025

RE: Amazon bouncing

by Steve Savoy

I thought it might have been fixed as it stayed steady all night but I see it just flapped again 2 hrs ago. Steve Savoy WCTA From: Jeremy Lumby <jlumby(a)mnvoip.com> Sent: Thursday, March 27, 2025 3:01 PM To: mice-discuss(a)lists.micemn.net Subject: RE: Amazon bouncing As some background information Amazon has been migrating from 2x4x100G to 2x2x400G. When they migrated their first link to 400G it worked fine until they linked their old 100G ports up again (the ones that they had appeared to have migrated away from), and then the loss started. I let Amazon know, but I did not react since they were still mid migration, and it only impacted Amazon. Last night they finished the migration, and I received an official completion notice. It was running fine until about an hour ago. Coincidentally at that exact time one of their old 4x100G LAGs came back online. Since I had received a completion notice, this time I went ahead and forcibly shutdown their old 100G ports, and things have seemed stable since then. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: Thursday, March 27, 2025 2:04 PM To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Amazon bouncing Has anyone else's Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 [cid:image001.gif@01DB9FB5.410EFD80]<https://youtu.be/A_tgcHZD6oA>[cid:image002.gif@01DB9FB5.410EFD80]<https://twitter.com/connectctc>[cid:image003.gif@01DB9FB5.410EFD80]<https://www.facebook.com/connectctc>goctc.com<http://www.goctc.com/> [cid:image004.gif@01DB9FB5.410EFD80]<https://www.goctc.com/> Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

March 28, 2025

RE: Amazon bouncing

by Jeremy Lumby

The MICE order form does encourage every member to signup for the mailing lists. MICE posts as many new member connections, as well as upgrades to the list as possible. Beyond that it would be up to the member to share their specific maintenance details on the list. From: nanog(a)jima.us <nanog(a)jima.us> Sent: Thursday, March 27, 2025 2:51 PM To: mice-discuss(a)lists.micemn.net Subject: RE: [EXTERNAL] Amazon bouncing Someone should come up with a way to preemptively give all potential peers at a given IX a heads-up as to maintenance events like that, so that all of the other participants didn’t have to individually troubleshoot the failure condition, work out amongst themselves what was going on, and possibly multiple of them reach out to inquire. I think something like that would really be popular. 👀 (Thanks for passing that along, Ryan!) - Jima From: Ryan Gunther via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> > Sent: Thursday, March 27, 2025 14:21 To: Shawn Rasmussen <shawnr(a)goctc.com <mailto:shawnr@goctc.com> >; mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: RE: [EXTERNAL] Amazon bouncing Hi Shawn – We have been seeing lots of the same over the past week or two. Bounces every minute for hours until we shut down our sessions. It took a few emails to Amazon to get a response – but I did receive the following back from them just the other day – Dear partner, Thank you for your inquiry. We're currently in the process of a significant infrastructure upgrade, which involves relocating this circuit to our new fabric. Our team has initiated the turndown process, which will be followed by a cable migration. While we don't have a specific ETA for the completion of this work, please be assured that our team is working diligently to restore capacity as quickly as possible. No action is required from your end at this time. We will keep you informed of any significant developments or if any action becomes necessary. We understand the inconvenience this may cause and sincerely appreciate your patience during this important upgrade process. If you have any questions or concerns, please don't hesitate to reach out. Regards, Ryan Gunther Network & Switching Engineer CCNA, MEF-CECP, A+ (406) 234-7458 105 N Montana Ave., Miles City, MT 59301 <https://www.midrivers.com/> www.midrivers.com <https://www.midrivers.com/> <https://facebook.com/midrivers> <https://www.instagram.com/midriverscommunications/> <https://www.linkedin.com/company/mid-rivers-communications/> <http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> Review Us on Google From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> > Sent: Thursday, March 27, 2025 1:04 PM To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> Subject: [EXTERNAL] Amazon bouncing Notice: External Source: Confirm authenticity, stay secure. Has anyone else’s Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 <http://www.goctc.com/> goctc.com _____

March 27, 2025

RE: Amazon bouncing

by Jeremy Lumby

As some background information Amazon has been migrating from 2x4x100G to 2x2x400G. When they migrated their first link to 400G it worked fine until they linked their old 100G ports up again (the ones that they had appeared to have migrated away from), and then the loss started. I let Amazon know, but I did not react since they were still mid migration, and it only impacted Amazon. Last night they finished the migration, and I received an official completion notice. It was running fine until about an hour ago. Coincidentally at that exact time one of their old 4x100G LAGs came back online. Since I had received a completion notice, this time I went ahead and forcibly shutdown their old 100G ports, and things have seemed stable since then. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, March 27, 2025 2:04 PM To: mice-discuss(a)lists.micemn.net Subject: Amazon bouncing Has anyone else's Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 <https://youtu.be/A_tgcHZD6oA> <https://twitter.com/connectctc> <https://www.facebook.com/connectctc> <http://www.goctc.com/> goctc.com- <https://www.goctc.com/> -

March 27, 2025

Re: [EXTERNAL] Amazon bouncing

by Andrew Hoyos

RFC9003 is a thing - https://www.rfc-editor.org/rfc/rfc9003.html Juniper example: https://www.juniper.net/documentation/us/en/software/junos/cli-reference/to… I know a lot of folks using this, and it is indeed very helpful to not waste resources chasing down BGP peers. Andrew Hoyos hoyosa(a)gmail.com > On Mar 27, 2025, at 14:51, nanog(a)jima.us wrote: > > Someone should come up with a way to preemptively give all potential peers at a given IX a heads-up as to maintenance events like that, so that all of the other participants didn’t have to individually troubleshoot the failure condition, work out amongst themselves what was going on, and possibly multiple of them reach out to inquire. > > I think something like that would really be popular. > > 👀 > > (Thanks for passing that along, Ryan!) > > - Jima > > From: Ryan Gunther via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net>> > Sent: Thursday, March 27, 2025 14:21 > To: Shawn Rasmussen <shawnr(a)goctc.com <mailto:shawnr@goctc.com>>; mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> > Subject: RE: [EXTERNAL] Amazon bouncing > > Hi Shawn – > We have been seeing lots of the same over the past week or two. Bounces every minute for hours until we shut down our sessions. It took a few emails to Amazon to get a response – but I did receive the following back from them just the other day – > > > Dear partner, > > Thank you for your inquiry. We're currently in the process of a significant infrastructure upgrade, which involves relocating this circuit to our new fabric. Our team has initiated the turndown process, which will be followed by a cable migration. > > While we don't have a specific ETA for the completion of this work, please be assured that our team is working diligently to restore capacity as quickly as possible. > > No action is required from your end at this time. We will keep you informed of any significant developments or if any action becomes necessary. > > We understand the inconvenience this may cause and sincerely appreciate your patience during this important upgrade process. If you have any questions or concerns, please don't hesitate to reach out. > > Regards, > > > <image001.png> > <image002.png> > Ryan Gunther > Network & Switching Engineer > CCNA, MEF-CECP, A+ > (406) 234-7458 > 105 N Montana Ave., Miles City, MT 59301 > www.midrivers.com <https://www.midrivers.com/> > <image003.png> <https://www.midrivers.com/> <image004.png> <image005.png> <https://facebook.com/midrivers> <image006.png> <https://www.instagram.com/midriverscommunications/> <image007.png> <https://www.linkedin.com/company/mid-rivers-communications/> > Review Us on Google <http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> > > > From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net>> > Sent: Thursday, March 27, 2025 1:04 PM > To: mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> > Subject: [EXTERNAL] Amazon bouncing > > Notice: External Source: Confirm authenticity, stay secure. > Has anyone else’s Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? > > Thanks, > > SHAWN > RASMUSSEN > Network Manager > D: (218) 454-1106 > <image008.gif> <https://youtu.be/A_tgcHZD6oA><image009.gif> <https://twitter.com/connectctc><image010.gif> <https://www.facebook.com/connectctc>goctc.com <http://www.goctc.com/> > <image011.gif> <https://www.goctc.com/> > > > > > > > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net <mailto:mice-discuss@lists.micemn.net> > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net <mailto:mice-discuss-leave@lists.micemn.net>

March 27, 2025

RE: [EXTERNAL] Amazon bouncing

by nanog＠jima.us

Someone should come up with a way to preemptively give all potential peers at a given IX a heads-up as to maintenance events like that, so that all of the other participants didn’t have to individually troubleshoot the failure condition, work out amongst themselves what was going on, and possibly multiple of them reach out to inquire. I think something like that would really be popular. 👀 (Thanks for passing that along, Ryan!) - Jima From: Ryan Gunther via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, March 27, 2025 14:21 To: Shawn Rasmussen <shawnr(a)goctc.com>; mice-discuss(a)lists.micemn.net Subject: RE: [EXTERNAL] Amazon bouncing Hi Shawn – We have been seeing lots of the same over the past week or two. Bounces every minute for hours until we shut down our sessions. It took a few emails to Amazon to get a response – but I did receive the following back from them just the other day – Dear partner, Thank you for your inquiry. We're currently in the process of a significant infrastructure upgrade, which involves relocating this circuit to our new fabric. Our team has initiated the turndown process, which will be followed by a cable migration. While we don't have a specific ETA for the completion of this work, please be assured that our team is working diligently to restore capacity as quickly as possible. No action is required from your end at this time. We will keep you informed of any significant developments or if any action becomes necessary. We understand the inconvenience this may cause and sincerely appreciate your patience during this important upgrade process. If you have any questions or concerns, please don't hesitate to reach out. Regards, [signature] Ryan Gunther Network & Switching Engineer CCNA, MEF-CECP, A+ (406) 234-7458 105 N Montana Ave., Miles City, MT 59301 www.midrivers.com<https://www.midrivers.com/> [logo]<https://www.midrivers.com/> [facebook] <https://facebook.com/midrivers> [instagram] <https://www.instagram.com/midriverscommunications/> [linkedin] <https://www.linkedin.com/company/mid-rivers-communications/> Review Us on Google<http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: Thursday, March 27, 2025 1:04 PM To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: [EXTERNAL] Amazon bouncing Notice: External Source: Confirm authenticity, stay secure. Has anyone else’s Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 [cid:image008.gif@01DB9F27.AEA7B910]<https://youtu.be/A_tgcHZD6oA>[cid:image009.gif@01DB9F27.AEA7B910]<https://twitter.com/connectctc>[cid:image010.gif@01DB9F27.AEA7B910]<https://www.facebook.com/connectctc>goctc.com<http://www.goctc.com/> [cid:image011.gif@01DB9F27.AEA7B910]<https://www.goctc.com/> ________________________________

March 27, 2025

RE: [EXTERNAL] Amazon bouncing

by Steve Savoy

We have witnessed this as well. .111 has been up for 22 minutes .112 has been up for 11 hours [A logo with blue and green textDescription automatically generated]<https://www.wctatel.net/> Steve Savoy Network Administrator Winnebago Cooperative Telecom Association 704 E Main St | Lake Mills, IA 50450 Phone: 641.592.6105 | Toll Free: 800.592.6105 | Fax: 641.592.6102 Email: stevesavoy(a)wctatel.com<mailto:stevesavoy@wctatel.com> [cid:image015.png@01DB9F25.F4053570]<https://www.facebook.com/WCTAtel> [cid:image017.png@01DB9F25.F4053570]<https://www.instagram.com/wctatel/> [cid:image019.png@01DB9F25.F4053570]<https://www.linkedin.com/company/wctatel/> [cid:image021.png@01DB9F25.F4053570]<https://www.youtube.com/@wcta4892> [cid:image022.jpg@01DB9F25.F4053570] From: Nathan Beard <nbeard(a)7sigma.com> Sent: Thursday, March 27, 2025 2:30 PM To: Ryan Gunther <Ryan.Gunther(a)midrivers.coop> Cc: Shawn Rasmussen <shawnr(a)goctc.com>; mice-discuss(a)lists.micemn.net Subject: Re: [EXTERNAL] Amazon bouncing You don't often get email from nbeard(a)7sigma.com<mailto:nbeard@7sigma.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Stellar Association has also seen this same behavior since the MICE port moves last week, and I've reached out to Amazon a couple times and have yet to get a response. Things seemed to stabilize around 3:10am this morning so I was hopeful it was fixed, but it looks like the .111 bi-lateral is flapping again as of 1:42pm. It's worth noting that the second bi-lateral sessions appears to be stable today. If anybody else gets any detailed updates from Amazon and could post those to the list it would be much appreciated. Nathan Beard 7Sigma Systems, Inc. Senior Network Engineer (218) 820-7636 www.7sigma.com<http://www.7sigma.com/> On Thu, Mar 27, 2025 at 2:23 PM Ryan Gunther via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> wrote: Hi Shawn – We have been seeing lots of the same over the past week or two. Bounces every minute for hours until we shut down our sessions. It took a few emails to Amazon to get a response – but I did receive the following back from them just the other day – Dear partner, Thank you for your inquiry. We're currently in the process of a significant infrastructure upgrade, which involves relocating this circuit to our new fabric. Our team has initiated the turndown process, which will be followed by a cable migration. While we don't have a specific ETA for the completion of this work, please be assured that our team is working diligently to restore capacity as quickly as possible. No action is required from your end at this time. We will keep you informed of any significant developments or if any action becomes necessary. We understand the inconvenience this may cause and sincerely appreciate your patience during this important upgrade process. If you have any questions or concerns, please don't hesitate to reach out. Regards, [signature] Ryan Gunther Network & Switching Engineer CCNA, MEF-CECP, A+ (406) 234-7458 105 N Montana Ave., Miles City, MT 59301 www.midrivers.com<https://www.midrivers.com/> [logo]<https://www.midrivers.com/> [facebook] <https://facebook.com/midrivers> [instagram] <https://www.instagram.com/midriverscommunications/> [linkedin] <https://www.linkedin.com/company/mid-rivers-communications/> Review Us on Google<http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: Thursday, March 27, 2025 1:04 PM To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: [EXTERNAL] Amazon bouncing Notice: External Source: Confirm authenticity, stay secure. Has anyone else’s Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 [cid:image008.gif@01DB9F25.E3948790]<https://youtu.be/A_tgcHZD6oA>[cid:image009.gif@01DB9F25.E3948790]<https://twitter.com/connectctc>[cid:image010.gif@01DB9F25.E3948790]<https://www.facebook.com/connectctc>goctc.com<http://www.goctc.com/> [cid:image011.gif@01DB9F25.E3948790]<https://www.goctc.com/> ________________________________ _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

March 27, 2025

Re: [EXTERNAL] Amazon bouncing

by Nathan Beard

Stellar Association has also seen this same behavior since the MICE port moves last week, and I've reached out to Amazon a couple times and have yet to get a response. Things seemed to stabilize around 3:10am this morning so I was hopeful it was fixed, but it looks like the .111 bi-lateral is flapping again as of 1:42pm. It's worth noting that the second bi-lateral sessions appears to be stable today. If anybody else gets any detailed updates from Amazon and could post those to the list it would be much appreciated. *Nathan Beard* *7Sigma Systems, Inc.* Senior Network Engineer (218) 820-7636 www.7sigma.com On Thu, Mar 27, 2025 at 2:23 PM Ryan Gunther via mice-discuss < mice-discuss(a)lists.micemn.net> wrote: > Hi Shawn – > > We have been seeing lots of the same over the past week or > two. Bounces every minute for hours until we shut down our sessions. It > took a few emails to Amazon to get a response – but I did receive the > following back from them just the other day – > > > > > > Dear partner, > > > > Thank you for your inquiry. We're currently in the process of a > significant infrastructure upgrade, which involves relocating this circuit > to our new fabric. Our team has initiated the turndown process, which will > be followed by a cable migration. > > > > While we don't have a specific ETA for the completion of this work, please > be assured that our team is working diligently to restore capacity as > quickly as possible. > > > > No action is required from your end at this time. We will keep you > informed of any significant developments or if any action becomes necessary. > > > > We understand the inconvenience this may cause and sincerely appreciate > your patience during this important upgrade process. If you have any > questions or concerns, please don't hesitate to reach out. > > > > Regards, > > > > > > [image: signature] > > [image: divider] > > Ryan Gunther > > Network & Switching Engineer > > CCNA, MEF-CECP, A+ > > (406) 234-7458 > > 105 N Montana Ave., Miles City, MT 59301 > > www.midrivers.com > > [image: logo] <https://www.midrivers.com/> [image: divider] [image: > facebook] <https://facebook.com/midrivers> [image: instagram] > <https://www.instagram.com/midriverscommunications/> [image: linkedin] > <https://www.linkedin.com/company/mid-rivers-communications/> > > Review Us on Google > <http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> > > > > > > *From:* Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net> > *Sent:* Thursday, March 27, 2025 1:04 PM > *To:* mice-discuss(a)lists.micemn.net > *Subject:* [EXTERNAL] Amazon bouncing > > > > *Notice:* External Source: Confirm authenticity, stay secure. > > Has anyone else’s Amazon peering been bouncing a lot lately? They are > bouncing right now for us and have sporadic periods over the last week. Are > any of you seeing the same? Anyone know anything about it? > > > > Thanks, > > > > *SHAWN* > > *RASMUSSEN* > > Network Manager > > D: (218) 454-1106 > > <https://youtu.be/A_tgcHZD6oA> <https://twitter.com/connectctc> > <https://www.facebook.com/connectctc>*goctc.com* <http://www.goctc.com/> > ** > > <https://www.goctc.com/> > > > > > > > > > > > > ------------------------------ > > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

March 27, 2025

RE: Amazon bouncing

by Randy Lehs - CarverLink

Yes, we have seen the same, with the bouncing starting late last week. Randy Lehs | CarverLink Fiber Manager CarverLink - A Department of Carver County Public Services 600 East Fourth Street Chaska, MN 55318-2102 www.carverlink.com<http://www.carverlink.com/> | rlehs(a)carverlink.com<mailto:rlehs@carverlink.com> Office: 952-361-1494 | Fax: 952-361-1582 | Cell: 612-719-1656 | Standard and Emergency Support 24/7/365: 952-361-1200 From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, March 27, 2025 2:04 PM To: mice-discuss(a)lists.micemn.net Subject: Amazon bouncing Has anyone else's Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 [cid:image005.gif@01DB9F24.442E84E0]<https://youtu.be/A_tgcHZD6oA>[cid:image006.gif@01DB9F24.442E84E0]<https://twitter.com/connectctc>[cid:image007.gif@01DB9F24.442E84E0]<https://www.facebook.com/connectctc>goctc.com<http://www.goctc.com/> [cid:image008.gif@01DB9F24.442E84E0]<https://www.goctc.com/>

March 27, 2025

RE: [EXTERNAL] Amazon bouncing

by Ryan Gunther

Hi Shawn - We have been seeing lots of the same over the past week or two. Bounces every minute for hours until we shut down our sessions. It took a few emails to Amazon to get a response - but I did receive the following back from them just the other day - Dear partner, Thank you for your inquiry. We're currently in the process of a significant infrastructure upgrade, which involves relocating this circuit to our new fabric. Our team has initiated the turndown process, which will be followed by a cable migration. While we don't have a specific ETA for the completion of this work, please be assured that our team is working diligently to restore capacity as quickly as possible. No action is required from your end at this time. We will keep you informed of any significant developments or if any action becomes necessary. We understand the inconvenience this may cause and sincerely appreciate your patience during this important upgrade process. If you have any questions or concerns, please don't hesitate to reach out. Regards, [signature] Ryan Gunther Network & Switching Engineer CCNA, MEF-CECP, A+ (406) 234-7458 105 N Montana Ave., Miles City, MT 59301 www.midrivers.com<https://www.midrivers.com/> [logo]<https://www.midrivers.com/> [facebook] <https://facebook.com/midrivers> [instagram] <https://www.instagram.com/midriverscommunications/> [linkedin] <https://www.linkedin.com/company/mid-rivers-communications/> Review Us on Google<http://www.google.com/search?q=Mid-Rivers+Communications,1101%20Main%20St.,…> From: Shawn Rasmussen via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, March 27, 2025 1:04 PM To: mice-discuss(a)lists.micemn.net Subject: [EXTERNAL] Amazon bouncing Notice: External Source: Confirm authenticity, stay secure. Has anyone else's Amazon peering been bouncing a lot lately? They are bouncing right now for us and have sporadic periods over the last week. Are any of you seeing the same? Anyone know anything about it? Thanks, SHAWN RASMUSSEN Network Manager D: (218) 454-1106 [cid:image012.gif@01DB9F1A.F436CDC0]<https://youtu.be/A_tgcHZD6oA>[cid:image013.gif@01DB9F1A.F436CDC0]<https://twitter.com/connectctc>[cid:image014.gif@01DB9F1A.F436CDC0]<https://www.facebook.com/connectctc>goctc.com<http://www.goctc.com/> [cid:image015.gif@01DB9F1A.F436CDC0]<https://www.goctc.com/> ________________________________ This email has been scanned for spam and viruses by Proofpoint Essentials. Click here<https://us3.proofpointessentials.com/app/report_spam.php?mod_id=11&mod_opti…> to report this email as spam.

March 27, 2025