- mice-discuss - lists.micemn.net

Re: Zayo Belle Plaine - April 15th

by Jeremy Lumby

Frank, I have been gradually implementing the unknown unicast rate limiting (starting with my own extension) to make sure it does not cause any unexpected problems. I had not applied it to the route servers yet, so I can understand why you saw another blip when a member with a significant amount of traffic had their port go down unexpectedly today. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, April 21, 2021 3:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We saw another BFD event at 2:04 pm today – was there a disturbance in the force? 😉 And to follow up on my previous question, is Arista falling bit short in our situation, by not supporting a receive interval of 10 msec? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Sent: Friday, April 16, 2021 4:43 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [External GPC] Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th In this case it was BFD that kicked in – I have BFD only set up for the two route reflectors. Other BGP sessions stayed up. I also got this alert from our Arista 7504: Apr 15 17:37:34 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) Bfd: %BFD-5-INCOMPATIBLE_RX_INTERVAL: Received a BFD rx interval from peer (vrf:default, ip:206.108.255.2 (AS53679-MRS-2.micemn.net) intf:Port-Channel2, srcIp:0.0.0.0, type:normal) of 10 milliseconds, outside of the supported range of 50-60000 milliseconds. (message repeated 3 times in 1.18783e+06 secs) Is Arista falling bit short in this situation, by not supporting 10 msec? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 15, 2021 10:52 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. >From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 21, 2021

Re: Zayo Belle Plaine - April 15th

by Frank Bulk

We saw another BFD event at 2:04 pm today – was there a disturbance in the force? 😉 And to follow up on my previous question, is Arista falling bit short in our situation, by not supporting a receive interval of 10 msec? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Sent: Friday, April 16, 2021 4:43 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [External GPC] Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th In this case it was BFD that kicked in – I have BFD only set up for the two route reflectors. Other BGP sessions stayed up. I also got this alert from our Arista 7504: Apr 15 17:37:34 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) Bfd: %BFD-5-INCOMPATIBLE_RX_INTERVAL: Received a BFD rx interval from peer (vrf:default, ip:206.108.255.2 (AS53679-MRS-2.micemn.net) intf:Port-Channel2, srcIp:0.0.0.0, type:normal) of 10 milliseconds, outside of the supported range of 50-60000 milliseconds. (message repeated 3 times in 1.18783e+06 secs) Is Arista falling bit short in this situation, by not supporting 10 msec? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Thursday, April 15, 2021 10:52 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 21, 2021

WiscNet MICE switch maintenance 2021-04-27 7am - Noon

by Chris Wopat

Hi Folks, Heads up that the WiscNet remote MICE switch will be going down for maintenance a week from today - Tuesday April 27th from 7am until approximately noon. We'll gracefully tear down BGP Sessions in advance and will chime in when complete. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

April 20, 2021

Re: Zayo Belle Plaine - April 15th

by Jeremy Lumby

It is interesting that it is stated in the text, but not shown in the sample port config. I think it probably got missed when migrating from the Juniper core. I can correct it. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Friday, April 16, 2021 4:45 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th If the MICE’s technical page documented limiting unknown unicast to 100 Mbps, shouldn’t its gear be set up that way? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, April 16, 2021 7:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Right now it is setup for broadcast, and multicast, however I am in favor of adding unknown unicast. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Friday, April 16, 2021 12:38 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Is there anything configured on the MICE core switches to police/drop unknown unicast? I saw mention on the MICE technical page of 100mbps limit of unknown unicast, seems like maybe that could be more aggressive (or at a minimum, on route server ports and/or interfaces facing the Juniper?) On Apr 15, 2021, at 10:52 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. >From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Frank Bulk

If the MICE’s technical page documented limiting unknown unicast to 100 Mbps, shouldn’t its gear be set up that way? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, April 16, 2021 7:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Right now it is setup for broadcast, and multicast, however I am in favor of adding unknown unicast. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Friday, April 16, 2021 12:38 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Is there anything configured on the MICE core switches to police/drop unknown unicast? I saw mention on the MICE technical page of 100mbps limit of unknown unicast, seems like maybe that could be more aggressive (or at a minimum, on route server ports and/or interfaces facing the Juniper?) On Apr 15, 2021, at 10:52 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM<mailto:jlumby@MNVOIP.COM>> wrote: Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Frank Bulk

In this case it was BFD that kicked in – I have BFD only set up for the two route reflectors. Other BGP sessions stayed up. I also got this alert from our Arista 7504: Apr 15 17:37:34 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) Bfd: %BFD-5-INCOMPATIBLE_RX_INTERVAL: Received a BFD rx interval from peer (vrf:default, ip:206.108.255.2 (AS53679-MRS-2.micemn.net) intf:Port-Channel2, srcIp:0.0.0.0, type:normal) of 10 milliseconds, outside of the supported range of 50-60000 milliseconds. (message repeated 3 times in 1.18783e+06 secs) Is Arista falling bit short in this situation, by not supporting 10 msec? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 15, 2021 10:52 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

WISCNET Additional IP

by Jay Hanke

WISCNET has been allocated 206.108.255.213 for testing of the ARP timeout issues on the WiscNet remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 16, 2021

Re: MIDCO Port Move

by Jeremy Lumby

The core switch has been updated -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Friday, April 16, 2021 11:13 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MIDCO Port Move Midco will be moving from port 3/25/1 (100G) to port 4/20 (10G) on the main switch. Port provisioning please! All IP information will remain the same. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 16, 2021

MIDCO Port Move

by Jay Hanke

Midco will be moving from port 3/25/1 (100G) to port 4/20 (10G) on the main switch. Port provisioning please! All IP information will remain the same. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Jeremy Lumby

Right now it is setup for broadcast, and multicast, however I am in favor of adding unknown unicast. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Friday, April 16, 2021 12:38 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Is there anything configured on the MICE core switches to police/drop unknown unicast? I saw mention on the MICE technical page of 100mbps limit of unknown unicast, seems like maybe that could be more aggressive (or at a minimum, on route server ports and/or interfaces facing the Juniper?) On Apr 15, 2021, at 10:52 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. >From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Andrew Hoyos

Is there anything configured on the MICE core switches to police/drop unknown unicast? I saw mention on the MICE technical page of 100mbps limit of unknown unicast, seems like maybe that could be more aggressive (or at a minimum, on route server ports and/or interfaces facing the Juniper?) > On Apr 15, 2021, at 10:52 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Frank, > > Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. > > Jeremy > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk > Sent: Thursday, April 15, 2021 9:40 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th > > We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net <mailto:AnthonyAnderberg@nuvera.net> > Sent: Thursday, April 15, 2021 6:05 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th > > Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. > > From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. > > Best of luck everyone, > Anthony > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Jeremy Lumby

Frank, Ideally it should not have, however what tends to happen is that when large ports go down the MAC addresses leave the table, and therefore the traffic gets flooded to all ports until BGP times out on the networks sending the traffic. Since the route servers only have 1G ports, the 10G of traffic that typically is flowing to Belle Plaine probably saturated the route server ports for a short amount of time. If you have short timeouts, or use something more sensitive like BFD I could see how your sessions dropped from the saturation. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, April 15, 2021 9:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Zayo Belle Plaine - April 15th We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. >From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by Frank Bulk

We (AS18883) saw our BGP sessions with the MICE router reflectors bounce at that time – should that have happened? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, April 15, 2021 6:05 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo Belle Plaine - April 15th Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2021

Re: Zayo Belle Plaine - April 15th

by James Urwiller

I experienced the same and will open a ticket as well. thanks for the update! On Thu, Apr 15, 2021 at 18:04 AnthonyAnderberg(a)nuvera.net < AnthonyAnderberg(a)nuvera.net> wrote: > Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee > @ SW MN Broadband is opening a ticket now. > > > > From the MICE side we have light but no frames, and looking at other > unrelated systems it appears the cut is South of Hwy 212 someplace. > > > > Best of luck everyone, > > Anthony > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 15, 2021

Zayo Belle Plaine - April 15th

by AnthonyAnderberg＠nuvera.net

Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony

April 15, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

Other than splice multiple cuts in 288 strands of fiber? lol Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Tue, Apr 13, 2021 at 9:35 PM James Urwiller < james.urwiller(a)turnkey-telecom.com> wrote: > zayo is insisitng that they did nothing to bring up my circuit. :( > > On Tue, Apr 13, 2021 at 18:52 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > >> Tue Apr 13 23:25:20 GMT 2021 >> Current update: >> >> They have 73-74 spliced. 133-134 will be up shortly. They should have 36 >> fibers done in the next 20 minutes. No Change to ETR at this time. That was >> the last we got from them. >> >> >> Ben Wiechman >> Director of IP Strategy and Engineering >> Direct: 320.256.0184 >> Cell: 320.247.3224 >> ben.wiechman(a)arvig.com >> 150 Second Street SW | Perham, MN 56573 >> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >> | arvig.com >> >> On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: >> >>> This was our update at 5 PM. >>> Just hung up with Zayo - their 288F was hit by a plow paralleling the >>> Zayo cable and and cut in several spots making the restoration much more >>> difficult than expected. Zayo expects to see restoral on the cable in the >>> next 60-90 minutes... >>> >>> Working on an update from Zayo again given the timeline. >>> >>> Ben Wiechman >>> Director of IP Strategy and Engineering >>> Direct: 320.256.0184 >>> Cell: 320.247.3224 >>> ben.wiechman(a)arvig.com >>> 150 Second Street SW | Perham, MN 56573 >>> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >>> | arvig.com >>> >>> On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> >>> wrote: >>> >>>> At 3:15pm I got: ETR of 1-2 hours. >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of >>>> James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>>> *Sent:* Tuesday, April 13, 2021 3:31 PM >>>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>>> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >>>> >>>> has anyone heard anything? my ticket has gone nowhere, and they tell >>>> me they cant reach belle plaine, and are going to dispatch there. >>>> >>>> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >>>> >>>>> There are other circuits down on the Zayo path. Issues appear to be >>>>> South of Worthington someplace. >>>>> >>>>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>>>> james.urwiller(a)turnkey-telecom.com> wrote: >>>>> >>>>>> anyone else about 20 min ago now? >>>>>> ------------------------------ >>>>>> >>>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>> >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 14, 2021

Re: lost mice thur belle plaine

by James Urwiller

zayo is insisitng that they did nothing to bring up my circuit. :( On Tue, Apr 13, 2021 at 18:52 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > Tue Apr 13 23:25:20 GMT 2021 > Current update: > > They have 73-74 spliced. 133-134 will be up shortly. They should have 36 > fibers done in the next 20 minutes. No Change to ETR at this time. That was > the last we got from them. > > > Ben Wiechman > Director of IP Strategy and Engineering > Direct: 320.256.0184 > Cell: 320.247.3224 > ben.wiechman(a)arvig.com > 150 Second Street SW | Perham, MN 56573 > <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> > | arvig.com > > On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > >> This was our update at 5 PM. >> Just hung up with Zayo - their 288F was hit by a plow paralleling the >> Zayo cable and and cut in several spots making the restoration much more >> difficult than expected. Zayo expects to see restoral on the cable in the >> next 60-90 minutes... >> >> Working on an update from Zayo again given the timeline. >> >> Ben Wiechman >> Director of IP Strategy and Engineering >> Direct: 320.256.0184 >> Cell: 320.247.3224 >> ben.wiechman(a)arvig.com >> 150 Second Street SW | Perham, MN 56573 >> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >> | arvig.com >> >> On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: >> >>> At 3:15pm I got: ETR of 1-2 hours. >>> >>> >>> >>> ------------------------------ >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of >>> James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>> *Sent:* Tuesday, April 13, 2021 3:31 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >>> >>> has anyone heard anything? my ticket has gone nowhere, and they tell me >>> they cant reach belle plaine, and are going to dispatch there. >>> >>> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >>> >>>> There are other circuits down on the Zayo path. Issues appear to be >>>> South of Worthington someplace. >>>> >>>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>>> james.urwiller(a)turnkey-telecom.com> wrote: >>>> >>>>> anyone else about 20 min ago now? >>>>> ------------------------------ >>>>> >>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>> >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 14, 2021

Re: Google's new 100G from Chicago

by James Urwiller

Mine is back up as of about an hour and 16 minutes ago (18:40 CST) *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Tuesday, April 13, 2021 7:10 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago So that’s three different Zayo issues in two weeks: - Under-communicated maintenance starting at 10:02 pm - Google 100G between Chicago and Minneapolis - 288 fiber cut between Alton, IA and Worthington, MN Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Andrew Hoyos *Sent:* Tuesday, April 13, 2021 6:53 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago Ironically, also a Zayo issue with a DWDM node in Chicago. On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Sent:* Saturday, April 10, 2021 11:47 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Jay Hanke *Sent:* Saturday, April 10, 2021 11:05 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf Of *Frank Bulk *Sent:* Saturday, April 10, 2021 10:52 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: · Clean the interface optic and your patching path to the MMR. · Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 14, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

So that’s three different Zayo issues in two weeks: * Under-communicated maintenance starting at 10:02 pm * Google 100G between Chicago and Minneapolis * 288 fiber cut between Alton, IA and Worthington, MN Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Tuesday, April 13, 2021 6:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Ironically, also a Zayo issue with a DWDM node in Chicago. On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Sent: Saturday, April 10, 2021 11:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 14, 2021

Re: Google's new 100G from Chicago

by Andrew Hoyos

Ironically, also a Zayo issue with a DWDM node in Chicago. > On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > > The Google NOC reported to me that the 100G link was back up this afternoon. > > According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > Sent: Saturday, April 10, 2021 11:47 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago > > Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke > Sent: Saturday, April 10, 2021 11:05 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago > > Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. > > On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Frank, > > Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. > > Jeremy > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk > Sent: Saturday, April 10, 2021 10:52 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] Google's new 100G from Chicago > > Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: > > ======================== > We have informed you via G-127797702 > > Hello MICE (Midwest Internet Cooperative Exchange), > > We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). > Please check the below list: > Google ASN > Your ASN > Google Address > Your Address > Your Demarc > Metro > Problem > Speed > 15169 > PX.MICE.ORD > 206.108.255.167 > Peering Exchange > AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 > ORD > Down > 100 Gbps > > Tips for troubleshooting physical link problems: > > · Clean the interface optic and your patching path to the MMR. > · Replace the interface optic. > > Please check your link(s) and attempt to resolve the problems described above. > > Thanks! > ======================== > > Is someone from MICE performing the requested actions and/or looking into this? > > Kind regards, > > Frank Bulk > FiberNet (ASN18883) > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

Tue Apr 13 23:25:20 GMT 2021 Current update: They have 73-74 spliced. 133-134 will be up shortly. They should have 36 fibers done in the next 20 minutes. No Change to ETR at this time. That was the last we got from them. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > This was our update at 5 PM. > Just hung up with Zayo - their 288F was hit by a plow paralleling the Zayo > cable and and cut in several spots making the restoration much more > difficult than expected. Zayo expects to see restoral on the cable in the > next 60-90 minutes... > > Working on an update from Zayo again given the timeline. > > Ben Wiechman > Director of IP Strategy and Engineering > Direct: 320.256.0184 > Cell: 320.247.3224 > ben.wiechman(a)arvig.com > 150 Second Street SW | Perham, MN 56573 | arvig.com > > On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: > >> At 3:15pm I got: ETR of 1-2 hours. >> >> >> >> ------------------------------ >> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James >> Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >> *Sent:* Tuesday, April 13, 2021 3:31 PM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >> >> has anyone heard anything? my ticket has gone nowhere, and they tell me >> they cant reach belle plaine, and are going to dispatch there. >> >> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >> >>> There are other circuits down on the Zayo path. Issues appear to be >>> South of Worthington someplace. >>> >>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>> james.urwiller(a)turnkey-telecom.com> wrote: >>> >>>> anyone else about 20 min ago now? >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >

April 13, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

This was our update at 5 PM. Just hung up with Zayo - their 288F was hit by a plow paralleling the Zayo cable and and cut in several spots making the restoration much more difficult than expected. Zayo expects to see restoral on the cable in the next 60-90 minutes... Working on an update from Zayo again given the timeline. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: > At 3:15pm I got: ETR of 1-2 hours. > > > > ------------------------------ > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James > Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> > *Sent:* Tuesday, April 13, 2021 3:31 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine > > has anyone heard anything? my ticket has gone nowhere, and they tell me > they cant reach belle plaine, and are going to dispatch there. > > On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: > >> There are other circuits down on the Zayo path. Issues appear to be South >> of Worthington someplace. >> >> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >> james.urwiller(a)turnkey-telecom.com> wrote: >> >>> anyone else about 20 min ago now? >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Sent: Saturday, April 10, 2021 11:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Frank Bulk

>From our third-party BGP monitoring it looks like Great Plains' BGP session came up at 6:36 pm, so I'm guessing fiber is getting spliced. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Tuesday, April 13, 2021 3:35 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine At 3:15pm I got: ETR of 1-2 hours. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM<mailto:james.urwiller@TURNKEY-TELECOM.COM>> Sent: Tuesday, April 13, 2021 3:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine has anyone heard anything? my ticket has gone nowhere, and they tell me they cant reach belle plaine, and are going to dispatch there. On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: There are other circuits down on the Zayo path. Issues appear to be South of Worthington someplace. On Mon, Apr 12, 2021, 9:22 PM James Urwiller <james.urwiller(a)turnkey-telecom.com<mailto:james.urwiller@turnkey-telecom.com>> wrote: anyone else about 20 min ago now? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Justin Krejci

At 3:15pm I got: ETR of 1-2 hours. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Sent: Tuesday, April 13, 2021 3:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine has anyone heard anything? my ticket has gone nowhere, and they tell me they cant reach belle plaine, and are going to dispatch there. On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: There are other circuits down on the Zayo path. Issues appear to be South of Worthington someplace. On Mon, Apr 12, 2021, 9:22 PM James Urwiller <james.urwiller(a)turnkey-telecom.com<mailto:james.urwiller@turnkey-telecom.com>> wrote: anyone else about 20 min ago now? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by James Urwiller

has anyone heard anything? my ticket has gone nowhere, and they tell me they cant reach belle plaine, and are going to dispatch there. On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: > There are other circuits down on the Zayo path. Issues appear to be South > of Worthington someplace. > > On Mon, Apr 12, 2021, 9:22 PM James Urwiller < > james.urwiller(a)turnkey-telecom.com> wrote: > >> anyone else about 20 min ago now? >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2021

Re: lost mice thur belle plaine

by Jay Hanke

There are other circuits down on the Zayo path. Issues appear to be South of Worthington someplace. On Mon, Apr 12, 2021, 9:22 PM James Urwiller < james.urwiller(a)turnkey-telecom.com> wrote: > anyone else about 20 min ago now? > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

FYI big fiber cut with Zayo fiber between Alton, IA and Sioux Falls IA. The Arvig NOC has sent OTDR shots to Zayo indicating a general location. They have a case open and the last update we got was about 30 min ago. I don't have the ticket number handy or ETR. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Mon, Apr 12, 2021, 21:22 James Urwiller < james.urwiller(a)turnkey-telecom.com> wrote: > anyone else about 20 min ago now? > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2021

lost mice thur belle plaine

by James Urwiller

anyone else about 20 min ago now?

April 13, 2021

Re: MICE Issues

by Jeremy Lumby

Possibly. I have only ever used them for devices that have only one power supply. All of the impacted equipment had 2 supplies. I think breaker tripping below it's rating is the biggest issue. One of the Cologix techs mentioned to me that since the power strip was on the back (hot) side of the rack, the increased temp might have allowed it to trip easier than expected. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Monday, April 12, 2021 3:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues Would an ATS that was between the power strips and each piece of gear have helped? https://www.wti.com/collections/power-transfer-switch We use them quite successfully. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, April 12, 2021 9:24 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues The powerstrip that had the breaker trip is being replaced with one that is slower acting to last through the surge that happens when all power transferred from the B to the A side. I also agree with Dean that we need to diversify MICE some more. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Monday, April 12, 2021 9:13 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Frank Bulk

Would an ATS that was between the power strips and each piece of gear have helped? https://www.wti.com/collections/power-transfer-switch We use them quite successfully. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, April 12, 2021 9:24 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues The powerstrip that had the breaker trip is being replaced with one that is slower acting to last through the surge that happens when all power transferred from the B to the A side. I also agree with Dean that we need to diversify MICE some more. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Monday, April 12, 2021 9:13 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Brady Kittel

We saw similar issues with the power work and had a rack completely lose power during the maintenance despite being dual fed and well under 50% load. On Mon, Apr 12, 2021, 9:24 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > The powerstrip that had the breaker trip is being replaced with one that > is slower acting to last through the surge that happens when all power > transferred from the B to the A side. I also agree with Dean that we need > to diversify MICE some more. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Frank Bulk > Sent: Monday, April 12, 2021 9:13 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] MICE Issues > > That's unfortunate! Since this could easily happen again, what could be > done to troubleshoot and/or mitigate the issue? > > Frank > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Sunday, April 11, 2021 8:25 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] MICE Issues > > When half of the power was taken down for maintenance, it caused a surge > on the remaining side that tripped the breaker even though the circuit runs > at less than 80% under failure conditions. The breaker has been reset, and > things are coming back online. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Jay Hanke > Sent: Sunday, April 11, 2021 8:00 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] MICE Issues > > Cologix has done power maintenance in Suite 100 that has inadvertently > affected the MN VoIP remote switch as well as most of the MICE > monitoring. > > This has isolated the SFN Remote Switch in Albert Lea. Cologix is > investigating. No ETTR yet. > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 >

April 12, 2021

Re: MICE Issues

by Jeremy Lumby

The powerstrip that had the breaker trip is being replaced with one that is slower acting to last through the surge that happens when all power transferred from the B to the A side. I also agree with Dean that we need to diversify MICE some more. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Monday, April 12, 2021 9:13 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Dean Bahls

Building diversity. Having all of our eggs in one basket is just asking for trouble. We were affected as well, but thankfully we have battery in our cage and were able to run until Cologix was able to amend the issue. I haven't requested the RFO from them yet but will be doing so shortly. Thanks, Dean -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, April 12, 2021 9:13 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Frank Bulk

That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Jeremy Lumby

When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 11, 2021

MICE Issues

by Jay Hanke

Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 11, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 10, 2021

Re: Google's new 100G from Chicago

by Jay Hanke

Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Frank, > > > > Yes, we received a notification from Google last night. > We are looking into it. We are receiving light but no Ethernet framing. I > am 99% sure the issue lies with their wave provider. > > > > Jeremy > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Frank Bulk > *Sent:* Saturday, April 10, 2021 10:52 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Google's new 100G from Chicago > > > > Our BGP session over this new link went down Friday just before 6 pm. > When I opened a ticket this was Google’s response: > > > > ======================== > > We have informed you via G-127797702 > > Hello MICE (Midwest Internet Cooperative Exchange), > > We have encountered issues with the following physical network link(s) you > have with Google, to start troubleshooting and prevent any further impact > to traffic we have removed traffic from the link(s). > Please check the below list: > > Google ASN > > Your ASN > > Google Address > > Your Address > > Your Demarc > > Metro > > Problem > > Speed > > 15169 > > PX.MICE.ORD > > 206.108.255.167 > > Peering Exchange > > AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 > > ORD > > Down > > 100 Gbps > > > Tips for troubleshooting physical link problems: > > · Clean the interface optic and your patching path to the MMR. > > · Replace the interface optic. > > > Please check your link(s) and attempt to resolve the problems described > above. > > Thanks! > ======================== > > > > Is someone from MICE performing the requested actions and/or looking into > this? > > > > Kind regards, > > > > Frank Bulk > > FiberNet (ASN18883) > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 10, 2021

Re: Google's new 100G from Chicago

by Jeremy Lumby

Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASNYour ASNGoogle AddressYour AddressYour DemarcMetroProblemSpeed 15169PX.MICE.ORD206.108.255.167Peering ExchangeAS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32ORDDown100 Gbps Tips for troubleshooting physical link problems: · Clean the interface optic and your patching path to the MMR. · Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 10, 2021

Google's new 100G from Chicago

by Frank Bulk

Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google's response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: * Clean the interface optic and your patching path to the MMR. * Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883)

April 10, 2021

MICE UG 39

by Richard Laager

With NANOG happening this fall, it may make sense to schedule MICE UG 39 at the same time (e.g. Wednesday November 3, time TBD). That's not official yet, but tentatively "save the date". Thanks to the person who suggested this (off-list)! -- Richard

April 10, 2021

SFN Remote Uplink Change

by Jay Hanke

SFN has upgraded the uplink LAG group on the SFN MICE remote switch to 12x10G from 8x10G. We were getting a small amount of tail drops where we convert from 100GE to the LAG group. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 1, 2021

Re: belle plaine

by James Urwiller

Tried probably 8 times already.. never seem to get any notifications. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Tim Fjeld *Sent:* Thursday, April 1, 2021 08:04 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] belle plaine We received the following notice on 16MAR2021, If you guys are on legacy Neutral Path circuits you might need to have Zayo add your circuits to their notification list. *Zayo Maintenance Notification This email serves as official notification that Zayo and/or one of its providers will be performing maintenance on its network as described below. This maintenance may affect services you have with us. * *Maintenance Ticket #: *TTN-0004856934 *Urgency: *Planned *Date Notice Sent: *16-Mar-2021 *Customer: *City of Windom - Windomnet *Maintenance Window * *1st Activity Date * 31-Mar-2021 22:00 to 01-Apr-2021 05:00 ( Central ) 01-Apr-2021 03:00 to 01-Apr-2021 10:00 ( GMT ) *Location of Maintenance: *5th Ave & Excelesior Blvd Hopkins, MN *Reason for Maintenance: *Zayo will perform of Neutral Path fiber for new storm sewer construction. *Expected Impact: *Service Affecting Activity: Any Maintenance Activity directly impacting the service(s) of customers. Service(s) are expected to go down as a result of these activities. *Circuit(s) Affected: * *Circuit Id* *Expected Impact* *A Location CLLI* *Z Location CLLI* *Legacy Circuit Id* Thanks, Tim Fjeld Network Operations City of Windom Telecom 443 10th Street, P.O. Box 38 Windom, MN 56101 Phone: 507-832-8676 *From:* Tim Fjeld *Sent:* Thursday, April 1, 2021 6:48 AM *To:* MICE Discuss <MICE-DISCUSS(a)lists.iphouse.net> *Subject:* Re: [MICE-DISCUSS] belle plaine Zayo had scheduled maintenance last night at 22:00 to move a fiber for a sewer project in Hopkins. Sent from my iPhone On Apr 1, 2021, at 12:08 AM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: SDN confirmed that there was fiber cut with the “affiliate”. They presumably mean Zayo, but I don’t know for sure. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Jay Hanke *Sent:* Wednesday, March 31, 2021 10:44 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] belle plaine Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James Urwiller *Sent:* Wednesday, March 31, 2021 10:15 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021

Re: belle plaine

by Tim Fjeld

We received the following notice on 16MAR2021, If you guys are on legacy Neutral Path circuits you might need to have Zayo add your circuits to their notification list. Zayo Maintenance Notification This email serves as official notification that Zayo and/or one of its providers will be performing maintenance on its network as described below. This maintenance may affect services you have with us. Maintenance Ticket #: TTN-0004856934 Urgency: Planned Date Notice Sent: 16-Mar-2021 Customer: City of Windom - Windomnet Maintenance Window 1st Activity Date 31-Mar-2021 22:00 to 01-Apr-2021 05:00 ( Central ) 01-Apr-2021 03:00 to 01-Apr-2021 10:00 ( GMT ) Location of Maintenance: 5th Ave & Excelesior Blvd Hopkins, MN Reason for Maintenance: Zayo will perform of Neutral Path fiber for new storm sewer construction. Expected Impact: Service Affecting Activity: Any Maintenance Activity directly impacting the service(s) of customers. Service(s) are expected to go down as a result of these activities. Circuit(s) Affected: Circuit Id Expected Impact A Location CLLI Z Location CLLI Legacy Circuit Id Thanks, Tim Fjeld Network Operations City of Windom Telecom 443 10th Street, P.O. Box 38 Windom, MN 56101 Phone: 507-832-8676 From: Tim Fjeld Sent: Thursday, April 1, 2021 6:48 AM To: MICE Discuss <MICE-DISCUSS(a)lists.iphouse.net> Subject: Re: [MICE-DISCUSS] belle plaine Zayo had scheduled maintenance last night at 22:00 to move a fiber for a sewer project in Hopkins. Sent from my iPhone On Apr 1, 2021, at 12:08 AM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: SDN confirmed that there was fiber cut with the “affiliate”. They presumably mean Zayo, but I don’t know for sure. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jay Hanke Sent: Wednesday, March 31, 2021 10:44 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] belle plaine Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Wednesday, March 31, 2021 10:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021

Re: belle plaine

by Tim Fjeld

Zayo had scheduled maintenance last night at 22:00 to move a fiber for a sewer project in Hopkins. Sent from my iPhone On Apr 1, 2021, at 12:08 AM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: SDN confirmed that there was fiber cut with the “affiliate”. They presumably mean Zayo, but I don’t know for sure. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Wednesday, March 31, 2021 10:44 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] belle plaine Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Wednesday, March 31, 2021 10:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021

Re: belle plaine

by Frank Bulk

SDN confirmed that there was fiber cut with the “affiliate”. They presumably mean Zayo, but I don’t know for sure. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Wednesday, March 31, 2021 10:44 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] belle plaine Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Wednesday, March 31, 2021 10:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021

Re: belle plaine

by Jay Hanke

Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *James > Urwiller > *Sent:* Wednesday, March 31, 2021 10:15 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] belle plaine > > > > anyone on the belle plain remote just loose connection? we are coming in > from fremont, NE. > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 1, 2021

Re: belle plaine

by Jesse DuPont

April 1, 2021

Re: belle plaine

by Frank Bulk

Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Urwiller Sent: Wednesday, March 31, 2021 10:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021