- mice-discuss - lists.micemn.net

Re: lost mice thur belle plaine

by Ben Wiechman

Other than splice multiple cuts in 288 strands of fiber? lol Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Tue, Apr 13, 2021 at 9:35 PM James Urwiller < james.urwiller(a)turnkey-telecom.com> wrote: > zayo is insisitng that they did nothing to bring up my circuit. :( > > On Tue, Apr 13, 2021 at 18:52 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > >> Tue Apr 13 23:25:20 GMT 2021 >> Current update: >> >> They have 73-74 spliced. 133-134 will be up shortly. They should have 36 >> fibers done in the next 20 minutes. No Change to ETR at this time. That was >> the last we got from them. >> >> >> Ben Wiechman >> Director of IP Strategy and Engineering >> Direct: 320.256.0184 >> Cell: 320.247.3224 >> ben.wiechman(a)arvig.com >> 150 Second Street SW | Perham, MN 56573 >> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >> | arvig.com >> >> On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: >> >>> This was our update at 5 PM. >>> Just hung up with Zayo - their 288F was hit by a plow paralleling the >>> Zayo cable and and cut in several spots making the restoration much more >>> difficult than expected. Zayo expects to see restoral on the cable in the >>> next 60-90 minutes... >>> >>> Working on an update from Zayo again given the timeline. >>> >>> Ben Wiechman >>> Director of IP Strategy and Engineering >>> Direct: 320.256.0184 >>> Cell: 320.247.3224 >>> ben.wiechman(a)arvig.com >>> 150 Second Street SW | Perham, MN 56573 >>> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >>> | arvig.com >>> >>> On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> >>> wrote: >>> >>>> At 3:15pm I got: ETR of 1-2 hours. >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of >>>> James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>>> *Sent:* Tuesday, April 13, 2021 3:31 PM >>>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>>> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >>>> >>>> has anyone heard anything? my ticket has gone nowhere, and they tell >>>> me they cant reach belle plaine, and are going to dispatch there. >>>> >>>> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >>>> >>>>> There are other circuits down on the Zayo path. Issues appear to be >>>>> South of Worthington someplace. >>>>> >>>>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>>>> james.urwiller(a)turnkey-telecom.com> wrote: >>>>> >>>>>> anyone else about 20 min ago now? >>>>>> ------------------------------ >>>>>> >>>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>> >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 14, 2021

Re: lost mice thur belle plaine

by James Urwiller

zayo is insisitng that they did nothing to bring up my circuit. :( On Tue, Apr 13, 2021 at 18:52 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > Tue Apr 13 23:25:20 GMT 2021 > Current update: > > They have 73-74 spliced. 133-134 will be up shortly. They should have 36 > fibers done in the next 20 minutes. No Change to ETR at this time. That was > the last we got from them. > > > Ben Wiechman > Director of IP Strategy and Engineering > Direct: 320.256.0184 > Cell: 320.247.3224 > ben.wiechman(a)arvig.com > 150 Second Street SW | Perham, MN 56573 > <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> > | arvig.com > > On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > >> This was our update at 5 PM. >> Just hung up with Zayo - their 288F was hit by a plow paralleling the >> Zayo cable and and cut in several spots making the restoration much more >> difficult than expected. Zayo expects to see restoral on the cable in the >> next 60-90 minutes... >> >> Working on an update from Zayo again given the timeline. >> >> Ben Wiechman >> Director of IP Strategy and Engineering >> Direct: 320.256.0184 >> Cell: 320.247.3224 >> ben.wiechman(a)arvig.com >> 150 Second Street SW | Perham, MN 56573 >> <https://www.google.com/maps/search/150+Second+Street+SW+%7C+Perham,+MN+5657…> >> | arvig.com >> >> On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: >> >>> At 3:15pm I got: ETR of 1-2 hours. >>> >>> >>> >>> ------------------------------ >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of >>> James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>> *Sent:* Tuesday, April 13, 2021 3:31 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >>> >>> has anyone heard anything? my ticket has gone nowhere, and they tell me >>> they cant reach belle plaine, and are going to dispatch there. >>> >>> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >>> >>>> There are other circuits down on the Zayo path. Issues appear to be >>>> South of Worthington someplace. >>>> >>>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>>> james.urwiller(a)turnkey-telecom.com> wrote: >>>> >>>>> anyone else about 20 min ago now? >>>>> ------------------------------ >>>>> >>>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>>> >>>> >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 14, 2021

Re: Google's new 100G from Chicago

by James Urwiller

Mine is back up as of about an hour and 16 minutes ago (18:40 CST) *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Tuesday, April 13, 2021 7:10 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago So that’s three different Zayo issues in two weeks: - Under-communicated maintenance starting at 10:02 pm - Google 100G between Chicago and Minneapolis - 288 fiber cut between Alton, IA and Worthington, MN Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Andrew Hoyos *Sent:* Tuesday, April 13, 2021 6:53 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago Ironically, also a Zayo issue with a DWDM node in Chicago. On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Sent:* Saturday, April 10, 2021 11:47 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Jay Hanke *Sent:* Saturday, April 10, 2021 11:05 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf Of *Frank Bulk *Sent:* Saturday, April 10, 2021 10:52 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: · Clean the interface optic and your patching path to the MMR. · Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 14, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

So that’s three different Zayo issues in two weeks: * Under-communicated maintenance starting at 10:02 pm * Google 100G between Chicago and Minneapolis * 288 fiber cut between Alton, IA and Worthington, MN Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Tuesday, April 13, 2021 6:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Ironically, also a Zayo issue with a DWDM node in Chicago. On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Sent: Saturday, April 10, 2021 11:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 14, 2021

Re: Google's new 100G from Chicago

by Andrew Hoyos

Ironically, also a Zayo issue with a DWDM node in Chicago. > On Apr 13, 2021, at 6:48 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > > The Google NOC reported to me that the 100G link was back up this afternoon. > > According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > Sent: Saturday, April 10, 2021 11:47 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago > > Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke > Sent: Saturday, April 10, 2021 11:05 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago > > Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. > > On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Frank, > > Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. > > Jeremy > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk > Sent: Saturday, April 10, 2021 10:52 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] Google's new 100G from Chicago > > Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: > > ======================== > We have informed you via G-127797702 > > Hello MICE (Midwest Internet Cooperative Exchange), > > We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). > Please check the below list: > Google ASN > Your ASN > Google Address > Your Address > Your Demarc > Metro > Problem > Speed > 15169 > PX.MICE.ORD > 206.108.255.167 > Peering Exchange > AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 > ORD > Down > 100 Gbps > > Tips for troubleshooting physical link problems: > > · Clean the interface optic and your patching path to the MMR. > · Replace the interface optic. > > Please check your link(s) and attempt to resolve the problems described above. > > Thanks! > ======================== > > Is someone from MICE performing the requested actions and/or looking into this? > > Kind regards, > > Frank Bulk > FiberNet (ASN18883) > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

Tue Apr 13 23:25:20 GMT 2021 Current update: They have 73-74 spliced. 133-134 will be up shortly. They should have 36 fibers done in the next 20 minutes. No Change to ETR at this time. That was the last we got from them. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 13, 2021, 18:49 Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > This was our update at 5 PM. > Just hung up with Zayo - their 288F was hit by a plow paralleling the Zayo > cable and and cut in several spots making the restoration much more > difficult than expected. Zayo expects to see restoral on the cable in the > next 60-90 minutes... > > Working on an update from Zayo again given the timeline. > > Ben Wiechman > Director of IP Strategy and Engineering > Direct: 320.256.0184 > Cell: 320.247.3224 > ben.wiechman(a)arvig.com > 150 Second Street SW | Perham, MN 56573 | arvig.com > > On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: > >> At 3:15pm I got: ETR of 1-2 hours. >> >> >> >> ------------------------------ >> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James >> Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >> *Sent:* Tuesday, April 13, 2021 3:31 PM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine >> >> has anyone heard anything? my ticket has gone nowhere, and they tell me >> they cant reach belle plaine, and are going to dispatch there. >> >> On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: >> >>> There are other circuits down on the Zayo path. Issues appear to be >>> South of Worthington someplace. >>> >>> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >>> james.urwiller(a)turnkey-telecom.com> wrote: >>> >>>> anyone else about 20 min ago now? >>>> ------------------------------ >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >

April 13, 2021

Re: lost mice thur belle plaine

by Ben Wiechman

This was our update at 5 PM. Just hung up with Zayo - their 288F was hit by a plow paralleling the Zayo cable and and cut in several spots making the restoration much more difficult than expected. Zayo expects to see restoral on the cable in the next 60-90 minutes... Working on an update from Zayo again given the timeline. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 13, 2021, 15:35 Justin Krejci <JKrejci(a)usinternet.com> wrote: > At 3:15pm I got: ETR of 1-2 hours. > > > > ------------------------------ > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James > Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> > *Sent:* Tuesday, April 13, 2021 3:31 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost mice thur belle plaine > > has anyone heard anything? my ticket has gone nowhere, and they tell me > they cant reach belle plaine, and are going to dispatch there. > > On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io> wrote: > >> There are other circuits down on the Zayo path. Issues appear to be South >> of Worthington someplace. >> >> On Mon, Apr 12, 2021, 9:22 PM James Urwiller < >> james.urwiller(a)turnkey-telecom.com> wrote: >> >>> anyone else about 20 min ago now? >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

The Google NOC reported to me that the 100G link was back up this afternoon. According to our logs, our IPv4 and IPv6 BGP sessions came back up at 3:36 and 3:37 pm, respectively. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Sent: Saturday, April 10, 2021 11:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [External GPC] Re: [MICE-DISCUSS] Google's new 100G from Chicago Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Frank Bulk

>From our third-party BGP monitoring it looks like Great Plains' BGP session came up at 6:36 pm, so I'm guessing fiber is getting spliced. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Tuesday, April 13, 2021 3:35 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine At 3:15pm I got: ETR of 1-2 hours. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM<mailto:james.urwiller@TURNKEY-TELECOM.COM>> Sent: Tuesday, April 13, 2021 3:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine has anyone heard anything? my ticket has gone nowhere, and they tell me they cant reach belle plaine, and are going to dispatch there. On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: There are other circuits down on the Zayo path. Issues appear to be South of Worthington someplace. On Mon, Apr 12, 2021, 9:22 PM James Urwiller <james.urwiller(a)turnkey-telecom.com<mailto:james.urwiller@turnkey-telecom.com>> wrote: anyone else about 20 min ago now? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021

Re: lost mice thur belle plaine

by Justin Krejci

At 3:15pm I got: ETR of 1-2 hours. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Sent: Tuesday, April 13, 2021 3:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost mice thur belle plaine has anyone heard anything? my ticket has gone nowhere, and they tell me they cant reach belle plaine, and are going to dispatch there. On Mon, Apr 12, 2021 at 21:43 Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: There are other circuits down on the Zayo path. Issues appear to be South of Worthington someplace. On Mon, Apr 12, 2021, 9:22 PM James Urwiller <james.urwiller(a)turnkey-telecom.com<mailto:james.urwiller@turnkey-telecom.com>> wrote: anyone else about 20 min ago now? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2021