Any word on what's going on at Google? I'm guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls@cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>
We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago.
On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls@COOPERATIVE-NETWORKS.COM> wrote:
Any word on what’s going on at Google? I’m guessing someone had a bad day.
Dean
Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls@cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>
<image003.jpg>
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
I’m glad someone asked, because I was wondering why our peer went down. Makes sense. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Monday, November 18, 2019 2:04 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago. On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls@COOPERATIVE-NETWORKS.COM<mailto:dean.bahls@COOPERATIVE-NETWORKS.COM>> wrote: Any word on what’s going on at Google? I’m guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls@cooperative-networks.com<mailto:dean.bahls@cooperative-networks.com> <image003.jpg> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Did you first see issues around 12:30? Access to some Google services dropped for me around then. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman@arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Mon, Nov 18, 2019 at 3:11 PM Jeremy Schmeichel < Jeremy.Schmeichel@vastbroadband.com> wrote:
I’m glad someone asked, because I was wondering why our peer went down. Makes sense.
*From:* MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *On Behalf Of *Andrew Hoyos *Sent:* Monday, November 18, 2019 2:04 PM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Google
We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI).
I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago.
On Nov 18, 2019, at 3:01 PM, Dean Bahls < dean.bahls@COOPERATIVE-NETWORKS.COM> wrote:
Any word on what’s going on at Google? I’m guessing someone had a bad day.
Dean
*Dean Bahls*
Network Services Manager | Cooperative Network Services, LLC
14 Main St SW | PO Box J | Menahga, MN 56464
218.346.3635 (Direct) | 218.564.3000 (Main Office)
dean.bahls@cooperative-networks.com
<image003.jpg>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
The latest update we had from Level3 (hey, look, a non Zayo related fiber issue, haha!) *** CASCADED EXTERNAL NOTES 18-Nov-2019 20:39:54 GMT From CASE: 17573224 - Event The Transport NOC reports that light level testing indicates a potential fault 13.8 kilometers from the testing site. The Outside Plant Engineer (OSPE) was engaged, provided the test results, and requested to assist with locating the exact point of failure. The OSPE has dispatched Field Operations personnel to the approximate location of the fault to begin searching for the physical damage point. As a precaution, construction teams have been engaged and dispatched to the approximate location of the damage. An ETA of approximately 23:00 GMT has been provided for the construction teams.
On Nov 18, 2019, at 3:11 PM, Jeremy Schmeichel <Jeremy.Schmeichel@VASTBROADBAND.COM> wrote:
I’m glad someone asked, because I was wondering why our peer went down. Makes sense.
From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Andrew Hoyos Sent: Monday, November 18, 2019 2:04 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Google
We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago.
On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls@COOPERATIVE-NETWORKS.COM <mailto:dean.bahls@COOPERATIVE-NETWORKS.COM>> wrote:
Any word on what’s going on at Google? I’m guessing someone had a bad day.
Dean
Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls@cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>
<image003.jpg>
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
I wonder if Zayo cut it…. JUST KIDDING! 😊 23:00 ETA is not good. I’m hearing some pretty ugly stories of folks suffering with this cut. Tells you how much folks rely on MICE nowadays!! Dean From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Monday, November 18, 2019 3:18 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google The latest update we had from Level3 (hey, look, a non Zayo related fiber issue, haha!) *** CASCADED EXTERNAL NOTES 18-Nov-2019 20:39:54 GMT From CASE: 17573224 - Event The Transport NOC reports that light level testing indicates a potential fault 13.8 kilometers from the testing site. The Outside Plant Engineer (OSPE) was engaged, provided the test results, and requested to assist with locating the exact point of failure. The OSPE has dispatched Field Operations personnel to the approximate location of the fault to begin searching for the physical damage point. As a precaution, construction teams have been engaged and dispatched to the approximate location of the damage. An ETA of approximately 23:00 GMT has been provided for the construction teams. On Nov 18, 2019, at 3:11 PM, Jeremy Schmeichel <Jeremy.Schmeichel@VASTBROADBAND.COM <mailto:Jeremy.Schmeichel@VASTBROADBAND.COM> > wrote: I’m glad someone asked, because I was wondering why our peer went down. Makes sense. From: MICE Discuss < <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Monday, November 18, 2019 2:04 PM To: <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago. On Nov 18, 2019, at 3:01 PM, Dean Bahls < <mailto:dean.bahls@COOPERATIVE-NETWORKS.COM> dean.bahls@COOPERATIVE-NETWORKS.COM> wrote: Any word on what’s going on at Google? I’m guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) <mailto:dean.bahls@cooperative-networks.com> dean.bahls@cooperative-networks.com <image003.jpg> _____ To unsubscribe from the MICE-DISCUSS list, click the following link: <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. _____ To unsubscribe from the MICE-DISCUSS list, click the following link: <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
Ahh, thanks for setting me straight. That's actually pretty impressive if they can get it fixed that quickly. Dean -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Mike Johnston Sent: Monday, November 18, 2019 3:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google On 2019-11-18 15:23, Dean Bahls wrote:
23:00 ETA is not good.
The ETA was stated as GMT, not local time. 23:00 GMT is 17:00 (5:00pm) local time.
participants (5)
-
Andrew Hoyos -
Ben Wiechman -
Dean Bahls -
Jeremy Schmeichel -
Mike Johnston