December 2021 - mice-discuss - lists.micemn.net

FYI, inbound Google [15169] issues via MICE for AS3128 that started ~15:47 CST

by Michael Hare

Hi everyone- FYI in case others are impacted. We had many reports inside AS3128 of problems with Google services [AS15169] beginning around 15:47 CST. I won't bore folks with all details but in short, we observed what looked like a hashing issue. For example, our ip w.x.y.z could ping 8.8.8.8, but not w.x.y.z+1. I was able to reproduce easily myself with 8.8.4.4. Our primary outbound path to 8.8.4.0/24 was MICE. Swinging our outbound path didn't fix anything, but I was able to work around our issues at ~17:38CST by sending 0:15169 to the MICE RR. The relief was basically immediate, so the problem appears to have been inbound from Google via MICE. Hard to say where the issue is/was. I am on a remote switch, but no other issues with other MICE ASN were reported inside AS3128, FWIW. I would be interested to hear if anyone else had/has a problem or if anyone has a suggestion on how to make positive contact with Google on this as we are not direct GGC or direct PNI users. For example, is peering(a)google.com my best/only bet? I know some have suggested IRC in the past, I do not run in those circles but I guess I will learn to if that's my only real option. Thanks- -Michael

Dec. 10, 2021

Re: Several peers down since 3:16 am

by Frank Bulk

All six sessions came back up around 1:05 pm. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Thursday, December 9, 2021 11:16 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am Both of our yahoo peers are down since 3:16 AM as well. 206.108.255.58 10310 CONN 7h57m59s 0 0 0 23 206.108.255.59 10310 CONN 7h56m46s 0 0 0 23 Justin Huffman Long Lines Broadband 712-271-5902 (W) jhuffman(a)longlines.biz -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Thursday, December 9, 2021 11:15 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. For whatever reason, my google peering never went through, so, I couldn't tell you. I still get the maintenance notices. I suppose I should bug them again. On Thu, Dec 09, 2021 at 05:11:38PM +0000, Frank Bulk wrote: >Thanks for sharing what you're seeing. Is your Google peer down? > >My Google peer did not restore after their maintenance ... and it was scheduled to start 9:55 am, while we saw it go down at 3:16 am. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug >McIntyre >Sent: Thursday, December 9, 2021 11:00 AM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am > >On Thu, Dec 09, 2021 at 04:58:05PM +0000, Frank Bulk wrote: >>We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: >>206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >> >>206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >>2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >> >>Anyone else seeing this? > > >Only one of those Yahoo peers (.59) is down for me. > >Google had emergancy maintenance this morning. >Their notice said they had finished up recently though. >

Dec. 9, 2021

Re: Several peers down since 3:16 am

by Justin Huffman

Both of our yahoo peers are down since 3:16 AM as well. 206.108.255.58 10310 CONN 7h57m59s 0 0 0 23 206.108.255.59 10310 CONN 7h56m46s 0 0 0 23 Justin Huffman Long Lines Broadband 712-271-5902 (W) jhuffman(a)longlines.biz -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Thursday, December 9, 2021 11:15 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. For whatever reason, my google peering never went through, so, I couldn't tell you. I still get the maintenance notices. I suppose I should bug them again. On Thu, Dec 09, 2021 at 05:11:38PM +0000, Frank Bulk wrote: >Thanks for sharing what you're seeing. Is your Google peer down? > >My Google peer did not restore after their maintenance ... and it was scheduled to start 9:55 am, while we saw it go down at 3:16 am. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug >McIntyre >Sent: Thursday, December 9, 2021 11:00 AM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am > >On Thu, Dec 09, 2021 at 04:58:05PM +0000, Frank Bulk wrote: >>We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: >>206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >> >>206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >>2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >> >>Anyone else seeing this? > > >Only one of those Yahoo peers (.59) is down for me. > >Google had emergancy maintenance this morning. >Their notice said they had finished up recently though. >

Dec. 9, 2021

Re: Several peers down since 3:16 am

by Doug McIntyre

For whatever reason, my google peering never went through, so, I couldn't tell you. I still get the maintenance notices. I suppose I should bug them again. On Thu, Dec 09, 2021 at 05:11:38PM +0000, Frank Bulk wrote: >Thanks for sharing what you're seeing. Is your Google peer down? > >My Google peer did not restore after their maintenance ... and it was scheduled to start 9:55 am, while we saw it go down at 3:16 am. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre >Sent: Thursday, December 9, 2021 11:00 AM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am > >On Thu, Dec 09, 2021 at 04:58:05PM +0000, Frank Bulk wrote: >>We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: >>206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >>2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >> >>206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >>2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >> >>Anyone else seeing this? > > >Only one of those Yahoo peers (.59) is down for me. > >Google had emergancy maintenance this morning. >Their notice said they had finished up recently though. >

Dec. 9, 2021

Re: Several peers down since 3:16 am

by Frank Bulk

Thanks for sharing what you're seeing. Is your Google peer down? My Google peer did not restore after their maintenance ... and it was scheduled to start 9:55 am, while we saw it go down at 3:16 am. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Thursday, December 9, 2021 11:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Several peers down since 3:16 am On Thu, Dec 09, 2021 at 04:58:05PM +0000, Frank Bulk wrote: >We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: >206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) > >206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) > >Anyone else seeing this? Only one of those Yahoo peers (.59) is down for me. Google had emergancy maintenance this morning. Their notice said they had finished up recently though.

Dec. 9, 2021

Re: Several peers down since 3:16 am

by Doug McIntyre

On Thu, Dec 09, 2021 at 04:58:05PM +0000, Frank Bulk wrote: >We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: >206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) >2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) >2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) > >206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) >2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) > >Anyone else seeing this? Only one of those Yahoo peers (.59) is down for me. Google had emergancy maintenance this morning. Their notice said they had finished up recently though.

Dec. 9, 2021

Several peers down since 3:16 am

by Frank Bulk

We're seeing our BGP sessions to Yahoo and Google down since 3:16 am: 206.108.255.58 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) 206.108.255.59 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) 2001:504:27::2846:0:1 (AS10310-1.micemn.net) (AS 10310 YAHOO-1, US) 2001:504:27::2846:0:2 (AS10310-2.micemn.net) (AS 10310 YAHOO-1, US) 206.108.255.167 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) 2001:504:27::3b41:0:2 (AS15169-2.micemn.net) (AS 15169 GOOGLE, US) Anyone else seeing this? Frank

Dec. 9, 2021

Re: Netflix Peering Lost

by Travis Fincham

Looks good for us now as well Thanks, Travis From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Josh Kruckow <jkruckow(a)ACENTEK.NET> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Tuesday, December 7, 2021 at 2:17 PM To: "MICE-DISCUSS(a)LISTS.IPHOUSE.NET" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [EXTERNAL]Re: [MICE-DISCUSS] Netflix Peering Lost [EXTERNAL EMAIL]: This email originated from outside of Key Cooperative. Please verify that that you recognize the sender and know is the content is safe. I opened up a ticket with Netflix and they were looking into it. It now looks like they resolved the issue. Peering is up on v4 and v6 and routes are coming through. Thanks, Joshua Kruckow Network Operations Specialist [cid:image001.png@01D7EB7B.307A86C0] 888-404-4940 Office 507-896-6202 Desk jkruckow(a)AcenTek.net<mailto:jkruckow@acentek.net> www.AcenTek.net From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Tuesday, December 7, 2021 1:55 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Netflix Peering Lost EXTERNAL: This email originated from a sender outside AcenTek. Do not click links or open attachments unless you recognize the sender and know the content is safe. One of the two Netflix links to MICE is down. Times line up. On Tue, Dec 7, 2021 at 1:53 PM Josh Kruckow <jkruckow(a)acentek.net<mailto:jkruckow@acentek.net>> wrote: Anyone else having issues? We are seeing our Netflix peering with 206.108.255.35 and 2001:504:27::b5a:0:1 down. Occurred approximately 5:38 AM Central time. Regards, Joshua Kruckow Network Operations Specialist [cid:image001.png@01D7EB7B.307A86C0] 888-404-4940 Office 507-896-6202 Desk jkruckow(a)AcenTek.net<mailto:jkruckow@acentek.net> www.AcenTek.net<http://www.AcenTek.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ NOTICE: This message (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. § 2510-2521, is CONFIDENTIAL, and is intended only for the individual to whom it is addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you have received this message in error and that any review, retention, dissemination, distribution, copying or use of this communication is strictly prohibited. Please reply to the sender that you received this message in error and then delete it.

Dec. 7, 2021

Re: Netflix Peering Lost

by Josh Kruckow

I opened up a ticket with Netflix and they were looking into it. It now looks like they resolved the issue. Peering is up on v4 and v6 and routes are coming through. Thanks, Joshua Kruckow Network Operations Specialist [cid:image001.png@01D7EB75.2EC173D0] 888-404-4940 Office 507-896-6202 Desk jkruckow(a)AcenTek.net<mailto:jkruckow@acentek.net> www.AcenTek.net From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Tuesday, December 7, 2021 1:55 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Netflix Peering Lost EXTERNAL: This email originated from a sender outside AcenTek. Do not click links or open attachments unless you recognize the sender and know the content is safe. One of the two Netflix links to MICE is down. Times line up. On Tue, Dec 7, 2021 at 1:53 PM Josh Kruckow <jkruckow(a)acentek.net<mailto:jkruckow@acentek.net>> wrote: Anyone else having issues? We are seeing our Netflix peering with 206.108.255.35 and 2001:504:27::b5a:0:1 down. Occurred approximately 5:38 AM Central time. Regards, Joshua Kruckow Network Operations Specialist [cid:image001.png@01D7EB75.2EC173D0] 888-404-4940 Office 507-896-6202 Desk jkruckow(a)AcenTek.net<mailto:jkruckow@acentek.net> www.AcenTek.net<http://www.AcenTek.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 7, 2021

Re: Netflix Peering Lost

by Jay Hanke

One of the two Netflix links to MICE is down. Times line up. On Tue, Dec 7, 2021 at 1:53 PM Josh Kruckow <jkruckow(a)acentek.net> wrote: > > Anyone else having issues? > > We are seeing our Netflix peering with 206.108.255.35 and > 2001:504:27::b5a:0:1 down. > > Occurred approximately 5:38 AM Central time. > > > > Regards, > > > > *Joshua Kruckow* > > Network Operations Specialist > > 888-404-4940 Office > > 507-896-6202 Desk > > *jkruckow(a)AcenTek.net* <jkruckow(a)acentek.net> > > *www.AcenTek.net* <http://www.AcenTek.net> > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Dec. 7, 2021