April 2018 - mice-discuss - lists.micemn.net

Re: MICE Google Global cache issues?

by Jeremy Lumby

I noticed my Google v6 traffic shifted to IX-Denver. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Thursday, April 05, 2018 10:33 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Google Global cache issues? A heads up; We asked Google for an update on our ticket this evening. IPv6 problems for the MICE GGC seem to have returned today, based I our graphs probably around 10:00 am. Therefore, Google has temporarily redirected IPv6 traffic away from the MICE GGC, to Chicago in our case and I suspect most cases. Also, Google has escalated the issue internally, and with their switch vendor. We have some reports of IPv6 packet loss possibly in Chicago and probably earlier today. We'll be keeping an eye on this tomorrow. Also, I went back and looked at graphs for IPv6 traffic from the MICE GGC to us and this problem probably started last Wednesday. In most cases, I think Happy Eye Balls glossed over this, but if applications or OSes didn't do Happy Eye Balls then things probably broke. The saga continues... On Wed, Apr 4, 2018 at 12:17 PM, Colin Murphy <murphy(a)umn.edu> wrote: I saw that as well and we can now use YouTube and google play store. Thanks. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 On Wed, Apr 4, 2018 at 12:15 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: While I have not heard anything back from Google yet, I saw BGP to Google bounce, and I can now download app updates from the Google Play Store. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 04, 2018 9:32 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] MICE Google Global cache issues? I am seeing the same things. I just shot off an email to my contacts at Google From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Murphy Sent: Wednesday, April 04, 2018 9:16 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Google Global cache issues? Hi, Since Monday we've been having issues with mobile devices accessing YouTube and Google Play store via IPv6. Our symptoms with YouTube app is mostly with Apple IOS devices. When a user loads a video, the ads will play fine and then the video will start and play for 5 seconds and stop. Browsing youtube via a web browser on the IOS devices seems fine. With Android devices, users can't download apps from the play store. Is anyone else experiencing these issues with their peering with the Google Global Cache at MICE? Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 6, 2018

Re: MICE Google Global cache issues?

by David Farmer

A heads up; We asked Google for an update on our ticket this evening. IPv6 problems for the MICE GGC seem to have returned today, based I our graphs probably around 10:00 am. Therefore, Google has temporarily redirected IPv6 traffic away from the MICE GGC, to Chicago in our case and I suspect most cases. Also, Google has escalated the issue internally, and with their switch vendor. We have some reports of IPv6 packet loss possibly in Chicago and probably earlier today. We'll be keeping an eye on this tomorrow. Also, I went back and looked at graphs for IPv6 traffic from the MICE GGC to us and this problem probably started last Wednesday. In most cases, I think Happy Eye Balls glossed over this, but if applications or OSes didn't do Happy Eye Balls then things probably broke. The saga continues... On Wed, Apr 4, 2018 at 12:17 PM, Colin Murphy <murphy(a)umn.edu> wrote: > I saw that as well and we can now use YouTube and google play store. > Thanks. > > > Thanks, > > Colin Murphy > Sr. Network Engineer > Office of Information Technology > University of Minnesota > Northern Lights Gigapop > Desk: 612-626-0870 > > On Wed, Apr 4, 2018 at 12:15 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > >> While I have not heard anything back from Google yet, I saw BGP to Google >> bounce, and I can now download app updates from the Google Play Store. >> >> >> >> *From:* Jeremy Lumby [mailto:jlumby@mnvoip.com] >> *Sent:* Wednesday, April 04, 2018 9:32 AM >> *To:* MICE Discuss >> *Subject:* RE: [MICE-DISCUSS] MICE Google Global cache issues? >> >> >> >> I am seeing the same things. I just shot off an email to my contacts at >> Google >> >> >> >> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET >> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Colin Murphy >> *Sent:* Wednesday, April 04, 2018 9:16 AM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* [MICE-DISCUSS] MICE Google Global cache issues? >> >> >> >> Hi, >> >> Since Monday we've been having issues with mobile devices accessing >> YouTube and Google Play store via IPv6. Our symptoms with YouTube app is >> mostly with Apple IOS devices. When a user loads a video, the ads will >> play fine and then the video will start and play for 5 seconds and stop. >> Browsing youtube via a web browser on the IOS devices seems fine. >> >> >> >> With Android devices, users can't download apps from the play store. >> >> >> >> Is anyone else experiencing these issues with their peering with the >> Google Global Cache at MICE? >> >> >> >> >> Thanks, >> >> Colin Murphy >> Sr. Network Engineer >> Office of Information Technology >> University of Minnesota >> Northern Lights Gigapop >> Desk: 612-626-0870 >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 6, 2018

Re: MICE Google Global cache issues?

by David Farmer

FYI, there is a new URL to determine the GGC node you are redirected to; http://redirector.googlevideo.com/report_mapping The old URL has certificate issues, and should not be used any longer; Thought I would get the word out. On Wed, Apr 4, 2018 at 12:17 PM, Colin Murphy <murphy(a)umn.edu> wrote: > I saw that as well and we can now use YouTube and google play store. > Thanks. > > > Thanks, > > Colin Murphy > Sr. Network Engineer > Office of Information Technology > University of Minnesota > Northern Lights Gigapop > Desk: 612-626-0870 > > On Wed, Apr 4, 2018 at 12:15 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > >> While I have not heard anything back from Google yet, I saw BGP to Google >> bounce, and I can now download app updates from the Google Play Store. >> >> >> >> *From:* Jeremy Lumby [mailto:jlumby@mnvoip.com] >> *Sent:* Wednesday, April 04, 2018 9:32 AM >> *To:* MICE Discuss >> *Subject:* RE: [MICE-DISCUSS] MICE Google Global cache issues? >> >> >> >> I am seeing the same things. I just shot off an email to my contacts at >> Google >> >> >> >> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET >> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Colin Murphy >> *Sent:* Wednesday, April 04, 2018 9:16 AM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* [MICE-DISCUSS] MICE Google Global cache issues? >> >> >> >> Hi, >> >> Since Monday we've been having issues with mobile devices accessing >> YouTube and Google Play store via IPv6. Our symptoms with YouTube app is >> mostly with Apple IOS devices. When a user loads a video, the ads will >> play fine and then the video will start and play for 5 seconds and stop. >> Browsing youtube via a web browser on the IOS devices seems fine. >> >> >> >> With Android devices, users can't download apps from the play store. >> >> >> >> Is anyone else experiencing these issues with their peering with the >> Google Global Cache at MICE? >> >> >> >> >> Thanks, >> >> Colin Murphy >> Sr. Network Engineer >> Office of Information Technology >> University of Minnesota >> Northern Lights Gigapop >> Desk: 612-626-0870 >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 5, 2018

Re: MICE Google Global cache issues?

by Colin Murphy

I saw that as well and we can now use YouTube and google play store. Thanks. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 On Wed, Apr 4, 2018 at 12:15 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > While I have not heard anything back from Google yet, I saw BGP to Google > bounce, and I can now download app updates from the Google Play Store. > > > > *From:* Jeremy Lumby [mailto:jlumby@mnvoip.com] > *Sent:* Wednesday, April 04, 2018 9:32 AM > *To:* MICE Discuss > *Subject:* RE: [MICE-DISCUSS] MICE Google Global cache issues? > > > > I am seeing the same things. I just shot off an email to my contacts at > Google > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Colin Murphy > *Sent:* Wednesday, April 04, 2018 9:16 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] MICE Google Global cache issues? > > > > Hi, > > Since Monday we've been having issues with mobile devices accessing > YouTube and Google Play store via IPv6. Our symptoms with YouTube app is > mostly with Apple IOS devices. When a user loads a video, the ads will > play fine and then the video will start and play for 5 seconds and stop. > Browsing youtube via a web browser on the IOS devices seems fine. > > > > With Android devices, users can't download apps from the play store. > > > > Is anyone else experiencing these issues with their peering with the > Google Global Cache at MICE? > > > > > Thanks, > > Colin Murphy > Sr. Network Engineer > Office of Information Technology > University of Minnesota > Northern Lights Gigapop > Desk: 612-626-0870 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 4, 2018

Re: MICE Google Global cache issues?

by Jeremy Lumby

While I have not heard anything back from Google yet, I saw BGP to Google bounce, and I can now download app updates from the Google Play Store. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 04, 2018 9:32 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] MICE Google Global cache issues? I am seeing the same things. I just shot off an email to my contacts at Google From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Murphy Sent: Wednesday, April 04, 2018 9:16 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Google Global cache issues? Hi, Since Monday we've been having issues with mobile devices accessing YouTube and Google Play store via IPv6. Our symptoms with YouTube app is mostly with Apple IOS devices. When a user loads a video, the ads will play fine and then the video will start and play for 5 seconds and stop. Browsing youtube via a web browser on the IOS devices seems fine. With Android devices, users can't download apps from the play store. Is anyone else experiencing these issues with their peering with the Google Global Cache at MICE? Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 4, 2018

Re: MICE Google Global cache issues?

by Colin Murphy

Thanks Jeremy, please let me know if you hear anything back. I also sent a note to noc(a)google.com as well. If I hear anything, I'll send a note to this group. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 On Wed, Apr 4, 2018 at 9:32 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I am seeing the same things. I just shot off an email to my contacts at > Google > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Colin Murphy > *Sent:* Wednesday, April 04, 2018 9:16 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] MICE Google Global cache issues? > > > > Hi, > > Since Monday we've been having issues with mobile devices accessing > YouTube and Google Play store via IPv6. Our symptoms with YouTube app is > mostly with Apple IOS devices. When a user loads a video, the ads will > play fine and then the video will start and play for 5 seconds and stop. > Browsing youtube via a web browser on the IOS devices seems fine. > > > > With Android devices, users can't download apps from the play store. > > > > Is anyone else experiencing these issues with their peering with the > Google Global Cache at MICE? > > > > > Thanks, > > Colin Murphy > Sr. Network Engineer > Office of Information Technology > University of Minnesota > Northern Lights Gigapop > Desk: 612-626-0870 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 4, 2018

Re: MICE Google Global cache issues?

by Jeremy Lumby

I am seeing the same things. I just shot off an email to my contacts at Google From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Murphy Sent: Wednesday, April 04, 2018 9:16 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Google Global cache issues? Hi, Since Monday we've been having issues with mobile devices accessing YouTube and Google Play store via IPv6. Our symptoms with YouTube app is mostly with Apple IOS devices. When a user loads a video, the ads will play fine and then the video will start and play for 5 seconds and stop. Browsing youtube via a web browser on the IOS devices seems fine. With Android devices, users can't download apps from the play store. Is anyone else experiencing these issues with their peering with the Google Global Cache at MICE? Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 4, 2018

MICE Google Global cache issues?

by Colin Murphy

Hi, Since Monday we've been having issues with mobile devices accessing YouTube and Google Play store via IPv6. Our symptoms with YouTube app is mostly with Apple IOS devices. When a user loads a video, the ads will play fine and then the video will start and play for 5 seconds and stop. Browsing youtube via a web browser on the IOS devices seems fine. With Android devices, users can't download apps from the play store. Is anyone else experiencing these issues with their peering with the Google Global Cache at MICE? Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870

April 4, 2018

Re: Broadcast Traffic

by Ben Wiechman

Agreed on both the passive configuration for the route servers and implementation suggestions. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 3, 2018 at 9:23 AM, Michael Hare <michael.hare(a)wisc.edu> wrote: > My 2c is proceed but consider the following > > 1) space out changes the route servers by a day or so. > 2) force reset bgp on the changed route server. > > As someone who has previously screwed up control plane BGP filters (such > that sessions only came up if remove side was active), I believe step #2 > will shake out anyone who has their control plane filters wrong. Not > really MICE's fault per se, but a nice gesture. > > -Michael > > >>-----Original Message----- > >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf > >>Of Richard Laager > >>Sent: Tuesday, April 03, 2018 12:56 AM > >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > >>Subject: Re: [MICE-DISCUSS] Broadcast Traffic > >> > >>On 04/02/2018 11:23 PM, Andrew Hoyos wrote: > >>> I’d vote for passive BGP sessions (v4/v6) on the route servers across > the > >>board, already established or not. Easy solution to the issue at hand > and sane > >>default. “Quiet unless spoken to”. > >> > >>This is what I was thinking too. > >> > >>-- > >>Richard >

April 3, 2018

Re: Broadcast Traffic

by Michael Hare

My 2c is proceed but consider the following 1) space out changes the route servers by a day or so. 2) force reset bgp on the changed route server. As someone who has previously screwed up control plane BGP filters (such that sessions only came up if remove side was active), I believe step #2 will shake out anyone who has their control plane filters wrong. Not really MICE's fault per se, but a nice gesture. -Michael >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Richard Laager >>Sent: Tuesday, April 03, 2018 12:56 AM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Broadcast Traffic >> >>On 04/02/2018 11:23 PM, Andrew Hoyos wrote: >>> I’d vote for passive BGP sessions (v4/v6) on the route servers across the >>board, already established or not. Easy solution to the issue at hand and sane >>default. “Quiet unless spoken to”. >> >>This is what I was thinking too. >> >>-- >>Richard

April 3, 2018