And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back. But Google still seems to be down. On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu> wrote:

https://www.cloudflarestatus.com/

listed Minneapolis as Re-routed, but then a minute ago it changed to Operational

On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io> wrote:

...

Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on?

-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Since Cloudflare is in the building, I am assuming the issue impacted their cache fill side. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Keith Wirch Sent: Wednesday, August 05, 2020 3:17 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXT]: Re: [MICE-DISCUSS] MICE Content Issues Our direct peering with Cloudflare never went down. BGP state = Established, up for 12w0d Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of David Farmer <0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> Date: Wednesday, August 5, 2020 at 3:16 PM To: "MICE-DISCUSS@LISTS.IPHOUSE.NET" <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXT]: Re: [MICE-DISCUSS] MICE Content Issues And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back. But Google still seems to be down. On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu> wrote: https://www.cloudflarestatus.com/ listed Minneapolis as Re-routed, but then a minute ago it changed to Operational On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io> wrote: Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on? -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Yes, we also noticed our IPv4/IPv6 BGP peerings to Google bounced about 10 min ago (2:23 PM CST): BGP state = Established, up for 00:09:58 Jesse From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Austin Gillmann Sent: Wednesday, August 5, 2020 3:31 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXT]: Re: [MICE-DISCUSS] MICE Content Issues Cloudflare never went down for us as well, using Hurricane Electric's looking glass it shows they have the Google session established again as of 6 minutes ago On Wed, Aug 5, 2020 at 3:27 PM Keith Wirch <kwirch@atomicdata.com<mailto:kwirch@atomicdata.com>> wrote: Our direct peering with Cloudflare never went down. BGP state = Established, up for 12w0d Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of David Farmer <0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET<mailto:0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET>> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Date: Wednesday, August 5, 2020 at 3:16 PM To: "MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>" <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXT]: Re: [MICE-DISCUSS] MICE Content Issues And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back. But Google still seems to be down. On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu<mailto:farmer@umn.edu>> wrote: https://www.cloudflarestatus.com/ listed Minneapolis as Re-routed, but then a minute ago it changed to Operational On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io<mailto:jayhanke@southfront.io>> wrote: Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on? -- Jay Hanke, President South Front Networks jayhanke@southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Our Google BGP session came bak 10 minutes ago but not receiving any routes yet. On Wed, Aug 5, 2020 at 3:30 PM Austin Gillmann <ji@epik.com> wrote:

Cloudflare never went down for us as well, using Hurricane Electric's looking glass it shows they have the Google session established again as of 6 minutes ago

On Wed, Aug 5, 2020 at 3:27 PM Keith Wirch <kwirch@atomicdata.com> wrote:

...

Our direct peering with Cloudflare never went down.

BGP state = Established, up for 12w0d

* Keith Wirch* Infrastructure Engineer ---- * Atomic Data® *// *Safe. Simple. Smart.®* 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

*From: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of David Farmer <0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET> *Reply-To: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Date: *Wednesday, August 5, 2020 at 3:16 PM *To: *"MICE-DISCUSS@LISTS.IPHOUSE.NET" <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Subject: *[EXT]: Re: [MICE-DISCUSS] MICE Content Issues

And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back.

But Google still seems to be down.

On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu> wrote:

https://www.cloudflarestatus.com/

listed Minneapolis as Re-routed, but then a minute ago it changed to Operational

On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io> wrote:

Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on?

--

Jay Hanke, President

South Front Networks

jayhanke@southfront.io

Phone 612-204-0000

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

--

=============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

--

=============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

In the last couple of minutes, I'm seeing the same thing and the stats page changed to operational again for Minneapolis On Wed, Aug 5, 2020 at 3:38 PM Keith Wirch <kwirch@atomicdata.com> wrote:

For a while there I was only seeing 3 routes from CF. Now I am see 297 again.

* Keith Wirch* Infrastructure Engineer ---- * Atomic Data® *// *Safe. Simple. Smart.®* 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

*From: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of David Farmer <0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET> *Reply-To: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Date: *Wednesday, August 5, 2020 at 3:35 PM *To: *"MICE-DISCUSS@LISTS.IPHOUSE.NET" <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Subject: *Re: [MICE-DISCUSS] [EXT]: Re: [MICE-DISCUSS] MICE Content Issues

Our Google BGP session came bak 10 minutes ago but not receiving any routes yet.

On Wed, Aug 5, 2020 at 3:30 PM Austin Gillmann <ji@epik.com> wrote:

Cloudflare never went down for us as well, using Hurricane Electric's looking glass it shows they have the Google session established again as of 6 minutes ago

On Wed, Aug 5, 2020 at 3:27 PM Keith Wirch <kwirch@atomicdata.com> wrote:

Our direct peering with Cloudflare never went down.

BGP state = Established, up for 12w0d

* Keith Wirch* Infrastructure Engineer ---- * Atomic Data**®* // *Safe. Simple. Smart.**®* 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

*From: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of David Farmer <0000000e7948cb21-dmarc-request@LISTS.IPHOUSE.NET> *Reply-To: *MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Date: *Wednesday, August 5, 2020 at 3:16 PM *To: *"MICE-DISCUSS@LISTS.IPHOUSE.NET" <MICE-DISCUSS@LISTS.IPHOUSE.NET> *Subject: *[EXT]: Re: [MICE-DISCUSS] MICE Content Issues

And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back.

But Google still seems to be down.

On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu> wrote:

https://www.cloudflarestatus.com/

listed Minneapolis as Re-routed, but then a minute ago it changed to Operational

On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io> wrote:

Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on?

--

Jay Hanke, President

South Front Networks

jayhanke@southfront.io

Phone 612-204-0000

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

--

=============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

--

=============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

--

=============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

------------------------------

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

That’s what we saw, too. Frank From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Wednesday, August 05, 2020 3:33 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Content Issues https://www.cloudflarestatus.com/ Says Minneapolis is Re-routed again and the MICE graph plummeted again. BGP session is still up though, but only receiving 3 Ipv4 routes when we normally receive nearly 300. On Wed, Aug 5, 2020 at 3:15 PM David Farmer <farmer@umn.edu<mailto:farmer@umn.edu>> wrote: And Cloudflare's MICE graph has new signs of life, so I think Cloudflare is back. But Google still seems to be down. On Wed, Aug 5, 2020 at 3:09 PM David Farmer <farmer@umn.edu<mailto:farmer@umn.edu>> wrote: https://www.cloudflarestatus.com/ listed Minneapolis as Re-routed, but then a minute ago it changed to Operational On Wed, Aug 5, 2020 at 2:55 PM Jay Hanke <jayhanke@southfront.io<mailto:jayhanke@southfront.io>> wrote: Looks like some of the CDNs (google/cloudflare) just died at MICE. Is something big going on? -- Jay Hanke, President South Front Networks jayhanke@southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1