August 2020 - mice-discuss - lists.micemn.net

Re: Internet is down!

by Jeremy Lumby

A little more info on this is that I noticed outages to AS209 as well. I am directly connected to AS209 in Minneapolis, and experienced loss to some fiber/DSL connections on AS209 in MN on the outskirts of the Twin Cities metro. An interesting side note is that connections on AS209 in Minneapolis/St Paul proper stayed reachable. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Sunday, August 30, 2020 12:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Internet is down! On Sun, Aug 30, 2020 at 11:53:22AM -0500, Austin Gillmann wrote: >Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is >there a good article out there for those still living under a rock? Around 5:30AM CDT today, AS3356 stopped passing traffic to many (all?) of their Tier1 peers it seemed world-wide. The CL tickets say that they introduced a bad flowspec option into their global config dropping BGP sessions all around (I assume to their peers, as none of our transit L3 BGP sessions dropped). Even when one dropped their transit BGP peer into AS3356, they'd still were BGP announcing your prefixes out, causing further traffic disruption. Some people reported to keep your BGP session up, but depreference any learned routes from AS3356, and that seemed to help out with getting traffic. Around 10:00AM CDT, they fixed up their configs, and traffic started flowing normal.

Aug. 30, 2020

Re: Internet is down!

by Doug McIntyre

On Sun, Aug 30, 2020 at 11:53:22AM -0500, Austin Gillmann wrote: >Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is >there a good article out there for those still living under a rock? Around 5:30AM CDT today, AS3356 stopped passing traffic to many (all?) of their Tier1 peers it seemed world-wide. The CL tickets say that they introduced a bad flowspec option into their global config dropping BGP sessions all around (I assume to their peers, as none of our transit L3 BGP sessions dropped). Even when one dropped their transit BGP peer into AS3356, they'd still were BGP announcing your prefixes out, causing further traffic disruption. Some people reported to keep your BGP session up, but depreference any learned routes from AS3356, and that seemed to help out with getting traffic. Around 10:00AM CDT, they fixed up their configs, and traffic started flowing normal.

Aug. 30, 2020

Re: Internet is down!

by David Farmer

you can start with; https://downdetector.com/status/centurylink/map/ And here are a couple of articles; https://9to5mac.com/2020/08/30/centurylink-outage/ https://www.digitaltrends.com/web/centurylink-internet-service-outage/ On Sun, Aug 30, 2020 at 11:53 AM Austin Gillmann < 00000010dc53feca-dmarc-request(a)lists.iphouse.net> wrote: > Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is > there a good article out there for those still living under a rock? > > On Sun, Aug 30, 2020 at 11:24 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > >> That is why I am such a big fan of peering on multiple IXes, it gives you >> ultimate control of all of your BGP neighbors, and less reliance on the big >> transit providers. >> >> >> >> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf >> Of *Jay Hanke >> *Sent:* Sunday, August 30, 2020 11:02 AM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] Internet is down! >> >> >> >> On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman(a)longlines.biz> >> wrote: >> >> The internet actually broke! Yikes >> >> >> >> Seems like local peering networks seemed to be much less impacted.... >> >> >> >> https://ixpmgr.qcix.net/statistics/ixp >> >> https://ixpmgr.micemn.net/statistics/ixp >> >> >> >> :) >> >> >> >> -- >> >> Jay Hanke, President >> >> South Front Networks >> >> jayhanke(a)southfront.io >> >> Phone 612-204-0000 >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 30, 2020

Re: Internet is down!

by Austin Gillmann

Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is there a good article out there for those still living under a rock? On Sun, Aug 30, 2020 at 11:24 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > That is why I am such a big fan of peering on multiple IXes, it gives you > ultimate control of all of your BGP neighbors, and less reliance on the big > transit providers. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Jay Hanke > *Sent:* Sunday, August 30, 2020 11:02 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Internet is down! > > > > On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman(a)longlines.biz> > wrote: > > The internet actually broke! Yikes > > > > Seems like local peering networks seemed to be much less impacted.... > > > > https://ixpmgr.qcix.net/statistics/ixp > > https://ixpmgr.micemn.net/statistics/ixp > > > > :) > > > > -- > > Jay Hanke, President > > South Front Networks > > jayhanke(a)southfront.io > > Phone 612-204-0000 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 30, 2020

Re: Internet is down!

by Jeremy Lumby

That is why I am such a big fan of peering on multiple IXes, it gives you ultimate control of all of your BGP neighbors, and less reliance on the big transit providers. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, August 30, 2020 11:02 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Internet is down! On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman(a)longlines.biz> wrote: The internet actually broke! Yikes Seems like local peering networks seemed to be much less impacted.... https://ixpmgr.qcix.net/statistics/ixp https://ixpmgr.micemn.net/statistics/ixp :) -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 30, 2020

Re: Internet is down!

by Jay Hanke

On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman(a)longlines.biz> wrote: > The internet actually broke! Yikes > Seems like local peering networks seemed to be much less impacted.... https://ixpmgr.qcix.net/statistics/ixp https://ixpmgr.micemn.net/statistics/ixp :) -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 30, 2020

Re: Internet is down!

by Justin Huffman

The internet actually broke! Yikes Justin ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Mike Horwath <drechsau(a)GEEKS.ORG> Sent: Sunday, August 30, 2020 9:29:32 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Internet is down! WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. On Sun, Aug 30, 2020 at 08:37:52AM -0500, Mike Horwath wrote: > But I bet you knew that already. > > **WHEE** These are just canary messages to see when things start to flow. 0929 hours central, 1429 hours UTC for this message -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Aug. 30, 2020

Re: Internet is down!

by Mike Horwath

On Sun, Aug 30, 2020 at 08:37:52AM -0500, Mike Horwath wrote: > But I bet you knew that already. > > **WHEE** These are just canary messages to see when things start to flow. 0929 hours central, 1429 hours UTC for this message -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Aug. 30, 2020

Internet is down!

by Mike Horwath

But I bet you knew that already. **WHEE** -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Aug. 30, 2020

Re: MICE hits a 1T

by Jay Hanke

Confirmed in IXP Mgr as well! Congrats everyone!!! [image: image.png] On Fri, Aug 28, 2020 at 8:25 AM David Farmer < 0000000e7948cb21-dmarc-request(a)lists.iphouse.net> wrote: > MICE hits a 1T or 500G depending on how you want to measure it. > > [image: image.png] > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 28, 2020