But I bet you knew that already. **WHEE** -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Sun, Aug 30, 2020 at 08:37:52AM -0500, Mike Horwath wrote:
But I bet you knew that already.
**WHEE**
These are just canary messages to see when things start to flow. 0929 hours central, 1429 hours UTC for this message -- Mike Horwath, reachable via drechsau@Geeks.ORG
The internet actually broke! Yikes Justin ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Mike Horwath <drechsau@GEEKS.ORG> Sent: Sunday, August 30, 2020 9:29:32 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Internet is down! WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. On Sun, Aug 30, 2020 at 08:37:52AM -0500, Mike Horwath wrote:
But I bet you knew that already.
**WHEE**
These are just canary messages to see when things start to flow. 0929 hours central, 1429 hours UTC for this message -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman@longlines.biz> wrote:
The internet actually broke! Yikes
Seems like local peering networks seemed to be much less impacted.... https://ixpmgr.qcix.net/statistics/ixp https://ixpmgr.micemn.net/statistics/ixp :) -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
That is why I am such a big fan of peering on multiple IXes, it gives you ultimate control of all of your BGP neighbors, and less reliance on the big transit providers. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, August 30, 2020 11:02 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Internet is down! On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman@longlines.biz> wrote: The internet actually broke! Yikes Seems like local peering networks seemed to be much less impacted.... https://ixpmgr.qcix.net/statistics/ixp https://ixpmgr.micemn.net/statistics/ixp :) -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is there a good article out there for those still living under a rock? On Sun, Aug 30, 2020 at 11:24 AM Jeremy Lumby <jlumby@mnvoip.com> wrote:
That is why I am such a big fan of peering on multiple IXes, it gives you ultimate control of all of your BGP neighbors, and less reliance on the big transit providers.
*From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf Of *Jay Hanke *Sent:* Sunday, August 30, 2020 11:02 AM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Internet is down!
On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman@longlines.biz> wrote:
The internet actually broke! Yikes
Seems like local peering networks seemed to be much less impacted....
https://ixpmgr.qcix.net/statistics/ixp
https://ixpmgr.micemn.net/statistics/ixp
:)
--
Jay Hanke, President
South Front Networks
jayhanke@southfront.io
Phone 612-204-0000
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
you can start with; https://downdetector.com/status/centurylink/map/ And here are a couple of articles; https://9to5mac.com/2020/08/30/centurylink-outage/ https://www.digitaltrends.com/web/centurylink-internet-service-outage/ On Sun, Aug 30, 2020 at 11:53 AM Austin Gillmann < 00000010dc53feca-dmarc-request@lists.iphouse.net> wrote:
Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is there a good article out there for those still living under a rock?
On Sun, Aug 30, 2020 at 11:24 AM Jeremy Lumby <jlumby@mnvoip.com> wrote:
That is why I am such a big fan of peering on multiple IXes, it gives you ultimate control of all of your BGP neighbors, and less reliance on the big transit providers.
*From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf Of *Jay Hanke *Sent:* Sunday, August 30, 2020 11:02 AM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Internet is down!
On Sun, Aug 30, 2020 at 9:31 AM Justin Huffman <jhuffman@longlines.biz> wrote:
The internet actually broke! Yikes
Seems like local peering networks seemed to be much less impacted....
https://ixpmgr.qcix.net/statistics/ixp
https://ixpmgr.micemn.net/statistics/ixp
:)
--
Jay Hanke, President
South Front Networks
jayhanke@southfront.io
Phone 612-204-0000
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
On Sun, Aug 30, 2020 at 11:53:22AM -0500, Austin Gillmann wrote:
Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is there a good article out there for those still living under a rock?
Around 5:30AM CDT today, AS3356 stopped passing traffic to many (all?) of their Tier1 peers it seemed world-wide. The CL tickets say that they introduced a bad flowspec option into their global config dropping BGP sessions all around (I assume to their peers, as none of our transit L3 BGP sessions dropped). Even when one dropped their transit BGP peer into AS3356, they'd still were BGP announcing your prefixes out, causing further traffic disruption. Some people reported to keep your BGP session up, but depreference any learned routes from AS3356, and that seemed to help out with getting traffic. Around 10:00AM CDT, they fixed up their configs, and traffic started flowing normal.
A little more info on this is that I noticed outages to AS209 as well. I am directly connected to AS209 in Minneapolis, and experienced loss to some fiber/DSL connections on AS209 in MN on the outskirts of the Twin Cities metro. An interesting side note is that connections on AS209 in Minneapolis/St Paul proper stayed reachable. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Sunday, August 30, 2020 12:13 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Internet is down! On Sun, Aug 30, 2020 at 11:53:22AM -0500, Austin Gillmann wrote:
Mmmm happy I wasn't on-call this weekend :). Woke up late this Sunday is there a good article out there for those still living under a rock?
Around 5:30AM CDT today, AS3356 stopped passing traffic to many (all?) of their Tier1 peers it seemed world-wide. The CL tickets say that they introduced a bad flowspec option into their global config dropping BGP sessions all around (I assume to their peers, as none of our transit L3 BGP sessions dropped). Even when one dropped their transit BGP peer into AS3356, they'd still were BGP announcing your prefixes out, causing further traffic disruption. Some people reported to keep your BGP session up, but depreference any learned routes from AS3356, and that seemed to help out with getting traffic. Around 10:00AM CDT, they fixed up their configs, and traffic started flowing normal.
participants (7)
-
Austin Gillmann -
David Farmer -
Doug McIntyre -
Jay Hanke -
Jeremy Lumby -
Justin Huffman -
Mike Horwath