Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well. -James Berbee/CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Saw same, bgp poof, and mac-flap ick... Ryan ----- Reply message ----- From: "James Stahr" <stahr@MAILBAG.COM> To: "MICE-DISCUSS@LISTS.IPHOUSE.NET" <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE incident @11:47 CST Date: Tue, Nov 22, 2011 1:54 pm Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well. -James Berbee/CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Judging by the graphs, it was something between Cooperative Net and HE… --- Eric F Crist System Administrator ClaimLynx, Inc (952) 593-5969 x2301 On Nov 22, 2011, at 13:54:12, James Stahr wrote:
Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well.
-James Berbee/CDW
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Can we revisit the whole spanning tree on the IX thing again? My two cents last time was for 1 mac address per participant facing port, proper storm control limts, bpdufilter, etc. Also, I'd encourage all participants to review their config's, *especially* if they have layer2 devices in the mix somewhere. AMS-IX has put together very nice and self-explainatory guides about how to keep your layer2 devices acting nice and not have a negative effect on the entire MICE IX. That configuration guide is located here: http://www.ams-ix.net/config-guide -- Andrew Hoyos hoyosa@gmail.com On Nov 22, 2011, at 2:06 PM, Eric Crist wrote:
Judging by the graphs, it was something between Cooperative Net and HE…
--- Eric F Crist System Administrator ClaimLynx, Inc (952) 593-5969 x2301
On Nov 22, 2011, at 13:54:12, James Stahr wrote:
Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well.
-James Berbee/CDW
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
My favorite quote from their config guide: 4.2.6. Non-unicast IPv4: IGMP, DHCP, TFTP On the ISP peering LAN, the only non-unicast traffic that is allowed is the ARP query. Sometimes we see equipment trying to get a configuration through broadcast TFTP, or configure themselves through DHCP. We will leave it to the reader to consider why this is a bad idea. On Tue, 2011-11-22 at 14:14 -0600, Andrew Hoyos wrote:
Can we revisit the whole spanning tree on the IX thing again?
My two cents last time was for 1 mac address per participant facing port, proper storm control limts, bpdufilter, etc.
Also, I'd encourage all participants to review their config's, *especially* if they have layer2 devices in the mix somewhere. AMS-IX has put together very nice and self-explainatory guides about how to keep your layer2 devices acting nice and not have a negative effect on the entire MICE IX.
That configuration guide is located here: http://www.ams-ix.net/config-guide
-- Andrew Hoyos hoyosa@gmail.com
On Nov 22, 2011, at 2:06 PM, Eric Crist wrote:
Judging by the graphs, it was something between Cooperative Net and HE…
--- Eric F Crist System Administrator ClaimLynx, Inc (952) 593-5969 x2301
On Nov 22, 2011, at 13:54:12, James Stahr wrote:
Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well.
-James Berbee/CDW
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Tue, Nov 22, 2011 at 1:54 PM, James Stahr <stahr@mailbag.com> wrote:
Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well.
I apologize, that was my mess up. I was working on a customer transport link and didn't realize I had a bridge-domain built on the port. It created a loop when the port came up within the customer network and it looks like it spewed garbage on to the MICE vlan. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Looks like the routes came back up for everyone but DCN. If anyone needs a hand just let us know. Anthony Anderberg Sr. Systems Analyst - NU-Telecom O: 320-234-5239 C: 320-234-5539 anthonyanderberg@nu-telecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Tuesday, November 22, 2011 2:10 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE incident @11:47 CST On Tue, Nov 22, 2011 at 1:54 PM, James Stahr <stahr@mailbag.com> wrote:
Anyone else have an issue? stats.micemn.net show spikes which would likely indicate a spanning tree loop or a broadcast storm. Whatever the traffic was, it caused a CPU spike and upset all of our other BGP sessions on our ASR as well.
I apologize, that was my mess up. I was working on a customer transport link and didn't realize I had a bridge-domain built on the port. It created a loop when the port came up within the customer network and it looks like it spewed garbage on to the MICE vlan. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (7)
-
Andrew Hoyos -
Anthony Anderberg -
Eric Crist -
James Stahr -
Jay Hanke -
Justin Krejci -
Ryan Goldberg