- mice-discuss - lists.micemn.net

Re: Mice Issues

by Richard Laager

On Wed, 2014-02-19 at 16:21 -0600, Doug McIntyre wrote: > On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: > > Can someone look at the route server logs and see who the new IPv4 peer > > was that showed up just before this problem started? > > Nobody turned up recently. > > A better measure is who turned up on the switch rather than the route-server. > This was a l2 flood, not anything in routing. Yes, I know it was at layer 2, but if you look at the graph, there was one new *peer* (not route) that showed up just before this happened. It seems worth investigating that a little, as it *might* be related. In the graph tree, click "MICE Route Server". The first graph, "MICE IP4 Peers" is the graph to which I'm referring. -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jay Hanke

There are two Extreme networks on the Mankato Remote switch, Vaultas and Revnet. I don't have ssh right now, can someone check? On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: >> Can someone look at the route server logs and see who the new IPv4 peer >> was that showed up just before this problem started? > > Nobody turned up recently. > > A better measure is who turned up on the switch rather than the route-server. > This was a l2 flood, not anything in routing. > > No ports were up/down on the switch until the flood triggered > my port and Implex's port offline (probably the same class of cisco switch > protecting themselves from the l2 flood). > > Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 > Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 > > So, it probably was from one of the existing remote switches. > > The MAC address I posted about before is an Extreme? switch. > > > Also, the l2 flood seems abated. > > I've been pinging clear to the route servers for 5 minutes now. > > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Doug McIntyre

On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: > Can someone look at the route server logs and see who the new IPv4 peer > was that showed up just before this problem started? Nobody turned up recently. A better measure is who turned up on the switch rather than the route-server. This was a l2 flood, not anything in routing. No ports were up/down on the switch until the flood triggered my port and Implex's port offline (probably the same class of cisco switch protecting themselves from the l2 flood). Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 So, it probably was from one of the existing remote switches. The MAC address I posted about before is an Extreme? switch. Also, the l2 flood seems abated. I've been pinging clear to the route servers for 5 minutes now. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Danny Meister

We were seeing strange traffic making it to our MICE port before we shut it down. Other MICE members' http traffic destined to addresses not on our network. As if unicast packets were getting flooded. -Danny Danny Meister Network Engineer, Level II Atomic Data 615 North 3rd Street Minneapolis, MN 55401 612.466.2000 Main Line 612.466.2071 Direct Dial Simple. Safe. Smart. www.atomicdata.com On 2/19/14, 4:04 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: >On 02/19/2014 03:48 PM, Doug McIntyre wrote: >> On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: >>> Did somebody make a loop/packet storm? >>> Going through the switch now, very slow to see if I can identify >>>who/what/where. >> >> At the moment, it looks to be coming from Mankato, unless somebody >> can tell me I'm reading things wrong. >> >> The logs are filling up with >> >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> >> >> That MAC address is learned from Mankato's remote 10G link >> >> dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match >>00:04:96:37:14:c3 >> Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was >>deleted on xe-0/0/37.0 >> Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was >>learned on xe-0/0/37.0 >> >> >> But, I'm not getting an extraordinary level of input traffic from >>there. >> >> Juniper PR695200 says this error message can be harmless though, so I'm >>not >> convinced this is the root cause. >> > >Are you a bunch of broadcast/multicast traffic? The CNS switch is >seeing ~5000pps of broadcast traffic from the MICE main switch. I don't >know what the normal rate is, but it seems like it is a bit high. >Especially given that the average broadcast cost over the uptime of the >switch is about 20pps. > >If the Mankato switch has a high broadcast rate, I'd shut down the port >and see if things stabilize. > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Steve Howard

On 02/19/2014 03:48 PM, Doug McIntyre wrote: > On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: >> Did somebody make a loop/packet storm? >> Going through the switch now, very slow to see if I can identify who/what/where. > > At the moment, it looks to be coming from Mankato, unless somebody > can tell me I'm reading things wrong. > > The logs are filling up with > > Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > > > That MAC address is learned from Mankato's remote 10G link > > dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match 00:04:96:37:14:c3 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was deleted on xe-0/0/37.0 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was learned on xe-0/0/37.0 > > > But, I'm not getting an extraordinary level of input traffic from there. > > Juniper PR695200 says this error message can be harmless though, so I'm not > convinced this is the root cause. > Are you a bunch of broadcast/multicast traffic? The CNS switch is seeing ~5000pps of broadcast traffic from the MICE main switch. I don't know what the normal rate is, but it seems like it is a bit high. Especially given that the average broadcast cost over the uptime of the switch is about 20pps. If the Mankato switch has a high broadcast rate, I'd shut down the port and see if things stabilize. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Richard Laager

Can someone look at the route server logs and see who the new IPv4 peer was that showed up just before this problem started? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Nathan Beard

I've shutdown my port as well until things are resolved. On Wed, Feb 19, 2014 at 3:48 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: > > Did somebody make a loop/packet storm? > > Going through the switch now, very slow to see if I can identify > who/what/where. > > > At the moment, it looks to be coming from Mankato, unless somebody > can tell me I'm reading things wrong. > > The logs are filling up with > > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > > > That MAC address is learned from Mankato's remote 10G link > > dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match > 00:04:96:37:14:c3 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was > deleted on xe-0/0/37.0 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was > learned on xe-0/0/37.0 > > > But, I'm not getting an extraordinary level of input traffic from there. > > Juniper PR695200 says this error message can be harmless though, so I'm not > convinced this is the root cause. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Doug McIntyre

On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: > Did somebody make a loop/packet storm? > Going through the switch now, very slow to see if I can identify who/what/where. At the moment, it looks to be coming from Mankato, unless somebody can tell me I'm reading things wrong. The logs are filling up with Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 That MAC address is learned from Mankato's remote 10G link dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match 00:04:96:37:14:c3 Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was deleted on xe-0/0/37.0 Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was learned on xe-0/0/37.0 But, I'm not getting an extraordinary level of input traffic from there. Juniper PR695200 says this error message can be harmless though, so I'm not convinced this is the root cause. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Chase Rydberg

We're shut down as well until things calm down. Chase Rydberg *MDU Ethernet Solutions* On Wed, Feb 19, 2014 at 3:37 PM, Larry Patterson <larry(a)atomicdata.com>wrote: > Atomic has proactively shutdown our port to MICE due to heavy traffic > influx/reliability issues across the switch fabric. We will re-enable our > peering after the issues have subsided. > > On 2/19/14 3:31 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: > > >On 02/19/2014 03:23 PM, Jeremy Lumby wrote: > >> I have been experiencing packet loss, as well as seen many of my BGP > >>sessions go down across MICE. Has anyone else seen issues? > >> > > > >The first log message in the CNS switch was at Feb 19 15:11:44.238 > >CST. Did anybody plug something in then? > > > >######################################################################## > > > >To unsubscribe from the MICE-DISCUSS list, click the following link: > >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Larry Patterson

Atomic has proactively shutdown our port to MICE due to heavy traffic influx/reliability issues across the switch fabric. We will re-enable our peering after the issues have subsided. On 2/19/14 3:31 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: >On 02/19/2014 03:23 PM, Jeremy Lumby wrote: >> I have been experiencing packet loss, as well as seen many of my BGP >>sessions go down across MICE. Has anyone else seen issues? >> > >The first log message in the CNS switch was at Feb 19 15:11:44.238 >CST. Did anybody plug something in then? > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014