February 2014 - mice-discuss - lists.micemn.net

Re: Mice Issues

by Richard Laager

Can someone look at the route server logs and see who the new IPv4 peer was that showed up just before this problem started? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Nathan Beard

I've shutdown my port as well until things are resolved. On Wed, Feb 19, 2014 at 3:48 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: > > Did somebody make a loop/packet storm? > > Going through the switch now, very slow to see if I can identify > who/what/where. > > > At the moment, it looks to be coming from Mankato, unless somebody > can tell me I'm reading things wrong. > > The logs are filling up with > > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 master > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > Feb 19 15:34:18 MICE-SW1 backup > MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for > VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 > > > That MAC address is learned from Mankato's remote 10G link > > dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match > 00:04:96:37:14:c3 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was > deleted on xe-0/0/37.0 > Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was > learned on xe-0/0/37.0 > > > But, I'm not getting an extraordinary level of input traffic from there. > > Juniper PR695200 says this error message can be harmless though, so I'm not > convinced this is the root cause. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Doug McIntyre

On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: > Did somebody make a loop/packet storm? > Going through the switch now, very slow to see if I can identify who/what/where. At the moment, it looks to be coming from Mankato, unless somebody can tell me I'm reading things wrong. The logs are filling up with Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 master MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 Feb 19 15:34:18 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 That MAC address is learned from Mankato's remote 10G link dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match 00:04:96:37:14:c3 Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was deleted on xe-0/0/37.0 Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was learned on xe-0/0/37.0 But, I'm not getting an extraordinary level of input traffic from there. Juniper PR695200 says this error message can be harmless though, so I'm not convinced this is the root cause. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Chase Rydberg

We're shut down as well until things calm down. Chase Rydberg *MDU Ethernet Solutions* On Wed, Feb 19, 2014 at 3:37 PM, Larry Patterson <larry(a)atomicdata.com>wrote: > Atomic has proactively shutdown our port to MICE due to heavy traffic > influx/reliability issues across the switch fabric. We will re-enable our > peering after the issues have subsided. > > On 2/19/14 3:31 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: > > >On 02/19/2014 03:23 PM, Jeremy Lumby wrote: > >> I have been experiencing packet loss, as well as seen many of my BGP > >>sessions go down across MICE. Has anyone else seen issues? > >> > > > >The first log message in the CNS switch was at Feb 19 15:11:44.238 > >CST. Did anybody plug something in then? > > > >######################################################################## > > > >To unsubscribe from the MICE-DISCUSS list, click the following link: > >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Larry Patterson

Atomic has proactively shutdown our port to MICE due to heavy traffic influx/reliability issues across the switch fabric. We will re-enable our peering after the issues have subsided. On 2/19/14 3:31 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: >On 02/19/2014 03:23 PM, Jeremy Lumby wrote: >> I have been experiencing packet loss, as well as seen many of my BGP >>sessions go down across MICE. Has anyone else seen issues? >> > >The first log message in the CNS switch was at Feb 19 15:11:44.238 >CST. Did anybody plug something in then? > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Doug McIntyre

On Wed, Feb 19, 2014 at 09:34:40PM +0000, Justin Krejci wrote: > Is someone advertising v6 routes they should not be onto the Mice servers? > > 2001:504:27::D1AF:0:1 4 53679 7060055 721494 80179512 0 13 00:03:02 244 > 2001:504:27::D1AF:0:2 4 53679 7073987 724693 80179512 0 0 00:11:10 9020 > > I see a lot of HE routes coming from route server 02 That is normal, HE is peering their whole IPv6 table to the route servers. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Justin Krejci

Is someone advertising v6 routes they should not be onto the Mice servers? 2001:504:27::D1AF:0:1 4 53679 7060055 721494 80179512 0 13 00:03:02 244 2001:504:27::D1AF:0:2 4 53679 7073987 724693 80179512 0 0 00:11:10 9020 I see a lot of HE routes coming from route server 02 ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of James Stahr [stahr(a)MAILBAG.COM] Sent: Wednesday, February 19, 2014 3:28 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues On 02/19/2014 3:23 PM, Jeremy Lumby wrote: > I have been experiencing packet loss, as well as seen many of my BGP sessions go down across MICE. Has anyone else seen issues? > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > Yes, spanning tree loop? 30 second input rate 766296000 bits/sec, 118912 packets/sec -James ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Steve Howard

On 02/19/2014 03:23 PM, Jeremy Lumby wrote: > I have been experiencing packet loss, as well as seen many of my BGP sessions go down across MICE. Has anyone else seen issues? > The first log message in the CNS switch was at Feb 19 15:11:44.238 CST. Did anybody plug something in then? ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Doug McIntyre

On Wed, Feb 19, 2014 at 03:23:00PM -0600, Jeremy Lumby wrote: > I have been experiencing packet loss, as well as seen many of my BGP sessions go down across MICE. Has anyone else seen issues? I have a weird error on my MICE facing port. Feb 19 15:11:44.976 CST: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on GigabitEthernet0/4. Feb 19 15:11:44.976 CST: %PM-4-ERR_DISABLE: loopback error detected on Gi0/4, putting Gi0/4 in err-disable state I cleared that, but still having big packet loss across MICE. Did somebody make a loop/packet storm? Going through the switch now, very slow to see if I can identify who/what/where. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Steve Howard

On 02/19/2014 03:23 PM, Jeremy Lumby wrote: > I have been experiencing packet loss, as well as seen many of my BGP sessions go down across MICE. Has anyone else seen issues? > Yes, something is broken. Loop? The CNS switch has many log entries like the following: Feb 19 15:27:50.787 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 10f3.1134.e6de in vlan 847 is flapping between port Te0/10 and port Te0/8 Feb 19 15:28:04.578 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 001b.0de7.15c0 in vlan 847 is flapping between port Te0/10 and port Te0/11 Feb 19 15:28:04.779 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.9999.2345 in vlan 847 is flapping between port Te0/11 and port Te0/10 Feb 19 15:28:04.846 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 0021.a00a.a040 in vlan 847 is flapping between port Te0/10 and port Gi0/1 Feb 19 15:28:05.081 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 001a.a10e.9f05 in vlan 847 is flapping between port Te0/10 and port Te0/12 Feb 19 15:28:05.383 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 10f3.1134.e6de in vlan 847 is flapping between port Te0/10 and port Te0/8 Feb 19 15:28:19.862 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 001b.0de7.15c0 in vlan 847 is flapping between port Te0/10 and port Te0/11 Feb 19 15:28:19.963 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 001a.a10e.9f05 in vlan 847 is flapping between port Te0/10 and port Te0/12 Feb 19 15:28:20.030 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 0000.9999.2345 in vlan 847 is flapping between port Te0/10 and port Te0/11 Feb 19 15:28:20.433 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 0021.a00a.a040 in vlan 847 is flapping between port Te0/10 and port Gi0/1 Feb 19 15:28:22.798 CST: %SW_MATM-4-MACFLAP_NOTIF: Host 10f3.1134.e6de in vlan 847 is flapping between port Te0/10 and port Te0/8 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014