- mice-discuss - lists.micemn.net

Re: Mice Issues

by Jeremy Lumby

Is anyone else having issues with connectivity to Charter? I have re-established connectivity to every one else through my other upstreams, other than Charter. I am keeping my MICE port down overnight for good measure. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dan Boehlke Sent: Wednesday, February 19, 2014 5:41 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues The Vaultas switch has been configured to participate in spanning-tree, so it won't blindly go along with loops any more. Sorry. The Vaultas switch didn't make the loop, it just made it worse. Any word on the origination of the loop? -- Dan Boehlke dboehlke(a)mac.com On Feb 19, 2014, at 4:47 PM, Jay Hanke <jayhanke(a)MANKATONETWORKS.NET> wrote: Got ssh, that mac doug sent out belongs to Vaultas. On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: There are two Extreme networks on the Mankato Remote switch, Vaultas and Revnet. I don't have ssh right now, can someone check? On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: Can someone look at the route server logs and see who the new IPv4 peer was that showed up just before this problem started? Nobody turned up recently. A better measure is who turned up on the switch rather than the route-server. This was a l2 flood, not anything in routing. No ports were up/down on the switch until the flood triggered my port and Implex's port offline (probably the same class of cisco switch protecting themselves from the l2 flood). Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 So, it probably was from one of the existing remote switches. The MAC address I posted about before is an Extreme? switch. Also, the l2 flood seems abated. I've been pinging clear to the route servers for 5 minutes now. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Richard Laager

On Wed, 2014-02-19 at 17:40 -0600, Dan Boehlke wrote: > The Vaultas switch has been configured to participate in spanning-tree Is that really what we want? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Dan Boehlke

The Vaultas switch has been configured to participate in spanning-tree, so it won't blindly go along with loops any more. Sorry. The Vaultas switch didn't make the loop, it just made it worse. Any word on the origination of the loop? -- Dan Boehlke dboehlke(a)mac.com On Feb 19, 2014, at 4:47 PM, Jay Hanke <jayhanke(a)MANKATONETWORKS.NET> wrote: > Got ssh, that mac doug sent out belongs to Vaultas. > > On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: >> There are two Extreme networks on the Mankato Remote switch, Vaultas >> and Revnet. I don't have ssh right now, can someone check? >> >> On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >>> On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: >>>> Can someone look at the route server logs and see who the new IPv4 peer >>>> was that showed up just before this problem started? >>> >>> Nobody turned up recently. >>> >>> A better measure is who turned up on the switch rather than the route-server. >>> This was a l2 flood, not anything in routing. >>> >>> No ports were up/down on the switch until the flood triggered >>> my port and Implex's port offline (probably the same class of cisco switch >>> protecting themselves from the l2 flood). >>> >>> Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 >>> Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 >>> >>> So, it probably was from one of the existing remote switches. >>> >>> The MAC address I posted about before is an Extreme? switch. >>> >>> >>> Also, the l2 flood seems abated. >>> >>> I've been pinging clear to the route servers for 5 minutes now. >>> >>> >>> >>> -- >>> Doug McIntyre <merlyn(a)iphouse.net> >>> ~.~ ipHouse ~.~ >>> Network Engineer/Provisioning/Jack of all Trades >>> >>> ######################################################################## >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke >> CTO, CCIE #19093 >> Mankato Networks LLC >> PO Box 54 >> 619 S Front St >> Mankato, MN 56001-3838 >> Google 530-618-2398 >> jayhanke(a)mankatonetworks.net >> http://www.mankatonetworks.com >> http://www.neutralpath.net > > > > -- > Jay Hanke > CTO, CCIE #19093 > Mankato Networks LLC > PO Box 54 > 619 S Front St > Mankato, MN 56001-3838 > Google 530-618-2398 > jayhanke(a)mankatonetworks.net > http://www.mankatonetworks.com > http://www.neutralpath.net > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jay Hanke

Got ssh, that mac doug sent out belongs to Vaultas. On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: > There are two Extreme networks on the Mankato Remote switch, Vaultas > and Revnet. I don't have ssh right now, can someone check? > > On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: >>> Can someone look at the route server logs and see who the new IPv4 peer >>> was that showed up just before this problem started? >> >> Nobody turned up recently. >> >> A better measure is who turned up on the switch rather than the route-server. >> This was a l2 flood, not anything in routing. >> >> No ports were up/down on the switch until the flood triggered >> my port and Implex's port offline (probably the same class of cisco switch >> protecting themselves from the l2 flood). >> >> Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 >> Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 >> >> So, it probably was from one of the existing remote switches. >> >> The MAC address I posted about before is an Extreme? switch. >> >> >> Also, the l2 flood seems abated. >> >> I've been pinging clear to the route servers for 5 minutes now. >> >> >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke > CTO, CCIE #19093 > Mankato Networks LLC > PO Box 54 > 619 S Front St > Mankato, MN 56001-3838 > Google 530-618-2398 > jayhanke(a)mankatonetworks.net > http://www.mankatonetworks.com > http://www.neutralpath.net -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Larry Patterson

Did they plug in a switch or a router? Was it a loop? Are there loop mitigation techniques in place to automatically shutdown offending ports? When would you suggest that it's safe to come back online? On 2/19/14 4:40 PM, "Jay Hanke" <jayhanke(a)MANKATONETWORKS.NET> wrote: >HCMC turned up peering this afternoon, l2 has been up for several days. > >I think the timing was maybe 20 minutes prior to issue. > > > >On Wed, Feb 19, 2014 at 2:03 PM, Richard Laager <rlaager(a)wiktel.com> >wrote: >> Can someone look at the route server logs and see who the new IPv4 peer >> was that showed up just before this problem started? >> >> -- >> Richard >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > >-- >Jay Hanke >CTO, CCIE #19093 >Mankato Networks LLC >PO Box 54 >619 S Front St >Mankato, MN 56001-3838 >Google 530-618-2398 >jayhanke(a)mankatonetworks.net >http://www.mankatonetworks.com >http://www.neutralpath.net > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jeff Anderson

Our port flipped to err-dis due to a detected loop. Jeff Anderson jeffa(a)implex.net 612.799.7277 Cell 612.339.8255 x200 Office 612.339.3310 Fax -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Wednesday, February 19, 2014 4:34 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues On 2/19/14, 16:21 , Doug McIntyre wrote: > Also, the l2 flood seems abated. > > I've been pinging clear to the route servers for 5 minutes now. From looking at my CPU history, the appears to started 85 minutes ago, peaked about 75 minutes ago, subsided about 25 minutes ago. We saw BGP flaps all over the place, all peerings are back now except our peering with Implex, which has been down for 80 minutes or so. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jay Hanke

HCMC turned up peering this afternoon, l2 has been up for several days. I think the timing was maybe 20 minutes prior to issue. On Wed, Feb 19, 2014 at 2:03 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > Can someone look at the route server logs and see who the new IPv4 peer > was that showed up just before this problem started? > > -- > Richard > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by David Farmer

On 2/19/14, 16:33 , David Farmer wrote: > On 2/19/14, 16:21 , Doug McIntyre wrote: > >> Also, the l2 flood seems abated. >> >> I've been pinging clear to the route servers for 5 minutes now. > > From looking at my CPU history, the appears to started 85 minutes ago, > peaked about 75 minutes ago, subsided about 25 minutes ago. > > We saw BGP flaps all over the place, all peerings are back now except > our peering with Implex, which has been down for 80 minutes or so. Implex is back now too, so all our direct peerings are back up. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by David Farmer

On 2/19/14, 16:21 , Doug McIntyre wrote: > Also, the l2 flood seems abated. > > I've been pinging clear to the route servers for 5 minutes now. From looking at my CPU history, the appears to started 85 minutes ago, peaked about 75 minutes ago, subsided about 25 minutes ago. We saw BGP flaps all over the place, all peerings are back now except our peering with Implex, which has been down for 80 minutes or so. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Bertsch, Steven

We were seeing elevated levels of multicast, but most of what we were receiving was unicast well in excess of what we normally see. If MACs were flapping between ports due to a bridging loop, the switch may have disabled MAC learning in the VLAN, leading to unicast flooding after learned MACs timed out. Steven Bertsch Engineer III OneNeck(r) IT Solutions, a TDS(r) Company Direct: 612.395.8966 Cell: 612.232.1181 Email: steven.bertsch(a)oneneck.com Website: www.oneneck.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Danny Meister Sent: Wednesday, February 19, 2014 4:08 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues We were seeing strange traffic making it to our MICE port before we shut it down. Other MICE members' http traffic destined to addresses not on our network. As if unicast packets were getting flooded. -Danny Danny Meister Network Engineer, Level II Atomic Data 615 North 3rd Street Minneapolis, MN 55401 612.466.2000 Main Line 612.466.2071 Direct Dial Simple. Safe. Smart. www.atomicdata.com On 2/19/14, 4:04 PM, "Steve Howard" <showard(a)PAULBUNYAN.NET> wrote: >On 02/19/2014 03:48 PM, Doug McIntyre wrote: >> On Wed, Feb 19, 2014 at 03:30:09PM -0600, Doug McIntyre wrote: >>> Did somebody make a loop/packet storm? >>> Going through the switch now, very slow to see if I can identify >>>who/what/where. >> >> At the moment, it looks to be coming from Mankato, unless somebody >> can tell me I'm reading things wrong. >> >> The logs are filling up with >> >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 master >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> Feb 19 15:34:18 MICE-SW1 backup >>MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) >>for VLANIdx=3:00:04:96:37:14:c3/48:IFL=105 >> >> >> That MAC address is learned from Mankato's remote 10G link >> >> dmcintyre@MICE-SW1> show ethernet-switching mac-learning-log | match >>00:04:96:37:14:c3 >> Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was >>deleted on xe-0/0/37.0 >> Wed Feb 19 15:11:44 2014 vlan_name default mac 00:04:96:37:14:c3 was >>learned on xe-0/0/37.0 >> >> >> But, I'm not getting an extraordinary level of input traffic from >>there. >> >> Juniper PR695200 says this error message can be harmless though, so I'm >>not >> convinced this is the root cause. >> > >Are you a bunch of broadcast/multicast traffic? The CNS switch is >seeing ~5000pps of broadcast traffic from the MICE main switch. I don't >know what the normal rate is, but it seems like it is a bit high. >Especially given that the average broadcast cost over the uptime of the >switch is about 20pps. > >If the Mankato switch has a high broadcast rate, I'd shut down the port >and see if things stabilize. > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014