February 2014 - mice-discuss - lists.micemn.net

Re: Mice Issues

by Jeremy Lumby

Charter Update: I am providing this update to let others know what I went through after the outage to get re-connected to Charter in the event that others are seeing similar issues, hopefully it can cut a lot of troubleshooting time. -When the outage happened yesterday I shut down my MICE port. -All traffic shifted to my other upstreams as expected, other than Charter -I could not pass 2 way traffic with anyone on Charter's network. -Some traceroutes from Charter's side showed that traffic heading to me was hitting their peering router that was connected to MICE, even though my BGP session with them had been down for hours -Charter's peering router showed that my BGP session was still established through MICE from prior to the outage starting. -Charter attempted to clear the BGP connection to me as well as do a shutdown on my neighbor statement, however whenever they looked at it, it still showed established with the timer reflecting it being up prior to the MICE outage. -I turned my MICE port back up, and return traffic from Charter started making it to me, despite BGP being down from my point of view. -It took Charter removing all of their neighbor statements to me, and re-entering them before BGP properly re-established in both directions. -The entire time our IPv6 session worked as expected -We are both running Cisco hardware, I believe they have a 9K and I have a 7206VXR-G2 I hope this helps others out with troubleshooting in the future Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Wednesday, February 19, 2014 5:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues On 2/19/14, 17:45 , Jeremy Lumby wrote: > Is anyone else having issues with connectivity to Charter? I have > re-established connectivity to every one else through my other > upstreams, other than Charter. I am keeping my MICE port down > overnight for good measure. Our peering with Charter has reestablished via MICE, everything looks good here. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 20, 2014

Re: Mice Issues

by David Farmer

On 2/19/14, 17:45 , Jeremy Lumby wrote: > Is anyone else having issues with connectivity to Charter? I have > re-established connectivity to every one else through my other > upstreams, other than Charter. I am keeping my MICE port down overnight > for good measure. Our peering with Charter has reestablished via MICE, everything looks good here. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Dan Boehlke

So next question, which flavor of spanning tree is best. I configured PVST on a reflex, but remembered that the MICE switch is a Juniper, not a Cisco. Simple dot1d? -- Dan B. On Feb 19, 2014, at 5:50 PM, David Farmer <farmer(a)UMN.EDU> wrote: > On 2/19/14, 17:43 , Richard Laager wrote: >> On Wed, 2014-02-19 at 17:40 -0600, Dan Boehlke wrote: >>> The Vaultas switch has been configured to participate in spanning-tree >> >> Is that really what we want? > > There are philosophical arguments both ways on this one. Given our cooperative management there is an argument to having spanning tree there to catch us if/when we screw up. > > In a very tightly managed situation where there can be no loops then there is an argument to turn-off spanning tree, but god help you if there ever is a loop. > > > -- > ================================================ > David Farmer Email: farmer(a)umn.edu > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 1-612-626-0815 > Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 > ================================================ > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by David Farmer

On 2/19/14, 17:43 , Richard Laager wrote: > On Wed, 2014-02-19 at 17:40 -0600, Dan Boehlke wrote: >> The Vaultas switch has been configured to participate in spanning-tree > > Is that really what we want? There are philosophical arguments both ways on this one. Given our cooperative management there is an argument to having spanning tree there to catch us if/when we screw up. In a very tightly managed situation where there can be no loops then there is an argument to turn-off spanning tree, but god help you if there ever is a loop. -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Dan Boehlke

I was just thinking that, because generally exchange point best practice is to turn spanning tree off. So maybe I shouldn't be embarrassed for not having done it.... :-( -- Dan B. On Feb 19, 2014, at 5:43 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > On Wed, 2014-02-19 at 17:40 -0600, Dan Boehlke wrote: >> The Vaultas switch has been configured to participate in spanning-tree > > Is that really what we want? > > -- > Richard > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jeremy Lumby

Is anyone else having issues with connectivity to Charter? I have re-established connectivity to every one else through my other upstreams, other than Charter. I am keeping my MICE port down overnight for good measure. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dan Boehlke Sent: Wednesday, February 19, 2014 5:41 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Mice Issues The Vaultas switch has been configured to participate in spanning-tree, so it won't blindly go along with loops any more. Sorry. The Vaultas switch didn't make the loop, it just made it worse. Any word on the origination of the loop? -- Dan Boehlke dboehlke(a)mac.com On Feb 19, 2014, at 4:47 PM, Jay Hanke <jayhanke(a)MANKATONETWORKS.NET> wrote: Got ssh, that mac doug sent out belongs to Vaultas. On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: There are two Extreme networks on the Mankato Remote switch, Vaultas and Revnet. I don't have ssh right now, can someone check? On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: Can someone look at the route server logs and see who the new IPv4 peer was that showed up just before this problem started? Nobody turned up recently. A better measure is who turned up on the switch rather than the route-server. This was a l2 flood, not anything in routing. No ports were up/down on the switch until the flood triggered my port and Implex's port offline (probably the same class of cisco switch protecting themselves from the l2 flood). Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 So, it probably was from one of the existing remote switches. The MAC address I posted about before is an Extreme? switch. Also, the l2 flood seems abated. I've been pinging clear to the route servers for 5 minutes now. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Richard Laager

On Wed, 2014-02-19 at 17:40 -0600, Dan Boehlke wrote: > The Vaultas switch has been configured to participate in spanning-tree Is that really what we want? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Dan Boehlke

The Vaultas switch has been configured to participate in spanning-tree, so it won't blindly go along with loops any more. Sorry. The Vaultas switch didn't make the loop, it just made it worse. Any word on the origination of the loop? -- Dan Boehlke dboehlke(a)mac.com On Feb 19, 2014, at 4:47 PM, Jay Hanke <jayhanke(a)MANKATONETWORKS.NET> wrote: > Got ssh, that mac doug sent out belongs to Vaultas. > > On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: >> There are two Extreme networks on the Mankato Remote switch, Vaultas >> and Revnet. I don't have ssh right now, can someone check? >> >> On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >>> On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: >>>> Can someone look at the route server logs and see who the new IPv4 peer >>>> was that showed up just before this problem started? >>> >>> Nobody turned up recently. >>> >>> A better measure is who turned up on the switch rather than the route-server. >>> This was a l2 flood, not anything in routing. >>> >>> No ports were up/down on the switch until the flood triggered >>> my port and Implex's port offline (probably the same class of cisco switch >>> protecting themselves from the l2 flood). >>> >>> Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 >>> Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 >>> >>> So, it probably was from one of the existing remote switches. >>> >>> The MAC address I posted about before is an Extreme? switch. >>> >>> >>> Also, the l2 flood seems abated. >>> >>> I've been pinging clear to the route servers for 5 minutes now. >>> >>> >>> >>> -- >>> Doug McIntyre <merlyn(a)iphouse.net> >>> ~.~ ipHouse ~.~ >>> Network Engineer/Provisioning/Jack of all Trades >>> >>> ######################################################################## >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke >> CTO, CCIE #19093 >> Mankato Networks LLC >> PO Box 54 >> 619 S Front St >> Mankato, MN 56001-3838 >> Google 530-618-2398 >> jayhanke(a)mankatonetworks.net >> http://www.mankatonetworks.com >> http://www.neutralpath.net > > > > -- > Jay Hanke > CTO, CCIE #19093 > Mankato Networks LLC > PO Box 54 > 619 S Front St > Mankato, MN 56001-3838 > Google 530-618-2398 > jayhanke(a)mankatonetworks.net > http://www.mankatonetworks.com > http://www.neutralpath.net > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Jay Hanke

Got ssh, that mac doug sent out belongs to Vaultas. On Wed, Feb 19, 2014 at 2:27 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: > There are two Extreme networks on the Mankato Remote switch, Vaultas > and Revnet. I don't have ssh right now, can someone check? > > On Wed, Feb 19, 2014 at 2:21 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> On Wed, Feb 19, 2014 at 04:03:15PM -0600, Richard Laager wrote: >>> Can someone look at the route server logs and see who the new IPv4 peer >>> was that showed up just before this problem started? >> >> Nobody turned up recently. >> >> A better measure is who turned up on the switch rather than the route-server. >> This was a l2 flood, not anything in routing. >> >> No ports were up/down on the switch until the flood triggered >> my port and Implex's port offline (probably the same class of cisco switch >> protecting themselves from the l2 flood). >> >> Feb 19 15:11:45 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/3 >> Feb 19 15:11:48 MICE-SW1 mib2d[1107]: SNMP_TRAP_LINK_DOWN: ifIndex 606, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/5 >> >> So, it probably was from one of the existing remote switches. >> >> The MAC address I posted about before is an Extreme? switch. >> >> >> Also, the l2 flood seems abated. >> >> I've been pinging clear to the route servers for 5 minutes now. >> >> >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke > CTO, CCIE #19093 > Mankato Networks LLC > PO Box 54 > 619 S Front St > Mankato, MN 56001-3838 > Google 530-618-2398 > jayhanke(a)mankatonetworks.net > http://www.mankatonetworks.com > http://www.neutralpath.net -- Jay Hanke CTO, CCIE #19093 Mankato Networks LLC PO Box 54 619 S Front St Mankato, MN 56001-3838 Google 530-618-2398 jayhanke(a)mankatonetworks.net http://www.mankatonetworks.com http://www.neutralpath.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014

Re: Mice Issues

by Larry Patterson

Did they plug in a switch or a router? Was it a loop? Are there loop mitigation techniques in place to automatically shutdown offending ports? When would you suggest that it's safe to come back online? On 2/19/14 4:40 PM, "Jay Hanke" <jayhanke(a)MANKATONETWORKS.NET> wrote: >HCMC turned up peering this afternoon, l2 has been up for several days. > >I think the timing was maybe 20 minutes prior to issue. > > > >On Wed, Feb 19, 2014 at 2:03 PM, Richard Laager <rlaager(a)wiktel.com> >wrote: >> Can someone look at the route server logs and see who the new IPv4 peer >> was that showed up just before this problem started? >> >> -- >> Richard >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > >-- >Jay Hanke >CTO, CCIE #19093 >Mankato Networks LLC >PO Box 54 >619 S Front St >Mankato, MN 56001-3838 >Google 530-618-2398 >jayhanke(a)mankatonetworks.net >http://www.mankatonetworks.com >http://www.neutralpath.net > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2014