January 2015 - mice-discuss - lists.micemn.net

Re: Packet loss across the exchange

by Mike Horwath

On Sat, Jan 31, 2015 at 02:55:12AM +0000, Anthony Anderberg wrote: [wrote a reasonable response] Thanks Mr. Anderberg. ObTroll: if there were port fees this issue of money wouldn't be an issue -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Jan. 31, 2015

Re: Packet loss across the exchange

by Levi Pederson

Well defined and thought out response On Jan 30, 2015 8:55 PM, "Anthony Anderberg" < anthonyanderberg(a)nu-telecom.net> wrote: > I was on the road this afternoon and unable to safely contribute or > comment, but it looks like everyone came together and correctly diagnosed > and resolved the issues. I thought I'd follow up to a few > questions/comments to make sure all is clear. > > > clear ethernet-switching table > > From the logs, comments, and graphs I'm sure we ran into that software bug > again, regardless of any original leaf-node issues. The switch gets into a > state where MAC addresses are not all correctly added to the L2 forwarding > database so some destinations are treated as broadcasts. After issuing the > command above the switch recovers from its funk and the table populates > normally. Once our maintenance is reinstated we'll try to carve out some > time to open a ticket, and upgrade software if needed. > > Thanks to our friends at Wistrom Telephone for stepping up to the plate > once again on that. Speaking of which, given how important the exchange > seems to be to members I assume everyone will be eager to offer both time > and financial support as needs arise in the future. There's nothing magic > happening here, just hard work and capital... same as any endeavor. > > > It doesn't look like Anthony applied the controls onto > > the leaf switches, like Mankato Networks', that is > > currently not applied. > > Also correct, our L2 security plans centered around member facing ports - > the definition of which may shift over time, along with configuration. > > Note that we do have spanning tree running between the exchange's three > switches, and it appears to have worked properly to change the state of the > Mankato switch port, but switching-table bug reared its ugly > head at the same time. > > > Only suggestion would be to make the storm control limits > > consistent (i.e.: Juniper == 10% of 1g port right now, > > Cisco = 20%) > > That's certainly something that can be revisited, the current values were > somewhat arbitrary - our goal was to start off liberal to avoid any chance > of tripping up normal traffic. > > As a side note, we've had several times where the L2 security configs have > shutdown individual member ports because of ingress BPDUs or MAC address > limits so I'm sure they're working correctly. The storm > control is a little harder to judge, logs show it kicking in during this > afternoon's incident but the underlying bug made it ineffective. > > Thanks everyone, > anthony >

Jan. 31, 2015

Re: Packet loss across the exchange

by Anthony Anderberg

I don’t have a case yet, without money we’d let our support contract lapse – that seems to be getting resolved though. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Adam White Sent: Friday, January 30, 2015 9:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange Thanks for the info Anthony. Do you know if Juniper has a PR # (or a case #) for the bug you describe? I'd like to get some more info on it. On Jan 30, 2015 8:55 PM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET<mailto:anthonyanderberg@NU-TELECOM.NET>> wrote: I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 31, 2015

Re: Packet loss across the exchange

by Adam White

Thanks for the info Anthony. Do you know if Juniper has a PR # (or a case #) for the bug you describe? I'd like to get some more info on it. On Jan 30, 2015 8:55 PM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET> wrote: I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony

Jan. 31, 2015

Re: Packet loss across the exchange

by Anthony Anderberg

I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table >From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony

Jan. 31, 2015

Re: Packet loss across the exchange

by Hannigan, Martin

The table switching question (and others) below? The internet is helpful sometimes. Best, -M< > On Jan 30, 2015, at 19:17, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > In response to what question? > >> On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote: >> Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. >> >> Best, >> >> Marty >> >> >> >> >> >>>> On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: >>>> >>>> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: >>>> Does anyone know the status of the service contract renewal post UG15? >>>> I feel strongly that the switching table issue should be run past >>>> Juniper tech support. >>> >>> I've sent the Mankato Networks invoice to our (Wiktel's) accounts >>> payable. From Dan's reply, it seems we haven't paid it yet. >>> >>> -- >>> Richard > > > -- > Richard

Jan. 31, 2015

Re: Packet loss across the exchange

by Richard Laager

In response to what question? On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote: > Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. > > Best, > > Marty > > > > > > > On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > > >> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: > >> Does anyone know the status of the service contract renewal post UG15? > >> I feel strongly that the switching table issue should be run past > >> Juniper tech support. > > > > I've sent the Mankato Networks invoice to our (Wiktel's) accounts > > payable. From Dan's reply, it seems we haven't paid it yet. > > > > -- > > Richard -- Richard

Jan. 31, 2015

Re: Packet loss across the exchange

by Hannigan, Martin

Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. Best, Marty > On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > >> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: >> Does anyone know the status of the service contract renewal post UG15? >> I feel strongly that the switching table issue should be run past >> Juniper tech support. > > I've sent the Mankato Networks invoice to our (Wiktel's) accounts > payable. From Dan's reply, it seems we haven't paid it yet. > > -- > Richard

Jan. 30, 2015

Re: Packet loss across the exchange

by Richard Laager

On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: > Does anyone know the status of the service contract renewal post UG15? > I feel strongly that the switching table issue should be run past > Juniper tech support. I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet. -- Richard

Jan. 30, 2015

Re: Packet loss across the exchange

by dangieser

It's in the works, but I haven't seen payment or a PO to order it yet. Dan Gieser Mankato Networks Cell: 507-327-5341 Desk: 507-242-6469 dangieser(a)mankatonetworks.net ------ Original Message ------ From: "Reid Fishler" <rfishler(a)he.net> To: MICE-DISCUSS(a)lists.iphouse.net Sent: 1/30/2015 5:34:03 PM Subject: Re: [MICE-DISCUSS] Packet loss across the exchange >I believe it has been paid now? > >Reid > > >On Fri, Jan 30, 2015 at 5:53 PM, Jeremy Lumby <jlumby(a)mnvoip.com> >wrote: >>Does anyone know the status of the service contract renewal post UG15? >> I feel strongly that the switching table issue should be run past >>Juniper tech support. >> >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Doug McIntyre >>Sent: Friday, January 30, 2015 4:30 PM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Packet loss across the exchange >> >>On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote: >> > Perhaps we need another >> clear ethernet-switching table >> >>I was able to get in there and do that.. >> >>-- >>Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> > > > >-- >Reid Fishler >Director >Hurricane Electric >+1-510-580-4178 > >-------------------------------------------------------------------------------- >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Jan. 30, 2015