- mice-discuss - lists.micemn.net

Re: Packet loss across the exchange

by Anthony Anderberg

I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table >From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony

Jan. 31, 2015

Re: Packet loss across the exchange

by Hannigan, Martin

The table switching question (and others) below? The internet is helpful sometimes. Best, -M< > On Jan 30, 2015, at 19:17, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > In response to what question? > >> On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote: >> Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. >> >> Best, >> >> Marty >> >> >> >> >> >>>> On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: >>>> >>>> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: >>>> Does anyone know the status of the service contract renewal post UG15? >>>> I feel strongly that the switching table issue should be run past >>>> Juniper tech support. >>> >>> I've sent the Mankato Networks invoice to our (Wiktel's) accounts >>> payable. From Dan's reply, it seems we haven't paid it yet. >>> >>> -- >>> Richard > > > -- > Richard

Jan. 31, 2015

Re: Packet loss across the exchange

by Richard Laager

In response to what question? On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote: > Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. > > Best, > > Marty > > > > > > > On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > > >> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: > >> Does anyone know the status of the service contract renewal post UG15? > >> I feel strongly that the switching table issue should be run past > >> Juniper tech support. > > > > I've sent the Mankato Networks invoice to our (Wiktel's) accounts > > payable. From Dan's reply, it seems we haven't paid it yet. > > > > -- > > Richard -- Richard

Jan. 31, 2015

Re: Packet loss across the exchange

by Hannigan, Martin

Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. Best, Marty > On Jan 30, 2015, at 18:40, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > >> On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: >> Does anyone know the status of the service contract renewal post UG15? >> I feel strongly that the switching table issue should be run past >> Juniper tech support. > > I've sent the Mankato Networks invoice to our (Wiktel's) accounts > payable. From Dan's reply, it seems we haven't paid it yet. > > -- > Richard

Jan. 30, 2015

Re: Packet loss across the exchange

by Richard Laager

On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: > Does anyone know the status of the service contract renewal post UG15? > I feel strongly that the switching table issue should be run past > Juniper tech support. I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet. -- Richard

Jan. 30, 2015

Re: Packet loss across the exchange

by dangieser

It's in the works, but I haven't seen payment or a PO to order it yet. Dan Gieser Mankato Networks Cell: 507-327-5341 Desk: 507-242-6469 dangieser(a)mankatonetworks.net ------ Original Message ------ From: "Reid Fishler" <rfishler(a)he.net> To: MICE-DISCUSS(a)lists.iphouse.net Sent: 1/30/2015 5:34:03 PM Subject: Re: [MICE-DISCUSS] Packet loss across the exchange >I believe it has been paid now? > >Reid > > >On Fri, Jan 30, 2015 at 5:53 PM, Jeremy Lumby <jlumby(a)mnvoip.com> >wrote: >>Does anyone know the status of the service contract renewal post UG15? >> I feel strongly that the switching table issue should be run past >>Juniper tech support. >> >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Doug McIntyre >>Sent: Friday, January 30, 2015 4:30 PM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Packet loss across the exchange >> >>On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote: >> > Perhaps we need another >> clear ethernet-switching table >> >>I was able to get in there and do that.. >> >>-- >>Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> > > > >-- >Reid Fishler >Director >Hurricane Electric >+1-510-580-4178 > >-------------------------------------------------------------------------------- >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Jan. 30, 2015

Re: Packet loss across the exchange

by Reid Fishler

I believe it has been paid now? Reid On Fri, Jan 30, 2015 at 5:53 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Does anyone know the status of the service contract renewal post UG15? I > feel strongly that the switching table issue should be run past Juniper > tech support. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Doug McIntyre > Sent: Friday, January 30, 2015 4:30 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Packet loss across the exchange > > On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote: > > Perhaps we need another > clear ethernet-switching table > > I was able to get in there and do that.. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > -- Reid Fishler Director Hurricane Electric +1-510-580-4178

Jan. 30, 2015

Re: Packet loss across the exchange

by Richard Laager

On Fri, 2015-01-30 at 16:55 -0600, Doug McIntyre wrote: > It doesn't look like Anthony applied the controls onto the leaf > switches, like Mankato Networks', that is currently not applied. 100 Mbps is a lot of broadcast traffic. It seems to me that it should be safe to apply this limit to ports facing remote switches. -- Richard

Jan. 30, 2015

Re: Packet loss across the exchange

by Doug McIntyre

On Fri, Jan 30, 2015 at 04:52:15PM -0600, Andrew Hoyos wrote: > Only suggestion would be to make the storm control limits consistent (i.e.: Juniper == 10% of 1g port right now, Cisco = 20%) > > Would be interested to see if same (or similar) config is on the main MICE switch ports facing the leaf switches. Per the discussion at the last member meeting about how the treat the leaf switches (as a member connection), would seem appropriate to have these limits there too (and hope that similar config is in place downstream on member run leaf switches too). It doesn't look like Anthony applied the controls onto the leaf switches, like Mankato Networks', that is currently not applied. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Jan. 30, 2015

Re: Packet loss across the exchange

by Jeremy Lumby

Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, January 30, 2015 4:30 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote: > Perhaps we need another clear ethernet-switching table I was able to get in there and do that.. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Jan. 30, 2015