Packet loss across the exchange
Just wondering if others are seeing packet loss across the exchange that just started Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
Indeed...seeing it here as well. BGP sessions are bouncing. *Matthew Beckwell* - Director of Network Operations Advanced Integrated Technologies Technical Support: 1-800-300-5408 Office: 1-952-829-5511 x204 E-Mail: matthewb@aitech.net On Fri, Jan 30, 2015 at 4:09 PM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
Just wondering if others are seeing packet loss across the exchange that just started
Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, January 30, 2015 4:30 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote:
Perhaps we need another clear ethernet-switching table
I was able to get in there and do that.. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
I believe it has been paid now? Reid On Fri, Jan 30, 2015 at 5:53 PM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, January 30, 2015 4:30 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange
On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote:
Perhaps we need another clear ethernet-switching table
I was able to get in there and do that..
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
-- Reid Fishler Director Hurricane Electric +1-510-580-4178
It's in the works, but I haven't seen payment or a PO to order it yet. Dan Gieser Mankato Networks Cell: 507-327-5341 Desk: 507-242-6469 dangieser@mankatonetworks.net ------ Original Message ------ From: "Reid Fishler" <rfishler@he.net> To: MICE-DISCUSS@lists.iphouse.net Sent: 1/30/2015 5:34:03 PM Subject: Re: [MICE-DISCUSS] Packet loss across the exchange
I believe it has been paid now?
Reid
On Fri, Jan 30, 2015 at 5:53 PM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, January 30, 2015 4:30 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange
On Fri, Jan 30, 2015 at 04:19:07PM -0600, Steve Howard wrote:
Perhaps we need another clear ethernet-switching table
I was able to get in there and do that..
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
-- Reid Fishler Director Hurricane Electric +1-510-580-4178
-------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote:
Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet. -- Richard
Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer. Best, Marty
On Jan 30, 2015, at 18:40, Richard Laager <rlaager@WIKTEL.COM> wrote:
On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet.
-- Richard
In response to what question? On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote:
Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer.
Best,
Marty
On Jan 30, 2015, at 18:40, Richard Laager <rlaager@WIKTEL.COM> wrote:
On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet.
-- Richard
-- Richard
The table switching question (and others) below? The internet is helpful sometimes. Best, -M<
On Jan 30, 2015, at 19:17, Richard Laager <rlaager@WIKTEL.COM> wrote:
In response to what question?
On Fri, 2015-01-30 at 23:55 +0000, Hannigan, Martin wrote: Open IX has a public group with all the IXP ops on it. You could go there and ask and probably get a real answer.
Best,
Marty
On Jan 30, 2015, at 18:40, Richard Laager <rlaager@WIKTEL.COM> wrote:
On Fri, 2015-01-30 at 16:53 -0600, Jeremy Lumby wrote: Does anyone know the status of the service contract renewal post UG15? I feel strongly that the switching table issue should be run past Juniper tech support.
I've sent the Mankato Networks invoice to our (Wiktel's) accounts payable. From Dan's reply, it seems we haven't paid it yet.
-- Richard
-- Richard
I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear.
clear ethernet-switching table
From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed.
Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor.
It doesn't look like Anthony applied the controls onto the leaf switches, like Mankato Networks', that is currently not applied.
Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time.
Only suggestion would be to make the storm control limits consistent (i.e.: Juniper == 10% of 1g port right now, Cisco = 20%)
That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony
Well defined and thought out response On Jan 30, 2015 8:55 PM, "Anthony Anderberg" < anthonyanderberg@nu-telecom.net> wrote:
I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear.
clear ethernet-switching table
From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed.
Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor.
It doesn't look like Anthony applied the controls onto the leaf switches, like Mankato Networks', that is currently not applied.
Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration.
Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time.
Only suggestion would be to make the storm control limits consistent (i.e.: Juniper == 10% of 1g port right now, Cisco = 20%)
That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic.
As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective.
Thanks everyone, anthony
On Sat, Jan 31, 2015 at 02:55:12AM +0000, Anthony Anderberg wrote: [wrote a reasonable response] Thanks Mr. Anderberg. ObTroll: if there were port fees this issue of money wouldn't be an issue -- Mike Horwath, reachable via drechsau@Geeks.ORG
All, We're shifting some traffic onto our 10Gig link, and accidentally created an L2 Loop within the Exchange. We've since rectified it. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net On Fri, Jan 30, 2015 at 4:09 PM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
Just wondering if others are seeing packet loss across the exchange that just started
Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
I thought some sort of loop detection / storm control was configured recently? Do we need to revisit that? Seems like shutting down a port (even if a leaf node) at 10% broadcast limit would be better than effecting the whole exchange. With proper config, this shouldn’t be possible (or at least the effects minimized to a single port). I’d be happy to volunteer time/join a team to help accomplish. -- Andrew Hoyos hoyosa@gmail.com
On Jan 30, 2015, at 4:17 PM, Levi Pederson <levipederson@MANKATONETWORKS.NET> wrote:
All,
We're shifting some traffic onto our 10Gig link, and accidentally created an L2 Loop within the Exchange. We've since rectified it.
Thank you,
Levi Pederson Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net <mailto:levipederson@mankatonetworks.net>
On Fri, Jan 30, 2015 at 4:09 PM, Jeremy Lumby <jlumby@mnvoip.com <mailto:jlumby@mnvoip.com>> wrote: Just wondering if others are seeing packet loss across the exchange that just started
Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 <tel:612-355-7740%20x211> Direct: 612-392-6814 <tel:612-392-6814> EFax: 952-873-7425 <tel:952-873-7425> jlumby@mnvoip.com <mailto:jlumby@mnvoip.com>
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
I agree. We all need to make efforts to improve the stability of the exchange. This is causing Atomic issues with reliability and we are considering bypassing the exchange for our critical traffic. ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Andrew Hoyos <hoyosa@GMAIL.COM> Sent: Friday, January 30, 2015 4:24 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange I thought some sort of loop detection / storm control was configured recently? Do we need to revisit that? Seems like shutting down a port (even if a leaf node) at 10% broadcast limit would be better than effecting the whole exchange. With proper config, this shouldn't be possible (or at least the effects minimized to a single port). I'd be happy to volunteer time/join a team to help accomplish. -- Andrew Hoyos hoyosa@gmail.com<mailto:hoyosa@gmail.com> Larry Patterson CTO ---- Atomic Data // Safe. Simple. Smart.(tm) 615 North 3rd Street Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2080 Facebook<http://www.facebook.com/AtomicDataCenters> // Twitter<http://twitter.com/AtomicData> // LinkedIn<http://www.linkedin.com/company/atomic-data> On Jan 30, 2015, at 4:17 PM, Levi Pederson <levipederson@MANKATONETWORKS.NET<mailto:levipederson@MANKATONETWORKS.NET>> wrote: All, We're shifting some traffic onto our 10Gig link, and accidentally created an L2 Loop within the Exchange. We've since rectified it. Thank you, Levi Pederson Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net<mailto:levipederson@mankatonetworks.net> [http://www.mankatonetworks.com/images/mn_logo_email.png] On Fri, Jan 30, 2015 at 4:09 PM, Jeremy Lumby <jlumby@mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Just wondering if others are seeing packet loss across the exchange that just started Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211<tel:612-355-7740%20x211> Direct: 612-392-6814<tel:612-392-6814> EFax: 952-873-7425<tel:952-873-7425> jlumby@mnvoip.com<mailto:jlumby@mnvoip.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Fri, 2015-01-30 at 16:24 -0600, Andrew Hoyos wrote:
I thought some sort of loop detection / storm control was configured recently? Do we need to revisit that? Seems like shutting down a port (even if a leaf node) at 10% broadcast limit would be better than effecting the whole exchange. With proper config, this shouldn’t be possible (or at least the effects minimized to a single port).
The config is listed here: http://micemn.net/technical.html Specifically: set ethernet-switching-options storm-control interface X/X/X bandwidth 100000 - and on Cisco - storm-control broadcast level 20.00 I can't say personally whether that's actually applied, but I believe that's been completed. Maybe it wasn't applied to the ports facing remote switches. Can anyone with access to the switches comment on that? I'm not sure off the top of my head what level of traffic these numbers represent. -- Richard
On Jan 30, 2015, at 4:39 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
The config is listed here: http://micemn.net/technical.html <http://micemn.net/technical.html> Ahh, cool. Didn’t realize this was out there now. This looks totally sane and awesome, and what would be expected. Only suggestion would be to make the storm control limits consistent (i.e.: Juniper == 10% of 1g port right now, Cisco = 20%)
Would be interested to see if same (or similar) config is on the main MICE switch ports facing the leaf switches. Per the discussion at the last member meeting about how the treat the leaf switches (as a member connection), would seem appropriate to have these limits there too (and hope that similar config is in place downstream on member run leaf switches too). -- Andrew Hoyos hoyosa@gmail.com
On Fri, Jan 30, 2015 at 04:52:15PM -0600, Andrew Hoyos wrote:
Only suggestion would be to make the storm control limits consistent (i.e.: Juniper == 10% of 1g port right now, Cisco = 20%)
Would be interested to see if same (or similar) config is on the main MICE switch ports facing the leaf switches. Per the discussion at the last member meeting about how the treat the leaf switches (as a member connection), would seem appropriate to have these limits there too (and hope that similar config is in place downstream on member run leaf switches too).
It doesn't look like Anthony applied the controls onto the leaf switches, like Mankato Networks', that is currently not applied. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Fri, 2015-01-30 at 16:55 -0600, Doug McIntyre wrote:
It doesn't look like Anthony applied the controls onto the leaf switches, like Mankato Networks', that is currently not applied.
100 Mbps is a lot of broadcast traffic. It seems to me that it should be safe to apply this limit to ports facing remote switches. -- Richard
Something is still wrong with the exchange. I wonder if the loop tripped off the forwarding issue. There is no reason for there to be hundreds of megs of traffic hitting the route servers From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Levi Pederson Sent: Friday, January 30, 2015 4:18 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange All, We're shifting some traffic onto our 10Gig link, and accidentally created an L2 Loop within the Exchange. We've since rectified it. Thank you, Levi Pederson Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net On Fri, Jan 30, 2015 at 4:09 PM, Jeremy Lumby <jlumby@mnvoip.com> wrote: Just wondering if others are seeing packet loss across the exchange that just started Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (13)
-
Andrew Hoyos -
Anthony Anderberg -
dangieser -
Doug McIntyre -
Hannigan, Martin -
Jeremy Lumby -
Larry Patterson -
Levi Pederson -
Matthew Beckwell -
Mike Horwath -
Reid Fishler -
Richard Laager -
Steve Howard