- mice-discuss - lists.micemn.net

Re: Fwd: Case # 137226 - BGP Session - MICE RS 1 Down

by Mike Horwath

On Sun, Feb 08, 2015 at 10:41:56AM -0600, Nevin Lyne wrote: > Have some newer machines I can donate. I will touch bases with Doug > on my next trip to Minneapolis. You're the best non-participant member of MICE evar. And thanks for the $20 to say that! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Feb. 9, 2015

Re: Fwd: Case # 137226 - BGP Session - MICE RS 1 Down

by Nevin Lyne

On Sunday, February 8, 2015 9:29am, "Mike Horwath" <drechsau(a)GEEKS.ORG> said: > On Wed, Feb 04, 2015 at 07:59:27AM -0600, Doug McIntyre wrote: >> The machine hung again, and I remotely power cycled it. Not sure if >> the RAID system went through a battery cycle and made it hang, been >> seeing more and more Dell stuff do that. :-( > > Time for some new machines if you ask me. Have some newer machines I can donate. I will touch bases with Doug on my next trip to Minneapolis. -Nevin -- Nevin Lyne -- Founder & Director of Technology -- Arcustech, LLC. - arcustech.com -- Gippy's Internet Solutions, LLC. - enginehosting.com

Feb. 8, 2015

Re: Fwd: Case # 137226 - BGP Session - MICE RS 1 Down

by Mike Horwath

On Wed, Feb 04, 2015 at 07:59:27AM -0600, Doug McIntyre wrote: > The machine hung again, and I remotely power cycled it. Not sure if > the RAID system went through a battery cycle and made it hang, been > seeing more and more Dell stuff do that. :-( Time for some new machines if you ask me. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Feb. 8, 2015

FW: ARIN Fellowship Program

by Frank Bulk

FYI, just in case there are those who are interested in policy. _____ Time is running out! Don't miss your chance to submit an application for the ARIN 35 Fellowship program to take advantage of this great opportunity to join us at ARIN 35 in San Francisco from 12-15 April 2015. If you have never attended an ARIN meeting, we invite you to come and learn more about ARIN services, meet industry peers and voice your opinions in policy discussions. Take a few minutes and submit your application before next week's Feb. 13 deadline at: https://www.arin.net/participate/meetings/fellowship.html Selected Fellows will receive free airfare, hotel, and a stipend. Three fellowships are available, one for each area of the ARIN service region: Caribbean, USA, and Canada. This program was developed to encourage new participants, so if you regularly attend ARIN meetings please forward this information to a colleague in order to help ARIN cultivate new voices. Please direct any questions you have to info(a)arin.net<mailto:info@arin.net>. Regards, Communications and Member Services American Registry for Internet Numbers (ARIN)

Feb. 6, 2015

Re: Fwd: Case # 137226 - BGP Session - MICE RS 1 Down

by Doug McIntyre

> From: Support <Support(a)usinternet.com> > Subject: Case # 137226 - BGP Session - MICE RS 1 Down > > We are currently seeing one of our BGP Sessions to you as down using the IP > of 206.108.255.1. The machine hung again, and I remotely power cycled it. Not sure if the RAID system went through a battery cycle and made it hang, been seeing more and more Dell stuff do that. :-( -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Feb. 4, 2015

Fwd: Case # 137226 - BGP Session - MICE RS 1 Down

by Jason Hanke

---------- Forwarded message ---------- From: Support <Support(a)usinternet.com> Date: Wed, Feb 4, 2015 at 5:24 AM Subject: Case # 137226 - BGP Session - MICE RS 1 Down To: "peering(a)micemn.net" <peering(a)micemn.net> Hello, We are currently seeing one of our BGP Sessions to you as down using the IP of 206.108.255.1. Are you currently experiencing an outage or doing maintenance that would cause this? Please let us know. Thank you, *Grant Johnson* Technical Support 952-253-3290 ・ 1-800-US-INTERNET 12450 Wayzata Blvd. Suite 315 ・ Minnetonka, MN 55305 [image: Description: Sig Temp] <http://usinternet.com/> -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Feb. 4, 2015

Re: Packet loss across the exchange

by Mike Horwath

On Sat, Jan 31, 2015 at 02:55:12AM +0000, Anthony Anderberg wrote: [wrote a reasonable response] Thanks Mr. Anderberg. ObTroll: if there were port fees this issue of money wouldn't be an issue -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Jan. 31, 2015

Re: Packet loss across the exchange

by Levi Pederson

Well defined and thought out response On Jan 30, 2015 8:55 PM, "Anthony Anderberg" < anthonyanderberg(a)nu-telecom.net> wrote: > I was on the road this afternoon and unable to safely contribute or > comment, but it looks like everyone came together and correctly diagnosed > and resolved the issues. I thought I'd follow up to a few > questions/comments to make sure all is clear. > > > clear ethernet-switching table > > From the logs, comments, and graphs I'm sure we ran into that software bug > again, regardless of any original leaf-node issues. The switch gets into a > state where MAC addresses are not all correctly added to the L2 forwarding > database so some destinations are treated as broadcasts. After issuing the > command above the switch recovers from its funk and the table populates > normally. Once our maintenance is reinstated we'll try to carve out some > time to open a ticket, and upgrade software if needed. > > Thanks to our friends at Wistrom Telephone for stepping up to the plate > once again on that. Speaking of which, given how important the exchange > seems to be to members I assume everyone will be eager to offer both time > and financial support as needs arise in the future. There's nothing magic > happening here, just hard work and capital... same as any endeavor. > > > It doesn't look like Anthony applied the controls onto > > the leaf switches, like Mankato Networks', that is > > currently not applied. > > Also correct, our L2 security plans centered around member facing ports - > the definition of which may shift over time, along with configuration. > > Note that we do have spanning tree running between the exchange's three > switches, and it appears to have worked properly to change the state of the > Mankato switch port, but switching-table bug reared its ugly > head at the same time. > > > Only suggestion would be to make the storm control limits > > consistent (i.e.: Juniper == 10% of 1g port right now, > > Cisco = 20%) > > That's certainly something that can be revisited, the current values were > somewhat arbitrary - our goal was to start off liberal to avoid any chance > of tripping up normal traffic. > > As a side note, we've had several times where the L2 security configs have > shutdown individual member ports because of ingress BPDUs or MAC address > limits so I'm sure they're working correctly. The storm > control is a little harder to judge, logs show it kicking in during this > afternoon's incident but the underlying bug made it ineffective. > > Thanks everyone, > anthony >

Jan. 31, 2015

Re: Packet loss across the exchange

by Anthony Anderberg

I don’t have a case yet, without money we’d let our support contract lapse – that seems to be getting resolved though. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Adam White Sent: Friday, January 30, 2015 9:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss across the exchange Thanks for the info Anthony. Do you know if Juniper has a PR # (or a case #) for the bug you describe? I'd like to get some more info on it. On Jan 30, 2015 8:55 PM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET<mailto:anthonyanderberg@NU-TELECOM.NET>> wrote: I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 31, 2015

Re: Packet loss across the exchange

by Adam White

Thanks for the info Anthony. Do you know if Juniper has a PR # (or a case #) for the bug you describe? I'd like to get some more info on it. On Jan 30, 2015 8:55 PM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET> wrote: I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues. I thought I'd follow up to a few questions/comments to make sure all is clear. > clear ethernet-switching table From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues. The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts. After issuing the command above the switch recovers from its funk and the table populates normally. Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed. Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that. Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future. There's nothing magic happening here, just hard work and capital... same as any endeavor. > It doesn't look like Anthony applied the controls onto > the leaf switches, like Mankato Networks', that is > currently not applied. Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration. Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly head at the same time. > Only suggestion would be to make the storm control limits > consistent (i.e.: Juniper == 10% of 1g port right now, > Cisco = 20%) That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic. As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly. The storm control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective. Thanks everyone, anthony

Jan. 31, 2015