- mice-discuss - lists.micemn.net

Re: IPv6 routing issue

by Reid Fishler

If anyone is seeing problems, PLEASE email noc(a)he.net. Reid On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote: > > This morning at approximately 1:30 our peering session with HE > > reset, I assume for some kind of normal maintenance on their end. > > > > This triggered a weird issue on our end. Because of some crufty > > route policy we were preferring the routes we learned from the MICE > > route server over the routes we learned directly from HE. And for > > some reason the routes we were learning from the route server had a > > different link-local address associated with them than the routes we > > learned directly. And this address no longer seems to be valid, and > > this caused traffic to be black holed. > > > > From below > > FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS > > FE80::224:38FF:FEA7:EA36 - address on routes direct from HE > > > FIWW: We do buy transit from HE and did get the maintenance notice for > this morning. > > I've also seen this behavior from them before. I'm not sure if the link > local address changes on their end due to some change or what is going > on. It doesn't matter on our direct connect router, where we only > talk the public IPv6 address, but the MICE connection seems to have > stale link-local IPv6 addresses from time to time that I've had to > clear out due to some change. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > -- Reid Fishler Director Hurricane Electric +1-510-580-4178 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Doug McIntyre

On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote: > This morning at approximately 1:30 our peering session with HE > reset, I assume for some kind of normal maintenance on their end. > > This triggered a weird issue on our end. Because of some crufty > route policy we were preferring the routes we learned from the MICE > route server over the routes we learned directly from HE. And for > some reason the routes we were learning from the route server had a > different link-local address associated with them than the routes we > learned directly. And this address no longer seems to be valid, and > this caused traffic to be black holed. > > From below > FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS > FE80::224:38FF:FEA7:EA36 - address on routes direct from HE FIWW: We do buy transit from HE and did get the maintenance notice for this morning. I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Jeremy Lumby

I buy from them, and got 2 notices. The second one corrected the date. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Bushard, Jr Sent: Wednesday, April 03, 2013 7:48 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] IPv6 routing issue We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr(a)mailbag.com> wrote: On 4/3/2013 6:59 AM, Anthony Anderberg wrote: By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Mike Bushard Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Mike Bushard, Jr

We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr(a)mailbag.com> wrote: > On 4/3/2013 6:59 AM, Anthony Anderberg wrote: > >> By my notes HE did have a maintenance window scheduled this morning to do >> a "Router hardware upgrade to provide for expanded services and upgrades." >> >> >> > I saw the flaps this morning, but I don't recall receiving a maintenance > notification. Do you buy transit from HE? > > James Stahr > CDW > > > ##############################**##############################** > ############ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-**bin/wa?SUBED1=MICE-DISCUSS&A=1<http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > -- Mike Bushard Jr | Network Engineer IV *| *Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by James Stahr

On 4/3/2013 6:59 AM, Anthony Anderberg wrote: > By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." > > I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Anthony Anderberg

By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I see our traffic levels from them dropped quite a bit just after 1:00am and have more-or-less returned to normal now. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg(a)nu-telecom.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

IPv6 routing issue

by David Farmer

This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end. This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed. From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE I changed my route policy to ignore the routes from HE through the MICE Route Server and things are working again. I'm not sure what changed and why the routes from the MICE Route server have an bogus link-local address. But I though I would warn you all. MTC-GR-01#sh ipv6 ro 2001:470:0:1C2::1 Routing entry for 2001:470::/32 Known via "bgp 57", distance 20, metric 1, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA03, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:15 ago MTC-GR-01#sh ipv6 ro 2600:0:2:1239:144:232:26:11 Routing entry for 2600::/29 Known via "bgp 57", distance 20, metric 0, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA36, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:52 ago -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: Integra's Status

by Mike Bushard, Jr

Thanks Tony. On Mon, Mar 25, 2013 at 8:30 AM, Radzwon, Tony < Tony.Radzwon(a)integratelecom.com> wrote: > Jay is correct, it specifically affected Arvig and possibly others in > Northern Minnesota, we have an older Cisco 7600 in St.Cloud that needed to > have Next-Hop-Self turned on to be able to reach those specific customers > via BGP. The Integra NOC is suppose to make the addition during a > Maintenance window, sorry for this not going faster, we are trying to > slowly integrate the MICE & other Sites into being Maintained by our NOC > rather than Engineering. > > Regards, > > Tony > > Tony Radzwon | Network Engineer > direct 612.436.8009 | mobile 612.986.5715 > Integra | 511 11th ave. s. | Suite 340 > | Minneapolis, MN 55415 > www.integratelecom.com <http://www.integratelecom.com> > > <http://www.integratelecom.com/> <http://www.facebook.com/IntegraTelecom> > <http://twitter.com/Integra_Telecom> > <http://www.linkedin.com/company/integra> > > > > > > > > > > On 3/25/13 8:15 AM, "Jay Hanke" <jayhanke(a)MANKATONETWORKS.NET> wrote: > > >IIRC, the issue was with a specific prefix within Integra. The > >traceroutes indicated that the problem was after the MICE portion so > >one only saw the issue if they were talking to a specific IP range. > > > >Jay > > > >On Sun, Mar 24, 2013 at 1:11 PM, Mike Bushard, Jr > ><mike.bushard(a)arvig.com> wrote: > >> We did have complaints from customers. > >> > >> Mike Bushard Jr > >> | Network Engineer IV > >> | Arvig > >> | 320.256.0178 direct > >> | 320.429.0837 Mobile > >> > >> Sent from my Verizon Wireless DROID RAZR by Motorola > >> > >> Please excuse short responses and spelling errors. > >> > >> On Mar 24, 2013 12:47 PM, "Jeremy Lumby" <jlumby(a)mnvoip.com> wrote: > >>> > >>> Just checking to see if anyone knows what is going on with Integra. > >>>After > >>> the issues they had over a week ago, they have not responded to emails > >>>I > >>> sent to their peering address, as well as they never seemed to give a > >>>clear > >>> indication of what the issue really was in the first place. I was > >>>running > >>> RTP across the MICE connection to them at the time that they > >>>supposedly had > >>> 50% packet loss. I sure would have expected a whole lot of complaints > >>>if > >>> there was 50% loss. I never got a single complaint. Did anyone see > >>>this > >>> loss on their end? > >>> > >>> > >>>######################################################################## > >>> > >>> To unsubscribe from the MICE-DISCUSS list, click the following link: > >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > >> > >> > >> ________________________________ > >> > >> To unsubscribe from the MICE-DISCUSS list, click the following link: > >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > > > >-- > >Jay Hanke > >CTO, CCIE #19093 > >Mankato Networks LLC > >PO Box 54 > >619 S Front St > >Mankato, MN 56001-3838 > >Google 530-618-2398 > >jayhanke(a)mankatonetworks.net > >http://www.mankatonetworks.com > >http://www.neutralpath.net > > > >######################################################################## > > > >To unsubscribe from the MICE-DISCUSS list, click the following link: > >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Mike Bushard Jr | Network Engineer IV *| *Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 25, 2013

Re: Integra's Status

by David Farmer

Thanks for the explanation. However, is there an ETA for Integra to restore its peering at MICE? Thanks again. On 3/25/13 08:30 , Radzwon, Tony wrote: > Jay is correct, it specifically affected Arvig and possibly others in > Northern Minnesota, we have an older Cisco 7600 in St.Cloud that needed to > have Next-Hop-Self turned on to be able to reach those specific customers > via BGP. The Integra NOC is suppose to make the addition during a > Maintenance window, sorry for this not going faster, we are trying to > slowly integrate the MICE & other Sites into being Maintained by our NOC > rather than Engineering. > > Regards, > > Tony > > Tony Radzwon | Network Engineer > direct 612.436.8009 | mobile 612.986.5715 > Integra | 511 11th ave. s. | Suite 340 > | Minneapolis, MN 55415 > www.integratelecom.com <http://www.integratelecom.com> > > <http://www.integratelecom.com/> <http://www.facebook.com/IntegraTelecom> > <http://twitter.com/Integra_Telecom> > <http://www.linkedin.com/company/integra> > > > > > > > > > > On 3/25/13 8:15 AM, "Jay Hanke" <jayhanke(a)MANKATONETWORKS.NET> wrote: > >> IIRC, the issue was with a specific prefix within Integra. The >> traceroutes indicated that the problem was after the MICE portion so >> one only saw the issue if they were talking to a specific IP range. >> >> Jay >> >> On Sun, Mar 24, 2013 at 1:11 PM, Mike Bushard, Jr >> <mike.bushard(a)arvig.com> wrote: >>> We did have complaints from customers. >>> >>> Mike Bushard Jr >>> | Network Engineer IV >>> | Arvig >>> | 320.256.0178 direct >>> | 320.429.0837 Mobile >>> >>> Sent from my Verizon Wireless DROID RAZR by Motorola >>> >>> Please excuse short responses and spelling errors. >>> >>> On Mar 24, 2013 12:47 PM, "Jeremy Lumby" <jlumby(a)mnvoip.com> wrote: >>>> >>>> Just checking to see if anyone knows what is going on with Integra. >>>> After >>>> the issues they had over a week ago, they have not responded to emails >>>> I >>>> sent to their peering address, as well as they never seemed to give a >>>> clear >>>> indication of what the issue really was in the first place. I was >>>> running >>>> RTP across the MICE connection to them at the time that they >>>> supposedly had >>>> 50% packet loss. I sure would have expected a whole lot of complaints >>>> if >>>> there was 50% loss. I never got a single complaint. Did anyone see >>>> this >>>> loss on their end? >>>> >>>> >>>> ######################################################################## >>>> >>>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >>> >>> ________________________________ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke >> CTO, CCIE #19093 >> Mankato Networks LLC >> PO Box 54 >> 619 S Front St >> Mankato, MN 56001-3838 >> Google 530-618-2398 >> jayhanke(a)mankatonetworks.net >> http://www.mankatonetworks.com >> http://www.neutralpath.net >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 25, 2013

Re: Integra's Status

by Radzwon, Tony

Jay is correct, it specifically affected Arvig and possibly others in Northern Minnesota, we have an older Cisco 7600 in St.Cloud that needed to have Next-Hop-Self turned on to be able to reach those specific customers via BGP. The Integra NOC is suppose to make the addition during a Maintenance window, sorry for this not going faster, we are trying to slowly integrate the MICE & other Sites into being Maintained by our NOC rather than Engineering. Regards, Tony Tony Radzwon | Network Engineer direct 612.436.8009 | mobile 612.986.5715 Integra | 511 11th ave. s. | Suite 340 | Minneapolis, MN 55415 www.integratelecom.com <http://www.integratelecom.com> <http://www.integratelecom.com/> <http://www.facebook.com/IntegraTelecom> <http://twitter.com/Integra_Telecom> <http://www.linkedin.com/company/integra> On 3/25/13 8:15 AM, "Jay Hanke" <jayhanke(a)MANKATONETWORKS.NET> wrote: >IIRC, the issue was with a specific prefix within Integra. The >traceroutes indicated that the problem was after the MICE portion so >one only saw the issue if they were talking to a specific IP range. > >Jay > >On Sun, Mar 24, 2013 at 1:11 PM, Mike Bushard, Jr ><mike.bushard(a)arvig.com> wrote: >> We did have complaints from customers. >> >> Mike Bushard Jr >> | Network Engineer IV >> | Arvig >> | 320.256.0178 direct >> | 320.429.0837 Mobile >> >> Sent from my Verizon Wireless DROID RAZR by Motorola >> >> Please excuse short responses and spelling errors. >> >> On Mar 24, 2013 12:47 PM, "Jeremy Lumby" <jlumby(a)mnvoip.com> wrote: >>> >>> Just checking to see if anyone knows what is going on with Integra. >>>After >>> the issues they had over a week ago, they have not responded to emails >>>I >>> sent to their peering address, as well as they never seemed to give a >>>clear >>> indication of what the issue really was in the first place. I was >>>running >>> RTP across the MICE connection to them at the time that they >>>supposedly had >>> 50% packet loss. I sure would have expected a whole lot of complaints >>>if >>> there was 50% loss. I never got a single complaint. Did anyone see >>>this >>> loss on their end? >>> >>> >>>######################################################################## >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > >-- >Jay Hanke >CTO, CCIE #19093 >Mankato Networks LLC >PO Box 54 >619 S Front St >Mankato, MN 56001-3838 >Google 530-618-2398 >jayhanke(a)mankatonetworks.net >http://www.mankatonetworks.com >http://www.neutralpath.net > >######################################################################## > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 25, 2013