April 2013 - mice-discuss - lists.micemn.net

Re: IPv6 routing issue

by Reid Fishler

As announced at the MICE meeting, we are upgrading our MSP router to be 100G capable. This swap is now complete. If anyone is still seeing any problems, please email noc(a)he.net. Reid On Wed, Apr 3, 2013 at 10:01 AM, Reid Fishler <rfishler(a)he.net> wrote: > If anyone is seeing problems, PLEASE email noc(a)he.net. > > Reid > > > On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > >> On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote: >> > This morning at approximately 1:30 our peering session with HE >> > reset, I assume for some kind of normal maintenance on their end. >> > >> > This triggered a weird issue on our end. Because of some crufty >> > route policy we were preferring the routes we learned from the MICE >> > route server over the routes we learned directly from HE. And for >> > some reason the routes we were learning from the route server had a >> > different link-local address associated with them than the routes we >> > learned directly. And this address no longer seems to be valid, and >> > this caused traffic to be black holed. >> > >> > From below >> > FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS >> > FE80::224:38FF:FEA7:EA36 - address on routes direct from HE >> >> >> FIWW: We do buy transit from HE and did get the maintenance notice for >> this morning. >> >> I've also seen this behavior from them before. I'm not sure if the link >> local address changes on their end due to some change or what is going >> on. It doesn't matter on our direct connect router, where we only >> talk the public IPv6 address, but the MICE connection seems to have >> stale link-local IPv6 addresses from time to time that I've had to >> clear out due to some change. >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> -- ipHouse/Goldengate/Bitstream/ProNS -- >> Network Engineer/Provisioning/Jack of all Trades >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> > > > -- > Reid Fishler > Director > Hurricane Electric > +1-510-580-4178 > -- Reid Fishler Director Hurricane Electric +1-510-580-4178 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Reid Fishler

If anyone is seeing problems, PLEASE email noc(a)he.net. Reid On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote: > > This morning at approximately 1:30 our peering session with HE > > reset, I assume for some kind of normal maintenance on their end. > > > > This triggered a weird issue on our end. Because of some crufty > > route policy we were preferring the routes we learned from the MICE > > route server over the routes we learned directly from HE. And for > > some reason the routes we were learning from the route server had a > > different link-local address associated with them than the routes we > > learned directly. And this address no longer seems to be valid, and > > this caused traffic to be black holed. > > > > From below > > FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS > > FE80::224:38FF:FEA7:EA36 - address on routes direct from HE > > > FIWW: We do buy transit from HE and did get the maintenance notice for > this morning. > > I've also seen this behavior from them before. I'm not sure if the link > local address changes on their end due to some change or what is going > on. It doesn't matter on our direct connect router, where we only > talk the public IPv6 address, but the MICE connection seems to have > stale link-local IPv6 addresses from time to time that I've had to > clear out due to some change. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > -- Reid Fishler Director Hurricane Electric +1-510-580-4178 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Doug McIntyre

On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote: > This morning at approximately 1:30 our peering session with HE > reset, I assume for some kind of normal maintenance on their end. > > This triggered a weird issue on our end. Because of some crufty > route policy we were preferring the routes we learned from the MICE > route server over the routes we learned directly from HE. And for > some reason the routes we were learning from the route server had a > different link-local address associated with them than the routes we > learned directly. And this address no longer seems to be valid, and > this caused traffic to be black holed. > > From below > FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS > FE80::224:38FF:FEA7:EA36 - address on routes direct from HE FIWW: We do buy transit from HE and did get the maintenance notice for this morning. I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Jeremy Lumby

I buy from them, and got 2 notices. The second one corrected the date. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Bushard, Jr Sent: Wednesday, April 03, 2013 7:48 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] IPv6 routing issue We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr(a)mailbag.com> wrote: On 4/3/2013 6:59 AM, Anthony Anderberg wrote: By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Mike Bushard Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Mike Bushard, Jr

We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr(a)mailbag.com> wrote: > On 4/3/2013 6:59 AM, Anthony Anderberg wrote: > >> By my notes HE did have a maintenance window scheduled this morning to do >> a "Router hardware upgrade to provide for expanded services and upgrades." >> >> >> > I saw the flaps this morning, but I don't recall receiving a maintenance > notification. Do you buy transit from HE? > > James Stahr > CDW > > > ##############################**##############################** > ############ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-**bin/wa?SUBED1=MICE-DISCUSS&A=1<http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > -- Mike Bushard Jr | Network Engineer IV *| *Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by James Stahr

On 4/3/2013 6:59 AM, Anthony Anderberg wrote: > By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." > > I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

Re: IPv6 routing issue

by Anthony Anderberg

By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I see our traffic levels from them dropped quite a bit just after 1:00am and have more-or-less returned to normal now. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg(a)nu-telecom.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013

IPv6 routing issue

by David Farmer

This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end. This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed. From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE I changed my route policy to ignore the routes from HE through the MICE Route Server and things are working again. I'm not sure what changed and why the routes from the MICE Route server have an bogus link-local address. But I though I would warn you all. MTC-GR-01#sh ipv6 ro 2001:470:0:1C2::1 Routing entry for 2001:470::/32 Known via "bgp 57", distance 20, metric 1, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA03, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:15 ago MTC-GR-01#sh ipv6 ro 2600:0:2:1239:144:232:26:11 Routing entry for 2600::/29 Known via "bgp 57", distance 20, metric 0, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA36, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:52 ago -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 3, 2013