This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end. This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed. From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE I changed my route policy to ignore the routes from HE through the MICE Route Server and things are working again. I'm not sure what changed and why the routes from the MICE Route server have an bogus link-local address. But I though I would warn you all. MTC-GR-01#sh ipv6 ro 2001:470:0:1C2::1 Routing entry for 2001:470::/32 Known via "bgp 57", distance 20, metric 1, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA03, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:15 ago MTC-GR-01#sh ipv6 ro 2600:0:2:1239:144:232:26:11 Routing entry for 2600::/29 Known via "bgp 57", distance 20, metric 0, type external Route count is 1/1, share count 0 Routing paths: FE80::224:38FF:FEA7:EA36, Vlan300 MPLS label: nolabel MPLS Flags: NSF Last updated 01:43:52 ago -- ================================================ David Farmer Email: farmer@umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I see our traffic levels from them dropped quite a bit just after 1:00am and have more-or-less returned to normal now. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 4/3/2013 6:59 AM, Anthony Anderberg wrote:
By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades."
I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr@mailbag.com> wrote:
On 4/3/2013 6:59 AM, Anthony Anderberg wrote:
By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades."
I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE?
James Stahr CDW
##############################**##############################** ############
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-**bin/wa?SUBED1=MICE-DISCUSS&A=1<http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
-- Mike Bushard Jr | Network Engineer IV *| *Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I buy from them, and got 2 notices. The second one corrected the date. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Bushard, Jr Sent: Wednesday, April 03, 2013 7:48 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] IPv6 routing issue We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr@mailbag.com> wrote: On 4/3/2013 6:59 AM, Anthony Anderberg wrote: By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Mike Bushard Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Anyone that buys or peers with us who did not receive notice of the maintenance (and wants future notices) should send a request to support@he.net requesting to be notified of future maintenance events. To amplify what Reid said, any time you see any issue with your HE peering or transit, please email noc@he.net and we will work to address it promptly. Owen On Apr 3, 2013, at 05:47 , "Mike Bushard, Jr" <mike.bushard@ARVIG.COM> wrote:
We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also.
On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr@mailbag.com> wrote: On 4/3/2013 6:59 AM, Anthony Anderberg wrote: By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades."
I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE?
James Stahr CDW
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Mike Bushard Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We buy from the and did receive two notices. Second one corrected the date as well.. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Bushard, Jr Sent: Wednesday, April 03, 2013 7:48 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] IPv6 routing issue We buy form HE, never got a notice. Looks like our traffic to/from them dropped off also. On Wed, Apr 3, 2013 at 7:16 AM, James Stahr <stahr@mailbag.com> wrote: On 4/3/2013 6:59 AM, Anthony Anderberg wrote: By my notes HE did have a maintenance window scheduled this morning to do a "Router hardware upgrade to provide for expanded services and upgrades." I saw the flaps this morning, but I don't recall receiving a maintenance notification. Do you buy transit from HE? James Stahr CDW ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Mike Bushard Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile | 320.256.7555 Fax <http://ipstats.arvig.net/arviglogo.jpeg> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote:
This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end.
This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed.
From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE
FIWW: We do buy transit from HE and did get the maintenance notice for this morning. I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change. -- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
If anyone is seeing problems, PLEASE email noc@he.net. Reid On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote:
This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end.
This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed.
From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE
FIWW: We do buy transit from HE and did get the maintenance notice for this morning.
I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change.
-- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Reid Fishler Director Hurricane Electric +1-510-580-4178 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
As announced at the MICE meeting, we are upgrading our MSP router to be 100G capable. This swap is now complete. If anyone is still seeing any problems, please email noc@he.net. Reid On Wed, Apr 3, 2013 at 10:01 AM, Reid Fishler <rfishler@he.net> wrote:
If anyone is seeing problems, PLEASE email noc@he.net.
Reid
On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote:
This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end.
This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed.
From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE
FIWW: We do buy transit from HE and did get the maintenance notice for this morning.
I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change.
-- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Reid Fishler Director Hurricane Electric +1-510-580-4178
-- Reid Fishler Director Hurricane Electric +1-510-580-4178 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
To be clear I'm not saying this was HE's problem. I'm not sure it wasn't an issue with the route server, itself. I just wanted to give people a heads up that we had an issue. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== On Apr 3, 2013, at 9:01, Reid Fishler <rfishler@HE.NET> wrote:
If anyone is seeing problems, PLEASE email noc@he.net.
Reid
On Wed, Apr 3, 2013 at 9:01 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Wed, Apr 03, 2013 at 06:45:15AM -0500, David Farmer wrote:
This morning at approximately 1:30 our peering session with HE reset, I assume for some kind of normal maintenance on their end.
This triggered a weird issue on our end. Because of some crufty route policy we were preferring the routes we learned from the MICE route server over the routes we learned directly from HE. And for some reason the routes we were learning from the route server had a different link-local address associated with them than the routes we learned directly. And this address no longer seems to be valid, and this caused traffic to be black holed.
From below FE80::224:38FF:FEA7:EA03 - address on routes learned from MICE RS FE80::224:38FF:FEA7:EA36 - address on routes direct from HE
FIWW: We do buy transit from HE and did get the maintenance notice for this morning.
I've also seen this behavior from them before. I'm not sure if the link local address changes on their end due to some change or what is going on. It doesn't matter on our direct connect router, where we only talk the public IPv6 address, but the MICE connection seems to have stale link-local IPv6 addresses from time to time that I've had to clear out due to some change.
-- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Reid Fishler Director Hurricane Electric +1-510-580-4178
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (9)
-
Anthony Anderberg -
David Farmer -
Doug McIntyre -
James Stahr -
Jeremy Lumby -
John Unger -
Mike Bushard, Jr -
Owen DeLong -
Reid Fishler