- mice-discuss - lists.micemn.net

Re: Multiple members reporting no light on the main switch

by Justin Krejci

Our 10gig link to MICE is down therefore the cacti server has no path to the switches/prefix. -----Original Message----- From: Jason Hanke [jayhanke(a)NEUTRALPATH.NET] Received: Friday, 24 Jul 2015, 7:25PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Subject: [MICE-DISCUSS] Multiple members reporting no light on the main switch There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net<http://www.neutralpath.net>

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Shaun Carlson

We're seeing the same at Arvig. Mike Hemphill is seeing if someone from Cologix can get eyes on the switch. In the meantime, is there anyone else who is able to get to the 511 in a timely way? s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" <jayhanke(a)neutralpath.net> wrote: There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Multiple members reporting no light on the main switch

by Jason Hanke

There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Jeremy Lumby

>From what I can tell remotely the Juniper 4500 (10 gig switch) is down. I have BGP established to the route servers (they and I are both on the 4200) Is anyone close by that can take a look? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, July 24, 2015 7:12 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN All of my MICE BGP sessions have been down since approximately 18:41. For reference, I'm connected to the Mankato Networks switch. -- Richard

July 25, 2015

Re: Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Richard Laager

All of my MICE BGP sessions have been down since approximately 18:41. For reference, I'm connected to the Mankato Networks switch. -- Richard

July 25, 2015

Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Jason Hanke

can someone take a peek? ---------- Forwarded message ---------- From: NOC IP <svc_NNOCIP(a)charter.com> Date: Fri, Jul 24, 2015 at 7:04 PM Subject: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN To: "peering(a)micemn.net" <peering(a)micemn.net> MICE, Charter Communications is not getting any incoming light at the Colo link. Could you please check this link and reply with any issues from your end? Thank you, *[image: CharterCommunicatio#B375E30]* *Katie McCarthy, *CCNA | Network Operations Technician II | 866.248.7662 Opt. 2 9333 Dielman Industrial Drive | St Louis, MO 63132 *Please do your part to reduce unnecessary email volume by directing your request to the appropriate NOC group using the following distribution lists:* · *Critical Facilities*: DLCriticalFacilities(a)Charter.com · *IP, CMTS, Optical*: DLNOCIP(a)Charter.Com · *Video*: DLNOCVideo(a)Charter.com · *Voice*: DLNOCVoice(a)Charter.com -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

CNS Switch - Cacti Graph Anomalies

by Justin Krejci

Looks like there is maybe some kind of trouble with the CNS Switch. There are a lot of cacti graphing gaps displayed but only for the ports being queried on the CNS switch itself. Looks like it started around 2pm yesterday. Looking at the LAG port graph off the main switch, it does not appear to be affecting actual member traffic so it may be a management issue with the CNS switch or some other control plane issue possibly. Could be indicative of a larger, brewing problem.

July 23, 2015

Re: MICE Switch Upgrade

by Jeff Bartig

The Seattle SIX exchange tried out this technique for their last software upgrade. Seemed like an interesting solution. https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf On Mon, Jul 20, 2015 at 01:11:34PM -0500, Andrew Hoyos wrote: > Shutting down the route server sessions would at least let BGP hold timers expire gracefully while traffic can still be forwarded for the duration of the hold timer. > > Probably more important for those folks that are layer 2 extended into the IX and don't necessarily have any notion of physical link to the switch. > > -- > Andrew Hoyos > hoyosa(a)gmail.com -- Jeff Bartig University of Wisconsin jeffb(a)doit.wisc.edu (608) 262-8336

July 21, 2015

Re: MICE Switch Upgrade

by Andrew Hoyos

> On Jul 20, 2015, at 12:54 PM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > > Wouldn't that be just as disruptive as just doing the switch reboot after upgrade? Shutting down the route server sessions would at least let BGP hold timers expire gracefully while traffic can still be forwarded for the duration of the hold timer. Probably more important for those folks that are layer 2 extended into the IX and don't necessarily have any notion of physical link to the switch. -- Andrew Hoyos hoyosa(a)gmail.com

July 20, 2015

Re: MICE Switch Upgrade

by Jeremy Lumby

I am in favor of turning down the route servers since it would gracefully stop the traffic flow allowing the members routers to automatically reroute traffic before the switch reboot cuts everyone off. The only issue is it does not address the bi-lateral traffic, and therefore the members that care would be forced to turn down BGP on their own anyway. So for simplicity sake, it would be easiest to leave it up to the individual members. I could see it working either way. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Monday, July 20, 2015 12:55 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Switch Upgrade On Mon, Jul 20, 2015 at 12:48:16PM -0500, Richard Laager wrote: > Will you be turning down the route server BGP sessions in advance of > rebooting the switch? Wouldn't that be just as disruptive as just doing the switch reboot after upgrade? Ie. individual members could do so on the schedule they wanted to control the schedule themselves, but I guess I don't see a benefit for the central route servers. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 20, 2015