July 2015 - mice-discuss - lists.micemn.net

Re: Multiple members reporting no light on the main switch

by Frank Bulk

It failed at 6:41 pm -- what brought it back up around ~7:40 pm -- the Cologix tech powering it on? Or cycling power? Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, July 24, 2015 10:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multiple members reporting no light on the main switch On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: > Can someone log in and pull the alarms? The alarm lit right now is just "boot from backup root". The software upgrade should clear that. This is the start of the logs around the time the 4500 rebooted tonight Lots of info (pages and pages more), probably none too relevent to any issue. Jul 24 18:38:15 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:19 MICE-SW1 sfid[1022]: sfi_pfepeer_grace_timer expired Jul 24 18:38:30 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:45 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/26 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/27 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/24 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/25 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c0 on vcp-1.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-1.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-1 went down Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c1 on vcp-0.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-0.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-0 went down .. Probably getting up to the current recommended code base for this hardware is the best course of action. (JunOS 12.3 has been working very well for our Juniper EX switch stacks). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Doug McIntyre

On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: > Can someone log in and pull the alarms? The alarm lit right now is just "boot from backup root". The software upgrade should clear that. This is the start of the logs around the time the 4500 rebooted tonight Lots of info (pages and pages more), probably none too relevent to any issue. Jul 24 18:38:15 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:19 MICE-SW1 sfid[1022]: sfi_pfepeer_grace_timer expired Jul 24 18:38:30 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:45 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/26 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/27 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/24 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/25 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c0 on vcp-1.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-1.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-1 went down Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c1 on vcp-0.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-0.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-0 went down .. Probably getting up to the current recommended code base for this hardware is the best course of action. (JunOS 12.3 has been working very well for our Juniper EX switch stacks). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Jason Hanke

Can someone log in and pull the alarms? On Fri, Jul 24, 2015 at 8:07 PM, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: > Mike called and the Cologix tech sees the following alarms on the switch: > SPD > DPX > ADM > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com > > > > > On Fri, Jul 24, 2015 at 5:44 PM -0700, "Jason Hanke" > <jayhanke(a)neutralpath.net> wrote: > >> Main switch services appear to be coming back online. Do we have a root >> cause? >> >> On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill >> wrote: >> > A Cologix tech is on his way. Our NOC was seeing another issue. He will >> > look >> > at the MICE switch as soon as he can and let me know. >> > Mike >> > >> > On Friday, July 24, 2015, Shaun Carlson wrote: >> >> >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone >> >> from >> >> Cologix can get eyes on the switch. >> >> >> >> In the meantime, is there anyone else who is able to get to the 511 in >> >> a >> >> timely way? >> >> >> >> s >> >> >> >> Shaun Carlson >> >> Senior Manager of Information Technology | Arvig >> >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> >> wrote: >> >> >> >>> There are multiple reports of members not getting light from the main >> >>> switch. >> >>> >> >>> Cacti graphing is down. >> >>> >> >>> -- >> >>> Jay Hanke >> >>> CTO >> >>> Neutral Path Communications >> >>> 3 Civic Center Plaza, Suite 204 >> >>> Mankato, MN 56001 >> >>> (507) 327-2398 mobile >> >>> jayhanke(a)neutralpath.net >> >>> www.neutralpath.net >> >> >> >> >> >> ________________________________ >> >> >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > >> > >> > >> > -- >> > Mike Hemphill >> > General Manager | Cologix, Inc. >> > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 >> > P: 1+612.333.1922 | M: 1+612.812.5242 >> > mike.hemphill(a)cologix.com >> > >> > Sent from my iPhone 6 Plus >> > >> > >> > ________________________________ >> > >> > To unsubscribe from the MICE-DISCUSS list, click the following link: >> > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke >> CTO >> Neutral Path Communications >> 3 Civic Center Plaza, Suite 204 >> Mankato, MN 56001 >> (507) 327-2398 mobile >> jayhanke(a)neutralpath.net >> www.neutralpath.net > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Shaun Carlson

Mike called and the Cologix tech sees the following alarms on the switch:SPDDPXADM s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Fri, Jul 24, 2015 at 5:44 PM -0700, "Jason Hanke" <jayhanke(a)neutralpath.net> wrote: Main switch services appear to be coming back online. Do we have a root cause? On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill wrote: > A Cologix tech is on his way. Our NOC was seeing another issue. He will look > at the MICE switch as soon as he can and let me know. > Mike > > On Friday, July 24, 2015, Shaun Carlson wrote: >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone from >> Cologix can get eyes on the switch. >> >> In the meantime, is there anyone else who is able to get to the 511 in a >> timely way? >> >> s >> >> Shaun Carlson >> Senior Manager of Information Technology | Arvig >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> wrote: >> >>> There are multiple reports of members not getting light from the main >>> switch. >>> >>> Cacti graphing is down. >>> >>> -- >>> Jay Hanke >>> CTO >>> Neutral Path Communications >>> 3 Civic Center Plaza, Suite 204 >>> Mankato, MN 56001 >>> (507) 327-2398 mobile >>> jayhanke(a)neutralpath.net >>> www.neutralpath.net >> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Mike Hemphill > General Manager | Cologix, Inc. > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 > P: 1+612.333.1922 | M: 1+612.812.5242 > mike.hemphill(a)cologix.com > > Sent from my iPhone 6 Plus > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Jason Hanke

Main switch services appear to be coming back online. Do we have a root cause? On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill <mike.hemphill(a)cologix.com> wrote: > A Cologix tech is on his way. Our NOC was seeing another issue. He will look > at the MICE switch as soon as he can and let me know. > Mike > > On Friday, July 24, 2015, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone from >> Cologix can get eyes on the switch. >> >> In the meantime, is there anyone else who is able to get to the 511 in a >> timely way? >> >> s >> >> Shaun Carlson >> Senior Manager of Information Technology | Arvig >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> <jayhanke(a)neutralpath.net> wrote: >> >>> There are multiple reports of members not getting light from the main >>> switch. >>> >>> Cacti graphing is down. >>> >>> -- >>> Jay Hanke >>> CTO >>> Neutral Path Communications >>> 3 Civic Center Plaza, Suite 204 >>> Mankato, MN 56001 >>> (507) 327-2398 mobile >>> jayhanke(a)neutralpath.net >>> www.neutralpath.net >> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Mike Hemphill > General Manager | Cologix, Inc. > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 > P: 1+612.333.1922 | M: 1+612.812.5242 > mike.hemphill(a)cologix.com > > Sent from my iPhone 6 Plus > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Mike Hemphill

A Cologix tech is on his way. Our NOC was seeing another issue. He will look at the MICE switch as soon as he can and let me know. Mike On Friday, July 24, 2015, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: > We're seeing the same at Arvig. Mike Hemphill is seeing if someone from > Cologix can get eyes on the switch. > > In the meantime, is there anyone else who is able to get to the 511 in a > timely way? > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com > <javascript:_e(%7B%7D,'cvml','shaun.carlson(a)arvig.com');> > > > > > On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" < > jayhanke(a)neutralpath.net > <javascript:_e(%7B%7D,'cvml','jayhanke(a)neutralpath.net');>> wrote: > > There are multiple reports of members not getting light from the main switch. >> >> Cacti graphing is down. >> >> -- >> Jay Hanke >> CTO >> Neutral Path Communications >> 3 Civic Center Plaza, Suite 204 >> Mankato, MN 56001 >> (507) 327-2398 mobilejayhanke(a)neutralpath.net <javascript:_e(%7B%7D,'cvml','jayhanke(a)neutralpath.net');>www.neutralpath.net >> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill(a)cologix.com Sent from my iPhone 6 Plus

July 25, 2015

Re: Multiple members reporting no light on the main switch

by David Farmer

I could get down there quick, still in my office, but don't have access to Cologix. On 7/24/15 19:26 , Shaun Carlson wrote: > We're seeing the same at Arvig. Mike Hemphill is seeing if someone from > Cologix can get eyes on the switch. > > In the meantime, is there anyone else who is able to get to the 511 in a > timely way? > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 <tel:(218)%20346-8673> | em: shaun.carlson(a)arvig.com > <mailto:shaun.carlson@arvig.com> > > > > > On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" > <jayhanke(a)neutralpath.net <mailto:jayhanke@neutralpath.net>> wrote: > > There are multiple reports of members not getting light from the main switch. > > Cacti graphing is down. > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Justin Krejci

Our 10gig link to MICE is down therefore the cacti server has no path to the switches/prefix. -----Original Message----- From: Jason Hanke [jayhanke(a)NEUTRALPATH.NET] Received: Friday, 24 Jul 2015, 7:25PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Subject: [MICE-DISCUSS] Multiple members reporting no light on the main switch There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net<http://www.neutralpath.net>

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Shaun Carlson

We're seeing the same at Arvig. Mike Hemphill is seeing if someone from Cologix can get eyes on the switch. In the meantime, is there anyone else who is able to get to the 511 in a timely way? s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" <jayhanke(a)neutralpath.net> wrote: There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Multiple members reporting no light on the main switch

by Jason Hanke

There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015