Problem at MICE? Physical link is up. Frank -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Thursday, August 04, 2016 4:53 PM To: Frank Bulk <fbulk@mypremieronline.com> Subject: PROBLEM: CRITICAL alert for Premiers Spencer-facing Brocade MLXe-4 router/BGP ***** Nagios ***** This is a PROBLEM notice that Premiers Spencer-facing Brocade MLXe-4 router/BGP is CRITICAL Date/Time: Thu Aug 4 16:52:57 CDT 2016 Additional Info: CRITICAL: critical 2, warning 6 Host 192.168.0.85 with interface to BGP peer AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: the BGP peer state is in state connect Host 192.168.0.85 with interface to BGP peer AS29040.micemn.net [206.108.255.84/AS 20940]: the BGP peer state is in state connect Host 192.168.0.85 with interface to BGP peer AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: the elapsed time the remote peer has been in the Established state or how long since this peer was last in the Established state changed from 39 days, 19 hours, 41 minutes, 12 seconds to 0 seconds Host 192.168.0.85 with interface to BGP peer AS29040.micemn.net [206.108.255.84/AS 20940]: the elapsed time the remote peer has been in the Established state or how long since this peer was last in the Established state changed from 23 days, 4 hours, 33 minutes, 7 seconds to 0 seconds Host 192.168.0.85 with interface to BGP peer AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: the number of received route prefixes changed from 1763 to 0 Host 192.168.0.85 with interface to BGP peer AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: 0 learned prefixes is less than the specified minimum of 1 Host 192.168.0.85 with interface to BGP peer AS29040.micemn.net [206.108.255.84/AS 20940]: the number of received route prefixes changed from 10 to 0 Host 192.168.0.85 with interface to BGP peer AS29040.micemn.net [206.108.255.84/AS 20940]: 0 learned prefixes is less than the specified minimum of 1 Host 192.168.0.85 with interface to BGP peer AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: the number of installed route prefixes changed from 1763 to 0 Host 192.168.0.85 with interface to BGP peer AS29040.micemn.net [206.108.255.84/AS 20940]: the number of installed route prefixes changed from 10 to 0 BGP state of premier.spnc-mlx.fbnt.netins.net [173.215.60.1/AS 18883]: established; prefixes (received/installed): 1/1 BGP state of AS53679-MRS-1.micemn.net [206.108.255.1/AS 53679]: established; prefixes (received/installed): 1769/1769 BGP state of AS53679-MRS-2.micemn.net [206.108.255.2/AS 53679]: connect; prefixes (received/installed): 0/0 BGP state of AS6939.micemn.net [206.108.255.52/AS 6939]: established; prefixes (received/installed): 92186/91624 BGP state of AS29040.micemn.net [206.108.255.84/AS 20940]: connect; prefixes (received/installed): 0/0
Thanks -- looks like it's down just to Akamai and MICE MRS #2. Since they all share the same physical path, not clear to me how this can be, but I'll start digging. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, August 04, 2016 5:01 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] BGP sessions down... On 08/04/2016 04:58 PM, Frank Bulk wrote:
Problem at MICE? Physical link is up.
All of my MICE sessions (including both route servers) are up, and I can still ping across the exchange. -- Richard
Looks like it was 4:52 to 5:02 pm. And it's back up to all but one, which I need to check. Aug 4 16:52:19 192.168.0.85 BGP: Peer 206.108.255.84 DOWN (Hold Timer Expired) Aug 4 16:52:23 192.168.0.85 BGP: Peer 206.108.255.2 DOWN (Hold Timer Expired) Aug 4 16:52:24 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 DOWN (Hold Timer Expired) Aug 4 16:54:56 192.168.0.85 BGP: Peer 206.108.255.1 DOWN (Hold Timer Expired) Aug 4 16:54:57 192.168.0.85 BGP: Peer 206.108.255.52 DOWN (Hold Timer Expired) Aug 4 17:02:11 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 UP (ESTABLISHED) Aug 4 17:02:11 192.168.0.85 BGP: Peer 206.108.255.1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 206.108.255.52 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 UP (ESTABLISHED) Aug 4 17:02:14 192.168.0.85 BGP: Peer 206.108.255.84 UP (ESTABLISHED) Aug 4 17:02:16 192.168.0.85 BGP: Peer 206.108.255.2 UP (ESTABLISHED) Aug 4 17:02:38 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 Aug 4 17:05:58 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 [cid:image001.png@01D1EE72.E89ADA40] -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, August 04, 2016 5:05 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] BGP sessions down... Thanks -- looks like it's down just to Akamai and MICE MRS #2. Since they all share the same physical path, not clear to me how this can be, but I'll start digging. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, August 04, 2016 5:01 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] BGP sessions down... On 08/04/2016 04:58 PM, Frank Bulk wrote:
Problem at MICE? Physical link is up.
All of my MICE sessions (including both route servers) are up, and I can still ping across the exchange. -- Richard
We were still doing active maintenance on COLOGIX-MSP3 (our clusters connected to MICE), so it may have been a brief outage on our part related to that. Owen On Aug 4, 2016, at 15:09 , Frank Bulk <fbulk@mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: Looks like it was 4:52 to 5:02 pm. And it's back up to all but one, which I need to check. Aug 4 16:52:19 192.168.0.85 BGP: Peer 206.108.255.84 DOWN (Hold Timer Expired) Aug 4 16:52:23 192.168.0.85 BGP: Peer 206.108.255.2 DOWN (Hold Timer Expired) Aug 4 16:52:24 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 DOWN (Hold Timer Expired) Aug 4 16:54:56 192.168.0.85 BGP: Peer 206.108.255.1 DOWN (Hold Timer Expired) Aug 4 16:54:57 192.168.0.85 BGP: Peer 206.108.255.52 DOWN (Hold Timer Expired) Aug 4 17:02:11 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 UP (ESTABLISHED) Aug 4 17:02:11 192.168.0.85 BGP: Peer 206.108.255.1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 206.108.255.52 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 UP (ESTABLISHED) Aug 4 17:02:14 192.168.0.85 BGP: Peer 206.108.255.84 UP (ESTABLISHED) Aug 4 17:02:16 192.168.0.85 BGP: Peer 206.108.255.2 UP (ESTABLISHED) Aug 4 17:02:38 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 Aug 4 17:05:58 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 <image001.png> -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, August 04, 2016 5:05 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@lists.iphouse.net> Subject: Re: [MICE-DISCUSS] BGP sessions down... Thanks -- looks like it's down just to Akamai and MICE MRS #2. Since they all share the same physical path, not clear to me how this can be, but I'll start digging. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, August 04, 2016 5:01 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] BGP sessions down... On 08/04/2016 04:58 PM, Frank Bulk wrote:
Problem at MICE? Physical link is up.
All of my MICE sessions (including both route servers) are up, and I can still ping across the exchange. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2Dbin_wa-3FSUBED1-3DMICE-2DDISCUSS-26A-3D1&d=DQMFAg&c=96ZbZZcaMF4w0F4jpN6LZg&r=MJ9m6mlVJrWIyxSYEMDah-0oNiPbOmFEMKuZCiZZwoI&m=7xVzF9svzGGKqTYZvqjK-iiGd0YIUE3dPrxuTMp7fV8&s=rdktmGLuhD1wzx9WQNOjWt1Rko4qHtvU6egdAlzm3Sw&e=>
Owen, Thanks - the reason I saw it and fews other did was because I connect through Belle Plaine and some equipment ran into a hiccup there ... talk to Jay offline if you want more details. Frank From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of DeLong, Owen Sent: Friday, August 05, 2016 7:59 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [Peering-lists] [MICE-DISCUSS] BGP sessions down... We were still doing active maintenance on COLOGIX-MSP3 (our clusters connected to MICE), so it may have been a brief outage on our part related to that. Owen On Aug 4, 2016, at 15:09 , Frank Bulk <fbulk@mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: Looks like it was 4:52 to 5:02 pm. And it's back up to all but one, which I need to check. Aug 4 16:52:19 192.168.0.85 BGP: Peer 206.108.255.84 DOWN (Hold Timer Expired) Aug 4 16:52:23 192.168.0.85 BGP: Peer 206.108.255.2 DOWN (Hold Timer Expired) Aug 4 16:52:24 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 DOWN (Hold Timer Expired) Aug 4 16:52:27 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 DOWN (Hold Timer Expired) Aug 4 16:54:56 192.168.0.85 BGP: Peer 206.108.255.1 DOWN (Hold Timer Expired) Aug 4 16:54:57 192.168.0.85 BGP: Peer 206.108.255.52 DOWN (Hold Timer Expired) Aug 4 17:02:11 192.168.0.85 BGP: Peer 2001:504:27::1b1b:0:1 UP (ESTABLISHED) Aug 4 17:02:11 192.168.0.85 BGP: Peer 206.108.255.1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 206.108.255.52 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::51cc:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:1 UP (ESTABLISHED) Aug 4 17:02:12 192.168.0.85 BGP: Peer 2001:504:27::d1af:0:2 UP (ESTABLISHED) Aug 4 17:02:14 192.168.0.85 BGP: Peer 206.108.255.84 UP (ESTABLISHED) Aug 4 17:02:16 192.168.0.85 BGP: Peer 206.108.255.2 UP (ESTABLISHED) Aug 4 17:02:38 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 Aug 4 17:05:58 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.52: exceeds warning limit 90000 <image001.png> -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, August 04, 2016 5:05 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@lists.iphouse.net> Subject: Re: [MICE-DISCUSS] BGP sessions down... Thanks -- looks like it's down just to Akamai and MICE MRS #2. Since they all share the same physical path, not clear to me how this can be, but I'll start digging. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, August 04, 2016 5:01 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] BGP sessions down... On 08/04/2016 04:58 PM, Frank Bulk wrote:
Problem at MICE? Physical link is up.
All of my MICE sessions (including both route servers) are up, and I can still ping across the exchange. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2Dbin_wa-3FSUBED1-3DMICE-2DDISCUSS-26A-3D1&d=DQMFAg&c=96ZbZZcaMF4w0F4jpN6LZg&r=MJ9m6mlVJrWIyxSYEMDah-0oNiPbOmFEMKuZCiZZwoI&m=7xVzF9svzGGKqTYZvqjK-iiGd0YIUE3dPrxuTMp7fV8&s=rdktmGLuhD1wzx9WQNOjWt1Rko4qHtvU6egdAlzm3Sw&e=> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (3)
-
DeLong, Owen -
Frank Bulk -
Richard Laager