For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. All our other peers with MICE are fine. I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. Any hints would be welcomed. I've shutdown those six sessions for now. Frank Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established ... Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established
Did this get resolved? On Fri, Sep 6, 2019 at 6:52 PM Frank Bulk <fbulk@mypremieronline.com> wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
Any hints would be welcomed.
I've shutdown those six sessions for now.
Frank
Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established ...
Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
It did not -- still waiting to hear from MICE admins on what the RR saw on its side. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Saturday, September 7, 2019 10:53 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Did this get resolved? On Fri, Sep 6, 2019 at 6:52 PM Frank Bulk <fbulk@mypremieronline.com> wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
Any hints would be welcomed.
I've shutdown those six sessions for now.
Frank
Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established ...
Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
We've been getting a lot of these.. Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution And these 2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 As well as I suppose the expected 2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown If you have 18883 link shutdown. But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side. Could these two be mixed up at all? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Doug, Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Saturday, September 7, 2019 4:15 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
We've been getting a lot of these.. Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution And these 2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 As well as I suppose the expected 2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown If you have 18883 link shutdown. But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side. Could these two be mixed up at all? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote:
Doug,
Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great.
Frank
-----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Saturday, September 7, 2019 4:15 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors?
On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
We've been getting a lot of these..
Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution
And these
2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883
As well as I suppose the expected
2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown
If you have 18883 link shutdown.
But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side.
Could these two be mixed up at all?
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Thanks, I will try to turn one of the two route-reflector sessions up this morning. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Sunday, September 08, 2019 11:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote:
Doug,
Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great.
Frank
-----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Saturday, September 7, 2019 4:15 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors?
On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
We've been getting a lot of these..
Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution
And these
2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883
As well as I suppose the expected
2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown
If you have 18883 link shutdown.
But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side.
Could these two be mixed up at all?
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Good news -- Jeremy identified an issue in his transport that looks very similar to a Zayo issue he worked through in another link a few months ago, so he's opened up a ticket with Zayo. Thanks, Doug and Jeremy, for your help! Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 09, 2019 8:33 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Thanks, I will try to turn one of the two route-reflector sessions up this morning. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Sunday, September 08, 2019 11:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote:
Doug,
Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great.
Frank
-----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Saturday, September 7, 2019 4:15 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors?
On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote:
For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then.
All our other peers with MICE are fine.
I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that.
We've been getting a lot of these..
Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution
And these
2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883
As well as I suppose the expected
2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown
If you have 18883 link shutdown.
But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side.
Could these two be mixed up at all?
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
participants (3)
-
Doug McIntyre -
Frank Bulk -
Jay Hanke