Anyone else seeing instability to MICE route reflector #2
Anyone else seeing instability to MICE route reflector #2? Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 18:41:44 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 19:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 01:41:31 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 03:41:06 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 04:42:11 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 05:39:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 07:42:54 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 08:40:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 09:40:25 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:10:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:41:29 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 13:39:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 14:22:19 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE? |? Sioux Center, IA 51250?? phone:?712.722.3451?web:?mypremieronline.com direct:?(712) 722-7152?cell:?(712) 441-6049
Yes, we have been seeing drops to RS2 for months now. We can't really see a reason why. We have noticed that they always seem to happen near the 40th minute of the hour. I didn't fully break out my secret decoder ring on your output below, but your results seem like they might be similar. RS1 has not been affected for us. Shawn Rasmussen Network Manager CTC -----Original Message----- From: Frank Bulk <fbulk@mypremieronline.com> Sent: Friday, October 10, 2025 2:42 PM To: Subject: Anyone else seeing instability to MICE route reflector #2 CAUTION: This email originated from outside CTC. Do not open links or attachments unless you are expecting this message and know they are safe. Anyone else seeing instability to MICE route reflector #2? Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 18:41:44 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 19:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 01:41:31 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 03:41:06 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 04:42:11 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 05:39:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 07:42:54 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 08:40:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 09:40:25 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:10:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:41:29 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 13:39:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 14:22:19 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE? |? Sioux Center, IA 51250?? phone:?712.722.3451?web:?mypremieronline.com direct:?(712) 722-7152?cell:?(712) 441-6049 _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Here's the last three days ... and good catch, always around 40 minutes past the hour! Oct 11 01:40:04 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 01:40:04 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 01:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 02:38:57 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 02:38:57 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 02:39:20 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 03:40:29 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 03:40:29 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 03:40:35 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 04:39:38 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 04:39:38 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 04:39:46 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 05:42:26 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 05:42:26 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 05:42:42 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 10:39:33 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 10:39:33 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 10:39:43 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 11:40:23 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 11:40:23 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 11:40:33 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 13:41:53 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 13:41:53 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 13:41:59 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 14:39:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 14:39:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 14:39:23 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 16:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 16:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 16:39:36 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 17:40:01 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 17:40:01 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 17:40:16 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 19:41:57 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 19:41:57 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 19:43:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 21:39:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 21:39:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 21:39:21 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 11 23:40:59 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 11 23:40:59 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 11 23:41:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 12 01:39:50 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 12 01:39:50 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 12 01:40:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 12 08:39:51 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 12 08:39:51 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 12 08:41:22 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 12 09:39:44 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 12 09:39:44 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 12 09:39:53 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 12 10:40:35 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 12 10:40:35 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 12 10:41:03 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 12 11:41:45 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 12 11:41:45 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 12 11:43:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 13 05:41:18 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 13 05:41:18 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 13 05:41:26 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 13 06:41:05 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 13 06:41:05 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 13 06:41:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 13 11:39:38 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 13 11:39:38 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 13 11:39:50 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Frank Frank Bulk -----Original Message----- From: Shawn Rasmussen via mice-discuss <mice-discuss@lists.micemn.net> Sent: Monday, October 13, 2025 8:22 AM To: mice-discuss@lists.micemn.net Subject: RE: Anyone else seeing instability to MICE route reflector #2 Yes, we have been seeing drops to RS2 for months now. We can't really see a reason why. We have noticed that they always seem to happen near the 40th minute of the hour. I didn't fully break out my secret decoder ring on your output below, but your results seem like they might be similar. RS1 has not been affected for us. Shawn Rasmussen Network Manager CTC -----Original Message----- From: Frank Bulk <fbulk@mypremieronline.com> Sent: Friday, October 10, 2025 2:42 PM To: Subject: Anyone else seeing instability to MICE route reflector #2 CAUTION: This email originated from outside CTC. Do not open links or attachments unless you are expecting this message and know they are safe. Anyone else seeing instability to MICE route reflector #2? Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 18:41:17 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 18:41:44 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 9 19:41:56 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 9 19:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Establ ished Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 01:41:12 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 01:41:31 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 03:40:52 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 03:41:06 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 04:41:58 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 04:42:11 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 05:38:55 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 05:39:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established ne w state Established Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 07:42:15 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 07:42:54 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 08:40:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 08:40:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Establi shed new state Established Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 09:40:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 09:40:25 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:10:10 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:10:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 10:41:13 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 10:41:29 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 13:39:28 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 13:39:37 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state Established event Stop new state Idle Oct 10 14:22:07 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.2 (VRF default AS 53679) 4/0 (Hold Timer Expired Error/None) 0 bytes Oct 10 14:22:19 CDT SiouxCity-Fibernet-Arista Bgp: %BGP-5-ADJCHANGE: peer 206.108.255.2 (VRF default AS 53679) old state OpenConfirm event Established new state Established Frank Frank Bulk Chief Technology Officer / Chief Operating Officer 339 1st Ave NE? |? Sioux Center, IA 51250?? phone:?712.722.3451?web:?mypremieronline.com direct:?(712) 722-7152?cell:?(712) 441-6049 _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
What do you have for hold timers? And is it the same for both route servers? rs2 is proposing 240 seconds, if I'm reading this packet capture correctly. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
On 2025-10-15 16:32, Richard Laager wrote:
What do you have for hold timers? And is it the same for both route servers?
rs2 is proposing 240 seconds, if I'm reading this packet capture correctly.
I can set a "min hold time" value in BIRD. If we find that raising the hold timer fixes the issues, I will probably set a minimum. If I set that, it will break the sessions with lower values immediately, so this would need some transition. That would probably look something like: 1) See who it would affect and email them. 2) Enable it on one route server. 3) After a bit, enable it on the other route server. -- Richard
What is the problem with shorter hold timers? Shorter hold timers allow BGP failures to be detected more quickly, reducing time to find an alternate, functional route. 60 second hold timers with 20 second keepalives are fine in the real world in my experience. Jonathan Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA On Fri, 2025-10-17 4:21 p.m., Richard Laager wrote:
On 2025-10-15 16:32, Richard Laager wrote:
What do you have for hold timers? And is it the same for both route servers?
rs2 is proposing 240 seconds, if I'm reading this packet capture correctly.
I can set a "min hold time" value in BIRD. If we find that raising the hold timer fixes the issues, I will probably set a minimum. If I set that, it will break the sessions with lower values immediately, so this would need some transition. That would probably look something like: 1) See who it would affect and email them. 2) Enable it on one route server. 3) After a bit, enable it on the other route server. -- Richard
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
BIRD 2.x is single-threaded. The config file is non-trivial. For IPv4, it is 359,018 lines and 38,367,665 bytes (37 MiB). This is due to the massive configuration generated by the IRR stuff. (This is especially true for HE, which is the majority of the config. HE's part of the config is 2,335 lines of ASNs and 217,914 lines of prefixes. Note that there is than one per line.) Since this problem occurs when BIRD is being reconfigured, my working theory is that BIRD is just getting overwhelmed for a bit. From the packet capture I have of CTC, it's definitely the CTC side tearing down the session due to the hold timer expiring. This will likely get better in BIRD 3.x. I think 3.x is multithreaded. On 2025-10-17 16:33, Jonathan Stewart wrote:
What is the problem with shorter hold timers? Shorter hold timers allow BGP failures to be detected more quickly, reducing time to find an alternate, functional route.
60 second hold timers with 20 second keepalives are fine in the real world in my experience.
-- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
We’re doing 10/30 against the two route servers. Frank [cid:premiercomm_logo_cmyk_thick_4b24da0e-a829-42f3-ba49-f303951df775.png] Frank Bulk Chief Technology Officer / Chief Operating Officer From: Richard Laager <rlaager@micemn.net> Sent: Wednesday, October 15, 2025 4:33 PM To: mice-discuss@lists.micemn.net Subject: Re: Anyone else seeing instability to MICE route reflector #2 What do you have for hold timers? And is it the same for both route servers? rs2 is proposing 240 seconds, if I'm reading this packet capture correctly. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
Let's see if the BIRD upgrade fixes this. It was suggested on the IXP Manager list that newer BIRD might fix this. If it does not, then I'd recommend you try raising your timers (if you haven't already). At a minimum, we should do that as a test. If it works, then that's probably at least a workaround. I'm hoping that BIRD 3 (which is multithreaded) will fix this, but IXP Manager doesn't support BIRD 3 yet (as they don't consider it production ready, though no reasons were given). On 2025-10-23 22:40, Frank Bulk wrote:
We’re doing 10/30 against the two route servers.
-- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
Our BGP with rs2 has been stable since last week Friday’s bounce, rs1 has been good since it was upgraded earlier today. Thanks, Frank [cid:premiercomm_logo_cmyk_thick_4b24da0e-a829-42f3-ba49-f303951df775.png] Frank Bulk Chief Technology Officer / Chief Operating Officer From: Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> Sent: Monday, October 27, 2025 2:47 PM To: mice-discuss@lists.micemn.net Subject: Re: Anyone else seeing instability to MICE route reflector #2 Any word since the upgrade of rs2? -- Richard
participants (5)
-
Frank Bulk -
Jonathan Stewart -
Richard Laager -
Richard Laager -
Shawn Rasmussen