- mice-discuss - lists.micemn.net

RE: Anyone else seeing instability to MICE route reflector #2

by Frank Bulk

We’re doing 10/30 against the two route servers. Frank [cid:premiercomm_logo_cmyk_thick_4b24da0e-a829-42f3-ba49-f303951df775.png] Frank Bulk Chief Technology Officer / Chief Operating Officer From: Richard Laager <rlaager(a)micemn.net> Sent: Wednesday, October 15, 2025 4:33 PM To: mice-discuss(a)lists.micemn.net Subject: Re: Anyone else seeing instability to MICE route reflector #2 What do you have for hold timers? And is it the same for both route servers? rs2 is proposing 240 seconds, if I'm reading this packet capture correctly. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by Richard Laager

I've built 2.17.1 for 22.04. I'm thinking we'll upgrade BIRD on rs2 tomorrow noon-ish, and apply regular updates while we are at it. It might help with the timeouts we are seeing there. Regardless, as long as it's not worse, then next week (tentatively Monday), we can upgrade BIRD on rs1. -- Richard

Oct. 24, 2025

RE: v6 on rs1/rs2 fubar?

by Michael Hare

JTAC pointed me to TSB76599. It’s behind a login, so while I won’t copy/paste the entire PDF, in summary: “The following Junos and Junos Evolved software releases are now fully compliant with RFC7606” 22.4R3, 22.4R3-EVO, and subsequent releases 23.2R2, 23.2R2-EVO, and subsequent releases 23.4R1, 23.4R1-EVO, and subsequent releases 24.1R1, 24.1R1-EVO, and all subsequent releases -Michael From: David Farmer via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, October 23, 2025 6:31 PM To: Justin Krejci <JKrejci(a)usinternet.com> Cc: mice-discuss(a)lists.micemn.net; rlaager(a)wiktel.com Subject: Re: v6 on rs1/rs2 fubar? According to the NA-IX Slack BIRD 2.17.1. fixes the issue, or at least doesn't propagate it. Thanks On Thu, Oct 23, 2025 at 6:23 PM Justin Krejci <JKrejci(a)usinternet.com<mailto:JKrejci@usinternet.com>> wrote: As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the default system packages, perhaps that will be sufficiently bug-fixed and is already very accessible to achieve with a simple upgrade to 24.04. FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without issue <include typical ymmv disclaimer here> -----Original Message----- From: Richard Laager via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss@lists.micemn.net%3e>> Reply-To: Richard Laager <rlaager(a)wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>> To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Re: v6 on rs1/rs2 fubar? Date: Thu, 23 Oct 2025 17:50:22 -0500 IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps. _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by David Farmer

According to the NA-IX Slack BIRD 2.17.1. fixes the issue, or at least doesn't propagate it. Thanks On Thu, Oct 23, 2025 at 6:23 PM Justin Krejci <JKrejci(a)usinternet.com> wrote: > As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the > default system packages, perhaps that will be sufficiently bug-fixed and is > already very accessible to achieve with a simple upgrade to 24.04. > > FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without > issue <include typical ymmv disclaimer here> > > > > -----Original Message----- > *From*: Richard Laager via mice-discuss <mice-discuss(a)lists.micemn.net > <Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss(a)lists.micemn.net%3e> > > > *Reply-To*: Richard Laager <rlaager(a)wiktel.com > <Richard%20Laager%20%3crlaager(a)wiktel.com%3e>> > *To*: mice-discuss(a)lists.micemn.net > *Subject*: Re: v6 on rs1/rs2 fubar? > *Date*: Thu, 23 Oct 2025 17:50:22 -0500 > > IXP Manager does not yet support BIRD 3. They don't feel it is > production ready. > > They suggested that maybe our issues would be fixed on newer BIRD 2. > > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we > can upgrade to 26.04 when it comes out, which will have at least BIRD > 2.17.1 (which is the current release right now). > > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then > we could see if that helps. > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 23, 2025

Re: v6 on rs1/rs2 fubar?

by Justin Krejci

As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the default system packages, perhaps that will be sufficiently bug-fixed and is already very accessible to achieve with a simple upgrade to 24.04. FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without issue <include typical ymmv disclaimer here> -----Original Message----- From: Richard Laager via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss@lists.micemn.net%3e>> Reply-To: Richard Laager <rlaager(a)wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>> To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Re: v6 on rs1/rs2 fubar? Date: Thu, 23 Oct 2025 17:50:22 -0500 IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.

Oct. 23, 2025

Re: v6 on rs1/rs2 fubar?

by Richard Laager

IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps. -- Richard

Oct. 23, 2025

RE: v6 on rs1/rs2 fubar?

by Michael Hare

It's probably not a secret that this recurred last evening between roughly 20:34 and 23:02 GMT -5/Central. With the offending prefix [per my logs] being 2a0c:b641:309::/48. Looks like I'll finally be contacting my var today [Juniper] to see specifically which versions will address this issue, but I'd also like to ask, based on this discussion from the first go around [I'm not in the correct circles to see any recent discussion] https://social.bgp.tools/@bgptools/statuses/01K7072TB9HS4JYN8E5Z01GEPK "The main cause of disruption peoples IX route server sessions going down (see image of MegaIXs looking glass), if IX's were running a modern version of bird/OpenBGPd this would have been filtered out!" Is there something "more" that could be done with bird on the IX to prevent future unknown unknowns from causing similar issues? -Michael > -----Original Message----- > From: Michael Hare via mice-discuss <mice-discuss(a)lists.micemn.net> > Sent: Tuesday, October 7, 2025 4:28 PM > To: Chris Wopat <wopat(a)wiscnet.net>; mice-discuss(a)lists.micemn.net > Subject: RE: v6 on rs1/rs2 fubar? > > I am late to the game. > > Me too, starting 14:49 Central/GMT-5 for us. Yes I'm a Juniper. > > = Yes, we use BFD. > = Yes, we have bgp-error-tolerance enabled. > "set protocols bgp bgp-error-tolerance" > [yes, my session is in global] > = No, I haven't contacted Juniper, but I seem to have a misunderstanding about > "bgp-error-tolerance" at the very least. > > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] > rpd[32044]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer > 2001:504:27::d1af:0:1 (External AS 53679) changed st > ate from Established to Idle (event RecvUpdate) (instance master) > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: > %DAEMON-3: BGP ERROR: Insufficient data for the packet > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] > rpd[32044]: %DAEMON-4: bgp_read_v4_update:13800: NOTIFICATION sent to > 2001:504:27::d1af:0:1 (External AS 53679): co > de 3 (Update Message Error) subcode 1 (invalid attribute list) > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: > %DAEMON-3: Received malformed update from 2001:504:27::d1af:0:1 (External > AS 53679) > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: > %DAEMON-3: Family inet6-unicast, prefix 2402:2de0::/32 > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: > %DAEMON-3: Malformed Attribute Unknown(35) flag 0xe0 length 1024. > Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: > %DAEMON-3: BGP ERROR: Insufficient data for the packet > > -Michael > > > -----Original Message----- > > From: Chris Wopat <wopat(a)wiscnet.net> > > Sent: Tuesday, October 7, 2025 4:10 PM > > To: mice-discuss(a)lists.micemn.net > > Subject: Re: v6 on rs1/rs2 fubar? > > > > A few years ago this came out: > > > > https://urldefense.com/v3/__https://blog.benjojo.co.uk/post/bgp-path- > > attributes-grave-error- > > > handling__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMY > > ZVmAbEhYBGcObRfkZtQ-jEoiojfbC9uW1cwqw$ > > > > And the Juniper specific link is to enable bgp-error-tolerance: > > > > https://urldefense.com/v3/__https://supportportal.juniper.net/s/article/202… > > 08-29-Out-of-Cycle-Security-Bulletin-Junos-OS-and-Junos-OS-Evolved-A- > crafted- > > BGP-UPDATE-message-allows-a-remote-attacker-to-de-peer-reset-BGP- > > sessions-CVE-2023- > > > 4481__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMYZV > > mAbEhYBGcObRfkZtQ-jEoiojfbC8RD26zjg$ > > > > > > > https://urldefense.com/v3/__https://www.juniper.net/documentation/us/en/soft > > ware/junos/bgp/topics/topic-map/bgp-error- > > > messages.html__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6I > > pTdMYZVmAbEhYBGcObRfkZtQ-jEoiojfbC9K24DUbQ$ > > > > We (as2381) are not having resets and do have this enabled, as a point > > of comparison. > > > > Cheers, > > -- > > Chris Wopat > > Network Engineer, WiscNet > > wopat(a)wiscnet.net 608-210-3965 > > _______________________________________________ > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 23, 2025

Re: Anyone else seeing instability to MICE route reflector #2

by Richard Laager

BIRD 2.x is single-threaded. The config file is non-trivial. For IPv4, it is 359,018 lines and 38,367,665 bytes (37 MiB). This is due to the massive configuration generated by the IRR stuff. (This is especially true for HE, which is the majority of the config. HE's part of the config is 2,335 lines of ASNs and 217,914 lines of prefixes. Note that there is than one per line.) Since this problem occurs when BIRD is being reconfigured, my working theory is that BIRD is just getting overwhelmed for a bit. From the packet capture I have of CTC, it's definitely the CTC side tearing down the session due to the hold timer expiring. This will likely get better in BIRD 3.x. I think 3.x is multithreaded. On 2025-10-17 16:33, Jonathan Stewart wrote: > What is the problem with shorter hold timers? Shorter hold timers > allow BGP failures to be detected more quickly, reducing time to find > an alternate, functional route. > > 60 second hold timers with 20 second keepalives are fine in the real > world in my experience. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Oct. 17, 2025

Re: Anyone else seeing instability to MICE route reflector #2

by Jonathan Stewart

What is the problem with shorter hold timers? Shorter hold timers allow BGP failures to be detected more quickly, reducing time to find an alternate, functional route. 60 second hold timers with 20 second keepalives are fine in the real world in my experience. Jonathan Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA On Fri, 2025-10-17 4:21 p.m., Richard Laager wrote: > On 2025-10-15 16:32, Richard Laager wrote: >> What do you have for hold timers? And is it the same for both route >> servers? >> >> rs2 is proposing 240 seconds, if I'm reading this packet capture >> correctly. > > I can set a "min hold time" value in BIRD. If we find that raising the > hold timer fixes the issues, I will probably set a minimum. If I set > that, it will break the sessions with lower values immediately, so > this would need some transition. That would probably look something > like: 1) See who it would affect and email them. 2) Enable it on one > route server. 3) After a bit, enable it on the other route server. > -- > Richard > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 17, 2025

Re: Anyone else seeing instability to MICE route reflector #2

by Richard Laager

On 2025-10-15 16:32, Richard Laager wrote: > What do you have for hold timers? And is it the same for both route > servers? > > rs2 is proposing 240 seconds, if I'm reading this packet capture > correctly. I can set a "min hold time" value in BIRD. If we find that raising the hold timer fixes the issues, I will probably set a minimum. If I set that, it will break the sessions with lower values immediately, so this would need some transition. That would probably look something like: 1) See who it would affect and email them. 2) Enable it on one route server. 3) After a bit, enable it on the other route server. -- Richard

Oct. 17, 2025