- mice-discuss - lists.micemn.net

Re: Anyone else seeing instability to MICE route reflector #2

by Richard Laager

Let's see if the BIRD upgrade fixes this. It was suggested on the IXP Manager list that newer BIRD might fix this. If it does not, then I'd recommend you try raising your timers (if you haven't already). At a minimum, we should do that as a test. If it works, then that's probably at least a workaround. I'm hoping that BIRD 3 (which is multithreaded) will fix this, but IXP Manager doesn't support BIRD 3 yet (as they don't consider it production ready, though no reasons were given). On 2025-10-23 22:40, Frank Bulk wrote: > > We’re doing 10/30 against the two route servers. > -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by Richard Laager

On 2025-10-24 08:40, Mike Horwath wrote: > On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote: >> MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can >> upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 >> (which is the current release right now). > > why not just do-release-upgrade to 24.04 and snag the newerver version > that way? As a general rule, I don't upgrade* production systems in place. To upgrade, I build a new VM, configure it (using Ansible), test, then cut over. This allows for more testing before going live, preserves the option to revert to the old VM as it was, generally reduces the downtime of the upgrade, forces/proves you have everything in Ansible, etc. It is more work, though. But not a lot more work, when it's VMs and configurations in Ansible. In this specific case, upgrading to 24.04 is also more heavy handed and more work than we need. We just need newer BIRD. As it turns out (Barry from IXP Manager pointed this out), the BIRD team has repositories for Ubuntu (and Debian) with newer versions available: https://pkg.labs.nic.cz/doc/?project=bird So I just used that. That was super easy, and it also means we can stay current on BIRD on an ongoing basis. * I'm using the word "upgrade" to indicate moving to a new distro, as opposed to "update" to indicate applying security patches. I imagine full devops would be to rebuild the whole thing (probably using containers) every time, even for patches. -- Richard

Oct. 24, 2025

Re: MICE RS2 Bird Update

by Richard Laager

I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now. Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable. On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote: > MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central > time today to resolve the malformed BGP update issues causing flapping > BGP sessions. > > We expect the upgrade to be less than 30 minutes. All sessions to RS2 > will drop during this time. Sessions to RS1 will be unaffected. > > MICE is planning to apply the same patch to RS1 (511) on Monday. -- Richard

Oct. 24, 2025

MICE RS2 Bird Update

by Jay Hanke

MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by Jay Hanke

It looks like up to ~25% of the networks attached to the route server are affected. I'm showing about 60 sessions down/flapping. On Fri, Oct 24, 2025 at 9:18 AM Michael Hare <michael.hare(a)wisc.edu> wrote: > > Jay, > > Thanks. To be super clear, I don't want anyone jumping through hoops they otherwise would not on behalf of AS3128. It's pretty obvious this is the new norm at this point. > > FWIW, the main reason I haven't pushed a corrected version of JunOS [yet] > > 1] My target version is affected by Juniper TSB103739.. So I have another case open on that. > 2] We're fortunate to have other options for where traffic is going + five minute holddown timers = churn is not exceptionally impacting > > -Michael > > > -----Original Message----- > > From: Jay Hanke <jayhanke(a)southfront.io> > > Sent: Friday, October 24, 2025 9:06 AM > > To: Michael Hare <michael.hare(a)wisc.edu> > > Cc: Mike Horwath <drechsau(a)geeks.org>; Richard Laager > > <rlaager(a)wiktel.com>; mice-discuss(a)lists.micemn.net > > Subject: Re: v6 on rs1/rs2 fubar? > > > > I have a ticket open with HE asking for filters. HE#6787436 > > > > On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss > > <mice-discuss(a)lists.micemn.net> wrote: > > > > > > I don't want to jeopardize the IX, but I'm game for whatever people are > > comfortable with. > > > > > > Cool story, as of writing, there is an event going on =right now= for > > 139.64.160.0/22 since roughly 01:04 local > > > > > > -Michael > > > > > > > -----Original Message----- > > > > From: Mike Horwath <drechsau(a)geeks.org> > > > > Sent: Friday, October 24, 2025 8:40 AM > > > > To: Richard Laager <rlaager(a)wiktel.com> > > > > Cc: mice-discuss(a)lists.micemn.net > > > > Subject: Re: v6 on rs1/rs2 fubar? > > > > > > > > On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice- > > discuss > > > > wrote: > > > > > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, > > we > > > > can > > > > > upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 > > > > > (which is the current release right now). > > > > > > > > why not just do-release-upgrade to 24.04 and snag the newerver version > > > > that way? > > > > > > > > > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. > > Then > > > > we > > > > > could see if that helps. > > > > > > > > This is not the devops way, baby! > > > > > > > > -- > > > > Mike Horwath, reachable via drechsau(a)Geeks.ORG > > > > _______________________________________________ > > > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > > > _______________________________________________ > > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 24, 2025

RE: v6 on rs1/rs2 fubar?

by Michael Hare

Jay, Thanks. To be super clear, I don't want anyone jumping through hoops they otherwise would not on behalf of AS3128. It's pretty obvious this is the new norm at this point. FWIW, the main reason I haven't pushed a corrected version of JunOS [yet] 1] My target version is affected by Juniper TSB103739.. So I have another case open on that. 2] We're fortunate to have other options for where traffic is going + five minute holddown timers = churn is not exceptionally impacting -Michael > -----Original Message----- > From: Jay Hanke <jayhanke(a)southfront.io> > Sent: Friday, October 24, 2025 9:06 AM > To: Michael Hare <michael.hare(a)wisc.edu> > Cc: Mike Horwath <drechsau(a)geeks.org>; Richard Laager > <rlaager(a)wiktel.com>; mice-discuss(a)lists.micemn.net > Subject: Re: v6 on rs1/rs2 fubar? > > I have a ticket open with HE asking for filters. HE#6787436 > > On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss > <mice-discuss(a)lists.micemn.net> wrote: > > > > I don't want to jeopardize the IX, but I'm game for whatever people are > comfortable with. > > > > Cool story, as of writing, there is an event going on =right now= for > 139.64.160.0/22 since roughly 01:04 local > > > > -Michael > > > > > -----Original Message----- > > > From: Mike Horwath <drechsau(a)geeks.org> > > > Sent: Friday, October 24, 2025 8:40 AM > > > To: Richard Laager <rlaager(a)wiktel.com> > > > Cc: mice-discuss(a)lists.micemn.net > > > Subject: Re: v6 on rs1/rs2 fubar? > > > > > > On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice- > discuss > > > wrote: > > > > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, > we > > > can > > > > upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 > > > > (which is the current release right now). > > > > > > why not just do-release-upgrade to 24.04 and snag the newerver version > > > that way? > > > > > > > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. > Then > > > we > > > > could see if that helps. > > > > > > This is not the devops way, baby! > > > > > > -- > > > Mike Horwath, reachable via drechsau(a)Geeks.ORG > > > _______________________________________________ > > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > > _______________________________________________ > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by Jay Hanke

I have a ticket open with HE asking for filters. HE#6787436 On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss <mice-discuss(a)lists.micemn.net> wrote: > > I don't want to jeopardize the IX, but I'm game for whatever people are comfortable with. > > Cool story, as of writing, there is an event going on =right now= for 139.64.160.0/22 since roughly 01:04 local > > -Michael > > > -----Original Message----- > > From: Mike Horwath <drechsau(a)geeks.org> > > Sent: Friday, October 24, 2025 8:40 AM > > To: Richard Laager <rlaager(a)wiktel.com> > > Cc: mice-discuss(a)lists.micemn.net > > Subject: Re: v6 on rs1/rs2 fubar? > > > > On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss > > wrote: > > > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we > > can > > > upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 > > > (which is the current release right now). > > > > why not just do-release-upgrade to 24.04 and snag the newerver version > > that way? > > > > > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then > > we > > > could see if that helps. > > > > This is not the devops way, baby! > > > > -- > > Mike Horwath, reachable via drechsau(a)Geeks.ORG > > _______________________________________________ > > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 24, 2025

RE: v6 on rs1/rs2 fubar?

by Michael Hare

I don't want to jeopardize the IX, but I'm game for whatever people are comfortable with. Cool story, as of writing, there is an event going on =right now= for 139.64.160.0/22 since roughly 01:04 local -Michael > -----Original Message----- > From: Mike Horwath <drechsau(a)geeks.org> > Sent: Friday, October 24, 2025 8:40 AM > To: Richard Laager <rlaager(a)wiktel.com> > Cc: mice-discuss(a)lists.micemn.net > Subject: Re: v6 on rs1/rs2 fubar? > > On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss > wrote: > > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we > can > > upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 > > (which is the current release right now). > > why not just do-release-upgrade to 24.04 and snag the newerver version > that way? > > > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then > we > > could see if that helps. > > This is not the devops way, baby! > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Oct. 24, 2025

Re: v6 on rs1/rs2 fubar?

by Mike Horwath

On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote: > MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can > upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 > (which is the current release right now). why not just do-release-upgrade to 24.04 and snag the newerver version that way? > I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we > could see if that helps. This is not the devops way, baby! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 24, 2025

RE: v6 on rs1/rs2 fubar?

by Frank Bulk

Thanks, really appreciate it. Frank [cid:a1ff5d57-4afc-4ec0-be61-51708bf94a4e.png] Frank Bulk Chief Technology Officer / Chief Operating Officer [cid:premiercomm_logo_cmyk_1388ffcb-8ec0-4842-9e40-a0e598482e7d.png] 339 1st Ave NE | Sioux Center, IA 51250 | Find us on Facebook<https://www.facebook.com/PremierCommunicationsSiouxCenter> phone: 712.722.3451 web: mypremieronline.com<https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049 From: Richard Laager via mice-discuss <mice-discuss(a)lists.micemn.net> Sent: Thursday, October 23, 2025 9:42 PM To: mice-discuss(a)lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar? I've built 2.17.1 for 22.04. I'm thinking we'll upgrade BIRD on rs2 tomorrow noon-ish, and apply regular updates while we are at it. It might help with the timeouts we are seeing there. Regardless, as long as it's not worse, then next week (tentatively Monday), we can upgrade BIRD on rs1. -- Richard

Oct. 24, 2025