MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday. -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 507-327-2398 https://southfront.io/
I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now. Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable. On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote:
MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions.
We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected.
MICE is planning to apply the same patch to RS1 (511) on Monday.
-- Richard
I’d be all for pushing updates to RS1 as well, and an alert/alarm free weekend :) Andrew Hoyos hoyosa@gmail.com
On Oct 24, 2025, at 12:51, Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now.
Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable.
On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote:
MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday.
-- Richard _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
I’m fine either way, but the counterpoint to Andrew’s comment is that if an ASN is affected [25%], has chosen not to upgrade yet [I’m in that boat], peered with both RS and doing next-hop-self, service impacts should be negligible other than annoying syslogs or perhaps undo burden on RS1 while things are in this state. It doesn’t seem like there are too many MICE members vocal about this within the last 24 hours so I’m not sure I would great it as an emergency. I wouldn’t upgrade both members of a redundancy pair essentially at the same time unless there was a critical impact to services. I’d hate to be the guy that says “update RS1 right now” and then things go south on both this weekend. -Michael From: Andrew Hoyos <hoyosa@gmail.com> Sent: Friday, October 24, 2025 1:58 PM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: MICE RS2 Bird Update I’d be all for pushing updates to RS1 as well, and an alert/alarm free weekend :) Andrew Hoyos hoyosa@gmail.com<mailto:hoyosa@gmail.com> On Oct 24, 2025, at 12:51, Richard Laager via mice-discuss <mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> wrote: I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now. Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable. On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote: MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday. -- Richard _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave@lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net>
I think we should do RS1 as well assuming the refresh worked without issue on RS2. j On Fri, Oct 24, 2025 at 2:29 PM Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I’m fine either way, but the counterpoint to Andrew’s comment is that if an ASN is affected [25%], has chosen not to upgrade yet [I’m in that boat], peered with both RS and doing next-hop-self, service impacts should be negligible other than annoying syslogs or perhaps undo burden on RS1 while things are in this state.
It doesn’t seem like there are too many MICE members vocal about this within the last 24 hours so I’m not sure I would great it as an emergency. I wouldn’t upgrade both members of a redundancy pair essentially at the same time unless there was a critical impact to services. I’d hate to be the guy that says “update RS1 right now” and then things go south on both this weekend.
-Michael
From: Andrew Hoyos <hoyosa@gmail.com> Sent: Friday, October 24, 2025 1:58 PM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: MICE RS2 Bird Update
I’d be all for pushing updates to RS1 as well, and an alert/alarm free weekend :)
Andrew Hoyos
hoyosa@gmail.com
On Oct 24, 2025, at 12:51, Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now.
Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable.
On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote:
MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday.
-- Richard _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Unless there is an emergency level condition on RS1, I'd prefer if it was left alone for the weekend. RS1 has been stable for us, while RS2 has not been. I'd hate to have the stable RS break over the weekend. Shawn Rasmussen -----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: Friday, October 24, 2025 3:43 PM To: Michael Hare <michael.hare@wisc.edu> Cc: Andrew Hoyos <hoyosa@gmail.com>; Richard Laager <rlaager@wiktel.com>; mice-discuss@lists.micemn.net Subject: Re: MICE RS2 Bird Update CAUTION: This email originated from outside CTC. Do not open links or attachments unless you are expecting this message and know they are safe. I think we should do RS1 as well assuming the refresh worked without issue on RS2. j On Fri, Oct 24, 2025 at 2:29 PM Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I’m fine either way, but the counterpoint to Andrew’s comment is that if an ASN is affected [25%], has chosen not to upgrade yet [I’m in that boat], peered with both RS and doing next-hop-self, service impacts should be negligible other than annoying syslogs or perhaps undo burden on RS1 while things are in this state.
It doesn’t seem like there are too many MICE members vocal about this within the last 24 hours so I’m not sure I would great it as an emergency. I wouldn’t upgrade both members of a redundancy pair essentially at the same time unless there was a critical impact to services. I’d hate to be the guy that says “update RS1 right now” and then things go south on both this weekend.
-Michael
From: Andrew Hoyos <hoyosa@gmail.com> Sent: Friday, October 24, 2025 1:58 PM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: MICE RS2 Bird Update
I’d be all for pushing updates to RS1 as well, and an alert/alarm free weekend :)
Andrew Hoyos
hoyosa@gmail.com
On Oct 24, 2025, at 12:51, Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I updated rs2. For IPv4, there are 40 more sessions up now than before the upgrade, so this clearly helped with the BGP attribute issue that's going on right now.
Opinions on whether I should upgrade rs1 sooner (i.e. today) vs waiting until Monday, given the active issue? At a minimum, I'll wait a few hours to ensure rs2 is stable.
On 2025-10-24 10:08, Jay Hanke via mice-discuss wrote:
MICE will be upgrading BIRD on RS2 (Ridgeview) at 12:00 PM Central time today to resolve the malformed BGP update issues causing flapping BGP sessions. We expect the upgrade to be less than 30 minutes. All sessions to RS2 will drop during this time. Sessions to RS1 will be unaffected. MICE is planning to apply the same patch to RS1 (511) on Monday.
-- Richard _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
On Fri, Oct 24, 2025 at 08:52:37PM +0000, Shawn Rasmussen via mice-discuss wrote:
Unless there is an emergency level condition on RS1, I'd prefer if it was left alone for the weekend. RS1 has been stable for us, while RS2 has not been. I'd hate to have the stable RS break over the weekend.
100% agree And changes on a friday?! Nah, dawg. -- Mike Horwath, reachable via drechsau@Geeks.ORG
I see the same number of sessions active on rs1 and rs2. So unless I'm missing something, it seems like the issue is no longer active. So at this point, I'd rather wait until Monday to update rs1. -- Richard
HE filltered the route with the invalid attribute. On Fri, Oct 24, 2025, 9:54 PM Richard Laager via mice-discuss < mice-discuss@lists.micemn.net> wrote:
I see the same number of sessions active on rs1 and rs2. So unless I'm missing something, it seems like the issue is no longer active. So at this point, I'd rather wait until Monday to update rs1.
-- Richard
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Hi folks, Just wanted to confirm a time for today this upgrade on rs1 will happen? Given the upgrade didn’t happen Friday, we just shutdown sessions to rs1 to avoid flapping BGP sessions (and alerts) all weekend. Andrew Hoyos hoyosa@gmail.com
On Oct 24, 2025, at 21:48, Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I see the same number of sessions active on rs1 and rs2. So unless I'm missing something, it seems like the issue is no longer active. So at this point, I'd rather wait until Monday to update rs1. -- Richard _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Did the upgrade on RS2 affect the scanner issue? On Mon, Oct 27, 2025 at 1:03 PM Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I just finished upgrading rs1.
-- Richard
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
scanner? On 2025-10-27 13:47, Jay Hanke wrote:
Did the upgrade on RS2 affect the scanner issue?
On Mon, Oct 27, 2025 at 1:03 PM Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I just finished upgrading rs1.
-- Richard
_______________________________________________ mice-discuss mailing list --mice-discuss@lists.micemn.net To unsubscribe send an email tomice-discuss-leave@lists.micemn.net
-- Richard
The hold-time expiration issues some members were having. On Mon, Oct 27, 2025 at 2:06 PM Richard Laager <rlaager@wiktel.com> wrote:
scanner?
On 2025-10-27 13:47, Jay Hanke wrote:
Did the upgrade on RS2 affect the scanner issue?
On Mon, Oct 27, 2025 at 1:03 PM Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I just finished upgrading rs1.
-- Richard
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
-- Richard
participants (6)
-
Andrew Hoyos -
Jay Hanke -
Michael Hare -
Mike Horwath -
Richard Laager -
Shawn Rasmussen