Hi folks, We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment? Andrew Hoyos hoyosa@gmail.com
Looks like it might be a worldwide issue. Running Juniper? On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
I've seen BGP sessions resetting this afternoon on Juniper. What more do we know from the global community? Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Confirmed at FLIX/CIX-ATL/PIT-IX so far. All juniper but not all junipers affected If someone affected can you send me a trace it would be helpful. On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
FWIW, we saw this from rs1/rs2 on a Cisco: Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Similarly: Oct 7 20:43:21 rpd[32393]: bgp_read_v4_update:12939: NOTIFICATION sent to 2001:504:27::d1af:0:2 (External AS 53679): code 3 (Update Message Error) subcode 1 (invalid attribute list) We do have bgp-error-tolerance enabled, however, so in theory, this should throw out update itself, not teardown session. We’re actually seeing BFD die to route servers as well. Any commonality w/ the IX’s below and route server software? We’re only seeing this on MICE, and v6, FWIW. Andrew Hoyos hoyosa@gmail.com
On Oct 7, 2025, at 15:40, Mark Kane <markk@honeycomb.net> wrote:
FWIW, we saw this from rs1/rs2 on a Cisco:
Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Looks like bird 1.x isn't affected 2.x is. Initial indications are that the route with the malformed attribute is coming from the philippines. On Tue, Oct 7, 2025 at 3:46 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Similarly:
Oct 7 20:43:21 rpd[32393]: bgp_read_v4_update:12939: NOTIFICATION sent to 2001:504:27::d1af:0:2 (External AS 53679): code 3 (Update Message Error) subcode 1 (invalid attribute list)
We do have bgp-error-tolerance enabled, however, so in theory, this should throw out update itself, not teardown session.
We’re actually seeing BFD die to route servers as well.
Any commonality w/ the IX’s below and route server software? We’re only seeing this on MICE, and v6, FWIW.
Andrew Hoyos hoyosa@gmail.com
On Oct 7, 2025, at 15:40, Mark Kane <markk@honeycomb.net> wrote:
FWIW, we saw this from rs1/rs2 on a Cisco:
Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we
manage, and same deal.
Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
This is what I'm seeing on my Cisco NCS edge routers in case it's helpful to anybody: RP/0/RP0/CPU0:Oct 7 14:49:10.478 CDT: bgp[1089]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 RP/0/RP0/CPU0:Oct 7 15:18:11.888 CDT: bgp[1089]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 The router is discarding the malformed update so I'm not seeing any actual issues based on what appears to be a correct RFC7606 error handling implementation by IOS-XR. *Nathan Beard* *7Sigma Systems, Inc.* Senior Network Engineer (218) 820-7636 www.7sigma.com On Tue, Oct 7, 2025 at 4:03 PM Jay Hanke via mice-discuss < mice-discuss@lists.micemn.net> wrote:
Looks like bird 1.x isn't affected 2.x is.
Initial indications are that the route with the malformed attribute is coming from the philippines.
On Tue, Oct 7, 2025 at 3:46 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Similarly:
Oct 7 20:43:21 rpd[32393]: bgp_read_v4_update:12939: NOTIFICATION sent
to 2001:504:27::d1af:0:2 (External AS 53679): code 3 (Update Message Error) subcode 1 (invalid attribute list)
We do have bgp-error-tolerance enabled, however, so in theory, this
should throw out update itself, not teardown session.
We’re actually seeing BFD die to route servers as well.
Any commonality w/ the IX’s below and route server software? We’re only seeing this on MICE, and v6, FWIW.
Andrew Hoyos hoyosa@gmail.com
On Oct 7, 2025, at 15:40, Mark Kane <markk@honeycomb.net> wrote:
FWIW, we saw this from rs1/rs2 on a Cisco:
Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE :
Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we
manage, and same deal.
Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
HE filtered, sessions are starting to come back. On Tue, Oct 7, 2025 at 4:05 PM Nathan Beard <nbeard@7sigma.com> wrote:
This is what I'm seeing on my Cisco NCS edge routers in case it's helpful to anybody:
RP/0/RP0/CPU0:Oct 7 14:49:10.478 CDT: bgp[1089]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 RP/0/RP0/CPU0:Oct 7 15:18:11.888 CDT: bgp[1089]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
The router is discarding the malformed update so I'm not seeing any actual issues based on what appears to be a correct RFC7606 error handling implementation by IOS-XR.
Nathan Beard 7Sigma Systems, Inc. Senior Network Engineer (218) 820-7636 www.7sigma.com
On Tue, Oct 7, 2025 at 4:03 PM Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> wrote:
Looks like bird 1.x isn't affected 2.x is.
Initial indications are that the route with the malformed attribute is coming from the philippines.
On Tue, Oct 7, 2025 at 3:46 PM Andrew Hoyos <hoyosa@gmail.com> wrote:
Similarly:
Oct 7 20:43:21 rpd[32393]: bgp_read_v4_update:12939: NOTIFICATION sent to 2001:504:27::d1af:0:2 (External AS 53679): code 3 (Update Message Error) subcode 1 (invalid attribute list)
We do have bgp-error-tolerance enabled, however, so in theory, this should throw out update itself, not teardown session.
We’re actually seeing BFD die to route servers as well.
Any commonality w/ the IX’s below and route server software? We’re only seeing this on MICE, and v6, FWIW.
Andrew Hoyos hoyosa@gmail.com
On Oct 7, 2025, at 15:40, Mark Kane <markk@honeycomb.net> wrote:
FWIW, we saw this from rs1/rs2 on a Cisco:
Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we
manage, and same deal.
Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
I am getting this same error on my Cisco 9K. -----Original Message----- From: Mark Kane <markk@honeycomb.net> Sent: Tuesday, October 7, 2025 3:41 PM To: Jay Hanke <jayhanke@southfront.io> Cc: mice-discuss@lists.micemn.net Subject: RE: v6 on rs1/rs2 fubar? FWIW, we saw this from rs1/rs2 on a Cisco: Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
https://social.bgp.tools/@bgptools/statuses/01K7072TB9HS4JYN8E5Z01GEPK On Tue, Oct 7, 2025 at 3:50 PM Jon Forbes <jforbes@americomnetworks.com> wrote:
I am getting this same error on my Cisco 9K.
-----Original Message----- From: Mark Kane <markk@honeycomb.net> Sent: Tuesday, October 7, 2025 3:41 PM To: Jay Hanke <jayhanke@southfront.io> Cc: mice-discuss@lists.micemn.net Subject: RE: v6 on rs1/rs2 fubar?
FWIW, we saw this from rs1/rs2 on a Cisco:
Oct 7 14:49:10 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:2 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32 Oct 7 15:18:11 CDT: bgp[1087]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 2001:504:27::d1af:0:1 (VRF: default) - message length 171 bytes, error flags 0x00000100, action taken "TreatAsWdr". Error details: "Error 0x00000100, Field "Attr-length", Attribute 35 (Flags 0xe0, Length 1024), Data [f0230400]". NLRIs: [IPv6 Unicast] 2402:2de0::/32
Both sessions have remained up and appear normal.
-----Original Message----- From: Jay Hanke via mice-discuss <mice-discuss@lists.micemn.net> Sent: October 7, 2025 15:34 To: Jonathan Stewart <jonathan@les.net> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
Confirmed at FLIX/CIX-ATL/PIT-IX so far.
All juniper but not all junipers affected
If someone affected can you send me a trace it would be helpful.
On Tue, Oct 7, 2025 at 3:32 PM Jonathan Stewart <jonathan@les.net> wrote:
I've seen BGP sessions resetting this afternoon on Juniper.
What more do we know from the global community?
Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA
On Tue, 2025-10-07 3:11 p.m., Jay Hanke via mice-discuss wrote:
Looks like it might be a worldwide issue.
Running Juniper?
On Tue, Oct 7, 2025 at 3:07 PM Andrew Hoyos <hoyosa@gmail.com>
wrote:
Hi folks,
We dropped v6 to rs1/rs2. Spot checked a few other members we manage, and same deal. Any work taking place on route servers at the moment?
Andrew Hoyos hoyosa@gmail.com
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
The errors in the logs are: Received: Malformed attribute list and Received: Malformed attribute list: f0230400025a94000000000000000000 Most do NOT have the ": ..." part. Only one particular AS has that additional information. It's not clear to me if that indicates that particular AS is originating the malformed data or not. That AS is not connected to all of the affected IXes, so that may be a red-herring. I'm intentionally not naming the particular AS at this time, to avoid publicly shaming them in case it's nothing they are doing. We are reaching out to them directly, though. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
A few years ago this came out: https://blog.benjojo.co.uk/post/bgp-path-attributes-grave-error-handling And the Juniper specific link is to enable bgp-error-tolerance: https://supportportal.juniper.net/s/article/2023-08-29-Out-of-Cycle-Security... https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic-... We (as2381) are not having resets and do have this enabled, as a point of comparison. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
I am late to the game. Me too, starting 14:49 Central/GMT-5 for us. Yes I'm a Juniper. = Yes, we use BFD. = Yes, we have bgp-error-tolerance enabled. "set protocols bgp bgp-error-tolerance" [yes, my session is in global] = No, I haven't contacted Juniper, but I seem to have a misunderstanding about "bgp-error-tolerance" at the very least. Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] rpd[32044]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2001:504:27::d1af:0:1 (External AS 53679) changed st ate from Established to Idle (event RecvUpdate) (instance master) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: BGP ERROR: Insufficient data for the packet Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] rpd[32044]: %DAEMON-4: bgp_read_v4_update:13800: NOTIFICATION sent to 2001:504:27::d1af:0:1 (External AS 53679): co de 3 (Update Message Error) subcode 1 (invalid attribute list) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Received malformed update from 2001:504:27::d1af:0:1 (External AS 53679) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Family inet6-unicast, prefix 2402:2de0::/32 Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Malformed Attribute Unknown(35) flag 0xe0 length 1024. Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: BGP ERROR: Insufficient data for the packet -Michael
-----Original Message----- From: Chris Wopat <wopat@wiscnet.net> Sent: Tuesday, October 7, 2025 4:10 PM To: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
A few years ago this came out:
https://urldefense.com/v3/__https://blog.benjojo.co.uk/post/bgp-path- attributes-grave-error- handling__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMY ZVmAbEhYBGcObRfkZtQ-jEoiojfbC9uW1cwqw$
And the Juniper specific link is to enable bgp-error-tolerance:
https://urldefense.com/v3/__https://supportportal.juniper.net/s/article/2023... 08-29-Out-of-Cycle-Security-Bulletin-Junos-OS-and-Junos-OS-Evolved-A-crafted- BGP-UPDATE-message-allows-a-remote-attacker-to-de-peer-reset-BGP- sessions-CVE-2023- 4481__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMYZV mAbEhYBGcObRfkZtQ-jEoiojfbC8RD26zjg$
https://urldefense.com/v3/__https://www.juniper.net/documentation/us/en/soft ware/junos/bgp/topics/topic-map/bgp-error- messages.html__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6I pTdMYZVmAbEhYBGcObRfkZtQ-jEoiojfbC9K24DUbQ$
We (as2381) are not having resets and do have this enabled, as a point of comparison.
Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965 _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
It's probably not a secret that this recurred last evening between roughly 20:34 and 23:02 GMT -5/Central. With the offending prefix [per my logs] being 2a0c:b641:309::/48. Looks like I'll finally be contacting my var today [Juniper] to see specifically which versions will address this issue, but I'd also like to ask, based on this discussion from the first go around [I'm not in the correct circles to see any recent discussion] https://social.bgp.tools/@bgptools/statuses/01K7072TB9HS4JYN8E5Z01GEPK "The main cause of disruption peoples IX route server sessions going down (see image of MegaIXs looking glass), if IX's were running a modern version of bird/OpenBGPd this would have been filtered out!" Is there something "more" that could be done with bird on the IX to prevent future unknown unknowns from causing similar issues? -Michael
-----Original Message----- From: Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> Sent: Tuesday, October 7, 2025 4:28 PM To: Chris Wopat <wopat@wiscnet.net>; mice-discuss@lists.micemn.net Subject: RE: v6 on rs1/rs2 fubar?
I am late to the game.
Me too, starting 14:49 Central/GMT-5 for us. Yes I'm a Juniper.
= Yes, we use BFD. = Yes, we have bgp-error-tolerance enabled. "set protocols bgp bgp-error-tolerance" [yes, my session is in global] = No, I haven't contacted Juniper, but I seem to have a misunderstanding about "bgp-error-tolerance" at the very least.
Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] rpd[32044]: %DAEMON-4-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2001:504:27::d1af:0:1 (External AS 53679) changed st ate from Established to Idle (event RecvUpdate) (instance master) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: BGP ERROR: Insufficient data for the packet Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.warning'] rpd[32044]: %DAEMON-4: bgp_read_v4_update:13800: NOTIFICATION sent to 2001:504:27::d1af:0:1 (External AS 53679): co de 3 (Update Message Error) subcode 1 (invalid attribute list) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Received malformed update from 2001:504:27::d1af:0:1 (External AS 53679) Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Family inet6-unicast, prefix 2402:2de0::/32 Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: Malformed Attribute Unknown(35) flag 0xe0 length 1024. Oct 7 14:49:10 r-minneapolis-511.uwsys.net [LEVEL='daemon.err'] rpd[32044]: %DAEMON-3: BGP ERROR: Insufficient data for the packet
-Michael
-----Original Message----- From: Chris Wopat <wopat@wiscnet.net> Sent: Tuesday, October 7, 2025 4:10 PM To: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
A few years ago this came out:
https://urldefense.com/v3/__https://blog.benjojo.co.uk/post/bgp-path- attributes-grave-error-
handling__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMY
ZVmAbEhYBGcObRfkZtQ-jEoiojfbC9uW1cwqw$
And the Juniper specific link is to enable bgp-error-tolerance:
https://urldefense.com/v3/__https://supportportal.juniper.net/s/article/2023... 08-29-Out-of-Cycle-Security-Bulletin-Junos-OS-and-Junos-OS-Evolved-A- crafted- BGP-UPDATE-message-allows-a-remote-attacker-to-de-peer-reset-BGP- sessions-CVE-2023-
4481__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6IpTdMYZV
mAbEhYBGcObRfkZtQ-jEoiojfbC8RD26zjg$
https://urldefense.com/v3/__https://www.juniper.net/documentation/us/en/soft
ware/junos/bgp/topics/topic-map/bgp-error-
messages.html__;!!Mak6IKo!MP3CJWT3VXc9yyZOF25Zkl6ylazMccDKnoNw_w6I
pTdMYZVmAbEhYBGcObRfkZtQ-jEoiojfbC9K24DUbQ$
We (as2381) are not having resets and do have this enabled, as a point of comparison.
Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965 _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps. -- Richard
As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the default system packages, perhaps that will be sufficiently bug-fixed and is already very accessible to achieve with a simple upgrade to 24.04. FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without issue <include typical ymmv disclaimer here> -----Original Message----- From: Richard Laager via mice-discuss <mice-discuss@lists.micemn.net<mailto:Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss@lists.micemn.net%3e>> Reply-To: Richard Laager <rlaager@wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>> To: mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Re: v6 on rs1/rs2 fubar? Date: Thu, 23 Oct 2025 17:50:22 -0500 IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
According to the NA-IX Slack BIRD 2.17.1. fixes the issue, or at least doesn't propagate it. Thanks On Thu, Oct 23, 2025 at 6:23 PM Justin Krejci <JKrejci@usinternet.com> wrote:
As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the default system packages, perhaps that will be sufficiently bug-fixed and is already very accessible to achieve with a simple upgrade to 24.04.
FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without issue <include typical ymmv disclaimer here>
-----Original Message----- *From*: Richard Laager via mice-discuss <mice-discuss@lists.micemn.net <Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss@lists.micemn.net%3e>
*Reply-To*: Richard Laager <rlaager@wiktel.com <Richard%20Laager%20%3crlaager@wiktel.com%3e>> *To*: mice-discuss@lists.micemn.net *Subject*: Re: v6 on rs1/rs2 fubar? *Date*: Thu, 23 Oct 2025 17:50:22 -0500
IXP Manager does not yet support BIRD 3. They don't feel it is production ready.
They suggested that maybe our issues would be fixed on newer BIRD 2.
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
JTAC pointed me to TSB76599. It’s behind a login, so while I won’t copy/paste the entire PDF, in summary: “The following Junos and Junos Evolved software releases are now fully compliant with RFC7606” 22.4R3, 22.4R3-EVO, and subsequent releases 23.2R2, 23.2R2-EVO, and subsequent releases 23.4R1, 23.4R1-EVO, and subsequent releases 24.1R1, 24.1R1-EVO, and all subsequent releases -Michael From: David Farmer via mice-discuss <mice-discuss@lists.micemn.net> Sent: Thursday, October 23, 2025 6:31 PM To: Justin Krejci <JKrejci@usinternet.com> Cc: mice-discuss@lists.micemn.net; rlaager@wiktel.com Subject: Re: v6 on rs1/rs2 fubar? According to the NA-IX Slack BIRD 2.17.1. fixes the issue, or at least doesn't propagate it. Thanks On Thu, Oct 23, 2025 at 6:23 PM Justin Krejci <JKrejci@usinternet.com<mailto:JKrejci@usinternet.com>> wrote: As a point of comparison Ubuntu 24.04 LTS includes bird 2.14 in the default system packages, perhaps that will be sufficiently bug-fixed and is already very accessible to achieve with a simple upgrade to 24.04. FWIW, I have done the upgrade from 22.04 to 24.04 many times, all without issue <include typical ymmv disclaimer here> -----Original Message----- From: Richard Laager via mice-discuss <mice-discuss@lists.micemn.net<mailto:Richard%20Laager%20via%20mice-discuss%20%3cmice-discuss@lists.micemn.net%3e>> Reply-To: Richard Laager <rlaager@wiktel.com<mailto:Richard%20Laager%20%3crlaager@wiktel.com%3e>> To: mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Re: v6 on rs1/rs2 fubar? Date: Thu, 23 Oct 2025 17:50:22 -0500 IXP Manager does not yet support BIRD 3. They don't feel it is production ready. They suggested that maybe our issues would be fixed on newer BIRD 2. MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now). I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps. _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave@lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
I've built 2.17.1 for 22.04. I'm thinking we'll upgrade BIRD on rs2 tomorrow noon-ish, and apply regular updates while we are at it. It might help with the timeouts we are seeing there. Regardless, as long as it's not worse, then next week (tentatively Monday), we can upgrade BIRD on rs1. -- Richard
Thanks, really appreciate it. Frank [cid:a1ff5d57-4afc-4ec0-be61-51708bf94a4e.png] Frank Bulk Chief Technology Officer / Chief Operating Officer [cid:premiercomm_logo_cmyk_1388ffcb-8ec0-4842-9e40-a0e598482e7d.png] 339 1st Ave NE | Sioux Center, IA 51250 | Find us on Facebook<https://www.facebook.com/PremierCommunicationsSiouxCenter> phone: 712.722.3451 web: mypremieronline.com<https://www.mypremieronline.com/> direct: (712) 722-7152 cell: (712) 441-6049 From: Richard Laager via mice-discuss <mice-discuss@lists.micemn.net> Sent: Thursday, October 23, 2025 9:42 PM To: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar? I've built 2.17.1 for 22.04. I'm thinking we'll upgrade BIRD on rs2 tomorrow noon-ish, and apply regular updates while we are at it. It might help with the timeouts we are seeing there. Regardless, as long as it's not worse, then next week (tentatively Monday), we can upgrade BIRD on rs1. -- Richard
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
This is not the devops way, baby! -- Mike Horwath, reachable via drechsau@Geeks.ORG
I don't want to jeopardize the IX, but I'm game for whatever people are comfortable with. Cool story, as of writing, there is an event going on =right now= for 139.64.160.0/22 since roughly 01:04 local -Michael
-----Original Message----- From: Mike Horwath <drechsau@geeks.org> Sent: Friday, October 24, 2025 8:40 AM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
This is not the devops way, baby!
-- Mike Horwath, reachable via drechsau@Geeks.ORG _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
I have a ticket open with HE asking for filters. HE#6787436 On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I don't want to jeopardize the IX, but I'm game for whatever people are comfortable with.
Cool story, as of writing, there is an event going on =right now= for 139.64.160.0/22 since roughly 01:04 local
-Michael
-----Original Message----- From: Mike Horwath <drechsau@geeks.org> Sent: Friday, October 24, 2025 8:40 AM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
This is not the devops way, baby!
-- Mike Horwath, reachable via drechsau@Geeks.ORG _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Jay, Thanks. To be super clear, I don't want anyone jumping through hoops they otherwise would not on behalf of AS3128. It's pretty obvious this is the new norm at this point. FWIW, the main reason I haven't pushed a corrected version of JunOS [yet] 1] My target version is affected by Juniper TSB103739.. So I have another case open on that. 2] We're fortunate to have other options for where traffic is going + five minute holddown timers = churn is not exceptionally impacting -Michael
-----Original Message----- From: Jay Hanke <jayhanke@southfront.io> Sent: Friday, October 24, 2025 9:06 AM To: Michael Hare <michael.hare@wisc.edu> Cc: Mike Horwath <drechsau@geeks.org>; Richard Laager <rlaager@wiktel.com>; mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
I have a ticket open with HE asking for filters. HE#6787436
On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I don't want to jeopardize the IX, but I'm game for whatever people are
comfortable with.
Cool story, as of writing, there is an event going on =right now= for
139.64.160.0/22 since roughly 01:04 local
-Michael
-----Original Message----- From: Mike Horwath <drechsau@geeks.org> Sent: Friday, October 24, 2025 8:40 AM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-
discuss
wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
This is not the devops way, baby!
-- Mike Horwath, reachable via drechsau@Geeks.ORG _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
It looks like up to ~25% of the networks attached to the route server are affected. I'm showing about 60 sessions down/flapping. On Fri, Oct 24, 2025 at 9:18 AM Michael Hare <michael.hare@wisc.edu> wrote:
Jay,
Thanks. To be super clear, I don't want anyone jumping through hoops they otherwise would not on behalf of AS3128. It's pretty obvious this is the new norm at this point.
FWIW, the main reason I haven't pushed a corrected version of JunOS [yet]
1] My target version is affected by Juniper TSB103739.. So I have another case open on that. 2] We're fortunate to have other options for where traffic is going + five minute holddown timers = churn is not exceptionally impacting
-Michael
-----Original Message----- From: Jay Hanke <jayhanke@southfront.io> Sent: Friday, October 24, 2025 9:06 AM To: Michael Hare <michael.hare@wisc.edu> Cc: Mike Horwath <drechsau@geeks.org>; Richard Laager <rlaager@wiktel.com>; mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
I have a ticket open with HE asking for filters. HE#6787436
On Fri, Oct 24, 2025 at 8:51 AM Michael Hare via mice-discuss <mice-discuss@lists.micemn.net> wrote:
I don't want to jeopardize the IX, but I'm game for whatever people are
comfortable with.
Cool story, as of writing, there is an event going on =right now= for
139.64.160.0/22 since roughly 01:04 local
-Michael
-----Original Message----- From: Mike Horwath <drechsau@geeks.org> Sent: Friday, October 24, 2025 8:40 AM To: Richard Laager <rlaager@wiktel.com> Cc: mice-discuss@lists.micemn.net Subject: Re: v6 on rs1/rs2 fubar?
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-
discuss
wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
I could attempt to rebuild the BIRD 2.17.1 package on Ubuntu 22.04. Then we could see if that helps.
This is not the devops way, baby!
-- Mike Horwath, reachable via drechsau@Geeks.ORG _______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
On 2025-10-24 08:40, Mike Horwath wrote:
On Thu, Oct 23, 2025 at 05:50:22PM -0500, Richard Laager via mice-discuss wrote:
MICE is running Ubuntu 22.04, which has BIRD 2.0.8. In a few months, we can upgrade to 26.04 when it comes out, which will have at least BIRD 2.17.1 (which is the current release right now).
why not just do-release-upgrade to 24.04 and snag the newerver version that way?
As a general rule, I don't upgrade* production systems in place. To upgrade, I build a new VM, configure it (using Ansible), test, then cut over. This allows for more testing before going live, preserves the option to revert to the old VM as it was, generally reduces the downtime of the upgrade, forces/proves you have everything in Ansible, etc. It is more work, though. But not a lot more work, when it's VMs and configurations in Ansible. In this specific case, upgrading to 24.04 is also more heavy handed and more work than we need. We just need newer BIRD. As it turns out (Barry from IXP Manager pointed this out), the BIRD team has repositories for Ubuntu (and Debian) with newer versions available: https://pkg.labs.nic.cz/doc/?project=bird So I just used that. That was super easy, and it also means we can stay current on BIRD on an ongoing basis. * I'm using the word "upgrade" to indicate moving to a new distro, as opposed to "update" to indicate applying security patches. I imagine full devops would be to rebuild the whole thing (probably using containers) every time, even for patches. -- Richard
participants (14)
-
Andrew Hoyos -
Chris Wopat -
David Farmer -
Frank Bulk -
Jay Hanke -
Jon Forbes -
Jonathan Stewart -
Justin Krejci -
Mark Kane -
Michael Hare -
Mike Horwath -
Nathan Beard -
Richard Laager -
Richard Laager