- mice-discuss - lists.micemn.net

Re: MICE interface flap

by Frank Bulk

Thanks for sharing, we definitely saw this, too. We had a *somewhat* similar issue last year that when we slotted a card into our 7504 and the other card rebooted. Turned out to be some kind of mesh bug. Bug ID 765840 resolved in 4.30.1. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, February 14, 2024 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE interface flap I am still sorting out the details. But it was related to the card swap scheduled for today. The card was in up and running without an issue for several minutes, however when I programmed/moved the first link to it, all of the other cards in the core switch started rebooting themselves. I am opening a ticket with Arista since we seem to be stable at the moment. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Chris Wopat Sent: Wednesday, February 14, 2024 4:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE interface flap WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: https://link.edgepilot.com/s/5abcf6c9/_Ox0o9rFAUimx0FLu4zJ0g?u=http://micel… Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965 Links contained in this email have been replaced. If you click on a link in the email above, the link will be analyzed for known threats. If a known threat is found, you will not be able to proceed to the destination. If suspicious content is detected, you will see a warning.

Feb. 16, 2024

Re: MICE interface flap

by Jeremy Lumby

I just wanted to send over a quick update. I stayed at 511 for an hour after all of the line cards came back online. The core switch remained stable. I performed basic actions such as programming ports, inserting optics into the new card, and bringing links up on it. None of them triggered the cards to reboot again. At the moment I believe that the switch is stable. I will report back any findings from Arista as to the root cause when I have them, and so that this can be avoided in the future. -----Original Message----- From: Jeremy Lumby <jlumby(a)mnvoip.com> Sent: Wednesday, February 14, 2024 4:57 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: [MICE-DISCUSS] MICE interface flap I am still sorting out the details. But it was related to the card swap scheduled for today. The card was in up and running without an issue for several minutes, however when I programmed/moved the first link to it, all of the other cards in the core switch started rebooting themselves. I am opening a ticket with Arista since we seem to be stable at the moment. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Chris Wopat Sent: Wednesday, February 14, 2024 4:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE interface flap WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 15, 2024

Re: MICE interface flap

by Jeremy Lumby

I am still sorting out the details. But it was related to the card swap scheduled for today. The card was in up and running without an issue for several minutes, however when I programmed/moved the first link to it, all of the other cards in the core switch started rebooting themselves. I am opening a ticket with Arista since we seem to be stable at the moment. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Chris Wopat Sent: Wednesday, February 14, 2024 4:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE interface flap WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 14, 2024

MICE interface flap

by Chris Wopat

WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 14, 2024

Please welcome Integris to MICE!

by Jeremy Lumby

Name: Integris LLC (fka Compudyne) Switch: MICE Core - 511 Ports: E 3/34 in po 9 (100G) ASN: This is a Remote Switch uplink upgrade from 20G to 100G IPv4: NA IPv6: NA Peering Contact: peering(a)integrisit.com Route Servers: NA Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Feb. 14, 2024

Re: MICE Server Updates

by Cameron Skappel

This evening I applied server updates to the rest of the MICE servers including Route Server 1. Let me know if there are issues. -- Cameron

Feb. 9, 2024

Please welcome SpaceX to MICE!

by Jeremy Lumby

Name: Space Exploration Technologies Inc. Switch: MICE Core - Ridgeview Ports: E 3/7 and 3/8 in po 3 (200G) Strands 17-20 ASN: 14593 IPv4: 206.108.255.226/24 IPv6: 2001:504:27:0:0:3901::2/64 Peering Contact: starlink-peering(a)spacex.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Feb. 8, 2024

Re: MICE BFD Errors

by Andrew Hoyos

Just from our POV for a data point (timestamps in GMT) Feb 8 04:52:48 er1.511.msp bfdd[31763]: BFDD_STATE_UP_TO_DOWN: BFD Session 2001:504:27::d1af:0:2 (IFL 360) state Up -> Down LD/RD(228/3270602753) Up time:16:39:29 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry. Feb 8 04:52:48 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 228, new state: down, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 Feb 8 04:59:05 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_UP: local discriminator: 228, new state: up, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 Feb 8 05:12:26 er1.511.msp bfdd[31763]: BFDD_STATE_UP_TO_DOWN: BFD Session 2001:504:27::d1af:0:2 (IFL 360) state Up -> Down LD/RD(228/1591314386) Up time:00:13:22 Local diag: NbrSignal Remote diag: CtlExpire Reason: Received DOWN from PEER. Feb 8 05:12:26 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 228, new state: down, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 We saw BFD go down to rs2 when those updates took place an hour ago and hasn’t come back up, still down. Juniper here, but based on ARP/ND for the others, looks like a mix of Cisco/Juniper devices so perhaps not specific to any platform. Address State Interface Time Interval Multiplier 2001:504:27::d1af:0:2 Down et-0/0/1.0 3.000 2.000 3 Client BGP, TX interval 0.500, RX interval 0.500 Session down time 01:11:17, previous up time 00:13:22 Local diagnostic NbrSignal, remote diagnostic CtlExpire Remote state Down, version 1 Session type: Single hop BFD Min async interval 0.500, min slow interval 2.000 Adaptive async TX interval 2.000, RX interval 2.000 Local min TX interval 2.000, minimum RX interval 0.500, multiplier 3 Remote min TX interval 1.000, min RX interval 0.500, multiplier 3 Local discriminator 228, remote discriminator 1591314386 Echo TX interval 2.000, echo detection interval 6.000 Echo mode disabled/inactiveno-absorb, no-refresh Session ID: 0x0 > On Feb 8, 2024, at 00:01, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > BIRD is complaining that various peers are sending BFD packets with an unknown session id. > > Anyone have thoughts on this? > > > The logs look like this: > 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) > > Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. > > > rs1: > > Inteliquent (Neutral Tandem) > 206.108.255.137 * > > > rs2: > > les.net > 2001:504:27::4813:0:1 > > Savage Communications > 206.108.255.17 * > 2001:504:27::5adc:0:1 * > > Stellar Association > 206.108.255.53 > 2001:504:27::8e16:0:1 * > > Hoyos Consulting > 2001:504:27::d15d:0:1 > > Eastern Iowa Data Group > 206.108.255.168 * > 2001:504:27::f7a2:0:1 * > > Omnitel Communications > 206.108.255.148 > > > Here is for the entire month of January: > > rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c > 9 2001:504:27::3823:0:2 > 2 2001:504:27::8252:0:1 > 1183836 2001:504:27::b7f8:0:1 > 4 2001:504:27::d15d:0:1 > 1183834 206.108.255.101 > 7498 206.108.255.119 > 380 206.108.255.121 > 1463099 206.108.255.137 > 3 206.108.255.176 > 3 206.108.255.185 > > root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c > 12 2001:504:27::1e49:0:1 > 1 2001:504:27::3823:0:2 > 1 2001:504:27::39:0:1 > 1457433 2001:504:27::5adc:0:1 > 3 2001:504:27::6478:0:1 > 1457369 2001:504:27::8e16:0:1 > 1 2001:504:27::9dbe:0:1 > 5 2001:504:27::9dbe:0:2 > 1179733 2001:504:27::b7f8:0:1 > 1176975 2001:504:27::d15d:0:1 > 4 2001:504:27::e4f:0:1 > 1 2001:504:27::f756:0:1 > 1331747 2001:504:27::f7a2:0:1 > 1179735 206.108.255.101 > 7494 206.108.255.119 > 1954 206.108.255.120 > 3 206.108.255.137 > 3 206.108.255.161 > 1331745 206.108.255.168 > 1457437 206.108.255.17 > 2 206.108.255.176 > 8 206.108.255.185 > 11 206.108.255.3 > 6 206.108.255.46 > 1457345 206.108.255.53 > 4 206.108.255.92 > > > -- > Richard

Feb. 8, 2024

MICE BFD Errors

by Richard Laager

BIRD is complaining that various peers are sending BFD packets with an unknown session id. Anyone have thoughts on this? The logs look like this: 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. rs1: Inteliquent (Neutral Tandem) 206.108.255.137 * rs2: les.net 2001:504:27::4813:0:1 Savage Communications 206.108.255.17 * 2001:504:27::5adc:0:1 * Stellar Association 206.108.255.53 2001:504:27::8e16:0:1 * Hoyos Consulting 2001:504:27::d15d:0:1 Eastern Iowa Data Group 206.108.255.168 * 2001:504:27::f7a2:0:1 * Omnitel Communications 206.108.255.148 Here is for the entire month of January: rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c 9 2001:504:27::3823:0:2 2 2001:504:27::8252:0:1 1183836 2001:504:27::b7f8:0:1 4 2001:504:27::d15d:0:1 1183834 206.108.255.101 7498 206.108.255.119 380 206.108.255.121 1463099 206.108.255.137 3 206.108.255.176 3 206.108.255.185 root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c 12 2001:504:27::1e49:0:1 1 2001:504:27::3823:0:2 1 2001:504:27::39:0:1 1457433 2001:504:27::5adc:0:1 3 2001:504:27::6478:0:1 1457369 2001:504:27::8e16:0:1 1 2001:504:27::9dbe:0:1 5 2001:504:27::9dbe:0:2 1179733 2001:504:27::b7f8:0:1 1176975 2001:504:27::d15d:0:1 4 2001:504:27::e4f:0:1 1 2001:504:27::f756:0:1 1331747 2001:504:27::f7a2:0:1 1179735 206.108.255.101 7494 206.108.255.119 1954 206.108.255.120 3 206.108.255.137 3 206.108.255.161 1331745 206.108.255.168 1457437 206.108.255.17 2 206.108.255.176 8 206.108.255.185 11 206.108.255.3 6 206.108.255.46 1457345 206.108.255.53 4 206.108.255.92 -- Richard

Feb. 8, 2024

MICE Server Updates

by Cameron Skappel

This evening I applied server updates to some of the MICE servers including Router Server 2. Let me know if there are issues. -- Cameron

Feb. 8, 2024