February 2024 - mice-discuss - lists.micemn.net

Re: MICE interface flap

by Jeremy Lumby

I am still sorting out the details. But it was related to the card swap scheduled for today. The card was in up and running without an issue for several minutes, however when I programmed/moved the first link to it, all of the other cards in the core switch started rebooting themselves. I am opening a ticket with Arista since we seem to be stable at the moment. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Chris Wopat Sent: Wednesday, February 14, 2024 4:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE interface flap WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 14, 2024

MICE interface flap

by Chris Wopat

WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 14, 2024

Please welcome Integris to MICE!

by Jeremy Lumby

Name: Integris LLC (fka Compudyne) Switch: MICE Core - 511 Ports: E 3/34 in po 9 (100G) ASN: This is a Remote Switch uplink upgrade from 20G to 100G IPv4: NA IPv6: NA Peering Contact: peering(a)integrisit.com Route Servers: NA Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Feb. 14, 2024

Re: MICE Server Updates

by Cameron Skappel

This evening I applied server updates to the rest of the MICE servers including Route Server 1. Let me know if there are issues. -- Cameron

Feb. 9, 2024

Please welcome SpaceX to MICE!

by Jeremy Lumby

Name: Space Exploration Technologies Inc. Switch: MICE Core - Ridgeview Ports: E 3/7 and 3/8 in po 3 (200G) Strands 17-20 ASN: 14593 IPv4: 206.108.255.226/24 IPv6: 2001:504:27:0:0:3901::2/64 Peering Contact: starlink-peering(a)spacex.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Feb. 8, 2024

Re: MICE BFD Errors

by Andrew Hoyos

Just from our POV for a data point (timestamps in GMT) Feb 8 04:52:48 er1.511.msp bfdd[31763]: BFDD_STATE_UP_TO_DOWN: BFD Session 2001:504:27::d1af:0:2 (IFL 360) state Up -> Down LD/RD(228/3270602753) Up time:16:39:29 Local diag: CtlExpire Remote diag: None Reason: Detect Timer Expiry. Feb 8 04:52:48 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 228, new state: down, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 Feb 8 04:59:05 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_UP: local discriminator: 228, new state: up, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 Feb 8 05:12:26 er1.511.msp bfdd[31763]: BFDD_STATE_UP_TO_DOWN: BFD Session 2001:504:27::d1af:0:2 (IFL 360) state Up -> Down LD/RD(228/1591314386) Up time:00:13:22 Local diag: NbrSignal Remote diag: CtlExpire Reason: Received DOWN from PEER. Feb 8 05:12:26 er1.511.msp bfdd[31763]: BFDD_TRAP_SHOP_STATE_DOWN: local discriminator: 228, new state: down, interface: et-0/0/1.0, peer addr: 2001:504:27::d1af:0:2 We saw BFD go down to rs2 when those updates took place an hour ago and hasn’t come back up, still down. Juniper here, but based on ARP/ND for the others, looks like a mix of Cisco/Juniper devices so perhaps not specific to any platform. Address State Interface Time Interval Multiplier 2001:504:27::d1af:0:2 Down et-0/0/1.0 3.000 2.000 3 Client BGP, TX interval 0.500, RX interval 0.500 Session down time 01:11:17, previous up time 00:13:22 Local diagnostic NbrSignal, remote diagnostic CtlExpire Remote state Down, version 1 Session type: Single hop BFD Min async interval 0.500, min slow interval 2.000 Adaptive async TX interval 2.000, RX interval 2.000 Local min TX interval 2.000, minimum RX interval 0.500, multiplier 3 Remote min TX interval 1.000, min RX interval 0.500, multiplier 3 Local discriminator 228, remote discriminator 1591314386 Echo TX interval 2.000, echo detection interval 6.000 Echo mode disabled/inactiveno-absorb, no-refresh Session ID: 0x0 > On Feb 8, 2024, at 00:01, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > BIRD is complaining that various peers are sending BFD packets with an unknown session id. > > Anyone have thoughts on this? > > > The logs look like this: > 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) > > Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. > > > rs1: > > Inteliquent (Neutral Tandem) > 206.108.255.137 * > > > rs2: > > les.net > 2001:504:27::4813:0:1 > > Savage Communications > 206.108.255.17 * > 2001:504:27::5adc:0:1 * > > Stellar Association > 206.108.255.53 > 2001:504:27::8e16:0:1 * > > Hoyos Consulting > 2001:504:27::d15d:0:1 > > Eastern Iowa Data Group > 206.108.255.168 * > 2001:504:27::f7a2:0:1 * > > Omnitel Communications > 206.108.255.148 > > > Here is for the entire month of January: > > rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c > 9 2001:504:27::3823:0:2 > 2 2001:504:27::8252:0:1 > 1183836 2001:504:27::b7f8:0:1 > 4 2001:504:27::d15d:0:1 > 1183834 206.108.255.101 > 7498 206.108.255.119 > 380 206.108.255.121 > 1463099 206.108.255.137 > 3 206.108.255.176 > 3 206.108.255.185 > > root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c > 12 2001:504:27::1e49:0:1 > 1 2001:504:27::3823:0:2 > 1 2001:504:27::39:0:1 > 1457433 2001:504:27::5adc:0:1 > 3 2001:504:27::6478:0:1 > 1457369 2001:504:27::8e16:0:1 > 1 2001:504:27::9dbe:0:1 > 5 2001:504:27::9dbe:0:2 > 1179733 2001:504:27::b7f8:0:1 > 1176975 2001:504:27::d15d:0:1 > 4 2001:504:27::e4f:0:1 > 1 2001:504:27::f756:0:1 > 1331747 2001:504:27::f7a2:0:1 > 1179735 206.108.255.101 > 7494 206.108.255.119 > 1954 206.108.255.120 > 3 206.108.255.137 > 3 206.108.255.161 > 1331745 206.108.255.168 > 1457437 206.108.255.17 > 2 206.108.255.176 > 8 206.108.255.185 > 11 206.108.255.3 > 6 206.108.255.46 > 1457345 206.108.255.53 > 4 206.108.255.92 > > > -- > Richard

Feb. 8, 2024

MICE BFD Errors

by Richard Laager

BIRD is complaining that various peers are sending BFD packets with an unknown session id. Anyone have thoughts on this? The logs look like this: 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. rs1: Inteliquent (Neutral Tandem) 206.108.255.137 * rs2: les.net 2001:504:27::4813:0:1 Savage Communications 206.108.255.17 * 2001:504:27::5adc:0:1 * Stellar Association 206.108.255.53 2001:504:27::8e16:0:1 * Hoyos Consulting 2001:504:27::d15d:0:1 Eastern Iowa Data Group 206.108.255.168 * 2001:504:27::f7a2:0:1 * Omnitel Communications 206.108.255.148 Here is for the entire month of January: rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c 9 2001:504:27::3823:0:2 2 2001:504:27::8252:0:1 1183836 2001:504:27::b7f8:0:1 4 2001:504:27::d15d:0:1 1183834 206.108.255.101 7498 206.108.255.119 380 206.108.255.121 1463099 206.108.255.137 3 206.108.255.176 3 206.108.255.185 root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c 12 2001:504:27::1e49:0:1 1 2001:504:27::3823:0:2 1 2001:504:27::39:0:1 1457433 2001:504:27::5adc:0:1 3 2001:504:27::6478:0:1 1457369 2001:504:27::8e16:0:1 1 2001:504:27::9dbe:0:1 5 2001:504:27::9dbe:0:2 1179733 2001:504:27::b7f8:0:1 1176975 2001:504:27::d15d:0:1 4 2001:504:27::e4f:0:1 1 2001:504:27::f756:0:1 1331747 2001:504:27::f7a2:0:1 1179735 206.108.255.101 7494 206.108.255.119 1954 206.108.255.120 3 206.108.255.137 3 206.108.255.161 1331745 206.108.255.168 1457437 206.108.255.17 2 206.108.255.176 8 206.108.255.185 11 206.108.255.3 6 206.108.255.46 1457345 206.108.255.53 4 206.108.255.92 -- Richard

Feb. 8, 2024

MICE Server Updates

by Cameron Skappel

This evening I applied server updates to some of the MICE servers including Router Server 2. Let me know if there are issues. -- Cameron

Feb. 8, 2024

MICE Core Switch Card Upgrade

by Richard Laager

Date & Time Wednesday, February 14 at 15:00 US/Central Description Jeremy will swap a (now unused) 10G line card for a 100G line card in the Arista core switch at 511. There is no expected impact for most participants. In the (relatively unlikely) event that a fiber cable is physically in the way and he has to unplug it to remove/install the card, he will disable BGP for that participant first and wait (unless the port is part of a LAG with sufficient excess capacity at that time of day). The Fiber Minnesota and Minnesota VoIP remote switches are connected by 3x100G LAG. Jeremy will move one port from each LAG from card 6 to the new card. Rollback Plan If there is some problem, the new 100G card can be removed. The ports in the Fiber Minnesota and Minnesota VoIP remote switches can be moved back to card 6. -- Richard

Feb. 8, 2024

Please welcome Mark Twain Rural Telephone Company to MICE!

by Jeremy Lumby

Name: Mark Twain Rural Telephone Company Switch: MN VoIP 10G Switch Port: E 14 ASN: 10750 IPv4: 206.108.255.225/24 IPv6: 2001:504:27:0:0:29FE::1/64 Peering Contact: noc(a)marktwain.net Route Servers: Yes Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Feb. 7, 2024