January 2025 - mice-discuss - lists.micemn.net

Please welcome Fiber Minnesota to MICE!

by Jeremy Lumby

Name: Fiber Minnesota Switch: MICE 400G Core - 511 Ports: E 3/2 and E 4/2 in Po 4 (2x400G) ASN: This is a Remote Switch uplink upgrade from 400G to 800G IPv4: NA IPv6: NA Peering Contact: NA Route Servers: NA Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 31, 2025

Please welcome Minnesota VoIP to MICE!

by Jeremy Lumby

Name: Twin City VoIP d/b/a: Minnesota VoIP Switch: MICE 400G Core - 511 Ports: E 3/1 (1x400G) ASN: This is a Remote Switch uplink upgrade from 300G to 400G IPv4: NA IPv6: NA Peering Contact: peering(a)mnvoip.com Route Servers: NA Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 31, 2025

MICE Switch-Cabinet Migration Round 1

by Jeremy Lumby

On Wednesday February 5th at approximately 1PM CST I will be migrating the following networks off of the Arista 7504R (100Gig core switch) to the new 7804R3 (400Gig core switch) at 511. Prior to the migration BGP will be blocked on the specific participant's port to allow traffic to gracefully reroute. This will be the first round of weekly moves as everything is migrated out of the existing cabinet. If you are interested in upgrading to 400Gig let us know ASAP so we can avoid double moves, and facilitate an easier transition for everyone. Ridgeview core switch uplink Old Ports: Port-channel10 Ethernet3/35/1 Ethernet3/36/1 Ethernet5/14/1 Ethernet5/15/1 Ethernet6/35/1 Ethernet6/36/1 New Ports: Port-channel3 Ethernet3/33/1 Ethernet3/34/1 Ethernet4/32/1 Ethernet4/33/1 Ethernet4/34/1 Ethernet4/36/1 MN VoIP extension switch uplink Old Ports: Port-channel15 Ethernet3/6/1 Ethernet6/6/1 Ethernet6/13/1 New Ports: Port-channel1 Ethernet3/1/1 (Native 400G port) Fiber Minnesota extension switch uplink Old Ports: Port-channel22 Ethernet3/7/1 Ethernet5/13/1 Ethernet5/16/1 Ethernet6/3/1 New Ports: Port-channel4 Ethernet3/2/1 (Native 400G port) Ethernet4/2/1 (Native 400G port) Akamai Old Ports: Port-channel19 Ethernet6/8/1 Ethernet6/10/1 Ethernet6/12/1 Ethernet6/31/1 Ethernet5/7/1 New Ports: Port-channel5 Ethernet3/3/1 Ethernet3/4/1 Ethernet3/5/1 Ethernet3/6/1 Ethernet3/7/1 Amazon Old Ports: Port-channel5 Ethernet3/12/1 Ethernet6/4/1 Ethernet5/3/1 Ethernet5/4/1 New Ports: Port-channel6 Ethernet3/8/1 Ethernet3/9/1 Ethernet3/10/1 Ethernet3/11/1 Apple Old Ports: Port-channel30 Ethernet 3/14/1 New Ports: Port-channel7 Ethernet 3/12/1 Arvig Old Ports: Port-channel42 Ethernet3/8/1 Ethernet6/11/1 Ethernet5/1/1 New Ports: Port-channel8 Ethernet3/13/1 Ethernet4/13/1 Ethernet4/14/1 Fastly Old Ports: Port-channel18 Ethernet4/50/1 New Ports: Port-channel9 Ethernet3/14/1 Meta Old Ports: Port-channel26 Ethernet3/10/1 New Ports: Port-channel10 Ethernet3/15/1 Microsoft Old Ports: Port-channel34 Ethernet3/18/1 New Ports: Port-channel11 Ethernet3/16/1 Netflix Old Ports: Port-channel2 Ethernet3/11/1 Ethernet3/27/1 New Ports: Port-channel12 Ethernet3/17/1 Ethernet3/18/1 Northern Lights GigaPOP Old Ports: Ethernet6/14/1 New Ports: Ethernet3/19/1 Qwilt Old Ports: Port-channel12 Ethernet6/29/1 Ethernet5/2/1 New Ports: Port-channel13 Ethernet3/20/1 Ethernet4/20/1 T-Mobile Old Ports: Port-channel41 Ethernet3/2/1 Ethernet6/9/1 Ethernet6/15/1 New Ports: Port-channel14 Ethernet3/21/1 Ethernet4/21/1 Ethernet4/22/1 Let me know if you have any questions Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 eFax 952-873-7425 jlumby(a)mnvoip.com

Jan. 30, 2025

1G Disconnections

by Richard Laager

The following participants were disconnected yesterday, as they were still on 1G and the 1G switch is going away as part of the cabinet move and 400G switch upgrade. Coloplast [AS25111] Evolve IP (fka Advanced Integrated Tech / AI Tech) [AS13746] ipHouse [AS7753] ipHouse was going away in any event, due to acquisition activity as previously discussed. The other two did confirm they were aware of the disconnect. They elected not to connect at 10G or move to a remote. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Jan. 23, 2025

Re: 2025-01-20 MICE Maintenance - Route Server 2

by Corey Hines

Do note that Arista only supports passive CR DACs. Fixed length Active Optical Cables (AOC) are another option that can be leveraged for short distances but greater than a copper DAC. Here are a number of optical standards based options for 400G transceivers (attached screenshot or inline): [image: image.png] More detail: https://www.arista.com/en/products/transceivers-cables Thanks! Corey Hines Technical Lead, Systems Engineering ACE: L5 Certified ACE: CVP Specialist Arista Networks *612-209-6550chines(a)arista.com <chines(a)arista.com>* *TAC: support(a)arista.com <support(a)arista.com>* www.arista.com On Thu, Jan 23, 2025 at 9:58 AM Brady Kittel <bkittel(a)gmail.com> wrote: > As a server guy I'll mention two things. > > 1) If we're using SM fiber with LR optics I've seen issues with LR optics > burning out on the receive side when used with short distances to servers. > Depends on the optic it seems but it definitely happens. Fix for us was > just to either use SR optics with mm fiber or use a DAC. Definitely avoid > the use of active mode DACs too. I've seen all sorts of problems with > compatibility and even causing OS crashes. Never had problems with hardware > compatible passive mode DACs in my experience across our fleet of 350 > servers. > > On Sun, Jan 19, 2025, 7:50 PM Richard Laager <rlaager(a)micemn.net> wrote: > >> MICE physical server 2 (and thus by extension route server 2) is again >> having errors on its peering interface. We will be performing maintenance >> tomorrow to address this. Since this only affects one of the two route >> servers, participants should not be impacted. >> >> >> Details: >> >> Sometime tomorrow (exact time depending on day job schedules), Jeremy and >> I will: >> >> 1. Apply the usual security updates to VMs and the host, since the host >> is going to be rebooted anyway. >> >> 2. Convert server 2's port to a LAG. This will make future changes >> easier, especially if this error problem continues. >> >> This will involve a reboot of the server. The reboot is "necessary" >> because Linux's netplan does not remove VLAN configuration when changes are >> made (only adds it). Without removing the old >> direct-on-the-physical-interface VLAN config, we could (would?) create a >> loop. You may recall that I created a loop that blew up the exchange in >> more-or-less that exact way when setting up the servers originally. >> >> While I could remove the VLAN configuration manually, in this case, it's >> easier and safer to simply reboot. That also ensures our running >> configuration is fully consistent with the configuration files. >> >> 3. Replace the optic, also moving to the other port on the NIC. >> >> >> The underlying failure manifests as a decrease in optical transmit power >> (i.e. power received at the switch) over time. We have replaced the optic >> previously, which fixes it, but then it fails again after a while. We are >> open to suggestions about why this might be occurring. >> >> One theory is that this is due to heat. We are going to try the second >> port in case that makes a difference. We are also considering switching to >> a direct attach cable, as perhaps that would produce less heat. >> >> -- >> Richard Laager >> Chief Manager, Director >> Midwest Internet Cooperative Exchange LLC >> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >> > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

Jan. 23, 2025

Re: 2025-01-20 MICE Maintenance - Route Server 2

by Brady Kittel

As a server guy I'll mention two things. 1) If we're using SM fiber with LR optics I've seen issues with LR optics burning out on the receive side when used with short distances to servers. Depends on the optic it seems but it definitely happens. Fix for us was just to either use SR optics with mm fiber or use a DAC. Definitely avoid the use of active mode DACs too. I've seen all sorts of problems with compatibility and even causing OS crashes. Never had problems with hardware compatible passive mode DACs in my experience across our fleet of 350 servers. On Sun, Jan 19, 2025, 7:50 PM Richard Laager <rlaager(a)micemn.net> wrote: > MICE physical server 2 (and thus by extension route server 2) is again > having errors on its peering interface. We will be performing maintenance > tomorrow to address this. Since this only affects one of the two route > servers, participants should not be impacted. > > > Details: > > Sometime tomorrow (exact time depending on day job schedules), Jeremy and > I will: > > 1. Apply the usual security updates to VMs and the host, since the host is > going to be rebooted anyway. > > 2. Convert server 2's port to a LAG. This will make future changes easier, > especially if this error problem continues. > > This will involve a reboot of the server. The reboot is "necessary" > because Linux's netplan does not remove VLAN configuration when changes are > made (only adds it). Without removing the old > direct-on-the-physical-interface VLAN config, we could (would?) create a > loop. You may recall that I created a loop that blew up the exchange in > more-or-less that exact way when setting up the servers originally. > > While I could remove the VLAN configuration manually, in this case, it's > easier and safer to simply reboot. That also ensures our running > configuration is fully consistent with the configuration files. > > 3. Replace the optic, also moving to the other port on the NIC. > > > The underlying failure manifests as a decrease in optical transmit power > (i.e. power received at the switch) over time. We have replaced the optic > previously, which fixes it, but then it fails again after a while. We are > open to suggestions about why this might be occurring. > > One theory is that this is due to heat. We are going to try the second > port in case that makes a difference. We are also considering switching to > a direct attach cable, as perhaps that would produce less heat. > > -- > Richard Laager > Chief Manager, Director > Midwest Internet Cooperative Exchange LLC > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

Jan. 23, 2025

Re: MICE 400G Switch

by Mike Horwath

On Thu, Jan 23, 2025 at 09:26:48AM -0600, Jeremy Lumby wrote: > Here are some pics: Thanks! That's some chonk! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Jan. 23, 2025

RE: MICE 400G Switch

by Jeremy Lumby

Here are some pics: https://www.mnvoip.com/MICE400G/1.jpg https://www.mnvoip.com/MICE400G/2.jpg https://www.mnvoip.com/MICE400G/3.jpg https://www.mnvoip.com/MICE400G/4.jpg https://www.mnvoip.com/MICE400G/5.jpg https://www.mnvoip.com/MICE400G/6.jpg https://www.mnvoip.com/MICE400G/7.jpg https://www.mnvoip.com/MICE400G/8.jpg https://www.mnvoip.com/MICE400G/9.jpg -----Original Message----- From: Mike Horwath <drechsau(a)geeks.org> Sent: Thursday, January 23, 2025 7:37 AM To: MICE Discuss <mice-discuss(a)lists.micemn.net> Subject: Re: MICE 400G Switch On Thu, Jan 23, 2025 at 12:02:13AM -0600, Richard Laager wrote: > The 400G switch has arrived. whoop! > Jeremy racked the switch. It's a big one! Pics!? -- Mike Horwath, reachable via drechsau(a)Geeks.ORG _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net

Jan. 23, 2025

Re: MICE 400G Switch

by Mike Horwath

On Thu, Jan 23, 2025 at 12:02:13AM -0600, Richard Laager wrote: > The 400G switch has arrived. whoop! > Jeremy racked the switch. It's a big one! Pics!? -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Jan. 23, 2025

MICE 400G Switch

by Richard Laager

The 400G switch has arrived. Jeremy racked the switch. It's a big one! We don't have a firm go-live date at this point, but it will be soon. If you are interested in 400G ports, the Order Form has been updated: https://micemn.net/files/MICE%20Order%20Form.docx (I've just sent this to mice-discuss so far. I'm thinking we'll send something to mice-announce once the switch is live. But if people think I should be sending it there now, I'm open to the suggestion.) -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Jan. 23, 2025