MICE physical server 2 (and thus by extension route server 2) is again having errors on its peering interface. We will be performing maintenance tomorrow to address this. Since this only affects one of the two route servers, participants should not be impacted. Details: Sometime tomorrow (exact time depending on day job schedules), Jeremy and I will: 1. Apply the usual security updates to VMs and the host, since the host is going to be rebooted anyway. 2. Convert server 2's port to a LAG. This will make future changes easier, especially if this error problem continues. This will involve a reboot of the server. The reboot is "necessary" because Linux's netplan does not remove VLAN configuration when changes are made (only adds it). Without removing the old direct-on-the-physical-interface VLAN config, we could (would?) create a loop. You may recall that I created a loop that blew up the exchange in more-or-less that exact way when setting up the servers originally. While I could remove the VLAN configuration manually, in this case, it's easier and safer to simply reboot. That also ensures our running configuration is fully consistent with the configuration files. 3. Replace the optic, also moving to the other port on the NIC. The underlying failure manifests as a decrease in optical transmit power (i.e. power received at the switch) over time. We have replaced the optic previously, which fixes it, but then it fails again after a while. We are open to suggestions about why this might be occurring. One theory is that this is due to heat. We are going to try the second port in case that makes a difference. We are also considering switching to a direct attach cable, as perhaps that would produce less heat. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
As a server guy I'll mention two things. 1) If we're using SM fiber with LR optics I've seen issues with LR optics burning out on the receive side when used with short distances to servers. Depends on the optic it seems but it definitely happens. Fix for us was just to either use SR optics with mm fiber or use a DAC. Definitely avoid the use of active mode DACs too. I've seen all sorts of problems with compatibility and even causing OS crashes. Never had problems with hardware compatible passive mode DACs in my experience across our fleet of 350 servers. On Sun, Jan 19, 2025, 7:50 PM Richard Laager <rlaager@micemn.net> wrote:
MICE physical server 2 (and thus by extension route server 2) is again having errors on its peering interface. We will be performing maintenance tomorrow to address this. Since this only affects one of the two route servers, participants should not be impacted.
Details:
Sometime tomorrow (exact time depending on day job schedules), Jeremy and I will:
1. Apply the usual security updates to VMs and the host, since the host is going to be rebooted anyway.
2. Convert server 2's port to a LAG. This will make future changes easier, especially if this error problem continues.
This will involve a reboot of the server. The reboot is "necessary" because Linux's netplan does not remove VLAN configuration when changes are made (only adds it). Without removing the old direct-on-the-physical-interface VLAN config, we could (would?) create a loop. You may recall that I created a loop that blew up the exchange in more-or-less that exact way when setting up the servers originally.
While I could remove the VLAN configuration manually, in this case, it's easier and safer to simply reboot. That also ensures our running configuration is fully consistent with the configuration files.
3. Replace the optic, also moving to the other port on the NIC.
The underlying failure manifests as a decrease in optical transmit power (i.e. power received at the switch) over time. We have replaced the optic previously, which fixes it, but then it fails again after a while. We are open to suggestions about why this might be occurring.
One theory is that this is due to heat. We are going to try the second port in case that makes a difference. We are also considering switching to a direct attach cable, as perhaps that would produce less heat.
-- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
Do note that Arista only supports passive CR DACs. Fixed length Active Optical Cables (AOC) are another option that can be leveraged for short distances but greater than a copper DAC. Here are a number of optical standards based options for 400G transceivers (attached screenshot or inline): [image: image.png] More detail: https://www.arista.com/en/products/transceivers-cables Thanks! Corey Hines Technical Lead, Systems Engineering ACE: L5 Certified ACE: CVP Specialist Arista Networks *612-209-6550chines@arista.com <chines@arista.com>* *TAC: support@arista.com <support@arista.com>* www.arista.com On Thu, Jan 23, 2025 at 9:58 AM Brady Kittel <bkittel@gmail.com> wrote:
As a server guy I'll mention two things.
1) If we're using SM fiber with LR optics I've seen issues with LR optics burning out on the receive side when used with short distances to servers. Depends on the optic it seems but it definitely happens. Fix for us was just to either use SR optics with mm fiber or use a DAC. Definitely avoid the use of active mode DACs too. I've seen all sorts of problems with compatibility and even causing OS crashes. Never had problems with hardware compatible passive mode DACs in my experience across our fleet of 350 servers.
On Sun, Jan 19, 2025, 7:50 PM Richard Laager <rlaager@micemn.net> wrote:
MICE physical server 2 (and thus by extension route server 2) is again having errors on its peering interface. We will be performing maintenance tomorrow to address this. Since this only affects one of the two route servers, participants should not be impacted.
Details:
Sometime tomorrow (exact time depending on day job schedules), Jeremy and I will:
1. Apply the usual security updates to VMs and the host, since the host is going to be rebooted anyway.
2. Convert server 2's port to a LAG. This will make future changes easier, especially if this error problem continues.
This will involve a reboot of the server. The reboot is "necessary" because Linux's netplan does not remove VLAN configuration when changes are made (only adds it). Without removing the old direct-on-the-physical-interface VLAN config, we could (would?) create a loop. You may recall that I created a loop that blew up the exchange in more-or-less that exact way when setting up the servers originally.
While I could remove the VLAN configuration manually, in this case, it's easier and safer to simply reboot. That also ensures our running configuration is fully consistent with the configuration files.
3. Replace the optic, also moving to the other port on the NIC.
The underlying failure manifests as a decrease in optical transmit power (i.e. power received at the switch) over time. We have replaced the optic previously, which fixes it, but then it fails again after a while. We are open to suggestions about why this might be occurring.
One theory is that this is due to heat. We are going to try the second port in case that makes a difference. We are also considering switching to a direct attach cable, as perhaps that would produce less heat.
-- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
_______________________________________________ mice-discuss mailing list -- mice-discuss@lists.micemn.net To unsubscribe send an email to mice-discuss-leave@lists.micemn.net
participants (3)
-
Brady Kittel -
Corey Hines -
Richard Laager