Proposal: MICE Static MAC address Only
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port. Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event. -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
I agree. Connections should be going into to routers but not everyone does that. Single Mac protects everyones traffic. Gary Glissendorf | Network Architect gary.glissendorf@sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Monday, October 31, 2022 8:27 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port. Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event. -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***
I disagree with static MAC address filtering unless we beef up operational ability to change/adjust in a reasonable timeframe, or offer a self service portal to do so (ie IXP Manager implementation first) Sent from my iPhone
On Oct 31, 2022, at 9:59 PM, Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I have similar concerns as Andrew. However, I'm cognizant of Jeremy's concerns as well. Nevertheless, if we are going to fix our old problems, we have to be willing to accept that we are likely to create other new problems. The hard part is deciding if the old problems, the known problems, are worse than the potential new problems, the potentially unknown problems, and which of the potentially new problems we are willing to accept. We seem to have a consensus that this particular old problem needs to be addressed; what is not clear is if we have a consensus on accepting the potential new problems and which of those potential problems we are willing to accept. Note either way we are accepting new problems; we are either accepting potential new problems caused by the automation of IX Manager. Otherwise, we are accepting the potential new problems caused by a static MAC ACL list with only manual changes; I expect delays in changes primarily. Both options are going to produce problems. And if they are delays impacting one of the large content peers they could have as significant of an impact as the original problem we are trying to solve. Thanks On Mon, Oct 31, 2022 at 9:02 AM Andrew Hoyos <hoyosa@gmail.com> wrote:
I disagree with static MAC address filtering unless we beef up operational ability to change/adjust in a reasonable timeframe, or offer a self service portal to do so (ie IXP Manager implementation first)
Sent from my iPhone
On Oct 31, 2022, at 9:59 PM, Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants:https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
Well articulated, as always, David. Thank you. I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though.... Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please. Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored. I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl? On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston@wiktel.com> wrote:
Well articulated, as always, David. Thank you.
I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though....
Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please.
Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored.
I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic. On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke@southfront.io> wrote:
What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl?
On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston@wiktel.com> wrote:
Well articulated, as always, David. Thank you.
I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though....
Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please.
Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored.
I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
The hard limit is to keep someone from stacking networks behind a port. On Wed, Nov 2, 2022 at 2:58 PM David Farmer <0000000e7948cb21-dmarc-request@lists.iphouse.net> wrote:
Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic.
On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke@southfront.io> wrote:
What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl?
On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston@wiktel.com> wrote:
Well articulated, as always, David. Thank you.
I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though....
Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please.
Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored.
I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
And if we statically define the MAC to port mapping do we care? On Wed, Nov 2, 2022 at 3:01 PM Jay Hanke <jayhanke@southfront.io> wrote:
The hard limit is to keep someone from stacking networks behind a port.
On Wed, Nov 2, 2022 at 2:58 PM David Farmer <0000000e7948cb21-dmarc-request@lists.iphouse.net> wrote:
Maybe we don't need to define a hard limit of 1 or 2, but only that the
list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic.
On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke@southfront.io> wrote:
What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl?
On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston@wiktel.com>
Well articulated, as always, David. Thank you.
I am in favor of the use of static MAC ACL, even if that requires
manual
changes to that ACL, and the additional coordination that may consume. A few thoughts, though....
Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please.
Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change,
wrote: this
sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored.
I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
Yes. The remote switch terms cover this for sharing a single MICE uplink. Also, when a problem arises, we can kill a port without doing a packet capture and only the offending member can be shut off. On Wed, Nov 2, 2022 at 3:23 PM David Farmer <0000000e7948cb21-dmarc-request@lists.iphouse.net> wrote:
And if we statically define the MAC to port mapping do we care?
On Wed, Nov 2, 2022 at 3:01 PM Jay Hanke <jayhanke@southfront.io> wrote:
The hard limit is to keep someone from stacking networks behind a port.
On Wed, Nov 2, 2022 at 2:58 PM David Farmer <0000000e7948cb21-dmarc-request@lists.iphouse.net> wrote:
Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic.
On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke@southfront.io> wrote:
What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl?
On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston@wiktel.com> wrote:
Well articulated, as always, David. Thank you.
I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though....
Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please.
Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored.
I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix...
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphou...
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jLh%2Bb2qZqRJ43G%2BWgEaxAy40%2FUZKOHTosSLbtElpDYg%3D&reserved=0
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net<mailto:showard@paulbunyan.net>> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ClK95Clgn9oAYq4nWq6A1YkDyLJcFObldGJjl7wFeus%3D&reserved=0>
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
-- Jay Hanke, President South Front Networks jayhanke@southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VKCfG8XpulGAXH9GaLcgjJ%2FXHf2MTmblsy3Xtkt8CK0%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VKCfG8XpulGAXH9GaLcgjJ%2FXHf2MTmblsy3Xtkt8CK0%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.
I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jLh%2Bb2qZqRJ43G%2BWgEaxAy40%2FUZKOHTosSLbtElpDYg%3D&reserved=0
<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VKCfG8XpulGAXH9GaLcgjJ%2FXHf2MTmblsy3Xtkt8CK0%3D&reserved=0> -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VKCfG8XpulGAXH9GaLcgjJ%2FXHf2MTmblsy3Xtkt8CK0%3D&reserved=0> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C319e3ed73365457f979908dabb49f1c9%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638028223533480725%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VKCfG8XpulGAXH9GaLcgjJ%2FXHf2MTmblsy3Xtkt8CK0%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net<mailto:showard@paulbunyan.net>> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6YrWAdVf37aKK%2BNTxjGYIX6%2BfBCCZ63%2BTFDBDg2P%2BTg%3D&reserved=0>
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
-- Jay Hanke, President South Front Networks jayhanke@southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.
Honestly I think the amount of work is minimal. I say that as the person that will probably be doing most of the work. There is even more that we should be doing to keep the IX clean and stable (like adding a quarantine VLAN to the turn-up process) which will add even more labor to the turn-up process. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Thursday, November 03, 2022 8:27 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jLh%2Bb2qZqRJ43G%2BWgEaxAy40%2FUZKOHTosSLbtElpDYg%3D&reserved=0
<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C4c21d62743bf432dabfc08dabd26a236%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030270896421600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tO8Yfr%2BxvcQXZhEA0QkQBsziBRFHKZ9fc8lOj4zaUjQ%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Could someone confirm the larger scope of this? Our remote switch indeed has static mac ACLs. Unsure how others work. If the goal here is to keep a static mac list on all access ports AND remote switch ports (so my list must be replicated to the core switch)? If so, lets be sure to consider operational issues. I'm unsure how mac ACLs work on arista, but is there a risk to block all mac addresses while the changes is being made if done incorrectly? (think the classic IOS 'switchport trunk allowed vlan 123..')? If this is being done by humans, hoping the procedure is documented and repeatable (a copy paste to do it). Or better yet, but our machine overlords (via NAPALM?) Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
I do not know that there is an official scope. I think things are just in the discussion stage. I do believe we all know what needs to be done, it is just about being motivated to get it all done (in this case a switch issue pushed it back to the front). There are plenty of IXes that have laid out and implemented best practices that have been proven in reality to work for them. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Chris Wopat Sent: Thursday, November 03, 2022 9:21 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Could someone confirm the larger scope of this? Our remote switch indeed has static mac ACLs. Unsure how others work. If the goal here is to keep a static mac list on all access ports AND remote switch ports (so my list must be replicated to the core switch)? If so, lets be sure to consider operational issues. I'm unsure how mac ACLs work on arista, but is there a risk to block all mac addresses while the changes is being made if done incorrectly? (think the classic IOS 'switchport trunk allowed vlan 123..')? If this is being done by humans, hoping the procedure is documented and repeatable (a copy paste to do it). Or better yet, but our machine overlords (via NAPALM?) Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
Fair enough. You have always been fast with everything from what I've seen. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, November 3, 2022 8:53 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Honestly I think the amount of work is minimal. I say that as the person that will probably be doing most of the work. There is even more that we should be doing to keep the IX clean and stable (like adding a quarantine VLAN to the turn-up process) which will add even more labor to the turn-up process. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Thursday, November 03, 2022 8:27 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net<mailto:showard@paulbunyan.net>> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413473862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0%2FUrS%2BTVfsf9KMSAdoTp5a9DK0rSF4jAexOU3eF9Hg4%3D&reserved=0>
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
-- Jay Hanke, President South Front Networks jayhanke@southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413473862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DvZgyo%2FccOp5UogTL5H%2BZo0adLTbeTila9MrPydaB6Y%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413473862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DvZgyo%2FccOp5UogTL5H%2BZo0adLTbeTila9MrPydaB6Y%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413473862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DvZgyo%2FccOp5UogTL5H%2BZo0adLTbeTila9MrPydaB6Y%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413473862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DvZgyo%2FccOp5UogTL5H%2BZo0adLTbeTila9MrPydaB6Y%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413630546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7B4QeedCZPcyDp%2FYIl3%2BQsbHm9813ny9E0SGxGhXJyY%3D&reserved=0> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7Ctom.krenn%40HENNEPIN.US%7C54c5c75ec0594094e72708dabda2da2a%7C8aefdf9f878046bf8fb74c924653a8be%7C0%7C0%7C638030804413630546%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7B4QeedCZPcyDp%2FYIl3%2BQsbHm9813ny9E0SGxGhXJyY%3D&reserved=0> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.
That was a direct quote from Arista Support. Based on the context that it was used in I would say that means that the forwarding table stopped functioning and the default fallback is to flood everything. They recommended a software update since there were some forwarding table issues fixed in more recent versions, however they could not 100% guarantee the software update would provide complete protection. Arista Support pointed out how destabilizing a loop can be @ 100Gig even if it is only for the 2 seconds that we experienced before port security kicked in (especially with how much traffic an IX is carrying to flow through the loop). When talking with Chris @ the SIX he mentioned that they went to the static ACL method because the port security runs on the control plane, so it will always be slow. A MAC ACL runs on the ASIC so it is about as instantaneous as you are going to get. The port that caused the issue last week had a 1 MAC address limit on it through port security, and not through a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Wednesday, November 02, 2022 4:59 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only What does "crashed the forwarding table" actually mean? Does that mean that the mac address table filled and the switch then flooded to all ports? Was there a mac address limit or a static MAC ACL on the port that caused the problem? On 10/31/22 09:12, Jeremy Lumby wrote: The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss<MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke<jayhanke@SOUTHFRONT.IO> <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET<MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard<showard@paulbunyan.net> <showard@paulbunyan.net> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: >https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix.net%2Fams%2Fdocumentation%2Fconfig-guide&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NdK%2BQxJvhCPwQjSWZsa8z2CjMsOw%2BY0mFS8Z21li9N8%3D&reserved=0 > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: >https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1&data=05%7C01%7CBRADY.KITTEL%40HCMED.ORG%7Cdfc752f4190b41df5c1f08dabb48f46c%7Cada0782c5f344003b5d63187f30aecdd%7C0%7C0%7C638028219286453089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jLh%2Bb2qZqRJ43G%2BWgEaxAy40%2FUZKOHTosSLbtElpDYg%3D&reserved=0 -- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
No, the recent event was on the main switch. Historically, this has happened on most (if not all) of the MICE switches. On Mon, Oct 31, 2022 at 9:07 AM Kittel, Brady <Brady.Kittel@hcmed.org> wrote:
Was the source of the issue a remote switch?
"
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here."
-Brady ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke@SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only
CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe.
The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm.
The churn rate on members changing mac addresses is fairly low.
I agree that we should automate the process so it can be done via self-service by the members.
On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-ix...
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.iphou...
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
Confidentiality Notice:
Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
Regarding the method of 'updating' the mac address list, there should probably be the ability to address maintenance, ie temporarily add a 2nd mac address to permit list. See SIX's faq. Unshore how handled internally, but one can email with a pending change (ie router replacement) to have 2 permitted for a period so maintenance can go smoothly. https://www.seattleix.net/faq -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
I have mixed feelings about giving IXP manager access to the switch. The main reason is that if a member is proactive they can request the second MAC ahead of time, and it can be temporarily added for the transition. As it is most requests are processed in a couple hours. IXP manager is a program that none of the volunteers have a great grasp on, and lack of a timely security update on a portal that is made available to the entire internet in an invitation to get hacked. If it has direct access to the switch things can go wrong quickly. While I am concerned as it is about IXP manager having access to the route servers, they are an extra and not 100% necessary for the IX to function. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Monday, October 31, 2022 9:05 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
The switch access could be air-gapped. ie there could be a script that generates the new mac-address list and then it could be manually applied. On Mon, Oct 31, 2022 at 9:22 AM Jeremy Lumby <jlumby@mnvoip.com> wrote:
I have mixed feelings about giving IXP manager access to the switch. The main reason is that if a member is proactive they can request the second MAC ahead of time, and it can be temporarily added for the transition. As it is most requests are processed in a couple hours. IXP manager is a program that none of the volunteers have a great grasp on, and lack of a timely security update on a portal that is made available to the entire internet in an invitation to get hacked. If it has direct access to the switch things can go wrong quickly. While I am concerned as it is about IXP manager having access to the route servers, they are an extra and not 100% necessary for the IX to function.
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Monday, October 31, 2022 9:05 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only
The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm.
The churn rate on members changing mac addresses is fairly low.
I agree that we should automate the process so it can be done via self-service by the members.
On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
I like that idea -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Monday, October 31, 2022 9:27 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The switch access could be air-gapped. ie there could be a script that generates the new mac-address list and then it could be manually applied. On Mon, Oct 31, 2022 at 9:22 AM Jeremy Lumby <jlumby@mnvoip.com> wrote:
I have mixed feelings about giving IXP manager access to the switch. The main reason is that if a member is proactive they can request the second MAC ahead of time, and it can be temporarily added for the transition. As it is most requests are processed in a couple hours. IXP manager is a program that none of the volunteers have a great grasp on, and lack of a timely security update on a portal that is made available to the entire internet in an invitation to get hacked. If it has direct access to the switch things can go wrong quickly. While I am concerned as it is about IXP manager having access to the route servers, they are an extra and not 100% necessary for the IX to function.
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Monday, October 31, 2022 9:05 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only
The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm.
The churn rate on members changing mac addresses is fairly low.
I agree that we should automate the process so it can be done via self-service by the members.
On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard@paulbunyan.net> wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
On 10/31/22 08:59, Steve Howard wrote:
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
Quick note to consider. Yes, there should be a mac limit. Certain vendor hardware may restrict the way this is done - as in, some vendor kit may allow one to set a mac limit of 1 (to anything) and some may only permit static mac, and some may permit both. WiscNet's remote switch is currently set to static mac, this is because of a vendor change that prevented the mac limit from working. Forcing one way or the other may be difficult to achieve, but the limit should exist in some form. -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
I think the best way to phrase it is “static MAC ACL” applied to all member ports. There could be reasons to do static MAC entries in the forwarding table (so traffic does not flood when a large link drops) but we are not talking about that here/yet. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Monday, October 31, 2022 8:59 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? On 12/13/19 18:00, Richard Laager wrote: The board has approved the change in MAC address limit from 5 to 1. NOTE: This is orthogonal to the dedicated remote switch question, whichis still pending. Non-dedicated remote switches can, for example,enforce this per-VLAN. We have already confirmed that most participants are using only 1 MACaddress, and some that were using more than 1 have addressed that afterbeing contacted. As discussed below, this change will be rolled out in away to keep everyone working, by grandfathering if necessary. On 10/28/19 3:25 PM, Richard Laager wrote: We have discussed this a bit in the past, and this came up at the lastUG. I'm looking for feedback from the group before formally proposingthis to the board for a vote. I first brought this up to the technical committee, CCing the board. Ihave heard no objections. Jeremy is "in favor of it 100%". Based on our quick conversation after the UG, I _think_ Anthony is infavor as well; we both made notes to follow up on this. ----- I am proposing that MICE change its MAC address limit from 5 MACs perport to 1 MAC per port. 1 MAC address is enough for normal scenarios andthis would further limit the potential for problems on the fabric. Thisrestriction is one that SIX and AMS-IX both have, with the latter beingknown for their excellent configuration guide for participants:https://www.ams-ix.net/ams/documentation/config-guide To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here. If someone is swapping their MICE-facing router, the resulting port flapon the MICE/remote switch will clear the limit anyway, if they aredirectly connected. If they are unable to flap the port (e.g. becausethey have someone else's layer 2 gear in the middle), they will have toeither work with that carrier or their switch operator (MICE/remote) toflap the port. This is a non-zero burden, but I think it's prettyminimal in practice. Additionally, if someone plans to make an equipmentswap, we would increase the limit in advance, temporarily, upon request. For reference, SIX goes further and locks you to a particular MACaddress in an ACL, so you have to contact them to change your MAC. I amnot proposing that. Still, I've been through that a couple of times, andeven that isn't really too big of a deal. So I don't think the change toa limit of 1 MAC at MICE will be particularly onerous. There are a couple of participants who have two IP address sets (oneIPv4 and one IPv6) on the same port. For those ports, the limitobviously must be at least 2 MACs, but I propose 3 MACs as the limit. Ifthey're using two IPs, they probably are doing so to get some redundancyout of it. For this particular use case, needing to flap the portdefeats that redundancy goal, as it breaks the other router. Having aMAC limit of 3 would allow them to swap a device without needing to flapthe port. We would grandfather these setups until they are looking for aport-type upgrade; at that point they would need to get to one IPaddress set per port (by splitting into two ports or reducing to one IPaddress set) or request an exception as outlined below. The exceptionprocess would allow us to better evaluate this use case at that time. There are some members who have multiple MACs showing up now, despiteusing only one device. For those, I am proposing we set the MAC limit tohowever many MACs are currently showing up on their port. That is, theywould be grandfathered at their current situation for now. At the timeof this change, we would encourage them to investigate and fix themultiple MACs issue at their leisure. In the future, if they are doing aequipment swap (especially like-for-like), we would again encourage themto address it, but would otherwise leave their higher limit in place. Ifthey do a port-type upgrade, we would not bring the grandfathering over.If they needed, they could ask for an official exception at that time. There may be cases where people cannot reasonably fix their routers touse only 1 MAC, other scenarios I haven't considered, or things that maycome up in the future. For that reason, I think we should allow for thepossibility of exceptions upon request to the board. In practice, Iwould expect the board would confer with the technical committee and/orthe technical committee would be bringing these to the board anyway. Thegoal is to strike a reasonable balance between protecting the fabricwithout preventing networks from connecting. On 10/31/22 08:27, Jay Hanke wrote: I propose we change the MICE required port security settings to behard filtered for a single static mac address per port. Over the years, we've had several incidents where floods have occurredprior to port being error-disabled during a looping event. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Yes, that was the intention. On Mon, Oct 31, 2022 at 9:26 AM Jeremy Lumby <jlumby@mnvoip.com> wrote:
I think the best way to phrase it is “static MAC ACL” applied to all member ports. There could be reasons to do static MAC entries in the forwarding table (so traffic does not flood when a large link drops) but we are not talking about that here/yet.
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Monday, October 31, 2022 8:59 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only
Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1.
The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed?
On 12/13/19 18:00, Richard Laager wrote:
The board has approved the change in MAC address limit from 5 to 1.
NOTE: This is orthogonal to the dedicated remote switch question, which
is still pending. Non-dedicated remote switches can, for example,
enforce this per-VLAN.
We have already confirmed that most participants are using only 1 MAC
address, and some that were using more than 1 have addressed that after
being contacted. As discussed below, this change will be rolled out in a
way to keep everyone working, by grandfathering if necessary.
On 10/28/19 3:25 PM, Richard Laager wrote:
We have discussed this a bit in the past, and this came up at the last
UG. I'm looking for feedback from the group before formally proposing
this to the board for a vote.
I first brought this up to the technical committee, CCing the board. I
have heard no objections. Jeremy is "in favor of it 100%".
Based on our quick conversation after the UG, I _think_ Anthony is in
favor as well; we both made notes to follow up on this.
-----
I am proposing that MICE change its MAC address limit from 5 MACs per
port to 1 MAC per port. 1 MAC address is enough for normal scenarios and
this would further limit the potential for problems on the fabric. This
restriction is one that SIX and AMS-IX both have, with the latter being
known for their excellent configuration guide for participants:
https://www.ams-ix.net/ams/documentation/config-guide
To be clear, this is still only for end ports, not ports facing remote
switches, of course. The remote switches are responsible for enforcing
the current MAC limit, and would be responsible for changing those
settings as described here.
If someone is swapping their MICE-facing router, the resulting port flap
on the MICE/remote switch will clear the limit anyway, if they are
directly connected. If they are unable to flap the port (e.g. because
they have someone else's layer 2 gear in the middle), they will have to
either work with that carrier or their switch operator (MICE/remote) to
flap the port. This is a non-zero burden, but I think it's pretty
minimal in practice. Additionally, if someone plans to make an equipment
swap, we would increase the limit in advance, temporarily, upon request.
For reference, SIX goes further and locks you to a particular MAC
address in an ACL, so you have to contact them to change your MAC. I am
not proposing that. Still, I've been through that a couple of times, and
even that isn't really too big of a deal. So I don't think the change to
a limit of 1 MAC at MICE will be particularly onerous.
There are a couple of participants who have two IP address sets (one
IPv4 and one IPv6) on the same port. For those ports, the limit
obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If
they're using two IPs, they probably are doing so to get some redundancy
out of it. For this particular use case, needing to flap the port
defeats that redundancy goal, as it breaks the other router. Having a
MAC limit of 3 would allow them to swap a device without needing to flap
the port. We would grandfather these setups until they are looking for a
port-type upgrade; at that point they would need to get to one IP
address set per port (by splitting into two ports or reducing to one IP
address set) or request an exception as outlined below. The exception
process would allow us to better evaluate this use case at that time.
There are some members who have multiple MACs showing up now, despite
using only one device. For those, I am proposing we set the MAC limit to
however many MACs are currently showing up on their port. That is, they
would be grandfathered at their current situation for now. At the time
of this change, we would encourage them to investigate and fix the
multiple MACs issue at their leisure. In the future, if they are doing a
equipment swap (especially like-for-like), we would again encourage them
to address it, but would otherwise leave their higher limit in place. If
they do a port-type upgrade, we would not bring the grandfathering over.
If they needed, they could ask for an official exception at that time.
There may be cases where people cannot reasonably fix their routers to
use only 1 MAC, other scenarios I haven't considered, or things that may
come up in the future. For that reason, I think we should allow for the
possibility of exceptions upon request to the board. In practice, I
would expect the board would confer with the technical committee and/or
the technical committee would be bringing these to the board anyway. The
goal is to strike a reasonable balance between protecting the fabric
without preventing networks from connecting.
On 10/31/22 08:27, Jay Hanke wrote:
I propose we change the MICE required port security settings to be
hard filtered for a single static mac address per port.
Over the years, we've had several incidents where floods have occurred
prior to port being error-disabled during a looping event.
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
________________________________
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Jay Hanke, President South Front Networks jayhanke@southfront.io Phone 612-204-0000
participants (10)
-
Andrew Hoyos
-
Chris Wopat
-
David Farmer
-
Gary Glissendorf
-
Jay Hanke
-
Jeremy Lumby
-
Kittel, Brady
-
Mike Johnston
-
Steve Howard
-
Tom Krenn