- mice-discuss - lists.micemn.net

Re: MICE Remote Switch Policy

by Richard Laager

On 3/24/22 18:00, Jeremy Lumby wrote: > As for a disincentive for CDN's to connect, I have only seen the opposite. Most CDN's will only accept a connection to the core. The only time I have seen them connect to a remote was for a secondary connection to gain switch diversity. I wasn't talking about CDNs connecting to remotes. The concern, or at least how I understood it, was: Imagine we put a MICE extension in city X. In the immediate term, that's great, as now networks in city X can get content from Minneapolis CDNs. But in the longer-term, it may create a disincentive for CDNs to go to city X. Counter-point: Whether CDNs come to city X is not our problem. -- Richard

March 25, 2022

Re: MICE Remote Switch Policy

by Jeremy Lumby

A few of my opinions on the topics that Richard brought up/relayed for others: For the most part I think the approval of remote switches should be discretionary. With that being said I think there should be a few stipulations. There should be minimum criteria to be met in order to even consider the proposal. My reasoning for the minimum criteria is to save time and effort since MICE is almost all volunteer based, and I am in favor of keeping it that way. I am in favor of the following minimums for a remote proposal: -Dedicated switching hardware -Minimum of 5 participants interested in connecting immediately -66% utilization upgrade threshold -Agreement that the switch operator will enforce, and keep up with MICE technical requirements -Agreement that the switch operator will cover all costs of operation including uplink, and MICE fees -Agreement that the switch operator notifies all participants that they are connecting to a remote switch, as well as is the primary source of support for connected members. If the agreed upon minimums are met then the proposal should go to the membership for discussion (for a fixed amount of time), and then the board for approval. I do feel that if board members were to vote against it, it should be for a reason that was discussed publically on the list first. On a somewhat related topic I have had a few different datacenters inquire about getting a MICE managed core switch installed. I think there should be minimum standards established for a core switch as well to save time and effort. In general I think they should be much greater than the list above. With respect to the location of a remote switch, I think distance is not an issue, and in some ways it is an advantage. When someone connects to a remote that is far from the core, everyone knows it. The location of the switch is documented on the participants page, and all of the members connecting through it are indicated as so. This gives network admin the ability to easily identify, and adjust their BGP metrics accordingly. I also like the idea that Jay proposed for communities that identify remote switch participants. If someone were to argue that distance was a negative, they would first need to propose blocking anyone connecting to the core across a long-haul circuit from outside of the metro. In cases of long-haul to the core, the only one that has a clue that it is going on is the participant themselves leaving all other members clueless. I have received several inquiries from members about high latency to certain peers who long-haul into the MICE core. I am often unsure if I can divulge their remote location to the person asking since often times the only way I know is based on the carrier listed on the cross connect tag going into the core (which is not public knowledge like the participants page is). As to the dedicated hardware requirement I would also like to state that in general I feel that if someone is serious about increasing connectivity to MICE, they are willing to spend the money for the hardware (not just trying to save on a cross connect for a friend), and also willing to spend the time to be the first line of defense when it comes to troubleshooting. Not to mention the much simpler config/troubleshooting that comes along with dedicated hardware, this all keeps the load off of the volunteers running MICE. I do not feel that any NEW minimum requirements should apply to existing switches. I think they should still be bound to their original proposals (within reason). I believe that all of those proposals would include enforcing current MICE rules on their switch (such as number of MAC addresses, and BPDU error disables) As for a disincentive for CDN's to connect, I have only seen the opposite. Most CDN's will only accept a connection to the core. The only time I have seen them connect to a remote was for a secondary connection to gain switch diversity. As for broadcast traffic I agree that it can get more dangerous across a long-haul link, however I think a larger issue is the lack of enforcement of good router config hygiene. To that point, a quarantine VLAN helps detection/enforcement before the problem gets out of hand. The complexities it leads to would be another reason to support requiring dedicated hardware for remote switches. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, March 24, 2022 3:59 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Remote Switch Policy I've had some discussions with the board as well as with Jay and Jeremy on these topics. The board consensus was to bring this (in general) to the membership for more input. As to the specifics, while I know others agree with at least parts of this, I'm only speaking for myself here. I'll let everyone articulate their own positions. (This disclaimer should not be read as me signaling the existance of disagreement either. I just don't want to put words in other people's mouths.) Our current policy on remote switches is here: https://micemn.net/technical.html#remotes It has the proposal presented to the membership for discussion, then the board makes a final decision. Is this decision ministerial or discretionary? That is, if the remote switch proposal checks all the boxes in our policy, is MICE "required" (supposed to) always grant it, or is the board supposed to apply some discretion? If the decision is ministerial, then why bother bringing this to the board (or for that matter, the members) all? Couldn't we save a bunch of time and hassle and simply have management (in some form, whether that's me, Jay, and/or Jeremy) approve it? If the decision is discretionary, are there particular criteria that the board should consider (above and beyond the listed criteria)? One criteria used in a discussion I had (and I can't recall which of us said it first) is "MICE's strategic interests". What would that phrase mean to you; what are some strategic interests of MICE? For a bit of an absurd example for the thought experiment, imagine that someone was proposing a MICE remote switch, but we knew their goal was to attract a bunch of members and then convert that into a competing exchange. Is that something we would have to agree to simply because they met all the objective criteria? When we were new and little, MICE certainly had an interest in making every decision in a way that would maximize additional peering. However, at this point, the calculus may be (I'd argue is) different. We are moving a lot of traffic and are important to our members / in our region. We have to be careful that our decisions do not destabilize the exchange--in multiple ways: technical, financial, or political. Either way, should we expand the list of objective criteria in the policy? Some examples: • We have previously discussed dedicated vs non-dedicated switches. As time goes along, I am more convinced than ever that MICE remote switches should be required to be dedicated. Non-dedicated switches present extra complications for configuration and troubleshooting. (Jeremy has some additional insight on this that he will share.) I think we should make it a requirement that the switch be dedicated. (Perhaps the board could still grant an exception in exceptional cases.) • Should we require that a remote switch have X number of participants committed? And if so, what is X? In my view, it hardly makes sense to have a remote switch one or two participants. They could just as well backhaul to MICE directly. The criteria for allowing new remote switches vs disconnecting existing remotes need not be the same. If we set a minimum of e.g. 5 participants, we don't necessarily need to disconnect existing remotes that don't meet that. And I think the consensus is that we would not, barring them creating some significant problem. How do we feel about far-away remote switches? (This is a live issue in the context of the proposed Kansas City remote.) Some concerns: • At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me some issues. For example, latency-sensitive gaming traffic was tromboning Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than Wiktel-Chicago-Wiktel. • Is it safe to have a broadcast domain that stretches across multiple states (or half a continent, in the SIX case)? • If we take this to its logical extreme... Imagine we had a MICE extension in every datacenter in the U.S. I think that is pretty obviously untenable for a bunch of reasons. Something close to that is actually within the realm of possibility, with some of these virtual extension things that people are doing. (Reid would be able to cite who.) Granted, nobody is proposing that today, but where should we draw the line? • Far-away extensions may reduce the incentive for CDNs to install locally. Some counterpoints: • Nobody is forcing networks to use the far-away remotes. • If people choose to use them, they take their routing into their own hands. They need to understand the tromboning risk and set their own routing policy. o Counter-counterpoint: Do they? Especially smaller / less experienced networks? Have we adequately warned them? o Counter-counterpoint: The existence of these far-away peers doesn't affect just them. It also affects the other networks with which they peer. Everyone on the exchange needs to be aware of the existence of far-away participants and handle their routing policy accordingly. If there are enough far-away peers, this might tip networks into an opt-in route server policy, or even to only do bilaterals. This will disadvantage small participants. • Networks can backhaul into far-away exchanges directly. o Counter-counterpoint: But a remote switch makes this cheaper / more feasible / more common, which is literally the point of creating such a remote switch. • For a local eyeball network in Des Moines, neither MICE nor Kansas City are far-away from me. Even MICE via Kansas City is not likely to be problematic. This might be the only economically feasible way they could peer with Minneapolis content. -- Richard ________________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 24, 2022

Re: MICE Remote Switch Policy

by Reid Fishler

In this case, we really do want the memberships thoughts. Do we WANT to allow any and all remotes, or do we want to start to trim who we accept? Reid On Thu, Mar 24, 2022 at 5:25 PM Dean Bahls <dean.bahls(a)fiberminnesota.com> wrote: > I don’t think we can create enough check boxes to protect against all > combinations and permutations that nefarious folks can dream up….that’s why > we have a board. I think the board is elected to make sure the exchange is > being run properly….and that’s how I vote. If the board wants feedback on > any of the topics below, I’m sure all of us have opinions, and that would > probably provide guidance needed for the board to make an official ruling. > All within the realm of reasonableness. > > > > Keep up the good work! > > > > Dean > > > > > > > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Richard > Laager > *Sent:* Thursday, March 24, 2022 3:59 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] MICE Remote Switch Policy > > > > I've had some discussions with the board as well as with Jay and Jeremy on > these topics. The board consensus was to bring this (in general) to the > membership for more input. > > As to the specifics, while I know others agree with at least parts of > this, I'm only speaking for myself here. I'll let everyone articulate their > own positions. (This disclaimer should not be read as me signaling the > existance of disagreement either. I just don't want to put words in other > people's mouths.) > > > > Our current policy on remote switches is here: > https://micemn.net/technical.html#remotes It has the proposal presented > to the membership for discussion, then the board makes a final decision. > > Is this decision ministerial or discretionary? That is, if the remote > switch proposal checks all the boxes in our policy, is MICE "required" > (supposed to) always grant it, or is the board supposed to apply some > discretion? > > If the decision is ministerial, then why bother bringing this to the board > (or for that matter, the members) all? Couldn't we save a bunch of time and > hassle and simply have management (in some form, whether that's me, Jay, > and/or Jeremy) approve it? > > If the decision is discretionary, are there particular criteria that the > board should consider (above and beyond the listed criteria)? > > One criteria used in a discussion I had (and I can't recall which of us > said it first) is "MICE's strategic interests". What would that phrase mean > to you; what are some strategic interests of MICE? > > For a bit of an absurd example for the thought experiment, imagine that > someone was proposing a MICE remote switch, but we knew their goal was to > attract a bunch of members and then convert that into a competing exchange. > Is that something we would have to agree to simply because they met all the > objective criteria? > > When we were new and little, MICE certainly had an interest in making > every decision in a way that would maximize additional peering. However, at > this point, the calculus may be (I'd argue is) different. We are moving a > lot of traffic and are important to our members / in our region. We have to > be careful that our decisions do not destabilize the exchange--in multiple > ways: technical, financial, or political. > > > > Either way, should we expand the list of objective criteria in the policy? > Some examples: > > - We have previously discussed dedicated vs non-dedicated switches. As > time goes along, I am more convinced than ever that MICE remote switches > should be required to be dedicated. Non-dedicated switches present extra > complications for configuration and troubleshooting. (Jeremy has some > additional insight on this that he will share.) I think we should make it a > requirement that the switch be dedicated. (Perhaps the board could still > grant an exception in exceptional cases.) > - Should we require that a remote switch have X number of participants > committed? And if so, what is X? In my view, it hardly makes sense to have > a remote switch one or two participants. They could just as well backhaul > to MICE directly. > > The criteria for allowing new remote switches vs disconnecting existing > remotes need not be the same. If we set a minimum of e.g. 5 participants, > we don't necessarily need to disconnect existing remotes that don't meet > that. And I think the consensus is that we would not, barring them creating > some significant problem. > > > > How do we feel about far-away remote switches? (This is a live issue in > the context of the proposed Kansas City remote.) > > Some concerns: > > - At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. > For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me > some issues. For example, latency-sensitive gaming traffic was tromboning > Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than > Wiktel-Chicago-Wiktel. > - Is it safe to have a broadcast domain that stretches across multiple > states (or half a continent, in the SIX case)? > - If we take this to its logical extreme... Imagine we had a MICE > extension in every datacenter in the U.S. I think that is pretty obviously > untenable for a bunch of reasons. Something close to that is actually > within the realm of possibility, with some of these virtual extension > things that people are doing. (Reid would be able to cite who.) Granted, > nobody is proposing that today, but where should we draw the line? > - Far-away extensions may reduce the incentive for CDNs to install > locally. > > Some counterpoints: > > - Nobody is forcing networks to use the far-away remotes. > - If people choose to use them, they take their routing into their own > hands. They need to understand the tromboning risk and set their own > routing policy. > - Counter-counterpoint: Do they? Especially smaller / less > experienced networks? Have we adequately warned them? > - Counter-counterpoint: The existence of these far-away peers > doesn't affect just them. It also affects the other networks with which > they peer. Everyone on the exchange needs to be aware of the existence of > far-away participants and handle their routing policy accordingly. If there > are enough far-away peers, this might tip networks into an opt-in route > server policy, or even to only do bilaterals. This will disadvantage small > participants. > - Networks can backhaul into far-away exchanges directly. > - Counter-counterpoint: But a remote switch makes this cheaper / > more feasible / more common, which is literally the point of creating such > a remote switch. > - For a local eyeball network in Des Moines, neither MICE nor Kansas > City are far-away from me. Even MICE via Kansas City is not likely to be > problematic. This might be the only economically feasible way they could > peer with Minneapolis content. > > -- > > Richard > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178

March 24, 2022

Re: MICE Remote Switch Policy

by Jay Hanke

> Is this decision ministerial or discretionary? That is, if the remote switch proposal checks all the boxes in our policy, is MICE "required" (supposed to) always grant it, or is the board supposed to apply some discretion? I've understood the intention of the policy to be someplace in the middle with only the cases where some box wasn't checked going requiring deep board review and the "all boxes checked" being converted into a rubber stamp. > If the decision is ministerial, then why bother bringing this to the board (or for that matter, the members) all? Couldn't we save a bunch of time and hassle and simply have management (in some form, whether that's me, Jay, and/or Jeremy) approve it? yes, I think in almost all cases this is the case. > One criteria used in a discussion I had (and I can't recall which of us said it first) is "MICE's strategic interests". What would that phrase mean to you; what are some strategic interests of MICE? >From the before times, the founding MICE fathers stated when in doubt take the option that improves connectivity. Granted, we weren't one of the largest IXes in North America back then with our Cisco 4500. Overall the remote model has been very effective at growing MICE. There is a down side that it does likely reduce port revenue on the main switch. The primary mission for the IX has been resolving the "Chicago Problem" which has been somewhat successfully moved to MSP. :) > For a bit of an absurd example for the thought experiment, imagine that someone was proposing a MICE remote switch, but we knew their goal was to attract a bunch of members and then convert that into a competing exchange. Is that something we would have to agree to simply because they met all the objective criteria? We likely need to add a utility component to the remote switch calc. A new remote switch in the 511 building is not likely to add much utility to the members. > When we were new and little, MICE certainly had an interest in making every decision in a way that would maximize additional peering. However, at this point, the calculus may be (I'd argue is) different. We are moving a lot of traffic and are important to our members / in our region. We have to be careful that our decisions do not destabilize the exchange--in multiple ways: technical, financial, or political. It's a fair point. > Either way, should we expand the list of objective criteria in the policy? Some examples: Yes > We have previously discussed dedicated vs non-dedicated switches. As time goes along, I am more convinced than ever that MICE remote switches should be required to be dedicated. Non-dedicated switches present extra complications for configuration and troubleshooting. (Jeremy has some additional insight on this that he will share.) I think we should make it a requirement that the switch be dedicated. (Perhaps the board could still grant an exception in exceptional cases.) We should require all new switches dedicated to MICE. There's been several problems involving a lot of troubleshooting time and fabric risk. > Should we require that a remote switch have X number of participants committed? And if so, what is X? In my view, it hardly makes sense to have a remote switch one or two participants. They could just as well backhaul to MICE directly. I'd propose that MICE look into adding a "VLAN reseller port type" where each member comes in on their own vlan into a MICE owned box (Juniper MX/ASR9k et al). This would cover a number of the smaller situations as well as the big PacketFabric/MegaPort/Console types. > The criteria for allowing new remote switches vs disconnecting existing remotes need not be the same. If we set a minimum of e.g. 5 participants, we don't necessarily need to disconnect existing remotes that don't meet that. And I think the consensus is that we would not, barring them creating some significant problem. Agreed > How do we feel about far-away remote switches? (This is a live issue in the context of the proposed Kansas City remote.) This doesn't bother me much, however it would be nice to tag the routes in the route server to let the other party know where the traffic is coming from. If a network doesn't want to bounce their traffic through a long path they could pref or discard the route based on the community. There are situations where the Main Switch isn't the best path as well as there is a closer path via the remotes. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

March 24, 2022

Re: MICE Remote Switch Policy

by Dean Bahls

I don’t think we can create enough check boxes to protect against all combinations and permutations that nefarious folks can dream up….that’s why we have a board. I think the board is elected to make sure the exchange is being run properly….and that’s how I vote. If the board wants feedback on any of the topics below, I’m sure all of us have opinions, and that would probably provide guidance needed for the board to make an official ruling. All within the realm of reasonableness. Keep up the good work! Dean From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Richard Laager Sent: Thursday, March 24, 2022 3:59 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Remote Switch Policy I've had some discussions with the board as well as with Jay and Jeremy on these topics. The board consensus was to bring this (in general) to the membership for more input. As to the specifics, while I know others agree with at least parts of this, I'm only speaking for myself here. I'll let everyone articulate their own positions. (This disclaimer should not be read as me signaling the existance of disagreement either. I just don't want to put words in other people's mouths.) Our current policy on remote switches is here: https://micemn.net/technical.html#remotes It has the proposal presented to the membership for discussion, then the board makes a final decision. Is this decision ministerial or discretionary? That is, if the remote switch proposal checks all the boxes in our policy, is MICE "required" (supposed to) always grant it, or is the board supposed to apply some discretion? If the decision is ministerial, then why bother bringing this to the board (or for that matter, the members) all? Couldn't we save a bunch of time and hassle and simply have management (in some form, whether that's me, Jay, and/or Jeremy) approve it? If the decision is discretionary, are there particular criteria that the board should consider (above and beyond the listed criteria)? One criteria used in a discussion I had (and I can't recall which of us said it first) is "MICE's strategic interests". What would that phrase mean to you; what are some strategic interests of MICE? For a bit of an absurd example for the thought experiment, imagine that someone was proposing a MICE remote switch, but we knew their goal was to attract a bunch of members and then convert that into a competing exchange. Is that something we would have to agree to simply because they met all the objective criteria? When we were new and little, MICE certainly had an interest in making every decision in a way that would maximize additional peering. However, at this point, the calculus may be (I'd argue is) different. We are moving a lot of traffic and are important to our members / in our region. We have to be careful that our decisions do not destabilize the exchange--in multiple ways: technical, financial, or political. Either way, should we expand the list of objective criteria in the policy? Some examples: * We have previously discussed dedicated vs non-dedicated switches. As time goes along, I am more convinced than ever that MICE remote switches should be required to be dedicated. Non-dedicated switches present extra complications for configuration and troubleshooting. (Jeremy has some additional insight on this that he will share.) I think we should make it a requirement that the switch be dedicated. (Perhaps the board could still grant an exception in exceptional cases.) * Should we require that a remote switch have X number of participants committed? And if so, what is X? In my view, it hardly makes sense to have a remote switch one or two participants. They could just as well backhaul to MICE directly. The criteria for allowing new remote switches vs disconnecting existing remotes need not be the same. If we set a minimum of e.g. 5 participants, we don't necessarily need to disconnect existing remotes that don't meet that. And I think the consensus is that we would not, barring them creating some significant problem. How do we feel about far-away remote switches? (This is a live issue in the context of the proposed Kansas City remote.) Some concerns: * At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me some issues. For example, latency-sensitive gaming traffic was tromboning Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than Wiktel-Chicago-Wiktel. * Is it safe to have a broadcast domain that stretches across multiple states (or half a continent, in the SIX case)? * If we take this to its logical extreme... Imagine we had a MICE extension in every datacenter in the U.S. I think that is pretty obviously untenable for a bunch of reasons. Something close to that is actually within the realm of possibility, with some of these virtual extension things that people are doing. (Reid would be able to cite who.) Granted, nobody is proposing that today, but where should we draw the line? * Far-away extensions may reduce the incentive for CDNs to install locally. Some counterpoints: * Nobody is forcing networks to use the far-away remotes. * If people choose to use them, they take their routing into their own hands. They need to understand the tromboning risk and set their own routing policy. * Counter-counterpoint: Do they? Especially smaller / less experienced networks? Have we adequately warned them? * Counter-counterpoint: The existence of these far-away peers doesn't affect just them. It also affects the other networks with which they peer. Everyone on the exchange needs to be aware of the existence of far-away participants and handle their routing policy accordingly. If there are enough far-away peers, this might tip networks into an opt-in route server policy, or even to only do bilaterals. This will disadvantage small participants. * Networks can backhaul into far-away exchanges directly. * Counter-counterpoint: But a remote switch makes this cheaper / more feasible / more common, which is literally the point of creating such a remote switch. * For a local eyeball network in Des Moines, neither MICE nor Kansas City are far-away from me. Even MICE via Kansas City is not likely to be problematic. This might be the only economically feasible way they could peer with Minneapolis content. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 24, 2022

Re: MICE Remote Switch Policy

by Reid Fishler

Thanks Richard for this great summary of what the board has been pondering for a bit... Reid On Thu, Mar 24, 2022 at 4:58 PM Richard Laager <rlaager(a)wiktel.com> wrote: > I've had some discussions with the board as well as with Jay and Jeremy on > these topics. The board consensus was to bring this (in general) to the > membership for more input. > > As to the specifics, while I know others agree with at least parts of > this, I'm only speaking for myself here. I'll let everyone articulate their > own positions. (This disclaimer should not be read as me signaling the > existance of disagreement either. I just don't want to put words in other > people's mouths.) > > > Our current policy on remote switches is here: > https://micemn.net/technical.html#remotes It has the proposal presented > to the membership for discussion, then the board makes a final decision. > > Is this decision ministerial or discretionary? That is, if the remote > switch proposal checks all the boxes in our policy, is MICE "required" > (supposed to) always grant it, or is the board supposed to apply some > discretion? > > If the decision is ministerial, then why bother bringing this to the board > (or for that matter, the members) all? Couldn't we save a bunch of time and > hassle and simply have management (in some form, whether that's me, Jay, > and/or Jeremy) approve it? > > If the decision is discretionary, are there particular criteria that the > board should consider (above and beyond the listed criteria)? > > One criteria used in a discussion I had (and I can't recall which of us > said it first) is "MICE's strategic interests". What would that phrase mean > to you; what are some strategic interests of MICE? > > For a bit of an absurd example for the thought experiment, imagine that > someone was proposing a MICE remote switch, but we knew their goal was to > attract a bunch of members and then convert that into a competing exchange. > Is that something we would have to agree to simply because they met all the > objective criteria? > > When we were new and little, MICE certainly had an interest in making > every decision in a way that would maximize additional peering. However, at > this point, the calculus may be (I'd argue is) different. We are moving a > lot of traffic and are important to our members / in our region. We have to > be careful that our decisions do not destabilize the exchange--in multiple > ways: technical, financial, or political. > > > Either way, should we expand the list of objective criteria in the policy? > Some examples: > > - We have previously discussed dedicated vs non-dedicated switches. As > time goes along, I am more convinced than ever that MICE remote switches > should be required to be dedicated. Non-dedicated switches present extra > complications for configuration and troubleshooting. (Jeremy has some > additional insight on this that he will share.) I think we should make it a > requirement that the switch be dedicated. (Perhaps the board could still > grant an exception in exceptional cases.) > - Should we require that a remote switch have X number of participants > committed? And if so, what is X? In my view, it hardly makes sense to have > a remote switch one or two participants. They could just as well backhaul > to MICE directly. > > The criteria for allowing new remote switches vs disconnecting existing > remotes need not be the same. If we set a minimum of e.g. 5 participants, > we don't necessarily need to disconnect existing remotes that don't meet > that. And I think the consensus is that we would not, barring them creating > some significant problem. > > > How do we feel about far-away remote switches? (This is a live issue in > the context of the proposed Kansas City remote.) > > Some concerns: > > - At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. > For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me > some issues. For example, latency-sensitive gaming traffic was tromboning > Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than > Wiktel-Chicago-Wiktel. > - Is it safe to have a broadcast domain that stretches across multiple > states (or half a continent, in the SIX case)? > - If we take this to its logical extreme... Imagine we had a MICE > extension in every datacenter in the U.S. I think that is pretty obviously > untenable for a bunch of reasons. Something close to that is actually > within the realm of possibility, with some of these virtual extension > things that people are doing. (Reid would be able to cite who.) Granted, > nobody is proposing that today, but where should we draw the line? > - Far-away extensions may reduce the incentive for CDNs to install > locally. > > Some counterpoints: > > - Nobody is forcing networks to use the far-away remotes. > - If people choose to use them, they take their routing into their own > hands. They need to understand the tromboning risk and set their own > routing policy. > - Counter-counterpoint: Do they? Especially smaller / less > experienced networks? Have we adequately warned them? > - Counter-counterpoint: The existence of these far-away peers > doesn't affect just them. It also affects the other networks with which > they peer. Everyone on the exchange needs to be aware of the existence of > far-away participants and handle their routing policy accordingly. If there > are enough far-away peers, this might tip networks into an opt-in route > server policy, or even to only do bilaterals. This will disadvantage small > participants. > - Networks can backhaul into far-away exchanges directly. > - Counter-counterpoint: But a remote switch makes this cheaper / > more feasible / more common, which is literally the point of creating such > a remote switch. > - For a local eyeball network in Des Moines, neither MICE nor > Kansas City are far-away from me. Even MICE via Kansas City is not likely > to be problematic. This might be the only economically feasible way they > could peer with Minneapolis content. > > -- > Richard > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178

March 24, 2022

MICE Remote Switch Policy

by Richard Laager

I've had some discussions with the board as well as with Jay and Jeremy on these topics. The board consensus was to bring this (in general) to the membership for more input. As to the specifics, while I know others agree with at least parts of this, I'm only speaking for myself here. I'll let everyone articulate their own positions. (This disclaimer should not be read as me signaling the existance of disagreement either. I just don't want to put words in other people's mouths.) Our current policy on remote switches is here: https://micemn.net/technical.html#remotes It has the proposal presented to the membership for discussion, then the board makes a final decision. Is this decision ministerial or discretionary? That is, if the remote switch proposal checks all the boxes in our policy, is MICE "required" (supposed to) always grant it, or is the board supposed to apply some discretion? If the decision is ministerial, then why bother bringing this to the board (or for that matter, the members) all? Couldn't we save a bunch of time and hassle and simply have management (in some form, whether that's me, Jay, and/or Jeremy) approve it? If the decision is discretionary, are there particular criteria that the board should consider (above and beyond the listed criteria)? One criteria used in a discussion I had (and I can't recall which of us said it first) is "MICE's strategic interests". What would that phrase mean to you; what are some strategic interests of MICE? For a bit of an absurd example for the thought experiment, imagine that someone was proposing a MICE remote switch, but we knew their goal was to attract a bunch of members and then convert that into a competing exchange. Is that something we would have to agree to simply because they met all the objective criteria? When we were new and little, MICE certainly had an interest in making every decision in a way that would maximize additional peering. However, at this point, the calculus may be (I'd argue is) different. We are moving a lot of traffic and are important to our members / in our region. We have to be careful that our decisions do not destabilize the exchange--in multiple ways: technical, financial, or political. Either way, should we expand the list of objective criteria in the policy? Some examples: * We have previously discussed dedicated vs non-dedicated switches. As time goes along, I am more convinced than ever that MICE remote switches should be required to be dedicated. Non-dedicated switches present extra complications for configuration and troubleshooting. (Jeremy has some additional insight on this that he will share.) I think we should make it a requirement that the switch be dedicated. (Perhaps the board could still grant an exception in exceptional cases.) * Should we require that a remote switch have X number of participants committed? And if so, what is X? In my view, it hardly makes sense to have a remote switch one or two participants. They could just as well backhaul to MICE directly. The criteria for allowing new remote switches vs disconnecting existing remotes need not be the same. If we set a minimum of e.g. 5 participants, we don't necessarily need to disconnect existing remotes that don't meet that. And I think the consensus is that we would not, barring them creating some significant problem. How do we feel about far-away remote switches? (This is a live issue in the context of the proposed Kansas City remote.) Some concerns: * At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me some issues. For example, latency-sensitive gaming traffic was tromboning Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than Wiktel-Chicago-Wiktel. * Is it safe to have a broadcast domain that stretches across multiple states (or half a continent, in the SIX case)? * If we take this to its logical extreme... Imagine we had a MICE extension in every datacenter in the U.S. I think that is pretty obviously untenable for a bunch of reasons. Something close to that is actually within the realm of possibility, with some of these virtual extension things that people are doing. (Reid would be able to cite who.) Granted, nobody is proposing that today, but where should we draw the line? * Far-away extensions may reduce the incentive for CDNs to install locally. Some counterpoints: * Nobody is forcing networks to use the far-away remotes. * If people choose to use them, they take their routing into their own hands. They need to understand the tromboning risk and set their own routing policy. o Counter-counterpoint: Do they? Especially smaller / less experienced networks? Have we adequately warned them? o Counter-counterpoint: The existence of these far-away peers doesn't affect just them. It also affects the other networks with which they peer. Everyone on the exchange needs to be aware of the existence of far-away participants and handle their routing policy accordingly. If there are enough far-away peers, this might tip networks into an opt-in route server policy, or even to only do bilaterals. This will disadvantage small participants. * Networks can backhaul into far-away exchanges directly. o Counter-counterpoint: But a remote switch makes this cheaper / more feasible / more common, which is literally the point of creating such a remote switch. * For a local eyeball network in Des Moines, neither MICE nor Kansas City are far-away from me. Even MICE via Kansas City is not likely to be problematic. This might be the only economically feasible way they could peer with Minneapolis content. -- Richard

March 24, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

On 3/17/22 21:09, Richard Laager wrote: > I will provide a more detailed explanation of my mistakes later. The existing servers are _only_ route servers. The new servers will be virtualization hosts running multiple VMs. The route servers will be running on there, but so will other things: IXP Manager, something to answer ARP for the blackholing we want to setup, Cacti will be brought in house, etc. Therefore, the host has bridging configured. We are tentatively planning to setup a quarantine VLAN as some other exchanges have done. This would also have quarantine route servers on it, which would behave normally except they would not advertise any routes out. This quarantine VLAN obviously needs to exist on the exchange switches because participant ports would be put into it. Because there will be multiple VLANs, the plan was to use VLAN tagging between the Arista and the new physical servers. The MICE VLAN is setup as VLAN 1. Before you say it... I don't think anyone likes that. Jeremy and I are of the opinion that this should change at some point in the future, but it doesn't seem worth doing until we are taking down the fabric anyway (e.g. a reboot of the Arista). In light of all that, our plan was (and in hindsight, I believe this to be mistake #1) to configure the main MICE VLAN as untagged, with the idea of adding the quarantine VLAN as tagged in the future. Cameron and I felt it was important to confirm the network was working to the MICE fabric before leaving Minneapolis, since we both live so far away. In hindsight, this was a good idea; had I made the same configuration mistake while working remote, it would have been slower to fix (at a minimum). The new systems are running Ubuntu 22.04 LTS which is increasingly frozen by the day [1] and will have a final release on April 21. So by the time we get everything configured, it will likely be released. This avoids us installing 20.04 LTS now and then wanting an upgrade immediately. Ubuntu (whether 22.04 LTS, or even 20.04 LTS) uses netplan to configure network interfaces. netplan uses an exclusively [2] declarative configuration model (unlike ifupdown, which is mostly declarative but some things can only be done with imperative commands). I'm actually a fan of netplan; it has made network configuration quite a bit nicer than ifupdown for me. I was not sure how to configure an untagged VLAN in netplan. I reviewed the documentation and was still unsure. I saw that the vlan "id" parameter was documented [3] as accepting a value of 0-4094. I figured I'd try 0 to see if that meant untagged. That did not work. I did a packet capture and found that 0 meant an explicit tag of 0. Having no more ideas, I gave up and removed the VLAN configuration, going back to just a straight interface. I figured I'd look at it more later. One of the big advantages of netplan is that it is (supposed to be) idempotent. That is, you configure it the way you want, run `netplan apply` and it makes it so. You don't (usually) need to explicitly manage the transition from the old state to the new state. Unfortunately, that is not true for bridging and/or VLANs, at least in some cases. In hindsight, for bridging, that largely makes sense. After all, you wouldn't want a `netplan apply` to remove the VMs host-side interfaces from the bridge group(s). But for VLANs, it seems like this is probably just something nobody implemented, as opposed to being desirable in theory. This was mistake #2 and the immediate cause of the loop. I ended up with a bridge interface that consisted of the physical interface (facing the MICE switch) and a VLAN interface (with an ID of 0) on that same physical interface. So as traffic came in, it was bridged back out the same physical interface, this time with an explicit VLAN 0 tag. Based on the fact that the explicit 0 tag didn't pass traffic normally in the first place, I don't think I was necessarily looping traffic at layer 3. But broadcast traffic would have been looped back to the switch, with the source MAC staying the same. This would obviously mess up the switch's MAC table. I believe that was the immediate cause of the issue. I noticed that things were not behaving quite right (some packet loss). I started a packet capture. When that was very quickly 5 GB rather than some tiny amount, I knew I had made a serious error that likely created a loop. I reflexively confirmed this with a "brctl show bridge", issued a reboot (because why not), and immediate jumped up and unplugged the cables from the back of the servers. When the systems came back up, I confirmed the network configuration was now correct and reconnected the cables. Because the server was expected to have multiple VMs on it, we could not use `port-security maximum 1`. As a result, port-security was not configured. Jeremy and I never had an explicit conversation about this, which was certainly another mistake. Had port-security been enabled, this loo would have been arrested much faster. Additionally, while I mentioned the upcoming work at the UG meeting, not announcing it on MICE-DISCUSS was a mistake. I know we talked about this before. I just forgot to do it. On 3/17/22 21:53, Richard Laager wrote: > Out of an abundance of caution, he is shutting down the ports facing the > new servers, cutting them (the thing we changed today) off completely. The ports have been re-enabled. The ports are now configured with the MICE VLAN _tagged_, which plays well with Linux & netplan. The ports have `port-security maximum 6` configured. This gives us enough headroom to support two route servers (in the event of a hardware failure where both have to run on the same physical box temporarily), two quarantine route servers (same note), the ARP responder, and the host's MAC address (if it shows up somehow). [1] https://discourse.ubuntu.com/t/jammy-jellyfish-release-schedule/23906 [2] Backends, like networkd, can and do implement imperative hooks. But netplan itself is only declarative. And Ubuntu is adding, in multiple cases at my direct suggestion, additional declarative parameters to eliminate the need for hook scripts in many scenarios. [3] https://netplan.io/reference/#properties-for-device-type-vlans%3A -- Richard

March 24, 2022

Corrected Room Code for Summer Meeting

by Jay Hanke

The hotel fixed the booking issue with the block of rooms. The code is now working correctly. Here is a shortcut to the link, you can book one or two nights at the discounted rate. https://reservations.travelclick.com/109230?groupID=3466891 Also to attend you need to register in advance. https://www.eventcreate.com/e/mwps2022 Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

March 23, 2022

Re: MAC Address Filtering

by AnthonyAnderberg＠nuvera.net

I’m going to be in tonight working on MOPs of my own, contact me off-list and we can talk about timing. Cheers, Anthony From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Larry Larsen <llarsen(a)LONGLINES.BIZ> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Wednesday, March 23, 2022 at 3:19 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MAC Address Filtering Good Afternoon; We are going to be swapping out a router this evening at 511, and if I remember correctly you have a MAC Address filter on your connections. Who do we need to work with to get the new MAC addresses authorized? Thank you! [cid:image001.png@01D83EC9.8E076EB0] Larry L. Larsen Director of Information Technology Phone: 712.271.5920 Email: llarsen(a)longlines.biz<mailto:llarsen@longlines.biz> 504 4th Street Sergeant Bluff, IA 51054 https://www.longlines.com/ [Title: Facebook - Description: image of Facebook icon]<https://www.facebook.com/LongLines/> [Title: LinkedIn - Description: image of LinkedIn icon] <https://www.linkedin.com/company/long-lines> [Title: Twitter - Description: image of Twitter icon] <https://twitter.com/LongLinesBB> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 23, 2022

MAC Address Filtering

by Larry Larsen

Good Afternoon; We are going to be swapping out a router this evening at 511, and if I remember correctly you have a MAC Address filter on your connections. Who do we need to work with to get the new MAC addresses authorized? Thank you! [cid:image001.png@01D83EC9.4D210D70] Larry L. Larsen Director of Information Technology Phone: 712.271.5920 Email: llarsen(a)longlines.biz<mailto:llarsen@longlines.biz> 504 4th Street Sergeant Bluff, IA 51054 https://www.longlines.com/ [Title: Facebook - Description: image of Facebook icon]<https://www.facebook.com/LongLines/> [Title: LinkedIn - Description: image of LinkedIn icon] <https://www.linkedin.com/company/long-lines> [Title: Twitter - Description: image of Twitter icon] <https://twitter.com/LongLinesBB>

March 23, 2022

Midwest Peering Summit Info July 26, 2022

by Jay Hanke

As part of the MICE Summer outreach meeting. We have set up a Peering Summit with a number of neighboring Internet Exchanges. These include OmahaIX, KCIX, QCIX and several others. Each of the involved IXes will be attending and assisting in marketing the event to their users. Here's the link to the registration for the conference. https://www.eventcreate.com/e/mwps2022 Advanced registration is free for peering networks. We'll encounter an extra fee from the venue for attendees not pre registered. We'd ask that you pick up a table if you're going to bring sales folks. We have a block of rooms reserved with a discount at the hotel for rooms from the 25th through the 27th of July at $95 per night. Log into: https://www.stoneycreekhotels.com/ Click book now Des Moines Discount Code: 2207SOUTHF Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

March 21, 2022

MICE Secretary

by Richard Laager

This is going to MICE-ANNOUNCE too, in this interest of formalities. After the UG 40 meeting on March 11, the MICE board appointed Anthony Anderberg as Secretary. (This is separate from the Treasurer appointment which happened earlier and was mentioned during the meeting.) -- Richard

March 19, 2022

Re: Netnod Down?

by Jeremy Lumby

Last I spoke with them they had a series of software issues crop up with their equipment. I have not spoken with them in a while, It is probably worth reaching out to them to see if they have a new estimated timeframe to repair. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Friday, March 18, 2022 8:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Netnod Down? Is Netnod down at MICE? I'm not seeing any routes from them, at least via the route servers. Thanks -- =============================================== David Farmer Email:farmer@umn.edu <Email%3Afarmer(a)umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 19, 2022

Netnod Down?

by David Farmer

Is Netnod down at MICE? I'm not seeing any routes from them, at least via the route servers. Thanks -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 19, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by David Farmer

Michael, Actually, the credit for looking at Netflow goes to Colin. Richard, Thanks for getting that project moving forward, and as Michael said, it happens Everyone, Attached are the detailed reports for the 3 alerts we got in that timeframe last night if you want more details. On Fri, Mar 18, 2022 at 9:24 AM Michael Hare < 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > Dave, thanks for the cluebat about looking at netflow. I didn’t think > that one through myself. I included the top talker we saw below. > > > > Richard, thanks for responding and letting us know what happened. Having > been in similar situations myself, “it happens”. > > > > -Michael > > > > ===/========== > > > > ** nfdump -M /var/local/flows/live/core -T -r nfcapd.202203172000 -n 10 > -s record/packets -A srcip,dstip -6 > > nfdump filter: > > router ip 143.235.32.110 and proto icmp6 > > Aggregated flows 116 > > Top 10 flows ordered by packets: > > Date first seen Duration Src IP > Addr Dst IP Addr Packets Bytes bps > Bpp Flows > > 2022-03-17 19:59:59.104 19.456 fe80::1a2a:d300:64dd:ed24 > <https://flows-1.uwsys.net/nfsen/nfsen.php#null> > ff02::1:ff00:254 <https://flows-1.uwsys.net/nfsen/nfsen.php#null> 9.4 > M 714.9 M 294.0 M 76 2 > > > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *David > Farmer > *Sent:* Thursday, March 17, 2022 8:34 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > > > Yes we say it and it reset a bunch of our BGP session on MICE. > > > > Our Arbor Sightling Netflow say the sources were > > > > 2001:504:27::d1af:0:241/128 > > fe80::8618:88ff:fea4:d301/128 > > e80::a66c:2aff:fe76:b400/128 > > > > Destin to; > > All routers ff02::1 > > All MLD Routers ff02::16 > > > > And then a solicited-node address of > > ff02::1:ff00:254 > > Don't know the source of that > > > > > > On Thu, Mar 17, 2022 at 8:22 PM Michael Hare < > 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > > I presume I wasn't the only one that felt the arp/nd storm that began ~ > 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have > info about mac addrs but our peering device reported 20kpps of icmp > neighbor discovery. > > -Michael > [AS3128] > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Michael Hare

Dave, thanks for the cluebat about looking at netflow. I didn’t think that one through myself. I included the top talker we saw below. Richard, thanks for responding and letting us know what happened. Having been in similar situations myself, “it happens”. -Michael ===/========== ** nfdump -M /var/local/flows/live/core -T -r nfcapd.202203172000 -n 10 -s record/packets -A srcip,dstip -6 nfdump filter: router ip 143.235.32.110 and proto icmp6 Aggregated flows 116 Top 10 flows ordered by packets: Date first seen Duration Src IP Addr Dst IP Addr Packets Bytes bps Bpp Flows 2022-03-17 19:59:59.104 19.456 fe80::1a2a:d300:64dd:ed24<https://flows-1.uwsys.net/nfsen/nfsen.php#null> ff02::1:ff00:254<https://flows-1.uwsys.net/nfsen/nfsen.php#null> 9.4 M 714.9 M 294.0 M 76 2 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Thursday, March 17, 2022 8:34 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? Yes we say it and it reset a bunch of our BGP session on MICE. Our Arbor Sightling Netflow say the sources were 2001:504:27::d1af:0:241/128 fe80::8618:88ff:fea4:d301/128 e80::a66c:2aff:fe76:b400/128 Destin to; All routers ff02::1 All MLD Routers ff02::16 And then a solicited-node address of ff02::1:ff00:254 Don't know the source of that On Thu, Mar 17, 2022 at 8:22 PM Michael Hare <000000097dab80c5-dmarc-request(a)lists.iphouse.net<mailto:000000097dab80c5-dmarc-request@lists.iphouse.net>> wrote: I presume I wasn't the only one that felt the arp/nd storm that began ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have info about mac addrs but our peering device reported 20kpps of icmp neighbor discovery. -Michael [AS3128] -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Chris Wopat

Seems to jive with some traffic shift on the Compudyne 511 remote switch: http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

FWIW, traffic numbers overall look good. I called Jeremy to get his thoughts. He hasn’t seen anything concerning outside of the initial issue. Out of an abundance of caution, he is shutting down the ports facing the new servers, cutting them (the thing we changed today) off completely. They are not in production use, so there is no reason not to take this step. You might want to do some ping testing to various peers interface IPs and/or check any Juniper-specific storm control stuff. -- Richard > On Mar 17, 2022, at 21:37, Ryan Nelson <ryan.nelson(a)integrisit.com> wrote: > > > Something still doesn't seem right. Really slow traffic via MICE. I just shut down our MICE sessions and the "weirdness" cleared up. > > Ryan > Compudyne/Integris/AS47096 > Ryan Nelson > Network Services Senior Architect > Office: 218-336-2220 > ryan.nelson(a)integrisit.com > > Empowering people through technology. > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Richard Laager <rlaager(a)WIKTEL.COM> > Sent: Thursday, March 17, 2022 9:09 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > EXTERNAL MESSAGE > > We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. > > -- > Richard > > > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > > > That was me. I believe I created a loop with the new servers. > > > > As soon as I realized, I yanked the cables. > > > > Sorry everyone! > > > > -- > > Richard > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Ryan Nelson

Something still doesn't seem right. Really slow traffic via MICE. I just shut down our MICE sessions and the "weirdness" cleared up. Ryan Compudyne/Integris/AS47096 ________________________________ Ryan Nelson Network Services Senior Architect Office: 218-336-2220 ryan.nelson(a)integrisit.com Empowering people through technology. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Richard Laager <rlaager(a)WIKTEL.COM> Sent: Thursday, March 17, 2022 9:09 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? EXTERNAL MESSAGE We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. -- Richard > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > That was me. I believe I created a loop with the new servers. > > As soon as I realized, I yanked the cables. > > Sorry everyone! > > -- > Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. -- Richard > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > That was me. I believe I created a loop with the new servers. > > As soon as I realized, I yanked the cables. > > Sorry everyone! > > -- > Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Ryan Malek

We also saw it. We flapped with 2001:504:27::51cc:0:1 (Akamai) at the same time. Not sure they're related. Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 3/17/2022 8:29 PM, Jeff Wilde wrote: > Your correct we did > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > > -------- Original message -------- > From: Michael Hare <000000097dab80c5-dmarc-request(a)LISTS.IPHOUSE.NET> > Date: 3/17/22 8:22 PM (GMT-06:00) > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > I presume I wasn't the only one that felt the arp/nd storm that began > ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't > have info about mac addrs but our peering device reported 20kpps of > icmp neighbor discovery. > > -Michael > [AS3128] > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> >

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

That was me. I believe I created a loop with the new servers. As soon as I realized, I yanked the cables. Sorry everyone! -- Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by David Farmer

Yes we say it and it reset a bunch of our BGP session on MICE. Our Arbor Sightling Netflow say the sources were 2001:504:27::d1af:0:241/128 fe80::8618:88ff:fea4:d301/128 e80::a66c:2aff:fe76:b400/128 Destin to; All routers ff02::1 All MLD Routers ff02::16 And then a solicited-node address of ff02::1:ff00:254 Don't know the source of that On Thu, Mar 17, 2022 at 8:22 PM Michael Hare < 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > I presume I wasn't the only one that felt the arp/nd storm that began ~ > 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have > info about mac addrs but our peering device reported 20kpps of icmp > neighbor discovery. > > -Michael > [AS3128] > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Jeff Wilde

Your correct we did Sent from my Verizon, Samsung Galaxy smartphone -------- Original message -------- From: Michael Hare <000000097dab80c5-dmarc-request(a)LISTS.IPHOUSE.NET> Date: 3/17/22 8:22 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? I presume I wasn't the only one that felt the arp/nd storm that began ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have info about mac addrs but our peering device reported 20kpps of icmp neighbor discovery. -Michael [AS3128]

March 18, 2022