Hello, CNS has received a request from one of its customers for a direct connection to MICE. If the consensus is to move forward, we could haul the MICE VLAN to that customer (and presumably to any other CNS member or customer who wants a direct MICE connection). This could result in the MICE VLAN appearing at various places throughout the region. I think that this could be a good thing for MICE, but if not managed correctly, it has the potential to be troublesome for problem isolation. MICE already has remote switches at 511. Other exchanges often have switches connected throughout their city. In this case, it is a little of both – except that the individual remote connections could be farther away in different directions. Does that matter? Whether we add remote connections or not, I'd like to see us come up with a plan for spanning tree. It probably would be best to make sure that our spanning tree root is one of the MICE 10Gig switches (probably the main one?). Or, alternatively, we could disable/filter spanning tree altogether and do MAC filtering for loop prevention. I think that spanning tree has its limitations, but it is much easier to maintain then MAC filtering. It looks like the current spanning tree root is 00:0c:db:fe:af:00 which looks like a Brocade device. Hmmm... We might want to change that... What does everybody think? Are there better ways to do this? Thanks, Steve ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I would say that so long as we are careful to avoid creating loops in the topology and take proper steps to insure that fault isolation can be accomplished in the event of a problem, this is a good thing. Owen On Sep 20, 2011, at 12:25 PM, Steve Howard wrote:
Hello,
CNS has received a request from one of its customers for a direct connection to MICE.
If the consensus is to move forward, we could haul the MICE VLAN to that customer (and presumably to any other CNS member or customer who wants a direct MICE connection). This could result in the MICE VLAN appearing at various places throughout the region.
I think that this could be a good thing for MICE, but if not managed correctly, it has the potential to be troublesome for problem isolation. MICE already has remote switches at 511. Other exchanges often have switches connected throughout their city. In this case, it is a little of both – except that the individual remote connections could be farther away in different directions. Does that matter?
Whether we add remote connections or not, I'd like to see us come up with a plan for spanning tree. It probably would be best to make sure that our spanning tree root is one of the MICE 10Gig switches (probably the main one?). Or, alternatively, we could disable/filter spanning tree altogether and do MAC filtering for loop prevention. I think that spanning tree has its limitations, but it is much easier to maintain then MAC filtering. It looks like the current spanning tree root is 00:0c:db:fe:af:00 which looks like a Brocade device. Hmmm... We might want to change that...
What does everybody think? Are there better ways to do this?
Thanks, Steve
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
CNS has received a request from one of its customers for a direct connection to MICE.
This has come up before. I've been recommending that the carriers sell it as Ethernet transport and then pop into a "fresh" MICE access port so each carrier ends up with their own port. I don't see any problem placing a remote switch outside the Twin Cities. I would disable spanning tree. If we haven't already, we should disable spanning tree in the core switches or place a guard so the port goes down if a BPDU comes in. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Tue, 2011-09-20 at 14:37 -0500, Jay Hanke wrote:
CNS has received a request from one of its customers for a direct connection to MICE.
This has come up before. I've been recommending that the carriers sell it as Ethernet transport and then pop into a "fresh" MICE access port so each carrier ends up with their own port.
What is/was the original intention of the remote switches in the first place if not for extending the reach of the MICE network? Getting people onto the main/core switch is most preferably for flat simplicity... KISS IMO
I don't see any problem placing a remote switch outside the Twin Cities. I would disable spanning tree. If we haven't already, we should disable spanning tree in the core switches or place a guard so the port goes down if a BPDU comes in.
+1 for properly managed STP unless/until it crosses administrative boundaries. Managing a shared STP domain between unrelated entities is messy. It only takes one port with an stp bpdu filter to generate a bad day that even a bpdu guard can't protect against. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 09/20/2011 02:51 PM, Justin Krejci wrote:
What is/was the original intention of the remote switches in the first place if not for extending the reach of the MICE network? Getting people onto the main/core switch is most preferably for flat simplicity... KISS IMO
For the CNS switch, it was a requirement for the donation. CNS provides free use of the MICE 10Gig switch and power, provided that the switch stays in CNS space. CNS also pays for one cross-connect that was originally intended to connect the MICE switch to the main switch. I think that FWR donated the cross connect from CNS to the main switch (thanks Mike!) and CNS pays for the cross connect to the GigaPOP instead. (?) I can't recall how the Mankato Networks switch originally came online, but it was part of the plan from the beginning. Perhaps Jay can refresh our memory. I do like that it is there because it gives an option for people that might not otherwise be able to justify the cost of a cross connect to get connected to MICE. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Jay, On 09/20/2011 02:37 PM, Jay Hanke wrote:
CNS has received a request from one of its customers for a direct connection to MICE. This has come up before. I've been recommending that the carriers sell it as Ethernet transport and then pop into a "fresh" MICE access port so each carrier ends up with their own port.
Unfortunately, that results in a cross-connect charge in addition to the ethernet transport. Doesn't exactly help to encourage membership. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Unfortunately, that results in a cross-connect charge in addition to the ethernet transport. Doesn't exactly help to encourage membership.
I was thinking you would connect them to the switch in the CNS rack... Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Jay, On 09/20/2011 05:50 PM, Jay Hanke wrote:
Unfortunately, that results in a cross-connect charge in addition to the ethernet transport. Doesn't exactly help to encourage membership. I was thinking you would connect them to the switch in the CNS rack...
That works great for CNS! I was thinking of other providers that might want to do the same thing. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
That works great for CNS!
I was thinking of other providers that might want to do the same thing.
If you want to plop a switch in 511 or remotely, I don't think it's an issue to connect it into the MICE fabric. Actually there are a fair number of participants with similar setups who . Back to the original Grumpy's meeting we're going to err on the side of connectivity until it's a problem and then we can deal with it. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 9/20/11 14:37 CDT, Jay Hanke wrote:
CNS has received a request from one of its customers for a direct connection to MICE.
This has come up before. I've been recommending that the carriers sell it as Ethernet transport and then pop into a "fresh" MICE access port so each carrier ends up with their own port.
I don't see any problem placing a remote switch outside the Twin Cities. I would disable spanning tree. If we haven't already, we should disable spanning tree in the core switches or place a guard so the port goes down if a BPDU comes in.
NO!!!!! DON'T Disable Spanning Tree. But we do need to think about Root Bridge and Root Guard issues. But Spanning tree is important in loop prevention, accidents do occur and spanning tree and prevent one of these accidents from turning into a broadcast storm. The best Idea is to design a loop free topology have spanning tree running to ensure the no loops occur by accident, which they more than likely will.
Jay
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Sep 20, 2011, at 2:46 PM, David Farmer wrote:
On 9/20/11 14:37 CDT, Jay Hanke wrote:
CNS has received a request from one of its customers for a direct connection to MICE.
This has come up before. I've been recommending that the carriers sell it as Ethernet transport and then pop into a "fresh" MICE access port so each carrier ends up with their own port.
I don't see any problem placing a remote switch outside the Twin Cities. I would disable spanning tree. If we haven't already, we should disable spanning tree in the core switches or place a guard so the port goes down if a BPDU comes in.
NO!!!!! DON'T Disable Spanning Tree. But we do need to think about Root Bridge and Root Guard issues. But Spanning tree is important in loop prevention, accidents do occur and spanning tree and prevent one of these accidents from turning into a broadcast storm.
The best Idea is to design a loop free topology have spanning tree running to ensure the no loops occur by accident, which they more than likely will.
If you're going to do that, make sure that spanning tree is running in fast convergence mode or you are asking for different brands of trouble. Unfortunately, fast converging spanning tree tends to be vendor-specific. Owen
Jay
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
########################################################################
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Tue, 2011-09-20 at 15:13 -0700, Owen DeLong wrote:
Unfortunately, fast converging spanning tree tends to be vendor-specific.
RSTP and MSTP are both standards. Aren't they fast enough? Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Sep 20, 2011, at 4:57 PM, Richard Laager wrote:
On Tue, 2011-09-20 at 15:13 -0700, Owen DeLong wrote:
Unfortunately, fast converging spanning tree tends to be vendor-specific.
RSTP and MSTP are both standards. Aren't they fast enough?
If you have two vendors with compatible implementations of them, perhaps. Admittedly, it's been a while, but, last time I tried, the interoperability across multiple vendors wasn't there once you departed from normal 90-second spanning tree. Owen ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Tue, 2011-09-20 at 17:01 -0700, Owen DeLong wrote:
On Sep 20, 2011, at 4:57 PM, Richard Laager wrote:
On Tue, 2011-09-20 at 15:13 -0700, Owen DeLong wrote:
Unfortunately, fast converging spanning tree tends to be vendor-specific.
RSTP and MSTP are both standards. Aren't they fast enough?
If you have two vendors with compatible implementations of them, perhaps. Admittedly, it's been a while, but, last time I tried, the interoperability across multiple vendors wasn't there once you departed from normal 90-second spanning tree.
I haven't had any cross-vendor problems, even mixing RSTP and MSTP (in configurations for which they are designed to mix). I'm about to add another vendor to the mix, so we'll see if this blows up in my face now that I've publicly said it's been working. ;) Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We mix vendors a lot including on the Minneapolis wifi network using MST and rstp. So literally thousands of bridges talking, more or less, faithfully. Sure some bugs exist but when facing adversity one must adapt or flounder (Darwinism?). Getting vendors to fix bugs helps immensely too of course. And yes convergence is faster than vanilla STP. Sent via BlackBerry from T-Mobile -----Original Message----- From: Richard Laager <rlaager@WIKTEL.COM> Sender: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> Date: Tue, 20 Sep 2011 19:08:52 To: <MICE-DISCUSS@LISTS.IPHOUSE.NET> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Remote MICE members? On Tue, 2011-09-20 at 17:01 -0700, Owen DeLong wrote:
On Sep 20, 2011, at 4:57 PM, Richard Laager wrote:
On Tue, 2011-09-20 at 15:13 -0700, Owen DeLong wrote:
Unfortunately, fast converging spanning tree tends to be vendor-specific.
RSTP and MSTP are both standards. Aren't they fast enough?
If you have two vendors with compatible implementations of them, perhaps. Admittedly, it's been a while, but, last time I tried, the interoperability across multiple vendors wasn't there once you departed from normal 90-second spanning tree.
I haven't had any cross-vendor problems, even mixing RSTP and MSTP (in configurations for which they are designed to mix). I'm about to add another vendor to the mix, so we'll see if this blows up in my face now that I've publicly said it's been working. ;) Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I haven't had any cross-vendor problems, even mixing RSTP and MSTP (in configurations for which they are designed to mix).
I think the problems come in more on the mixing vendors and more importantly mixing administrative domains with spanning tree. There are more than a couple of vendors with random spanning-tree quirks. Frankly, I'm surprised at the outpouring of love for spanning-tree on the list. I've on more than one occasion selected a vendor with a ring or a virtual chassis technology to avoid spanning-tree in the core. I was also thrilled at the prospect of something like TRILL. Can someone take it on to make a design recommendation? Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Sep 20, 2011, at 7:54 PM, Jay Hanke wrote:
Frankly, I'm surprised at the outpouring of love for spanning-tree on the list. I've on more than one occasion selected a vendor with a ring or a virtual chassis technology to avoid spanning-tree in the core. I was also thrilled at the prospect of something like TRILL.
Can someone take it on to make a design recommendation? I'll throw out a vote for a spanning-tree free IX design.
I'd also throw up a flag of caution for allowing the IX network to be extended inside other carriers networks. Doing so only extends the failure domain of the IX, and makes it tougher to monitor and maintain. Now, OTOH, I think carrier sponsored "remote switches" with a connection into the core IX switches, and a prescribed participant facing config would be a better option, and better suited for the long haul. And to Jay's point, folks can still sell transport circuits to get folks connected into the exchange, just pop the other end into one of these "MICE Remote Switches". Honestly, I'm a little surprised that there is spanning tree running now across the IX. My vote would be for something along the lines of: - spanning tree free core/remote switches, managed under the guise of MICE - set config on participant facing ports - only allowing IPv4/IPv6/ARP - storm control set to low pps threshold or outright blocking of broadcast traffic besides ARP/ND packets - one MAC addr per participant facing port This recipe, IMHO, is the only way to avoid L2 meltdown in the future if more and more connection points are added. Thanks, Andrew ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 9/20/11 19:54 CDT, Jay Hanke wrote:
Frankly, I'm surprised at the outpouring of love for spanning-tree on the list. I've on more than one occasion selected a vendor with a ring or a virtual chassis technology to avoid spanning-tree in the core. I was also thrilled at the prospect of something like TRILL.
Can someone take it on to make a design recommendation?
Let me be clear, I don't love spanning-tree at all. But turning off spanning-tree and praying you never have a loop is worse than spanning-tree. My personal design preference is to design your network with a loop free topology at layer2. Then it is possible to run without spanning-tree. Any redundancy is provided using LAGs, GMPLS or MPLS transport technologies, not loops in the bridge topology. However, you still run spanning-tree to catch mistakes, they will happen and spanning tree will keep them from melting the network. When the network is built and is functioning as designed, spanning-tree has nothing to converge, the network has a loop free topology. In this design, spanning-tree is only there to ensure the network is actually built and actually functioning as designed. When you are driving your car, if you see head lights coming at you on the wrong side of the road, most people slow down and get ready to take the ditch. Rather than take a head on into the other guy, or into the ditch, at full speed. In this design, spanning-tree is looking for head light on the wrong side of the road, rather than assuming it will never happen, which is what you are doing if you just turn spanning-tree off. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Forwarded to my team. Nicely put. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Tuesday, September 20, 2011 10:18 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Remote MICE members? On 9/20/11 19:54 CDT, Jay Hanke wrote:
Frankly, I'm surprised at the outpouring of love for spanning-tree on the list. I've on more than one occasion selected a vendor with a ring or a virtual chassis technology to avoid spanning-tree in the core. I was also thrilled at the prospect of something like TRILL.
Can someone take it on to make a design recommendation?
Let me be clear, I don't love spanning-tree at all. But turning off spanning-tree and praying you never have a loop is worse than spanning-tree. My personal design preference is to design your network with a loop free topology at layer2. Then it is possible to run without spanning-tree. Any redundancy is provided using LAGs, GMPLS or MPLS transport technologies, not loops in the bridge topology. However, you still run spanning-tree to catch mistakes, they will happen and spanning tree will keep them from melting the network. When the network is built and is functioning as designed, spanning-tree has nothing to converge, the network has a loop free topology. In this design, spanning-tree is only there to ensure the network is actually built and actually functioning as designed. When you are driving your car, if you see head lights coming at you on the wrong side of the road, most people slow down and get ready to take the ditch. Rather than take a head on into the other guy, or into the ditch, at full speed. In this design, spanning-tree is looking for head light on the wrong side of the road, rather than assuming it will never happen, which is what you are doing if you just turn spanning-tree off. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Can someone take it on to make a design recommendation?
SNIP Spanning tree love-hate Monologue Excellent! I believe you just volunteered to propose a solution! :) Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Sep 20, 2011, at 10:18 PM, David Farmer wrote:
However, you still run spanning-tree to catch mistakes, they will happen and spanning tree will keep them from melting the network. When the network is built and is functioning as designed, spanning-tree has nothing to converge, the network has a loop free topology. In this design, spanning-tree is only there to ensure the network is actually built and actually functioning as designed.
I'd somewhat agree here, *but*, there are alternatives to spanning-tree in the IX network - like storm control, limiting to 1 mac addr per port, putting naughty ports in the penalty box (err-disable/shutdown), etc. As an additional comment to my previous e-mail, I don't see issues with spanning-tree as a safety net, but then the proper steps need to be taken on the participant L2 devices to limit the interaction between the STP domains. If we have (I'm assuming) a participant Brocade l2 device acting as the STP root for the IX network, that's an issue that needs to get resolved (ie: bpdufilter towards participant ports, etc). This is an IP exchange, so we really should only be seeing unicast/mcast IP traffic, not layer2 protocols and/or broadcast traffic (save for ARP/ND packets). AMS-IX has some very good guidelines about how to get your layer2 devices to be quiet, located here: http://www.ams-ix.net/config-guide/ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Sep 20, 2011, at 7:08 PM, Richard Laager wrote:
I haven't had any cross-vendor problems, even mixing RSTP and MSTP (in configurations for which they are designed to mix).
Same here, at least as it relates to standards based MST/RSTP implementations. Never had issues there. OTOH, when trying to interop between per VLAN STP implementations (like RPVST+/Cisco, and VSTP/Juniper type scenarios), things *do* get a bit dicey. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (8)
-
Andrew Hoyos -
David Farmer -
Jay Hanke -
Justin Krejci -
Owen DeLong -
Richard Laager -
Ryan Goldberg -
Steve Howard