- mice-discuss - lists.micemn.net

Re: Route reflector check

by Michael Hare

Frank- I may have found the answer to your question. Looking for tips on better ways to do this, but I did this. "show route community $ourCommunityForRoutesWeExport detail | match Age" and did a mix of lame exclusions and visual greps to find a few routes with an age of less than 60 minutes. specifically 184.164.250.0/24 *[BGP/170] 00:44:15 184.164.251.0/24 *[BGP/170] 00:44:15 184.164.252.0/24 *[BGP/170] 00:44:15 These are part of an anycast R&E project. We receive this over an eBGP peering with the project. It looks like they are setting community 47065:184 if you want to filter. Frank, how new is your detection software? This is an old project and I would call this normal behavior for them. -Michael From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Michael Hare Sent: Monday, December 30, 2019 3:59 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Route reflector check Hi everyone- Wanted to add a bit more detail to what we (AS 3128) checked on our side. Specifically, 1) our peering session(s) to the RS seems stable from an uptime perspective. 2) I don't see a large delta in outbound bgp messages from us to MICE. 3) we don't run BMP so I don't actually track outbound changes at a level that would conclusive prove what's going on. Frank, is it possible you are being notified about a single route flapping? I did check AS2381's looking glass, specifically a single route (143.235.0.0/16) and it looks stable. Admittedly I didn't look for LG across all MICE members but I'm not sure if any would allow me to run an as-path regexp to see if there is a specific flapping route that is flapping. If someone can help out on that front it could be useful. [output from AS2381 looking glass] [BGP/170] 1w6d 13:08:45, MED 10, localpref 200, from 206.108.255.1 AS path: 3128 3128 3128 3128 I, validation-state: unverified > to 206.108.255.66 via xe-0/1/5.300 [BGP/170] 3w3d 14:51:19, MED 10, localpref 200, from 206.108.255.2 AS path: 3128 3128 3128 3128 I, validation-state: unverified > to 206.108.255.66 via xe-0/1/5.300 Thanks- -Michael From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, December 30, 2019 3:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Route reflector check QRator has been alerting me that AS 3128 is coming and going. It just happened again and wondering if anyone else is seeing this behavior and also how things look on MICE's route reflectors. I already contacted the good folks at the University of Wisconsin and they're not seeing anything on their side. AS Number: AS3128 AS Name: # First seen Last seen 1 2019-12-29 21:49:00 2019-12-29 21:50:00 2 2019-12-29 18:42:00 2019-12-29 18:43:00 3 2019-12-29 17:57:00 2019-12-29 17:58:00 4 2019-12-29 16:27:00 2019-12-29 16:28:00 5 2019-12-29 14:12:00 2019-12-29 14:13:00 6 2019-12-29 13:27:00 2019-12-29 13:28:00 7 2019-12-29 12:42:00 2019-12-29 12:43:00 8 2019-12-29 11:57:00 2019-12-29 11:58:00 9 2019-12-29 11:12:00 2019-12-29 11:13:00 10 2019-12-29 11:04:00 2019-12-29 11:05:00 11 2019-12-29 10:27:00 2019-12-29 10:28:00 12 2019-12-29 08:57:00 2019-12-29 08:58:00 13 2019-12-29 08:12:00 2019-12-29 08:13:00 14 2019-12-29 07:53:00 2019-12-29 07:54:00 15 2019-12-29 07:27:00 2019-12-29 07:28:00 16 2019-12-29 06:42:00 2019-12-29 06:43:00 17 2019-12-29 05:57:00 2019-12-29 05:58:00 18 2019-12-29 05:12:00 2019-12-29 05:13:00 19 2019-12-29 04:40:00 2019-12-29 04:41:00 20 2019-12-29 04:27:00 2019-12-29 04:28:00 21 2019-12-29 00:05:00 2019-12-29 00:06:00 22 2019-12-28 23:57:00 2019-12-28 23:58:00 23 2019-12-28 23:12:00 2019-12-28 23:13:00 24 2019-12-28 23:04:00 2019-12-28 23:05:00 25 2019-12-28 22:27:00 2019-12-28 22:28:00 26 2019-12-28 21:42:00 2019-12-28 21:43:00 27 2019-12-28 20:57:00 2019-12-28 20:58:00 28 2019-12-28 20:12:00 2019-12-28 20:13:00 29 2019-12-28 07:27:00 2019-12-28 07:28:00 30 2019-12-28 06:42:00 2019-12-28 06:43:00 31 2019-12-28 05:57:00 2019-12-28 05:58:00 32 2019-12-28 05:12:00 2019-12-28 05:13:00 33 2019-12-27 17:21:00 2019-12-27 17:22:00 34 2019-12-06 07:01:00 2019-12-27 06:10:00 Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 30, 2019

Re: Route reflector check

by Michael Hare

Hi everyone- Wanted to add a bit more detail to what we (AS 3128) checked on our side. Specifically, 1) our peering session(s) to the RS seems stable from an uptime perspective. 2) I don't see a large delta in outbound bgp messages from us to MICE. 3) we don't run BMP so I don't actually track outbound changes at a level that would conclusive prove what's going on. Frank, is it possible you are being notified about a single route flapping? I did check AS2381's looking glass, specifically a single route (143.235.0.0/16) and it looks stable. Admittedly I didn't look for LG across all MICE members but I'm not sure if any would allow me to run an as-path regexp to see if there is a specific flapping route that is flapping. If someone can help out on that front it could be useful. [output from AS2381 looking glass] [BGP/170] 1w6d 13:08:45, MED 10, localpref 200, from 206.108.255.1 AS path: 3128 3128 3128 3128 I, validation-state: unverified > to 206.108.255.66 via xe-0/1/5.300 [BGP/170] 3w3d 14:51:19, MED 10, localpref 200, from 206.108.255.2 AS path: 3128 3128 3128 3128 I, validation-state: unverified > to 206.108.255.66 via xe-0/1/5.300 Thanks- -Michael From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, December 30, 2019 3:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Route reflector check QRator has been alerting me that AS 3128 is coming and going. It just happened again and wondering if anyone else is seeing this behavior and also how things look on MICE's route reflectors. I already contacted the good folks at the University of Wisconsin and they're not seeing anything on their side. AS Number: AS3128 AS Name: # First seen Last seen 1 2019-12-29 21:49:00 2019-12-29 21:50:00 2 2019-12-29 18:42:00 2019-12-29 18:43:00 3 2019-12-29 17:57:00 2019-12-29 17:58:00 4 2019-12-29 16:27:00 2019-12-29 16:28:00 5 2019-12-29 14:12:00 2019-12-29 14:13:00 6 2019-12-29 13:27:00 2019-12-29 13:28:00 7 2019-12-29 12:42:00 2019-12-29 12:43:00 8 2019-12-29 11:57:00 2019-12-29 11:58:00 9 2019-12-29 11:12:00 2019-12-29 11:13:00 10 2019-12-29 11:04:00 2019-12-29 11:05:00 11 2019-12-29 10:27:00 2019-12-29 10:28:00 12 2019-12-29 08:57:00 2019-12-29 08:58:00 13 2019-12-29 08:12:00 2019-12-29 08:13:00 14 2019-12-29 07:53:00 2019-12-29 07:54:00 15 2019-12-29 07:27:00 2019-12-29 07:28:00 16 2019-12-29 06:42:00 2019-12-29 06:43:00 17 2019-12-29 05:57:00 2019-12-29 05:58:00 18 2019-12-29 05:12:00 2019-12-29 05:13:00 19 2019-12-29 04:40:00 2019-12-29 04:41:00 20 2019-12-29 04:27:00 2019-12-29 04:28:00 21 2019-12-29 00:05:00 2019-12-29 00:06:00 22 2019-12-28 23:57:00 2019-12-28 23:58:00 23 2019-12-28 23:12:00 2019-12-28 23:13:00 24 2019-12-28 23:04:00 2019-12-28 23:05:00 25 2019-12-28 22:27:00 2019-12-28 22:28:00 26 2019-12-28 21:42:00 2019-12-28 21:43:00 27 2019-12-28 20:57:00 2019-12-28 20:58:00 28 2019-12-28 20:12:00 2019-12-28 20:13:00 29 2019-12-28 07:27:00 2019-12-28 07:28:00 30 2019-12-28 06:42:00 2019-12-28 06:43:00 31 2019-12-28 05:57:00 2019-12-28 05:58:00 32 2019-12-28 05:12:00 2019-12-28 05:13:00 33 2019-12-27 17:21:00 2019-12-27 17:22:00 34 2019-12-06 07:01:00 2019-12-27 06:10:00 Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 30, 2019

Re: [ADV] [MICE-DISCUSS] Route reflector check

by Doug McIntyre

On Mon, Dec 30, 2019 at 09:36:55PM +0000, Frank Bulk wrote: >QRator has been alerting me that AS 3128 is coming and going. It just happened again and wondering if anyone else is seeing this behavior and also how things look on MICE's route reflectors. The RR's show the connection to AS3128 has been up since 2019-12-17, no flapping. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 30, 2019

Route reflector check

by Frank Bulk

QRator has been alerting me that AS 3128 is coming and going. It just happened again and wondering if anyone else is seeing this behavior and also how things look on MICE's route reflectors. I already contacted the good folks at the University of Wisconsin and they're not seeing anything on their side. AS Number: AS3128 AS Name: # First seen Last seen 1 2019-12-29 21:49:00 2019-12-29 21:50:00 2 2019-12-29 18:42:00 2019-12-29 18:43:00 3 2019-12-29 17:57:00 2019-12-29 17:58:00 4 2019-12-29 16:27:00 2019-12-29 16:28:00 5 2019-12-29 14:12:00 2019-12-29 14:13:00 6 2019-12-29 13:27:00 2019-12-29 13:28:00 7 2019-12-29 12:42:00 2019-12-29 12:43:00 8 2019-12-29 11:57:00 2019-12-29 11:58:00 9 2019-12-29 11:12:00 2019-12-29 11:13:00 10 2019-12-29 11:04:00 2019-12-29 11:05:00 11 2019-12-29 10:27:00 2019-12-29 10:28:00 12 2019-12-29 08:57:00 2019-12-29 08:58:00 13 2019-12-29 08:12:00 2019-12-29 08:13:00 14 2019-12-29 07:53:00 2019-12-29 07:54:00 15 2019-12-29 07:27:00 2019-12-29 07:28:00 16 2019-12-29 06:42:00 2019-12-29 06:43:00 17 2019-12-29 05:57:00 2019-12-29 05:58:00 18 2019-12-29 05:12:00 2019-12-29 05:13:00 19 2019-12-29 04:40:00 2019-12-29 04:41:00 20 2019-12-29 04:27:00 2019-12-29 04:28:00 21 2019-12-29 00:05:00 2019-12-29 00:06:00 22 2019-12-28 23:57:00 2019-12-28 23:58:00 23 2019-12-28 23:12:00 2019-12-28 23:13:00 24 2019-12-28 23:04:00 2019-12-28 23:05:00 25 2019-12-28 22:27:00 2019-12-28 22:28:00 26 2019-12-28 21:42:00 2019-12-28 21:43:00 27 2019-12-28 20:57:00 2019-12-28 20:58:00 28 2019-12-28 20:12:00 2019-12-28 20:13:00 29 2019-12-28 07:27:00 2019-12-28 07:28:00 30 2019-12-28 06:42:00 2019-12-28 06:43:00 31 2019-12-28 05:57:00 2019-12-28 05:58:00 32 2019-12-28 05:12:00 2019-12-28 05:13:00 33 2019-12-27 17:21:00 2019-12-27 17:22:00 34 2019-12-06 07:01:00 2019-12-27 06:10:00 Frank

Dec. 30, 2019

Re: Nuvera Port Upgrade

by Richard Laager

On 12/19/19 1:09 PM, Doug McIntyre wrote: > On Thu, Dec 19, 2019 at 11:40:29AM -0600, Jay Hanke wrote: >> Nuvera will be upgrading to 2x10G LAG on new ports and then removing >> their existing port. >> >> The new port assignments are (Main Switch) >> >> Ethernet 4/28 >> Ethernet 5/28 >> >> This port will be vacated Ethernet 4/15 > > Okay, I setup Po33 for them bound to Eth4/28 & Eth5/28. I've updated the website accordingly. Someone (Anthony, probably), please me know when the move is complete so I can mark 4/15 as unused again. -- Richard

Dec. 20, 2019

Re: Nuvera Port Upgrade

by Doug McIntyre

On Thu, Dec 19, 2019 at 11:40:29AM -0600, Jay Hanke wrote: >Nuvera will be upgrading to 2x10G LAG on new ports and then removing >their existing port. > >The new port assignments are (Main Switch) > >Ethernet 4/28 >Ethernet 5/28 > >This port will be vacated Ethernet 4/15 Okay, I setup Po33 for them bound to Eth4/28 & Eth5/28. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 19, 2019

Nuvera Port Upgrade

by Jay Hanke

Nuvera will be upgrading to 2x10G LAG on new ports and then removing their existing port. The new port assignments are (Main Switch) Ethernet 4/28 Ethernet 5/28 This port will be vacated Ethernet 4/15 Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Dec. 19, 2019

Re: MICE MAC Address Limit

by Richard Laager

The board has approved the change in MAC address limit from 5 to 1. NOTE: This is orthogonal to the dedicated remote switch question, which is still pending. Non-dedicated remote switches can, for example, enforce this per-VLAN. We have already confirmed that most participants are using only 1 MAC address, and some that were using more than 1 have addressed that after being contacted. As discussed below, this change will be rolled out in a way to keep everyone working, by grandfathering if necessary. On 10/28/19 3:25 PM, Richard Laager wrote: > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: > https://www.ams-ix.net/ams/documentation/config-guide > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > -- Richard

Dec. 14, 2019

NebraskaLink Port Move

by Jay Hanke

Nebraskalink is moving their connection from the Zayo remote switch to the main switch. All IP address/Contact information will remain the same. The new port assignment is Main Switch: Ethernet4/13 Welcome aboard again! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Dec. 12, 2019

Re: MICE graphs - gap from today?

by Justin Krejci

US Internet experienced a server storage issue that affected the cacti server. Unfortunately the data in that time span was not captured and saved. [☹] ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Andrew Hoyos <hoyosa(a)GMAIL.COM> Sent: Tuesday, December 3, 2019 7:00 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE graphs - gap from today? Hi folks, Looks like the Cacti graphs are showing a gap from this afternoon. Anyone know what happened? — Andrew Hoyos hoyosa(a)gmail.com<mailto:hoyosa@gmail.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 4, 2019

MICE graphs - gap from today?

by Andrew Hoyos

Hi folks, Looks like the Cacti graphs are showing a gap from this afternoon. Anyone know what happened? — Andrew Hoyos hoyosa(a)gmail.com <mailto:hoyosa@gmail.com>

Dec. 4, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Brady Kittel

I agree with Richard that we shouldn't put the board in the position up have to make judgement calls on a case by case basis. I'd rather see clear guidelines be established along with good language around when a misbehaving member port can be shutdown for non-compliance that provides the board with a reasonable level of latitude with regard to enforcement. I don't think it's good practice to put board members in that position either as it may expose them to some level of legal liability if their judgement call is later challenged versus basing decisions on guidelines that were approved by a majority of members. None of this would prevent amendments to those guidelines from being voted on later either so I'd look at this as an effort at getting it reasonably correct initially fully expecting to tweak and refine things later as issues are encountered with the language or things aren't totally clear. -Brady Kittel HCMC On Tue, Dec 3, 2019, 10:46 AM Steve Howard <showard(a)paulbunyan.net> wrote: > > > On 12/03/2019 03:34 AM, Richard Laager wrote: > > On 12/2/19 10:30 AM, Steve Howard wrote: > > If supported by the remote switch, enforce a specific MAC address > requirement on the MICE VLAN for remote switches. > > > I'm not 100% sure I follow your example here. > > Enforcing a single MAC address is straightforward if the only thing > plugged into the non-dedicated switch (on the "downstream" side) are > routers. But what happens if hypothetically Wiktel and Paul Bunyan want > to exchange an Ethernet circuit VLAN over the CNS switch? The CNS switch > is going to see more than just our router MAC addresses. CNS can't limit > us to one MAC on a per-port basis. > > Are you saying that a remote switch would use a layer 2 ACL to limit the > source MAC transmitting into the MICE VLAN while allowing other MACs on > other VLANs? Is this a relatively common feature? Is this something that > you feel would be reasonable to _require_ of a non-dedicated switch? > > > > I was thinking of Cisco's port-security feature for the CNS remote. That > would limit each VLAN to specific mac address(es). Per Andrew's message, I > believe this feature is available from other manufacturers. > > Below is an example config that I installed on the CNS remote for > testing. I think requiring something like this would provide a reasonable > balance between protecting the exchange and allowing a switch to be > non-dedicated. Additionally, I'd be in favor of mac-address restrictions > on all of the MICE switches whether dedicated or not. > > interface Ethernet1/40 > description Test > switchport mode trunk > mtu 9216 > switchport port-security maximum 5 > switchport port-security > switchport port-security mac-address AAAA.AAAA.AAAA vlan 847 > switchport port-security mac-address AAAA.AAAA.AA01 vlan 1067 > switchport port-security mac-address AAAA.AAAA.AA02 vlan 1068 > switchport port-security mac-address AAAA.AAAA.AA03 vlan 1068 > switchport port-security mac-address AAAA.AAAA.AA04 vlan 1068 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Dec. 4, 2019

Re: Internet Wether Report

by Jeremy Schmeichel

Interesting. Halo: Reach released today for the PC at 10AM PST and is delivered via Steam, so I’d assume that was related to the Akamai traffic spike. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Gary Glissendorf Sent: Tuesday, December 3, 2019 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Internet Wether Report Thanks for the update. I saw Akamai and Edgecast blow up our links to them. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D27592.E0CCAA30] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D27592.E0CCAA30] <http://twitter.com/sdncomm> [cid:image003.jpg@01D27592.E0CCAA30] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D27592.E0CCAA30] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D27592.E0CCAA30] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of David Farmer Sent: Tuesday, December 3, 2019 1:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Internet Wether Report CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders. Anyone notice the somewhat impressive approximately 100G traffic spike on MICE just after 12 noon today, I'm told that is Halo for XBox One. https://www.express.co.uk/entertainment/gaming/1211738/Halo-Reach-Release-D… -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

Dec. 3, 2019

Re: Internet Wether Report

by Gary Glissendorf

Thanks for the update. I saw Akamai and Edgecast blow up our links to them. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D27592.E0CCAA30] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D27592.E0CCAA30] <http://twitter.com/sdncomm> [cid:image003.jpg@01D27592.E0CCAA30] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D27592.E0CCAA30] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D27592.E0CCAA30] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Tuesday, December 3, 2019 1:31 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Internet Wether Report CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders. Anyone notice the somewhat impressive approximately 100G traffic spike on MICE just after 12 noon today, I'm told that is Halo for XBox One. https://www.express.co.uk/entertainment/gaming/1211738/Halo-Reach-Release-D… -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

Dec. 3, 2019

Internet Wether Report

by David Farmer

Anyone notice the somewhat impressive approximately 100G traffic spike on MICE just after 12 noon today, I'm told that is Halo for XBox One. https://www.express.co.uk/entertainment/gaming/1211738/Halo-Reach-Release-D… -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Steve Howard

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Chris Wopat

Hi folks, thanks for the lively discussion so far. A few notes and comments about the more recent technical part of the discussion. In our shared switch scenario, we intend to limit mac addresses per our downstream customers on the MICE vlan itsself, not per physical interface. This seems fairly standard, we would hope that as many as possible are already doing this. We would advocate this as a requirement for all customer-facing ports on all switches, remote or otherwise, dedicated or not dedicated. We would also welcome an mac limit on the Arista Core set to something reasonable facing us and other remote switches as well. The current non-remote switch limit of 5 would work perfectly fine in our scenario. Perhaps a limit per remote switch based on N+something should also be enforced as long as it's not a large burden on the MICE operators. We also use BGP session culling, it's deactivated by default. We'd activate on the MICE vlan during maintenance. #### Junos config example for mac limits #### interfaces { ae0 { description "WiscNet Router"; unit 100 { description "MICE :RP:"; vlan-id 100; } } et-0/0/25 { description "MICE Arista Core"; unit 0 { family ethernet-switching { interface-mode access; vlan { members 100; } } } } } vlans { vlan-100 { description "MICE Peering"; vlan-id 100; interface ae0.100; inactive: forwarding-options { /* based on https://tinyurl.com/junos-bgp-culling */ filter { input CullBGP-MICE; output CullBGP-MICE; } } switch-options { interface ae0.100 { interface-mac-limit { 1; packet-action drop-and-log; } } } } } As far as graphing per-vlan, this works well on Q5 based systems such as our Juniper QFX10k. This + buffers is part of the reason we chose this platform instead of a cheap Broadcom device. This is also a part of the reason we would prefer to *not* shove a cheap Broadcom/chinaswitch device in the middle. We also support SNMP polling per interface. In our case we'd expose just the MICE interfaces using views. 630 below is a subinterface's snmp ifIndex. 1.10, 1.11, etc are ifOctests, ifUcastPkts, and so on for that interface. This is manual, but with only a handful of interfaces, totally do-able: snmp { view view-mice { oid .1 exclude; oid .1.3.6.1.2.1.2.2.1.10.630 include; oid .1.3.6.1.2.1.2.2.1.11.630 include; oid .1.3.6.1.2.1.2.2.1.13.630 include; oid .1.3.6.1.2.1.2.2.1.14.630 include; oid .1.3.6.1.2.1.2.2.1.16.630 include; oid .1.3.6.1.2.1.2.2.1.17.630 include; oid .1.3.6.1.2.1.2.2.1.18.630 include; oid .1.3.6.1.2.1.2.2.1.19.630 include; oid .1.3.6.1.2.1.2.2.1.20.630 include; } community wiscnet-mice { view view-mice; authorization read-only; } } Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Jay Hanke

MICE has had several requests from ethernet transport vendors to hand customers to MICE on individual vlans over a single port. It might be worth looking into a bridge domain offering where MICE pops the tags off and enforces the required security. If this were added MICE would have two types of remote connectivity, full remote switch and MICE transport provider. This service is already available from several of the remotes by stacking a transport switch in front of the remote and breaking the customers out into their own ports. To my knowledge, this setup has been very effective at isolating issues down to a single customer/port. I believe this is supported in newer Arista code but was a bit of a mess to get the graphing and security to to work. If we did something like this the best course of action might be to place a pizza box and do and uplink into the main fabric. Connecting to one of the "local" remote switches is also a very viable option. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Andrew Hoyos

On Dec 3, 2019, at 3:34 AM, Richard Laager <rlaager(a)wiktel.com> wrote: > > Enforcing a single MAC address is straightforward if the only thing > plugged into the non-dedicated switch (on the "downstream" side) are > routers. But what happens if hypothetically Wiktel and Paul Bunyan want > to exchange an Ethernet circuit VLAN over the CNS switch? The CNS switch > is going to see more than just our router MAC addresses. CNS can't limit > us to one MAC on a per-port basis. Juniper QFX and MX have options for limiting number of mac addresses per logical interface and/or VLAN. A quick scrub of other common vendors (Cisco, Arista) have the same. I wouldn’t see it as unreasonable to *require* a remote switch operator by whatever means necessary to enforce a one MAC address limit on their extension switch per logical participant handoff.

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Richard Laager

On 12/2/19 6:08 PM, Justin Krejci wrote: > I advocate an official MICE position that switch fabric hardware be > dedicated to MICE but allow the board to make exceptions to this > position on a case by case basis, where the requesting entity can plead > a case and provide justification/explanation on why they feel it > necessary to provide remote access via a shared switch fabric, eg as in > this case it would allow WiscNet to provide additional networks to > connect to MICE that otherwise would not, while WiscNet otherwise has a > clean reputation within MICE and nobody has any reason to suspect they > are up to anything shady, and they are intending to use modern hardware > not some ancient 2000's era hunk o junk. The board can then measure the > request and approve or reject the exception request as they deem best > for the exchange using their own prerogative. I think the board is hesitant to go down the road of making judgements based on _who_ is making a proposal. Treating all participants equally has been a very high priority. The "who", in terms of organization, can change through acquisitions. I realize that's not going to be the case with an organization like WiscNet, but we already saw this with the Neutral Path remote. The "who", in terms of people, can also change through staffing changes. -- Richard

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Richard Laager

On 12/2/19 10:30 AM, Steve Howard wrote: > If supported by the remote switch, enforce a specific MAC address > requirement on the MICE VLAN for remote switches. I'm not 100% sure I follow your example here. Enforcing a single MAC address is straightforward if the only thing plugged into the non-dedicated switch (on the "downstream" side) are routers. But what happens if hypothetically Wiktel and Paul Bunyan want to exchange an Ethernet circuit VLAN over the CNS switch? The CNS switch is going to see more than just our router MAC addresses. CNS can't limit us to one MAC on a per-port basis. Are you saying that a remote switch would use a layer 2 ACL to limit the source MAC transmitting into the MICE VLAN while allowing other MACs on other VLANs? Is this a relatively common feature? Is this something that you feel would be reasonable to _require_ of a non-dedicated switch? -- Richard

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Justin Krejci

I advocate an official MICE position that switch fabric hardware be dedicated to MICE but allow the board to make exceptions to this position on a case by case basis, where the requesting entity can plead a case and provide justification/explanation on why they feel it necessary to provide remote access via a shared switch fabric, eg as in this case it would allow WiscNet to provide additional networks to connect to MICE that otherwise would not, while WiscNet otherwise has a clean reputation within MICE and nobody has any reason to suspect they are up to anything shady, and they are intending to use modern hardware not some ancient 2000's era hunk o junk. The board can then measure the request and approve or reject the exception request as they deem best for the exchange using their own prerogative. As the request for remote switches is a fairly low volume in the first place and the request for remote switches that would be deployed as a mixed use switch is even lower, I suspect this would not cause an undue burden on the board to handle requests of this nature. The board can easily dismiss frivolous requests, citing the relevant MICE policy. I feel this would provide continuing good network stability as I agree that dedicated hardware would be less susceptible to problems stemming from mixed use conditions but still allowing the best benefit to the exchange as a whole by allowing it to happen with control when it makes sense. The requesting entity still needs to go through the regular process of discussion on the list and what not as well, it would just need the extra step of being pre-screened by the board for shared use fabric switches. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Gary Glissendorf <Gary.Glissendorf(a)SDNCOMMUNICATIONS.COM> Sent: Monday, December 2, 2019 4:27 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches SDN would be in support of a 1 mac per external interface from the a participant connected at the Core or a connected to a Remote switch. A standard practice is to isolate any broadcast chances leaving a providers network put placing a point-point subnet or point-point VLAN on the interfaces directly leaving the network. But in the case of connecting to a Exchange and the need to support connectivity into a L2 topology the opportunities is broadcast loops/storms does increase. SDN was a early participant in the Exchange and I recall outages that we started minimizing traffic that was traversing the exchange. In the recent years the exchange has become a important source for content and keeping the network stable and secure is a benefit to all of the participants. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D27592.E0CCAA30] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D27592.E0CCAA30] <http://twitter.com/sdncomm> [cid:image003.jpg@01D27592.E0CCAA30] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D27592.E0CCAA30] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D27592.E0CCAA30] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, December 2, 2019 4:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders. One reason I am very much in favor of dedicated hardware is isolation of the unknown. As any seasoned network professional knows, things rarely fail in a way that you expect. The more complicated things are, the more likely that unintended things can happen, especially in the case of a failure, and the more functions a piece of equipment is doing the more widespread the failure. I think those of us that have been on MICE for some time can even remember some examples. The one that comes to mind is when the MICE core was 100% Juniper. We would have IX wide outages caused by the Juniper switches flooding everyone’s ports to capacity with other member’s unicast traffic. This was obviously not a feature, user error, or a setting in the switch. On at least one occasion a software update was done that specifically stated it fixed the bug, and unfortunately it did not. Some of these Juniper switches are still in operation in the MICE core for the 1G ports. Since the switches were connected with proprietary Juniper stacking cables, they acted as one amplifying the problem. The switches still have the unicast flooding problem 1-2 times per year. The difference is that it is isolated to those 1G members, so it is not as noticeable since it is isolated, and no longer impacting the majority of members on the IX. We see that the larger networks/CDNs that connect to MICE always request as much diversity as possible whether it be on separate linecards, or switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Ben Wiechman Sent: Monday, December 02, 2019 2:18 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches My concern would be protecting the integrity of the exchange. Just because there have been no issues, doesn't mean there will be no issues at some point. This doesn't necessarily require dedicated hardware, nor does dedicated hardware necessarily guarantee this. A request to disclose practices in place today sent to those that currently maintain shared extension switches would be a good starting point in achieving a better understanding of the potential risks and help identify potential best practices that are already in place in addition to the suggestions already offered by Steve. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2farvig.com&c=E,1,Y6eUwUEC…> On Mon, Dec 2, 2019 at 12:09 PM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: In my opinion the single MAC address per participant facing MICE is the most important requirement for the protection of MICE. I feel it is so important that I had contacted the board/received approval so that I could restrict my extension switch to 1 MAC address per participant regardless of if MICE requires it or not. I think short term exceptions for maintenance/upgrades are fine. I feel that trying to implement the 1 MAC address rule on a shared extension switch would be very problematic (mostly for other traffic that could be going across the same switch). I do not feel that the added cost/complexity of requiring a dedicated extension switch would deter any network from connecting. As an extension switch operator that offers multiple different services I offer MICE completely for free to anyone with no requirements that they connect to anything other than MICE. With this being said they are responsible to getting one connection to me, however once I receive that connection all services are broken out onto separate dedicated hardware free of charge in a manner that would satisfy any IX that allows a connection to anything other than their own core switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Steve Howard Sent: Monday, December 02, 2019 10:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches I am not in favor of requiring MICE remote switches to be dedicated solely to MICE. I think the proposed policy adds unnecessary complexity and costs (one-time hardware, monthly cross-connects, space at 511, and environmental) for remote switch operators. It also discourages additional membership (WiscNet) and, in my opinion, results in very little real value to MICE. Why institute a policy with real costs to MICE members and not much real value? Sometimes the MICE community seems to worship larger exchanges, most notably, SIX and AMS-IX. MICE should make decisions based upon what is best for MICE. Just because the “big” exchanges do something doesn’t necessarily mean it is best for MICE. We are not SIX. We are not AMS-IX. Sometimes it is better to be a leader at not a follower! I’d rather not have unnecessary rules and burdens. I’m not exactly sure what real problem this proposed rule is trying to solve. But, if there truly is a technical problem here that needs to be solved, I’d like to propose a couple of suggestions for discussion that could help protect the exchange: 1) If supported by the remote switch, enforce a specific MAC address requirement on the MICE VLAN for remote switches. Typically this would be one MAC address on the MICE VLAN, but there are a few cases where an additional MAC would be required. This could create some challenges when changing routers, but, in many cases, these can be resolved with some prior planning. For example: Paul Bunyan uses a switched virtual interface (a Cisco BVI) for its connection to SIX. This gives us the flexibility to move that interface to any device without having to contact people at SIX. Yet, it still gives SIX the protection of a single MAC address per port. MICE could also have a policy to briefly allow an extra MAC address when there is planned maintenance. 2) MICE could institute a “probationary” status for remote switches that have a history of problems at MICE. MICE has had non-dedicated remote switches from its beginning. To the best of my knowledge, this has never caused a problem. However, we’ve all seen the recent history of a single MICE remote switch operator that has had issues with packet loss, apparently slow downtime responses, etc. Fortunately, those problems have only affected their customers. Remote switch operators that have a history of poor performance could have additional requirements placed upon them until they’ve operated well for a period of time. Additionally, if a remote switch operator’s non-dedicated use causes problems at MICE, the remote switch operator could lose the privilege of having a non-dedicated switch. 3) MICE could determine an appropriate level of broadcast traffic and institute broadscast storm control based upon a multiplier of of the typical levels. Disclaimers: I work for Paul Bunyan Communications which connects to MICE via the CNS remote switch; Paul Bunyan is a founder and part-owner of CNS; I help manage the CNS remote switch; I am one of the co-founders of MICE. I try to be loyal to Paul Bunyan, CNS, and MICE. It gets complicated sometimes! ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 3, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Gary Glissendorf

SDN would be in support of a 1 mac per external interface from the a participant connected at the Core or a connected to a Remote switch. A standard practice is to isolate any broadcast chances leaving a providers network put placing a point-point subnet or point-point VLAN on the interfaces directly leaving the network. But in the case of connecting to a Exchange and the need to support connectivity into a L2 topology the opportunities is broadcast loops/storms does increase. SDN was a early participant in the Exchange and I recall outages that we started minimizing traffic that was traversing the exchange. In the recent years the exchange has become a important source for content and keeping the network stable and secure is a benefit to all of the participants. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D27592.E0CCAA30] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D27592.E0CCAA30] <http://twitter.com/sdncomm> [cid:image003.jpg@01D27592.E0CCAA30] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D27592.E0CCAA30] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D27592.E0CCAA30] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, December 2, 2019 4:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders. One reason I am very much in favor of dedicated hardware is isolation of the unknown. As any seasoned network professional knows, things rarely fail in a way that you expect. The more complicated things are, the more likely that unintended things can happen, especially in the case of a failure, and the more functions a piece of equipment is doing the more widespread the failure. I think those of us that have been on MICE for some time can even remember some examples. The one that comes to mind is when the MICE core was 100% Juniper. We would have IX wide outages caused by the Juniper switches flooding everyone’s ports to capacity with other member’s unicast traffic. This was obviously not a feature, user error, or a setting in the switch. On at least one occasion a software update was done that specifically stated it fixed the bug, and unfortunately it did not. Some of these Juniper switches are still in operation in the MICE core for the 1G ports. Since the switches were connected with proprietary Juniper stacking cables, they acted as one amplifying the problem. The switches still have the unicast flooding problem 1-2 times per year. The difference is that it is isolated to those 1G members, so it is not as noticeable since it is isolated, and no longer impacting the majority of members on the IX. We see that the larger networks/CDNs that connect to MICE always request as much diversity as possible whether it be on separate linecards, or switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Ben Wiechman Sent: Monday, December 02, 2019 2:18 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches My concern would be protecting the integrity of the exchange. Just because there have been no issues, doesn't mean there will be no issues at some point. This doesn't necessarily require dedicated hardware, nor does dedicated hardware necessarily guarantee this. A request to disclose practices in place today sent to those that currently maintain shared extension switches would be a good starting point in achieving a better understanding of the potential risks and help identify potential best practices that are already in place in addition to the suggestions already offered by Steve. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2farvig.com&c=E,1,Y6eUwUEC…> On Mon, Dec 2, 2019 at 12:09 PM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: In my opinion the single MAC address per participant facing MICE is the most important requirement for the protection of MICE. I feel it is so important that I had contacted the board/received approval so that I could restrict my extension switch to 1 MAC address per participant regardless of if MICE requires it or not. I think short term exceptions for maintenance/upgrades are fine. I feel that trying to implement the 1 MAC address rule on a shared extension switch would be very problematic (mostly for other traffic that could be going across the same switch). I do not feel that the added cost/complexity of requiring a dedicated extension switch would deter any network from connecting. As an extension switch operator that offers multiple different services I offer MICE completely for free to anyone with no requirements that they connect to anything other than MICE. With this being said they are responsible to getting one connection to me, however once I receive that connection all services are broken out onto separate dedicated hardware free of charge in a manner that would satisfy any IX that allows a connection to anything other than their own core switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Steve Howard Sent: Monday, December 02, 2019 10:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches I am not in favor of requiring MICE remote switches to be dedicated solely to MICE. I think the proposed policy adds unnecessary complexity and costs (one-time hardware, monthly cross-connects, space at 511, and environmental) for remote switch operators. It also discourages additional membership (WiscNet) and, in my opinion, results in very little real value to MICE. Why institute a policy with real costs to MICE members and not much real value? Sometimes the MICE community seems to worship larger exchanges, most notably, SIX and AMS-IX. MICE should make decisions based upon what is best for MICE. Just because the “big” exchanges do something doesn’t necessarily mean it is best for MICE. We are not SIX. We are not AMS-IX. Sometimes it is better to be a leader at not a follower! I’d rather not have unnecessary rules and burdens. I’m not exactly sure what real problem this proposed rule is trying to solve. But, if there truly is a technical problem here that needs to be solved, I’d like to propose a couple of suggestions for discussion that could help protect the exchange: 1) If supported by the remote switch, enforce a specific MAC address requirement on the MICE VLAN for remote switches. Typically this would be one MAC address on the MICE VLAN, but there are a few cases where an additional MAC would be required. This could create some challenges when changing routers, but, in many cases, these can be resolved with some prior planning. For example: Paul Bunyan uses a switched virtual interface (a Cisco BVI) for its connection to SIX. This gives us the flexibility to move that interface to any device without having to contact people at SIX. Yet, it still gives SIX the protection of a single MAC address per port. MICE could also have a policy to briefly allow an extra MAC address when there is planned maintenance. 2) MICE could institute a “probationary” status for remote switches that have a history of problems at MICE. MICE has had non-dedicated remote switches from its beginning. To the best of my knowledge, this has never caused a problem. However, we’ve all seen the recent history of a single MICE remote switch operator that has had issues with packet loss, apparently slow downtime responses, etc. Fortunately, those problems have only affected their customers. Remote switch operators that have a history of poor performance could have additional requirements placed upon them until they’ve operated well for a period of time. Additionally, if a remote switch operator’s non-dedicated use causes problems at MICE, the remote switch operator could lose the privilege of having a non-dedicated switch. 3) MICE could determine an appropriate level of broadcast traffic and institute broadscast storm control based upon a multiplier of of the typical levels. Disclaimers: I work for Paul Bunyan Communications which connects to MICE via the CNS remote switch; Paul Bunyan is a founder and part-owner of CNS; I help manage the CNS remote switch; I am one of the co-founders of MICE. I try to be loyal to Paul Bunyan, CNS, and MICE. It gets complicated sometimes! ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.iphouse.net%2fcgi-…> ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

Dec. 2, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Andrew Hoyos

I’d argue that dedicated hardware could fail in an unknown way, just the same as a shared device. To the "MICE Core", the action to fix it would hopefully be the same - sever the connection to the downstream switch and isolate the failure area (whether automated - pps limits for bcast/mcast traffic, mac limits, etc, or manual port shutdown). MICE should enforce certain standards that ensure remote switch operators are following any requirements - same if dedicated hardware or shared hardware. — Andrew Hoyos hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > On Dec 2, 2019, at 4:13 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > One reason I am very much in favor of dedicated hardware is isolation of the unknown. As any seasoned network professional knows, things rarely fail in a way that you expect. The more complicated things are, the more likely that unintended things can happen, especially in the case of a failure, and the more functions a piece of equipment is doing the more widespread the failure. > > I think those of us that have been on MICE for some time can even remember some examples. The one that comes to mind is when the MICE core was 100% Juniper. We would have IX wide outages caused by the Juniper switches flooding everyone’s ports to capacity with other member’s unicast traffic. This was obviously not a feature, user error, or a setting in the switch. On at least one occasion a software update was done that specifically stated it fixed the bug, and unfortunately it did not. Some of these Juniper switches are still in operation in the MICE core for the 1G ports. Since the switches were connected with proprietary Juniper stacking cables, they acted as one amplifying the problem. The switches still have the unicast flooding problem 1-2 times per year. The difference is that it is isolated to those 1G members, so it is not as noticeable since it is isolated, and no longer impacting the majority of members on the IX. We see that the larger networks/CDNs that connect to MICE always request as much diversity as possible whether it be on separate linecards, or switches. > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Ben Wiechman > Sent: Monday, December 02, 2019 2:18 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches > > My concern would be protecting the integrity of the exchange. Just because there have been no issues, doesn't mean there will be no issues at some point. This doesn't necessarily require dedicated hardware, nor does dedicated hardware necessarily guarantee this. A request to disclose practices in place today sent to those that currently maintain shared extension switches would be a good starting point in achieving a better understanding of the potential risks and help identify potential best practices that are already in place in addition to the suggestions already offered by Steve. > > Ben Wiechman > Director of IP Strategy and Engineering > 320.247.3224 | ben.wiechman(a)arvig.com <mailto:ben.wiechman@arvig.com> > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com <http://arvig.com/> > > > > On Mon, Dec 2, 2019 at 12:09 PM Jeremy Lumby <jlumby(a)mnvoip.com <mailto:jlumby@mnvoip.com>> wrote: > In my opinion the single MAC address per participant facing MICE is the most important requirement for the protection of MICE. I feel it is so important that I had contacted the board/received approval so that I could restrict my extension switch to 1 MAC address per participant regardless of if MICE requires it or not. I think short term exceptions for maintenance/upgrades are fine. I feel that trying to implement the 1 MAC address rule on a shared extension switch would be very problematic (mostly for other traffic that could be going across the same switch). > > I do not feel that the added cost/complexity of requiring a dedicated extension switch would deter any network from connecting. As an extension switch operator that offers multiple different services I offer MICE completely for free to anyone with no requirements that they connect to anything other than MICE. With this being said they are responsible to getting one connection to me, however once I receive that connection all services are broken out onto separate dedicated hardware free of charge in a manner that would satisfy any IX that allows a connection to anything other than their own core switches. > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Steve Howard > Sent: Monday, December 02, 2019 10:31 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches > > I am not in favor of requiring MICE remote switches to be dedicated solely to MICE. > > I think the proposed policy adds unnecessary complexity and costs (one-time hardware, monthly cross-connects, space at 511, and environmental) for remote switch operators. It also discourages additional membership (WiscNet) and, in my opinion, results in very little real value to MICE. Why institute a policy with real costs to MICE members and not much real value? > > Sometimes the MICE community seems to worship larger exchanges, most notably, SIX and AMS-IX. MICE should make decisions based upon what is best for MICE. Just because the “big” exchanges do something doesn’t necessarily mean it is best for MICE. We are not SIX. We are not AMS-IX. Sometimes it is better to be a leader at not a follower! > > > I’d rather not have unnecessary rules and burdens. I’m not exactly sure what real problem this proposed rule is trying to solve. But, if there truly is a technical problem here that needs to be solved, I’d like to propose a couple of suggestions for discussion that could help protect the exchange: > > 1) If supported by the remote switch, enforce a specific MAC address requirement on the MICE VLAN for remote switches. Typically this would be one MAC address on the MICE VLAN, but there are a few cases where an additional MAC would be required. This could create some challenges when changing routers, but, in many cases, these can be resolved with some prior planning. For example: Paul Bunyan uses a switched virtual interface (a Cisco BVI) for its connection to SIX. This gives us the flexibility to move that interface to any device without having to contact people at SIX. Yet, it still gives SIX the protection of a single MAC address per port. MICE could also have a policy to briefly allow an extra MAC address when there is planned maintenance. > > 2) MICE could institute a “probationary” status for remote switches that have a history of problems at MICE. MICE has had non-dedicated remote switches from its beginning. To the best of my knowledge, this has never caused a problem. However, we’ve all seen the recent history of a single MICE remote switch operator that has had issues with packet loss, apparently slow downtime responses, etc. Fortunately, those problems have only affected their customers. Remote switch operators that have a history of poor performance could have additional requirements placed upon them until they’ve operated well for a period of time. Additionally, if a remote switch operator’s non-dedicated use causes problems at MICE, the remote switch operator could lose the privilege of having a non-dedicated switch. > > 3) MICE could determine an appropriate level of broadcast traffic and institute broadscast storm control based upon a multiplier of of the typical levels. > > Disclaimers: I work for Paul Bunyan Communications which connects to MICE via the CNS remote switch; Paul Bunyan is a founder and part-owner of CNS; I help manage the CNS remote switch; I am one of the co-founders of MICE. I try to be loyal to Paul Bunyan, CNS, and MICE. It gets complicated sometimes! > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

Dec. 2, 2019

Re: WiscNet Remote Switch / Dedicated Remote Switches

by Jeremy Lumby

One reason I am very much in favor of dedicated hardware is isolation of the unknown. As any seasoned network professional knows, things rarely fail in a way that you expect. The more complicated things are, the more likely that unintended things can happen, especially in the case of a failure, and the more functions a piece of equipment is doing the more widespread the failure. I think those of us that have been on MICE for some time can even remember some examples. The one that comes to mind is when the MICE core was 100% Juniper. We would have IX wide outages caused by the Juniper switches flooding everyone’s ports to capacity with other member’s unicast traffic. This was obviously not a feature, user error, or a setting in the switch. On at least one occasion a software update was done that specifically stated it fixed the bug, and unfortunately it did not. Some of these Juniper switches are still in operation in the MICE core for the 1G ports. Since the switches were connected with proprietary Juniper stacking cables, they acted as one amplifying the problem. The switches still have the unicast flooding problem 1-2 times per year. The difference is that it is isolated to those 1G members, so it is not as noticeable since it is isolated, and no longer impacting the majority of members on the IX. We see that the larger networks/CDNs that connect to MICE always request as much diversity as possible whether it be on separate linecards, or switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Ben Wiechman Sent: Monday, December 02, 2019 2:18 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches My concern would be protecting the integrity of the exchange. Just because there have been no issues, doesn't mean there will be no issues at some point. This doesn't necessarily require dedicated hardware, nor does dedicated hardware necessarily guarantee this. A request to disclose practices in place today sent to those that currently maintain shared extension switches would be a good starting point in achieving a better understanding of the potential risks and help identify potential best practices that are already in place in addition to the suggestions already offered by Steve. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Mon, Dec 2, 2019 at 12:09 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: In my opinion the single MAC address per participant facing MICE is the most important requirement for the protection of MICE. I feel it is so important that I had contacted the board/received approval so that I could restrict my extension switch to 1 MAC address per participant regardless of if MICE requires it or not. I think short term exceptions for maintenance/upgrades are fine. I feel that trying to implement the 1 MAC address rule on a shared extension switch would be very problematic (mostly for other traffic that could be going across the same switch). I do not feel that the added cost/complexity of requiring a dedicated extension switch would deter any network from connecting. As an extension switch operator that offers multiple different services I offer MICE completely for free to anyone with no requirements that they connect to anything other than MICE. With this being said they are responsible to getting one connection to me, however once I receive that connection all services are broken out onto separate dedicated hardware free of charge in a manner that would satisfy any IX that allows a connection to anything other than their own core switches. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Monday, December 02, 2019 10:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] WiscNet Remote Switch / Dedicated Remote Switches I am not in favor of requiring MICE remote switches to be dedicated solely to MICE. I think the proposed policy adds unnecessary complexity and costs (one-time hardware, monthly cross-connects, space at 511, and environmental) for remote switch operators. It also discourages additional membership (WiscNet) and, in my opinion, results in very little real value to MICE. Why institute a policy with real costs to MICE members and not much real value? Sometimes the MICE community seems to worship larger exchanges, most notably, SIX and AMS-IX. MICE should make decisions based upon what is best for MICE. Just because the “big” exchanges do something doesn’t necessarily mean it is best for MICE. We are not SIX. We are not AMS-IX. Sometimes it is better to be a leader at not a follower! I’d rather not have unnecessary rules and burdens. I’m not exactly sure what real problem this proposed rule is trying to solve. But, if there truly is a technical problem here that needs to be solved, I’d like to propose a couple of suggestions for discussion that could help protect the exchange: 1) If supported by the remote switch, enforce a specific MAC address requirement on the MICE VLAN for remote switches. Typically this would be one MAC address on the MICE VLAN, but there are a few cases where an additional MAC would be required. This could create some challenges when changing routers, but, in many cases, these can be resolved with some prior planning. For example: Paul Bunyan uses a switched virtual interface (a Cisco BVI) for its connection to SIX. This gives us the flexibility to move that interface to any device without having to contact people at SIX. Yet, it still gives SIX the protection of a single MAC address per port. MICE could also have a policy to briefly allow an extra MAC address when there is planned maintenance. 2) MICE could institute a “probationary” status for remote switches that have a history of problems at MICE. MICE has had non-dedicated remote switches from its beginning. To the best of my knowledge, this has never caused a problem. However, we’ve all seen the recent history of a single MICE remote switch operator that has had issues with packet loss, apparently slow downtime responses, etc. Fortunately, those problems have only affected their customers. Remote switch operators that have a history of poor performance could have additional requirements placed upon them until they’ve operated well for a period of time. Additionally, if a remote switch operator’s non-dedicated use causes problems at MICE, the remote switch operator could lose the privilege of having a non-dedicated switch. 3) MICE could determine an appropriate level of broadcast traffic and institute broadscast storm control based upon a multiplier of of the typical levels. Disclaimers: I work for Paul Bunyan Communications which connects to MICE via the CNS remote switch; Paul Bunyan is a founder and part-owner of CNS; I help manage the CNS remote switch; I am one of the co-founders of MICE. I try to be loyal to Paul Bunyan, CNS, and MICE. It gets complicated sometimes! To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 2, 2019