- mice-discuss - lists.micemn.net

Re: Zayo maintenance

by Richard Laager

On 1/27/23 21:23, Justin Huffman wrote: > Just curious, anyone else affected by a maintenance that Zayo decides to > do starting at 5 pm tonight? I didn't receive anything on it, so hopefully it's not affecting me. Do you have any more details you want to share? -- Richard

Jan. 28, 2023

Zayo maintenance

by Justin Huffman

Just curious, anyone else affected by a maintenance that Zayo decides to do starting at 5 pm tonight? Justin Long Lines Get Outlook for Android<https://aka.ms/AAb9ysg>

Jan. 28, 2023

Re: Any recent MICE network events?

by Jay Hanke

I've gotten multiple requests on 20940. If you have a bilateral session make sure to reach out to the peer directly as well if you haven't already. On Wed, Jan 25, 2023 at 1:34 PM Jeff Wilde <jeff.wilde(a)parkregion.com> wrote: > Jan 23 13:40:22.181 CST: neighbor 206.108.255.58 Down - (AS: 10310) > > Jan 23 13:40:35.995 CST: neighbor 2001:504:27::2846:0:2 Down - (AS: > 10310) > > Jan 24 09:16:07.476 CST: neighbor 206.108.255.84 Down - BGP Notification > sent (AS: 20940) > > Jan 24 11:44:06.080 CST: neighbor 206.108.255.59 Down - Peer closing down > the session (AS: 10310) > > > > [image: A picture containing icon Description automatically generated] > > Jeff Wilde · Network Engineer · Park Region > > Office 218.826.6161 · Direct 218.826.8330 · Fax 218.826.6298 > > > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *David > Farmer > *Sent:* Wednesday, January 25, 2023 12:58 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Any recent MICE network events? > > > > Yes, I see those in our logs too. I have no idea about the cause. > > > > On Wed, Jan 25, 2023 at 12:25 PM Frank Bulk <fbulk(a)mypremieronline.com> > wrote: > > We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, > 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > And then on Tuesday at 11:44 am to about 10 minutes: > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, > 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > Did anyone else see this? Do they share some transport back to Chicago or > something? > > Frank > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Jan. 25, 2023

Re: Any recent MICE network events?

by Jeremy Lumby

The MICE switch saw their links go down, and they do share a common carrier to Chicago. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Wilde Sent: Wednesday, January 25, 2023 1:34 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Any recent MICE network events? Jan 23 13:40:22.181 CST: neighbor 206.108.255.58 Down - (AS: 10310) Jan 23 13:40:35.995 CST: neighbor 2001:504:27::2846:0:2 Down - (AS: 10310) Jan 24 09:16:07.476 CST: neighbor 206.108.255.84 Down - BGP Notification sent (AS: 20940) Jan 24 11:44:06.080 CST: neighbor 206.108.255.59 Down - Peer closing down the session (AS: 10310) Jeff Wilde * Network Engineer * Park Region Office 218.826.6161 * Direct 218.826.8330 * Fax 218.826.6298 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of David Farmer Sent: Wednesday, January 25, 2023 12:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Any recent MICE network events? Yes, I see those in our logs too. I have no idea about the cause. On Wed, Jan 25, 2023 at 12:25 PM Frank Bulk <fbulk(a)mypremieronline.com <mailto:fbulk@mypremieronline.com> > wrote: We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 And then on Tuesday at 11:44 am to about 10 minutes: AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 Did anyone else see this? Do they share some transport back to Chicago or something? Frank -- =============================================== David Farmer Email:farmer@umn.edu <mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Jan. 25, 2023

Re: Any recent MICE network events?

by Jeff Wilde

Jan 23 13:40:22.181 CST: neighbor 206.108.255.58 Down - (AS: 10310) Jan 23 13:40:35.995 CST: neighbor 2001:504:27::2846:0:2 Down - (AS: 10310) Jan 24 09:16:07.476 CST: neighbor 206.108.255.84 Down - BGP Notification sent (AS: 20940) Jan 24 11:44:06.080 CST: neighbor 206.108.255.59 Down - Peer closing down the session (AS: 10310) [A picture containing icon Description automatically generated] Jeff Wilde • Network Engineer • Park Region Office 218.826.6161 • Direct 218.826.8330 • Fax 218.826.6298 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Wednesday, January 25, 2023 12:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Any recent MICE network events? Yes, I see those in our logs too. I have no idea about the cause. On Wed, Jan 25, 2023 at 12:25 PM Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 And then on Tuesday at 11:44 am to about 10 minutes: AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 Did anyone else see this? Do they share some transport back to Chicago or something? Frank -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 25, 2023

Re: Any recent MICE network events?

by August Yang

It seems like administrative change than incident from the notification they sent. > Jan 24 17:44:10.607: %BGP-3-NOTIFICATION: received from neighbor > 206.108.255.59 6/6 (Other Configuration Change) 0 bytes On 2023-01-25 1:58 p.m., David Farmer wrote: > Yes, I see those in our logs too. I have no idea about the cause. > > On Wed, Jan 25, 2023 at 12:25 PM Frank Bulk > <fbulk(a)mypremieronline.com> wrote: > > We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, > 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > And then on Tuesday at 11:44 am to about 10 minutes: > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, > 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > Did anyone else see this? Do they share some transport back to > Chicago or something? > > Frank > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu <mailto:Email%3Afarmer@umn.edu> > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > -- Best regards August Yang

Jan. 25, 2023

Re: Any recent MICE network events?

by David Farmer

Yes, I see those in our logs too. I have no idea about the cause. On Wed, Jan 25, 2023 at 12:25 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, > 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > And then on Tuesday at 11:44 am to about 10 minutes: > AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, > 2001:504:27::2846:0:2 > AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 > > Did anyone else see this? Do they share some transport back to Chicago or > something? > > Frank > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Jan. 25, 2023

Any recent MICE network events?

by Frank Bulk

We saw few peers briefly bounce Monday at 1:40 pm (U.S. Central): AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 And then on Tuesday at 11:44 am to about 10 minutes: AS 10310 (Yahoo): 206.108.255.58, 206.108.255.59, 2001:504:27::2846:0:1, 2001:504:27::2846:0:2 AS 15169 (Google): 206.108.255.167, 2001:504:27::3b41:0:2 Did anyone else see this? Do they share some transport back to Chicago or something? Frank

Jan. 25, 2023

PeeringDB Carrier Object Update

by Jay Hanke

There was a pretty big upgrade to PeeringDB this morning that allows networks to load locations where they can sell transport services. This is what $DAYJOB record looks like: https://www.peeringdb.com/carrier/6 You submit a request that needs to be approved by the facility owner before it goes live. I'd recommend if you sell transport services into buildings with Internet Exchanges to add your records. It will make it a lot easier for networks to find transport into MICE (or other) IXes. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 https://southfront.io/path.html

Jan. 25, 2023

Re: Please welcome Ausra LLC to MICE!

by Doug McIntyre

On Thu, Jan 19, 2023 at 08:15:27PM -0600, Jeremy Lumby wrote: >Name: Ausra LLC >Switch: NOCIX >Ports: E 5/6 >ASN: 400818 >IPv4: 206.108.255.205/24 >IPv6: 2001:504:27:0:6:1DB2::1/64 >Peering Contact: noc(a)ausra.cloud >Route Servers: YES >BFD: YES I've added them into the RR's. Welcome!

Jan. 20, 2023

Please welcome Ausra LLC to MICE!

by Jeremy Lumby

Name: Ausra LLC Switch: NOCIX Ports: E 5/6 ASN: 400818 IPv4: 206.108.255.205/24 IPv6: 2001:504:27:0:6:1DB2::1/64 Peering Contact: noc(a)ausra.cloud Route Servers: YES BFD: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 20, 2023

Please welcome Hiawatha Broadband Communications to MICE!

by Jeremy Lumby

Name: Hiawatha Broadband Communications Switch: MICE Core Port: E6/30/1 ASN: 14828 IPv4: IPs remain the same. This is a port upgrade to 100G IPv6: Peering Contact: noc(a)exchange.hbci.com Route Servers: YES BFD: NO Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 17, 2023

Re: Please welcome Minnesota Brain Injury Alliance to MICE!

by Doug McIntyre

On Thu, Jan 12, 2023 at 09:50:08AM -0600, Jeremy Lumby wrote: >Name: Minnesota Brain Injury Alliance >Switch: MN VoIP >Ports: E 36 >ASN: 19418 >IPv4: None (This member is IPv6 only) >IPv6: 2001:504:27:0:0:4BDA::1/64 >Peering Contact: peering(a)braininjurymn.org >Route Servers: YES >BFD: NO Added to the RRs. Welcome!

Jan. 12, 2023

Please welcome Minnesota Brain Injury Alliance to MICE!

by Jeremy Lumby

Name: Minnesota Brain Injury Alliance Switch: MN VoIP Ports: E 36 ASN: 19418 IPv4: None (This member is IPv6 only) IPv6: 2001:504:27:0:0:4BDA::1/64 Peering Contact: peering(a)braininjurymn.org Route Servers: YES BFD: NO Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 12, 2023

Please welcome ImOn Communications to MICE!

by Jeremy Lumby

Name: ImOn Communications Switch: MN VoIP Port: Adding E37 and E38 to Po5 for a total of 4x10G ASN: 393238 IPv4: This is a port addition. All IP info remains the same. IPv6: Peering Contact: imonadmins(a)imon.net Route Servers: Yes BFD: Yes Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Jan. 8, 2023

Google Emergency Maintenance Notification PCR/648233 (MICE)

by Jay Hanke

Details: Start Time: 2023-01-06 20:55:00 +0000 UTC End Time: 2023-01-07 05:05:00 +0000 UTC Duration: 8h10m0s Google ASN:15169Peer ASN:53679 Google Neighbor Address(es):2001:504:27::3b41:0:2Peer Neighbor Address(es):2001:504:27::d1af:0:1 Google Neighbor Address(es):2001:504:27::3b41:0:2Peer Neighbor Address(es):2001:504:27::d1af:0:2 Google Neighbor Address(es):206.108.255.167Peer Neighbor Address(es):206.108.255.1 Google Neighbor Address(es):206.108.255.167Peer Neighbor Address(es):206.108.255.2 Google Neighbor Address(es):2001:504:27::3b41:0:1Peer Neighbor Address(es):2001:504:27::d1af:0:1 Google Neighbor Address(es):2001:504:27::3b41:0:1Peer Neighbor Address(es):2001:504:27::d1af:0:2 Google Neighbor Address(es):206.108.255.141Peer Neighbor Address(es):206.108.255.1 Google Neighbor Address(es):206.108.255.141Peer Neighbor Address(es):206.108.255.2 Google systems will automatically re-route the traffic away from the peering link(s) under maintenance. Depending on the nature of the work, your BGP session with Google may stay up through the maintenance window. Traffic may shift to alternate sessions with Google and/or your upstream provider(s). Questions about this event can be sent to noc(a)google.com.

Jan. 6, 2023

MICE UG42 Reminder

by AnthonyAnderberg＠nuvera.net

I’m sure everyone saw David Farmer’s meeting invite, attached is the agenda. Let me know if you have any questions, Anthony

Jan. 4, 2023

Re: Please welcome INCX Global, LLC to MICE!

by Doug McIntyre

On Mon, Jan 02, 2023 at 04:23:05PM -0600, Jeremy Lumby wrote: >Name: INCX Global, LLC >Switch: NOCIX >Ports: 5/4 & 5/5 in Portchannel 100 >ASN: 13737 >IPv4: 206.108.255.204/24 >IPv6: 2001:504:27:0:0:35A9::1/64 >Peering Contact: sbutler(a)incx.net >Route Servers: YES >BFD: NO I've added them into the RR's. Welcome!

Jan. 3, 2023

Please welcome INCX Global, LLC to MICE!

by Jeremy Lumby

Name: INCX Global, LLC Switch: NOCIX Ports: 5/4 & 5/5 in Portchannel 100 ASN: 13737 IPv4: 206.108.255.204/24 IPv6: 2001:504:27:0:0:35A9::1/64 Peering Contact: sbutler(a)incx.net Route Servers: YES BFD: NO Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Jan. 2, 2023

MICE UG 42 Meeting Information

by David Farmer

MICE UG 42 will be held on January 10th, 3:00 PM, at the University’s Information Technology Building, 2218 University Ave SE, Minneapolis, MN 55414, Room 185. It is across University Ave SE from Huntington Bank Stadium (Formerly TCF Bank Stadium). See the link to the Campus Map below. This room has a video room system, and remote access will be available via Zoom video or phone audio, see Zoom details below. Please enter the building through the middle of three doors on the Ontario St SE side of the building. It will be open after 2:30 and will have a sign on it. If not, please knock hard, or call my cell phone (612-812-9952). This door is outside the meeting room, preventing everyone from traipsing through the building. The Stadium Village Green Line Light Rail Station is less than a block east of the building, kitty-corner at the intersection of University Ave SE and Huron Blvd SE. See the link below for the station's info page. Limited Street Parking is available on Ontario St SE and Beacon St SE. Otherwise, public parking is available in the Oak St and University Ave Ramps, each a little over two blocks away. Also, the Maroon lot is a little further on the Stadium's north side. See the link below for more parking details. DO NOT park in University Parking Lot 70 on the north of the building, by the front door, between University Ave SE and the building, or Parking Lot 71 near the loading dock to the east of the building, they are contract parking only, and you be ticketed. Also, DO NOT park in the private parking lot to the south of the building; you will likely be tagged, booted, and towed from that parking lot. https://campusmaps.umn.edu/information-technology https://www.metrotransit.org/stadium-village-station https://pts.umn.edu/Park/Basic-Parking-Information/Parking-Facilities ---------- David E Farmer III is inviting you to a scheduled Zoom meeting. Topic: MICE UG 42 Time: Jan 10, 2023 03:00 PM Central Time (US and Canada) Join Zoom Meeting https://umn.zoom.us/j/92302900851?pwd=azZqR08xMzZhUDVzY3BiVmFNT3Y4QT09 Meeting ID: 923 0290 0851 Passcode: JacR0t One tap mobile +16513728299,,92302900851#,,,,*876480# US (Minnesota) +13017158592,,92302900851#,,,,*876480# US (Washington DC) Dial by your location +1 651 372 8299 US (Minnesota) +1 301 715 8592 US (Washington DC) +1 312 626 6799 US (Chicago) +1 646 558 8656 US (New York) +1 253 215 8782 US (Tacoma) +1 346 248 7799 US (Houston) +1 669 900 6833 US (San Jose) Meeting ID: 923 0290 0851 Passcode: 876480 Find your local number: https://umn.zoom.us/u/a2FCO8ubU Join by SIP 92302900851(a)zoomcrc.com Join by H.323 162.255.37.11 (US West) 162.255.36.11 (US East) 221.122.88.195 (China) 115.114.131.7 (India Mumbai) 115.114.115.7 (India Hyderabad) 213.19.144.110 (Amsterdam Netherlands) 213.244.140.110 (Germany) 103.122.166.55 (Australia Sydney) 103.122.167.55 (Australia Melbourne) 209.9.211.110 (Hong Kong SAR) 149.137.40.110 (Singapore) 64.211.144.160 (Brazil) 69.174.57.160 (Canada Toronto) 65.39.152.160 (Canada Vancouver) 207.226.132.110 (Japan Tokyo) 149.137.24.110 (Japan Osaka) Meeting ID: 923 0290 0851 Passcode: 876480 -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Jan. 1, 2023

Re: Please welcome August Internet Limited to MICE!

by Doug McIntyre

On Wed, Dec 21, 2022 at 08:51:00PM -0600, Jeremy Lumby wrote: >Name: August Internet Limited >Switch: NOCIX >Port: E 5/2 >ASN: 50058 >IPv4: 206.108.255.203/24 >IPv6: 2001:504:27:0:0:C38A::1/64 >Peering Contact: peering(a)august.tw >Route Servers: YES >BFD: YES Added into the RR's. Welcome!

Dec. 22, 2022

Please welcome August Internet Limited to MICE!

by Jeremy Lumby

Name: August Internet Limited Switch: NOCIX Port: E 5/2 ASN: 50058 IPv4: 206.108.255.203/24 IPv6: 2001:504:27:0:0:C38A::1/64 Peering Contact: peering(a)august.tw Route Servers: YES BFD: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Dec. 22, 2022

Please welcome KC Fiber to MICE!

by Jeremy Lumby

Name: KC Fiber Switch: NOCIX Port: Ethernet 5/3 ASN: 11708 IPv4: This is a port/switch move, all IP info remains the same. IPv6: Peering Contact: peering(a)wholesaleinternet.net Route Servers: Yes BFD: No Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Dec. 22, 2022

Please welcome Wholesale Internet to MICE!

by Jeremy Lumby

Name: Wholesale Internet Switch: NOCIX Port: Ethernet 5/1 ASN: 32097 IPv4: This is a port move, all IP info remains the same. IPv6: Peering Contact: peering(a)wholesaleinternet.net Route Servers: Yes BFD: No Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Dec. 21, 2022

Please welcome Eastern Iowa Data Group to MICE!

by Jeremy Lumby

Name: Eastern Iowa Data Group Switch: Southfront Albert Lea Port: Ethernet 8/2 ASN: 63394 IPv4: This is a port addition to 3x10G. All IP info remains the same. IPv6: Peering Contact: EIDG(a)Vantagepnt.com Route Servers: No BFD: Yes Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Dec. 18, 2022

Please welcome OmniTel Communications to MICE!

by Jeremy Lumby

Name: OnmiTel Communications Switch: Southfront Albert Lea Port: Ethernet 10/1 ASN: 62886 IPv4: This is a port upgrade to 100G. All IP info remains the same. IPv6: Peering Contact: it(a)omnitel.biz Route Servers: Yes BFD: Yes Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Dec. 18, 2022

Re: Please welcome UnitedHealth Group to MICE!

by David Farmer

Do you know when UnitedHelath will be active? Thanks On Sat, Aug 6, 2022 at 5:15 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > UnitedHealth Group is being converted to 2 individual 100G links with > separate IPs instead of a single port channel. > > Name: UnitedHealth Group > Switch: MICE Main > Ports: 3/28/1 & 6/24/1 > ASN: 10879 > IPv4: 206.108.255.198/24 and 206.108.255.200/24 > IPv6: 2001:504:27:0:0:2A7F::1/64 and 2001:504:27:0:0:2A7F::2/64 > Peering Contact: inet_ops(a)uhc.com > Route Servers: YES > BFD: YES > > Welcome Aboard Again! > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > Main: 612-355-7740 > Direct: 612-392-6814 > Fax: 952-873-7425 > jlumby(a)mnvoip.com > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Dec. 12, 2022

Google Peering

by Tom Krenn

I just wanted to give a heads up to anyone else who might also have been waiting. I had a peering session with Google approved via their ISP portal yesterday. So after a couple years it looks like they have their new system in place and appear to be accepting requests again. Good luck! Tom Krenn Network Architect Enterprise Architecture - Information Technology Office: 612-596-8994 tom.krenn(a)hennepin.us<mailto:tom.krenn@hennepin.us> | hennepin.us<https://hennepin.us/> Hennepin County Government Center 300 South 6th Street, A-190 Minneapolis, MN, 55487 [Hennepin County logo] Connect: Facebook<https://www.facebook.com/hennepin> | Twitter<https://twitter.com/Hennepin> | YouTube<https://www.youtube.com/user/HennepinMN> | LinkedIn<https://www.linkedin.com/company/hennepin-county> | Instagram<https://www.instagram.com/hennepincounty> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

Nov. 25, 2022

Please welcome Express Data Solutions LLC to MICE!

by Jeremy Lumby

Name: Express Data Solutions LLC Switch: MN VoIP Port: Ethernet 6 ASN: 31834 IPv4: This is a switch/port move. All IP info remains the same. IPv6: Peering Contact: peering(a)express-ds.com Route Servers: Yes BFD: Yes Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Nov. 24, 2022

Please welcome Qwilt Inc. to MICE!

by Jeremy Lumby

Name: Qwilt Inc. Switch: MICE Main Ports: 6/29/1 ASN: 20253 IPv4: 206.108.255.202/24 IPv6: 2001:504:27:0:0:4F1D::1/64 Peering Contact: peering(a)qwilt.com Route Servers: YES BFD: NO Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Nov. 24, 2022

Please welcome Truestream to MICE!

by Jeremy Lumby

Name: Truestream Switch: MICE Core Port: E 6/28/1 ASN: 398143 IPv4: All IP info remains the same, this is just a port upgrade from 10G to 100G IPv6: Peering Contact: peering(a)truestreamfiber.com Route Servers: No BFD: No Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Nov. 21, 2022

Re: Anyone lose peering with CloudFlare this morning at 4:13 am?

by David Farmer

Just so you know, Cloudflare is back as of Friday. Thanks On Mon, Oct 31, 2022 at 2:55 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I have been told that they have replaced the same Juniper switch two > different times, and the two different replacements keep showing the same > errors as the original. This is corroborated with their MICE link coming > up briefly twice since they originally rerouted. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *David Farmer > *Sent:* Monday, October 31, 2022 2:48 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > Any word from Cloudflare? They will have been in re-routed mode for three > weeks on Wednesday. > > > > They have theoretically completed two maintenance events since then; > > > > https://www.cloudflarestatus.com/incidents/txnkvw6frp25 > > https://www.cloudflarestatus.com/incidents/pqwsjx82wdt2 > > > > Thanks > > > > > > On Thu, Oct 20, 2022 at 11:08 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > Yes, the last time they had a failure in Minneapolis it took them just shy > of 2 weeks to get back online. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Steve Savoy > *Sent:* Thursday, October 20, 2022 10:39 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > I am pretty new to the CloudFlare peering. Anyone else go through the > “Re-routed” scenario before? Will it eventually become “Operational”? > > > > Thank you, > > Steve > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *David > Farmer > *Sent:* Wednesday, October 12, 2022 11:59 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > If you check Cloudflare’s status page you will see Minneapolis is in > “Re-routed” state. > > > > https://www.cloudflarestatus.com > > > > > > On Wed, Oct 12, 2022 at 10:45 Steve Savoy <stevesavoy(a)wctatel.com> wrote: > > Yes – ours went down at the same time. > > > > Steve > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Wednesday, October 12, 2022 10:42 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > Both IPv4 and IPv6. I didn’t see any exceeding of prefix limits. > > > > Frank > > > > Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) > SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 206.108.255.45 ( > AS13335.micemn.net) (AS 13335 CLOUDFLARENET, US) old state Established > event HoldTime new state Idle > > Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) > SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::3417:0:1 > (AS13335.micemn.net) (AS 13335 CLOUDFLARENET, US) old state Established > event HoldTime new state Idle > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > Confidentiality Notice: This correspondence is the property of Winnebago > Cooperative Telecom Association and is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply email > and destroy all copies of the original message. > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Nov. 14, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Tom Krenn

Fair enough. You have always been fast with everything from what I've seen. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, November 3, 2022 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Honestly I think the amount of work is minimal. I say that as the person that will probably be doing most of the work. There is even more that we should be doing to keep the IX clean and stable (like adding a quarantine VLAN to the turn-up process) which will add even more labor to the turn-up process. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Thursday, November 03, 2022 8:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard(a)paulbunyan.net<mailto:showard@paulbunyan.net>> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…> > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

Nov. 3, 2022

Re: Proposal: MICE Static MAC address Only

by Jeremy Lumby

I do not know that there is an official scope. I think things are just in the discussion stage. I do believe we all know what needs to be done, it is just about being motivated to get it all done (in this case a switch issue pushed it back to the front). There are plenty of IXes that have laid out and implemented best practices that have been proven in reality to work for them. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Chris Wopat Sent: Thursday, November 03, 2022 9:21 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Could someone confirm the larger scope of this? Our remote switch indeed has static mac ACLs. Unsure how others work. If the goal here is to keep a static mac list on all access ports AND remote switch ports (so my list must be replicated to the core switch)? If so, lets be sure to consider operational issues. I'm unsure how mac ACLs work on arista, but is there a risk to block all mac addresses while the changes is being made if done incorrectly? (think the classic IOS 'switchport trunk allowed vlan 123..')? If this is being done by humans, hoping the procedure is documented and repeatable (a copy paste to do it). Or better yet, but our machine overlords (via NAPALM?) Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Nov. 3, 2022

Re: Proposal: MICE Static MAC address Only

by Chris Wopat

Could someone confirm the larger scope of this? Our remote switch indeed has static mac ACLs. Unsure how others work. If the goal here is to keep a static mac list on all access ports AND remote switch ports (so my list must be replicated to the core switch)? If so, lets be sure to consider operational issues. I'm unsure how mac ACLs work on arista, but is there a risk to block all mac addresses while the changes is being made if done incorrectly? (think the classic IOS 'switchport trunk allowed vlan 123..')? If this is being done by humans, hoping the procedure is documented and repeatable (a copy paste to do it). Or better yet, but our machine overlords (via NAPALM?) Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Nov. 3, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Jeremy Lumby

Honestly I think the amount of work is minimal. I say that as the person that will probably be doing most of the work. There is even more that we should be doing to keep the IX clean and stable (like adding a quarantine VLAN to the turn-up process) which will add even more labor to the turn-up process. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Thursday, November 03, 2022 8:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard(a)paulbunyan.net> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: >https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i… <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…> > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: >https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho… <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Nov. 3, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Tom Krenn

Depends on the goal right? Stopping the 2 seconds of damage or recovering from 2 seconds of damage. (Ideally probably both.) But just throwing ideas out given the concerns being brought up with a static MAC ACL. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 2, 2022 6:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard(a)paulbunyan.net<mailto:showard@paulbunyan.net>> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…> > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

Nov. 3, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Jeremy Lumby

I think the way to think about it was the damage was done in 2 seconds (according to the syslog), however the switch did not automatically recover from those 2 seconds. So we need a method that will react quicker than 2 seconds like a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Tom Krenn Sent: Wednesday, November 02, 2022 4:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard(a)paulbunyan.net> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: >https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i… <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…> > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: >https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho… <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Nov. 2, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Jeremy Lumby

That was a direct quote from Arista Support. Based on the context that it was used in I would say that means that the forwarding table stopped functioning and the default fallback is to flood everything. They recommended a software update since there were some forwarding table issues fixed in more recent versions, however they could not 100% guarantee the software update would provide complete protection. Arista Support pointed out how destabilizing a loop can be @ 100Gig even if it is only for the 2 seconds that we experienced before port security kicked in (especially with how much traffic an IX is carrying to flow through the loop). When talking with Chris @ the SIX he mentioned that they went to the static ACL method because the port security runs on the control plane, so it will always be slow. A MAC ACL runs on the ASIC so it is about as instantaneous as you are going to get. The port that caused the issue last week had a 1 MAC address limit on it through port security, and not through a MAC ACL. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Wednesday, November 02, 2022 4:59 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only What does "crashed the forwarding table" actually mean? Does that mean that the mac address table filled and the switch then flooded to all ports? Was there a mac address limit or a static MAC ACL on the port that caused the problem? On 10/31/22 09:12, Jeremy Lumby wrote: The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remoteswitches, of course. The remote switches are responsible for enforcingthe current MAC limit, and would be responsible for changing thosesettings as described here." -Brady --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From: MICE Discuss<MICE-DISCUSS(a)LISTS.IPHOUSE.NET> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jay Hanke<jayhanke(a)SOUTHFRONT.IO> <jayhanke(a)SOUTHFRONT.IO> Sent: Monday, October 31, 2022 9:05:08 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET<MICE-DISCUSS(a)LISTS.IPHOUSE.NET> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard<showard(a)paulbunyan.net> <showard(a)paulbunyan.net> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: >https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i… > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: >https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho… -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Nov. 2, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Steve Howard

Nov. 2, 2022

Re: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only

by Tom Krenn

I don't know the config of the switch but out of curiosity, would storm control help mitigate the impact on the forwarding table? We might still have 2-3 seconds of disruption but it may avoid the manual intervention and let us keep the more dynamic port security approach. Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, October 31, 2022 9:12 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only The issue was a member port. The port security took 2 seconds to shutdown the port, however 100G of traffic looped and crashed the forwarding table on the main switch, forcing manual intervention. A static MAC ACL on each port would not have taken time to react. This is what the SIX has been doing on their ports since 2010 (as well as multiple other layers of security). From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Kittel, Brady Sent: Monday, October 31, 2022 9:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only Was the source of the issue a remote switch? " To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here." -Brady ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO<mailto:jayhanke@SOUTHFRONT.IO>> Sent: Monday, October 31, 2022 9:05:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [EXTERNAL] Re: [MICE-DISCUSS] Proposal: MICE Static MAC address Only CAUTION: This email originated from outside of HCMC. DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe. The error disable takes 1-2 seconds to clamp down and disable the port. This results in a short, but severe, broadcast storm. The churn rate on members changing mac addresses is fairly low. I agree that we should automate the process so it can be done via self-service by the members. On Mon, Oct 31, 2022 at 8:59 AM Steve Howard <showard(a)paulbunyan.net<mailto:showard@paulbunyan.net>> wrote: > > > Below is a message indicating the current policy as decided by the MICE Board in December 2019 is that the MAC address limit is 1. > > > The subject of Jay's message indicates "Static" MAC, but, the text seems to indicate a limit of 1. I like the 1 address limit, but, am not a fan of Static. Was the current issue caused by somebody not having a proper MAC address limit installed? > > > On 12/13/19 18:00, Richard Laager wrote: > > The board has approved the change in MAC address limit from 5 to 1. > > NOTE: This is orthogonal to the dedicated remote switch question, which > is still pending. Non-dedicated remote switches can, for example, > enforce this per-VLAN. > > We have already confirmed that most participants are using only 1 MAC > address, and some that were using more than 1 have addressed that after > being contacted. As discussed below, this change will be rolled out in a > way to keep everyone working, by grandfathering if necessary. > > On 10/28/19 3:25 PM, Richard Laager wrote: > > We have discussed this a bit in the past, and this came up at the last > UG. I'm looking for feedback from the group before formally proposing > this to the board for a vote. > > I first brought this up to the technical committee, CCing the board. I > have heard no objections. Jeremy is "in favor of it 100%". > > Based on our quick conversation after the UG, I _think_ Anthony is in > favor as well; we both made notes to follow up on this. > > ----- > > I am proposing that MICE change its MAC address limit from 5 MACs per > port to 1 MAC per port. 1 MAC address is enough for normal scenarios and > this would further limit the potential for problems on the fabric. This > restriction is one that SIX and AMS-IX both have, with the latter being > known for their excellent configuration guide for participants: > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ams-i…> > > To be clear, this is still only for end ports, not ports facing remote > switches, of course. The remote switches are responsible for enforcing > the current MAC limit, and would be responsible for changing those > settings as described here. > > If someone is swapping their MICE-facing router, the resulting port flap > on the MICE/remote switch will clear the limit anyway, if they are > directly connected. If they are unable to flap the port (e.g. because > they have someone else's layer 2 gear in the middle), they will have to > either work with that carrier or their switch operator (MICE/remote) to > flap the port. This is a non-zero burden, but I think it's pretty > minimal in practice. Additionally, if someone plans to make an equipment > swap, we would increase the limit in advance, temporarily, upon request. > > For reference, SIX goes further and locks you to a particular MAC > address in an ACL, so you have to contact them to change your MAC. I am > not proposing that. Still, I've been through that a couple of times, and > even that isn't really too big of a deal. So I don't think the change to > a limit of 1 MAC at MICE will be particularly onerous. > > There are a couple of participants who have two IP address sets (one > IPv4 and one IPv6) on the same port. For those ports, the limit > obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If > they're using two IPs, they probably are doing so to get some redundancy > out of it. For this particular use case, needing to flap the port > defeats that redundancy goal, as it breaks the other router. Having a > MAC limit of 3 would allow them to swap a device without needing to flap > the port. We would grandfather these setups until they are looking for a > port-type upgrade; at that point they would need to get to one IP > address set per port (by splitting into two ports or reducing to one IP > address set) or request an exception as outlined below. The exception > process would allow us to better evaluate this use case at that time. > > There are some members who have multiple MACs showing up now, despite > using only one device. For those, I am proposing we set the MAC limit to > however many MACs are currently showing up on their port. That is, they > would be grandfathered at their current situation for now. At the time > of this change, we would encourage them to investigate and fix the > multiple MACs issue at their leisure. In the future, if they are doing a > equipment swap (especially like-for-like), we would again encourage them > to address it, but would otherwise leave their higher limit in place. If > they do a port-type upgrade, we would not bring the grandfathering over. > If they needed, they could ask for an official exception at that time. > > There may be cases where people cannot reasonably fix their routers to > use only 1 MAC, other scenarios I haven't considered, or things that may > come up in the future. For that reason, I think we should allow for the > possibility of exceptions upon request to the board. In practice, I > would expect the board would confer with the technical committee and/or > the technical committee would be bringing these to the board anyway. The > goal is to strike a reasonable balance between protecting the fabric > without preventing networks from connecting. > > > > > On 10/31/22 08:27, Jay Hanke wrote: > > I propose we change the MICE required port security settings to be > hard filtered for a single static mac address per port. > > Over the years, we've had several incidents where floods have occurred > prior to port being error-disabled during a looping event. > > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 Confidentiality Notice: Information contained in this e-mail is being sent to you after appropriate authorization or by legal exception. You are obligated to maintain it in a safe, secure and confidential manner. Re-disclosure without patient consent or as permitted by law is prohibited and may subject you to state and/or federal penalties. This information may also be legally privileged, the disclosure of which is governed by law. This information is intended for the use of the person or entity to which it is addressed. If you are not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any access, disclosure, copying or distribution of this information is strictly prohibited. If you have received this message by error, please notify the sender immediately to arrange for return or proof of destruction of the information contained in this message. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ipho…> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

Nov. 2, 2022

Re: Proposal: MICE Static MAC address Only

by Jay Hanke

Yes. The remote switch terms cover this for sharing a single MICE uplink. Also, when a problem arises, we can kill a port without doing a packet capture and only the offending member can be shut off. On Wed, Nov 2, 2022 at 3:23 PM David Farmer <0000000e7948cb21-dmarc-request(a)lists.iphouse.net> wrote: > > And if we statically define the MAC to port mapping do we care? > > On Wed, Nov 2, 2022 at 3:01 PM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> The hard limit is to keep someone from stacking networks behind a port. >> >> On Wed, Nov 2, 2022 at 2:58 PM David Farmer >> <0000000e7948cb21-dmarc-request(a)lists.iphouse.net> wrote: >> > >> > Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic. >> > >> > On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> >> >> What if we raised the maximum number of static mac addresses to 2 (to >> >> allow for migrations and to allow Jeremy to sleep) in the mac acl? >> >> >> >> On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston(a)wiktel.com> wrote: >> >> > >> >> > Well articulated, as always, David. Thank you. >> >> > >> >> > >> >> > I am in favor of the use of static MAC ACL, even if that requires manual >> >> > changes to that ACL, and the additional coordination that may consume. >> >> > A few thoughts, though.... >> >> > >> >> > Maybe 1G links are exempt from this? Since a couple second switching >> >> > loop from a 1G port is not as impactful? ????? Thoughts from others, >> >> > please. >> >> > >> >> > Using a static MAC ACL could make troubleshooting a failure more >> >> > problematic. For example, replacing a line card to "see if that fixes >> >> > the issue" could send a tech down a trail of red hearings. Even if the >> >> > member is fully aware that they need to coordinate the MAC change, this >> >> > sort work is often being done in response to an unexpected outage. The >> >> > added time spend on the coordination could be seen as frustrating. But, >> >> > for some members it's not a big deal, and can use other links until >> >> > their MICE connectivity is restored. >> >> > >> >> > I've heard that some members are using a virtual MAC address, which is >> >> > probably the way to avoid needing to coordinate in the first place. >> >> >> >> >> >> >> >> -- >> >> Jay Hanke, President >> >> South Front Networks >> >> jayhanke(a)southfront.io >> >> Phone 612-204-0000 >> > >> > >> > >> > -- >> > =============================================== >> > David Farmer Email:farmer@umn.edu >> > Networking & Telecommunication Services >> > Office of Information Technology >> > University of Minnesota >> > 2218 University Ave SE Phone: 612-626-0815 >> > Minneapolis, MN 55414-3029 Cell: 612-812-9952 >> > =============================================== >> > >> > ________________________________ >> > >> > To unsubscribe from the MICE-DISCUSS list, click the following link: >> > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke, President >> South Front Networks >> jayhanke(a)southfront.io >> Phone 612-204-0000 > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Nov. 2, 2022

Re: Proposal: MICE Static MAC address Only

by David Farmer

And if we statically define the MAC to port mapping do we care? On Wed, Nov 2, 2022 at 3:01 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > The hard limit is to keep someone from stacking networks behind a port. > > On Wed, Nov 2, 2022 at 2:58 PM David Farmer > <0000000e7948cb21-dmarc-request(a)lists.iphouse.net> wrote: > > > > Maybe we don't need to define a hard limit of 1 or 2, but only that the > list needs to be static, as in not auto-detected but a defined list; the > point is to be able to quash/filter unknown unicast traffic. > > > > On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > >> > >> What if we raised the maximum number of static mac addresses to 2 (to > >> allow for migrations and to allow Jeremy to sleep) in the mac acl? > >> > >> On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston(a)wiktel.com> > wrote: > >> > > >> > Well articulated, as always, David. Thank you. > >> > > >> > > >> > I am in favor of the use of static MAC ACL, even if that requires > manual > >> > changes to that ACL, and the additional coordination that may consume. > >> > A few thoughts, though.... > >> > > >> > Maybe 1G links are exempt from this? Since a couple second switching > >> > loop from a 1G port is not as impactful? ????? Thoughts from others, > >> > please. > >> > > >> > Using a static MAC ACL could make troubleshooting a failure more > >> > problematic. For example, replacing a line card to "see if that fixes > >> > the issue" could send a tech down a trail of red hearings. Even if the > >> > member is fully aware that they need to coordinate the MAC change, > this > >> > sort work is often being done in response to an unexpected outage. > The > >> > added time spend on the coordination could be seen as frustrating. > But, > >> > for some members it's not a big deal, and can use other links until > >> > their MICE connectivity is restored. > >> > > >> > I've heard that some members are using a virtual MAC address, which is > >> > probably the way to avoid needing to coordinate in the first place. > >> > >> > >> > >> -- > >> Jay Hanke, President > >> South Front Networks > >> jayhanke(a)southfront.io > >> Phone 612-204-0000 > > > > > > > > -- > > =============================================== > > David Farmer Email:farmer@umn.edu > > Networking & Telecommunication Services > > Office of Information Technology > > University of Minnesota > > 2218 University Ave SE Phone: 612-626-0815 > > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > > =============================================== > > > > ________________________________ > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Nov. 2, 2022

Re: Proposal: MICE Static MAC address Only

by Jay Hanke

The hard limit is to keep someone from stacking networks behind a port. On Wed, Nov 2, 2022 at 2:58 PM David Farmer <0000000e7948cb21-dmarc-request(a)lists.iphouse.net> wrote: > > Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic. > > On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> What if we raised the maximum number of static mac addresses to 2 (to >> allow for migrations and to allow Jeremy to sleep) in the mac acl? >> >> On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston(a)wiktel.com> wrote: >> > >> > Well articulated, as always, David. Thank you. >> > >> > >> > I am in favor of the use of static MAC ACL, even if that requires manual >> > changes to that ACL, and the additional coordination that may consume. >> > A few thoughts, though.... >> > >> > Maybe 1G links are exempt from this? Since a couple second switching >> > loop from a 1G port is not as impactful? ????? Thoughts from others, >> > please. >> > >> > Using a static MAC ACL could make troubleshooting a failure more >> > problematic. For example, replacing a line card to "see if that fixes >> > the issue" could send a tech down a trail of red hearings. Even if the >> > member is fully aware that they need to coordinate the MAC change, this >> > sort work is often being done in response to an unexpected outage. The >> > added time spend on the coordination could be seen as frustrating. But, >> > for some members it's not a big deal, and can use other links until >> > their MICE connectivity is restored. >> > >> > I've heard that some members are using a virtual MAC address, which is >> > probably the way to avoid needing to coordinate in the first place. >> >> >> >> -- >> Jay Hanke, President >> South Front Networks >> jayhanke(a)southfront.io >> Phone 612-204-0000 > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Nov. 2, 2022

Re: Proposal: MICE Static MAC address Only

by David Farmer

Maybe we don't need to define a hard limit of 1 or 2, but only that the list needs to be static, as in not auto-detected but a defined list; the point is to be able to quash/filter unknown unicast traffic. On Wed, Nov 2, 2022 at 9:33 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > What if we raised the maximum number of static mac addresses to 2 (to > allow for migrations and to allow Jeremy to sleep) in the mac acl? > > On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston(a)wiktel.com> > wrote: > > > > Well articulated, as always, David. Thank you. > > > > > > I am in favor of the use of static MAC ACL, even if that requires manual > > changes to that ACL, and the additional coordination that may consume. > > A few thoughts, though.... > > > > Maybe 1G links are exempt from this? Since a couple second switching > > loop from a 1G port is not as impactful? ????? Thoughts from others, > > please. > > > > Using a static MAC ACL could make troubleshooting a failure more > > problematic. For example, replacing a line card to "see if that fixes > > the issue" could send a tech down a trail of red hearings. Even if the > > member is fully aware that they need to coordinate the MAC change, this > > sort work is often being done in response to an unexpected outage. The > > added time spend on the coordination could be seen as frustrating. But, > > for some members it's not a big deal, and can use other links until > > their MICE connectivity is restored. > > > > I've heard that some members are using a virtual MAC address, which is > > probably the way to avoid needing to coordinate in the first place. > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Nov. 2, 2022

Re: Proposal: MICE Static MAC address Only

by Jay Hanke

What if we raised the maximum number of static mac addresses to 2 (to allow for migrations and to allow Jeremy to sleep) in the mac acl? On Mon, Oct 31, 2022 at 12:48 PM Mike Johnston <mjohnston(a)wiktel.com> wrote: > > Well articulated, as always, David. Thank you. > > > I am in favor of the use of static MAC ACL, even if that requires manual > changes to that ACL, and the additional coordination that may consume. > A few thoughts, though.... > > Maybe 1G links are exempt from this? Since a couple second switching > loop from a 1G port is not as impactful? ????? Thoughts from others, > please. > > Using a static MAC ACL could make troubleshooting a failure more > problematic. For example, replacing a line card to "see if that fixes > the issue" could send a tech down a trail of red hearings. Even if the > member is fully aware that they need to coordinate the MAC change, this > sort work is often being done in response to an unexpected outage. The > added time spend on the coordination could be seen as frustrating. But, > for some members it's not a big deal, and can use other links until > their MICE connectivity is restored. > > I've heard that some members are using a virtual MAC address, which is > probably the way to avoid needing to coordinate in the first place. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Nov. 2, 2022

Re: Anyone lose peering with CloudFlare this morning at 4:13 am?

by Jeremy Lumby

I have been told that they have replaced the same Juniper switch two different times, and the two different replacements keep showing the same errors as the original. This is corroborated with their MICE link coming up briefly twice since they originally rerouted. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Monday, October 31, 2022 2:48 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this morning at 4:13 am? Any word from Cloudflare? They will have been in re-routed mode for three weeks on Wednesday. They have theoretically completed two maintenance events since then; https://www.cloudflarestatus.com/incidents/txnkvw6frp25 https://www.cloudflarestatus.com/incidents/pqwsjx82wdt2 Thanks On Thu, Oct 20, 2022 at 11:08 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Yes, the last time they had a failure in Minneapolis it took them just shy of 2 weeks to get back online. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Savoy Sent: Thursday, October 20, 2022 10:39 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this morning at 4:13 am? I am pretty new to the CloudFlare peering. Anyone else go through the “Re-routed” scenario before? Will it eventually become “Operational”? Thank you, Steve From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Wednesday, October 12, 2022 11:59 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this morning at 4:13 am? If you check Cloudflare’s status page you will see Minneapolis is in “Re-routed” state. https://www.cloudflarestatus.com On Wed, Oct 12, 2022 at 10:45 Steve Savoy <stevesavoy(a)wctatel.com> wrote: Yes – ours went down at the same time. Steve From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Wednesday, October 12, 2022 10:42 AM To:MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone lose peering with CloudFlare this morning at 4:13 am? Both IPv4 and IPv6. I didn’t see any exceeding of prefix limits. Frank Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net <http://sxct-north-2.sxcy.fbnt.netins.net>) SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 206.108.255.45 (AS13335.micemn.net <http://AS13335.micemn.net>) (AS 13335 CLOUDFLARENET, US) old state Established event HoldTime new state Idle Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net <http://sxct-north-2.sxcy.fbnt.netins.net>) SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::3417:0:1 (AS13335.micemn.net <http://AS13335.micemn.net>) (AS 13335 CLOUDFLARENET, US) old state Established event HoldTime new state Idle ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu <Email%3Afarmer(a)umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu <Email%3Afarmer(a)umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Oct. 31, 2022

Re: Anyone lose peering with CloudFlare this morning at 4:13 am?

by David Farmer

Any word from Cloudflare? They will have been in re-routed mode for three weeks on Wednesday. They have theoretically completed two maintenance events since then; https://www.cloudflarestatus.com/incidents/txnkvw6frp25 https://www.cloudflarestatus.com/incidents/pqwsjx82wdt2 Thanks On Thu, Oct 20, 2022 at 11:08 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Yes, the last time they had a failure in Minneapolis it took them just shy > of 2 weeks to get back online. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Steve Savoy > *Sent:* Thursday, October 20, 2022 10:39 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > I am pretty new to the CloudFlare peering. Anyone else go through the > “Re-routed” scenario before? Will it eventually become “Operational”? > > > > Thank you, > > Steve > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *David > Farmer > *Sent:* Wednesday, October 12, 2022 11:59 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > If you check Cloudflare’s status page you will see Minneapolis is in > “Re-routed” state. > > > > https://www.cloudflarestatus.com > > > > > > On Wed, Oct 12, 2022 at 10:45 Steve Savoy <stevesavoy(a)wctatel.com> wrote: > > Yes – ours went down at the same time. > > > > Steve > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Wednesday, October 12, 2022 10:42 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Anyone lose peering with CloudFlare this > morning at 4:13 am? > > > > Both IPv4 and IPv6. I didn’t see any exceeding of prefix limits. > > > > Frank > > > > Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) > SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 206.108.255.45 ( > AS13335.micemn.net) (AS 13335 CLOUDFLARENET, US) old state Established > event HoldTime new state Idle > > Oct 12 04:13:10 207.32.15.14 (sxct-north-2.sxcy.fbnt.netins.net) > SiouxCity-Fibernet-Arista Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::3417:0:1 > (AS13335.micemn.net) (AS 13335 CLOUDFLARENET, US) old state Established > event HoldTime new state Idle > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > Confidentiality Notice: This correspondence is the property of Winnebago > Cooperative Telecom Association and is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply email > and destroy all copies of the original message. > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 31, 2022

Re: Proposal: MICE Static MAC address Only

by Mike Johnston

Well articulated, as always, David. Thank you. I am in favor of the use of static MAC ACL, even if that requires manual changes to that ACL, and the additional coordination that may consume. A few thoughts, though.... Maybe 1G links are exempt from this? Since a couple second switching loop from a 1G port is not as impactful? ????? Thoughts from others, please. Using a static MAC ACL could make troubleshooting a failure more problematic. For example, replacing a line card to "see if that fixes the issue" could send a tech down a trail of red hearings. Even if the member is fully aware that they need to coordinate the MAC change, this sort work is often being done in response to an unexpected outage. The added time spend on the coordination could be seen as frustrating. But, for some members it's not a big deal, and can use other links until their MICE connectivity is restored. I've heard that some members are using a virtual MAC address, which is probably the way to avoid needing to coordinate in the first place.

Oct. 31, 2022

MICE - IXP Manager - Change Control Access

by Justin Krejci

Spinning off a new thread from the static/mac-limit/port-security conversation. The IXP Manager can generate an instruction command and makes it available to the Arista(s). The switch(es) could pull the config using a script that further sanitizes/validates the config its pulling before applying. This would prevent IXP Manager from having its own direct access to the switch. This could also be the case for pulling data from the switch(es) as well. Patterned after a TR-069 or similar concept, the switch could phone home to the IXP manager (once a minute cronjob for example) and see if there is anything the IXP manager wants done. A simple text based file (json, etc) is generated by the IXP manager and the Arista would then just query this file and parse out instructions and perform actions as instructed or ignore them if malformed or otherwise not allowed. Obviously this is would be only for Arista switch(es) managed by MICE. I don't have any IXP Manager experience beyond simple user but I know Arista can do this fairly easily. It could also just be a RESTful API even. Maybe something like this already exists I have no idea but I would highlight the two things this concept does 1: Allows IXP manager to affect change without direct access to the switch(es) 2: Allows the Arista switch(es) to ultimately decide to accept or ignore instructions to make config changes, send info, perform other actions like bounce my member port, apply TCP/179 filtering to my port, show me some log data, etc. This concept could also apply to the route servers too since they also run a *nix OS like Arista. This would not be a simple/quick idea but I feel would check various boxes for people: air gap, allows automation/self-service, and if automation fails it results in a fallback to manual only changes like we are doing today anyways, etc. Since I know MICE does have money, it could be dev work hired out to someone or a team to produce and then MICE retains rights to the resultant code. We have enough people across our membership that I am sure we could validate the code is sound and also perform bugfixes as needed going forward, perhaps even make it public, GPL style. Maybe IXP Manager already has some of this functionality? Last second additional thought, simple instructions (eg add a second MAC address, show me log data for my port) get automatically approved. Bigger changes (remove/change my MAC address, some other action) need to get some sort of manual approval by the tech team so it can be scrutinized first. None of this needs to be done in IXP Manager either but some other custom, purpose built web portal. Maybe I'm way overthinking this. Food for thought at any rate. -Justin

Oct. 31, 2022