- mice-discuss - lists.micemn.net

Google's new 100G from Chicago
by Frank Bulk April 14, 2021

April 14, 2021

Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google's response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: * Clean the interface optic and your patching path to the MMR. * Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883)

5 7

MICE Issues
by Jay Hanke April 12, 2021

April 12, 2021

Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

5 7

MICE UG 39
by Richard Laager April 10, 2021

April 10, 2021

With NANOG happening this fall, it may make sense to schedule MICE UG 39 at the same time (e.g. Wednesday November 3, time TBD). That's not official yet, but tentatively "save the date". Thanks to the person who suggested this (off-list)! -- Richard

1 0

SFN Remote Uplink Change
by Jay Hanke April 1, 2021

April 1, 2021

SFN has upgraded the uplink LAG group on the SFN MICE remote switch to 12x10G from 8x10G. We were getting a small amount of tail drops where we convert from 100GE to the LAG group. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

belle plaine
by James Urwiller April 1, 2021

April 1, 2021

anyone on the belle plain remote just loose connection? we are coming in from fremont, NE.

5 7

Please welcome Xiber LLC to MICE!
by Jay Hanke March 30, 2021

March 30, 2021

Name: Xiber LLC Port: Main Ethernet 4/46 ASN: 393950 IPv4: 206.108.255.178/24 IPv6: 2001:504:27:0:6:2de:0:1/64 Peering Contact: peering(a)xiber.net Route Servers- Yes BFD: Yes This email has been filed with Cologix as your LOA/CFA. Members are no longer required to provide optics for installation in the main switch. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Nanog!
by Reid Fishler March 30, 2021

March 30, 2021

It was just announced the first in person Nanog is in MSP Nov 1-3! Reid -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178

2 1

Please Welcome Range (AS30110) to MICE!
by Jay Hanke March 24, 2021

March 24, 2021

Name: Range Port: Main Ethernet 5/38 ASN: 30110 IPv4: 206.108.255.176/24 IPv6: 2001:504:27::759e:0:1/64 Peering Contact: dave.berry(a)range.net Route Servers- Yes BFD: Yes This email has been filed with Cologix as your LOA/CFA. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Facebook Peers Drop
by Travis Fincham March 21, 2021

March 21, 2021

Did anyone else have their peering with Facebook drop this morning about 7:00 AM? Travis Fincham, CCNA ACS Network Engineer Email: Travis.Fincham(a)keycoop.com<mailto:Travis.Fincham@keycoop.com> Cell: (515)-735-5011 ________________________________ NOTICE: This message (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. § 2510-2521, is CONFIDENTIAL, and is intended only for the individual to whom it is addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you have received this message in error and that any review, retention, dissemination, distribution, copying or use of this communication is strictly prohibited. Please reply to the sender that you received this message in error and then delete it.

4 3

AS16842 Disconnect Notice
by Jay Hanke March 16, 2021

March 16, 2021

AS16842 has disconnected from MICE. Please remove their route server and port configurations. 206.108.255.39 will be returned to the free pool as well as ge-1/1/0 on 1G main switch. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

MICE Terabit Dreams?
by Jay Hanke March 15, 2021

March 15, 2021

When do you think we'll cross over 1 terabit? [image: image.png] Blue line is more accurate as IXP Manager isn't able to grab stats from the Zayo remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

3 2

Please welcome Les.net to MICE!
by Jay Hanke March 15, 2021

March 15, 2021

Name: Les.net Port: MN VoIP 100G Ethernet4/1 ASN: 18451 IPv4: 206.108.255.175/24 IPv6: 2001:504:27::4813:0:1/64 Peering Contact: arin(a)les.net Route Servers- Yes BFD: NO Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Supranet (AS4150) Port Move
by Jay Hanke March 13, 2021

March 13, 2021

Supra net has moved from the 1G main Switch to port Ethernet7 on the MN VoIP remote 10G switch. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Oneneck Port Upgrade to 10G
by Jay Hanke March 13, 2021

March 13, 2021

Oneneck has been allocated the following port on the MAIN Switch MICE MAIN :: Ethernet3/7/1 This will replace their existing 1G connection. All IP/Route Server information will remain the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 1

Lost MICE VMWare server
by Jay Hanke March 12, 2021

March 12, 2021

The VMWare server that hosts RS3 and IXP Manager for MICE died. We're onsite working on restoration now. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

4 5

Please Welcome Metronet to MICE!
by Jay Hanke March 9, 2021

March 9, 2021

Name: Metronet Port: Main Ethernet6/7/1 (100G New Single Member LAG) ASN: 30600 IPv4: 206.108.255.174/24 IPv6: 2001:504:27::7788:0:1/64 Peering Contact: peering(a)metronetinc.com Route Servers- NO BFD: NO Please note this is in addition to the existing Jaguar/Metronet ports not a replacement. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Re: Packet Loss between 00:00 and 05:00 this morning
by Jeremy Lumby March 6, 2021

March 6, 2021

They do come in on one of the Zayo circuits that was impacted last night. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> Date: 3/6/21 2:45 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 14:45:37 Our Qrator monitoring noticed that BANKONIT’s BGP session going in and out many times between 11:17 pm to 5:59 am Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Keith Wirch Sent: Saturday, March 6, 2021 9:29 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

1 0

Packet Loss between 00:00 and 05:00 this morning
by Keith Wirch March 6, 2021

March 6, 2021

2 1

Re: Packet Loss between 00:00 and 05:00 this morning
by Jeremy Lumby March 6, 2021

March 6, 2021

I had some Zayo circuits leaving Minneapolis that were experiencing high packet loss during that exact same time frame Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Keith Wirch <kwirch(a)ATOMICDATA.COM> Date: 3/6/21 9:29 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Keith Wirch [kwirch(a)ATOMICDATA.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 09:29:00 Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

1 0

Hulu Live/Fastly
by Frank Bulk March 2, 2021

March 2, 2021

We've received 5 to 7 reports about Hulu Live issues ("freezing up") in the last week or so. They're not all on Roku as we've had at least one person with an Amazon Firestick. For those where we've identified the source IP it's been Fastly, which is present at MICE. Curious if anyone else has received an uptick in Hulu Live complaints? We have not (yet) been able to reproduce the issue internally. Other streaming services at these same customers appear fine. Raw throughput in speed tests to our speed test server are fine. The one customer that we know did call Hulu was told by Hulu support that their issue was with their ISP (us). Frank

2 2

MPW Port Move
by Jay Hanke Feb. 26, 2021

Feb. 26, 2021

MPW has moved from Port Ethernet 7 on the MN VoIP switch to port Ethernet 12 on the SFN remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

MICE UG 37
by Richard Laager Feb. 26, 2021

Feb. 26, 2021

Reminder: UG 37 is on Wednesday, March 10, at 3:00pm This will be on Google Meet: https://meet.google.com/zgg-eeyu-dxz In the event of serious problems, we will fall back to: 952-562-5400 passcode MICE (6423) Agenda The agenda is attached. If you have something to add, let me know. The UG 36 draft minutes are also attached. -- Richard

1 0

Sling TV
by Jeremy Lumby Feb. 23, 2021

Feb. 23, 2021

Has anyone else been experiencing issues with Sling TV's cloud DVR service, especially during peak hours within the past couple months? While they are not directly connected to MICE, some of their transit providers are. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

SFN MICE Remote Scheduled Maintenance
by Jay Hanke Feb. 23, 2021

Feb. 23, 2021

The SFN Remote switch will be undergoing maintenance and will be power cycled during our transport provider doing software upgrades. The SFN remote switch will be hard down during the part of the maintenance and be isolated during part of the maintenance. Connectivity may be intermittent during the entire window as software updates are applied. This will also affect test filtering sessions to MICE-RS3. Primary Window: FROM: Tuesday, Feb 23, 2021 0030 hours (CST) TO: Tuesday, Feb 23, 2021 0230 hours (CST) Backup Window: FROM: Tuesday, Feb 24, 2021 0030 hours (CST) TO: Tuesday, Feb 24, 2021 0230 hours (CST) Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 1

Apple links filling up?
by Frank Bulk Feb. 23, 2021

Feb. 23, 2021

The last two nights our NAGIOS-based monitoring of HTTPv6 access to http://www.apple.com has been timing out from time to time. Am I interpreting MICE's graphs correctly to suggest that Apple's two (1x10G?) LAGs are filling up? Frank

2 2

HE.net Routes Lost - Around 15:05 CST ?
by Jesse Reed Feb. 19, 2021

Feb. 19, 2021

Anyone else lose routes via HE shortly after 15:00 CST today? It was brief for us, but wondering if anyone else experienced this? Out BGP neighbor(s) remained stable but it appears like we dropped traffic during this time. Thanks ahead of time, Jesse Reed Network Operations Specialist [cid:image001.jpg@01D706D4.B26536D0] jreed(a)AcenTek.net<mailto:jreed@AcenTek.net> www.AcenTek.net

3 2

WiscNet Remote Switch maintenance 2021-02-23 5pm - 10pm
by Chris Wopat Feb. 19, 2021

Feb. 19, 2021

Hi Folks, The WiscNet remote switch, which currently only has active sessions for WiscNet, will be under maintenance this Tuesday, Feb 23rd at 5pm local time. We have work impacting our backbone paths to Minneapolis, so will be draining traffic from peers at this location during this time beginning at 5pm. We expect things to be fully online around 10pm. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

1 0

Amazon LAG Augmentations
by Jay Hanke Feb. 16, 2021

Feb. 16, 2021

Amazon is adding 100G to both of their existing port channels. PO5: 3/12/1, 6/4/1 (New) PO24 6/1/1, 3/21/1 (New) All IP information will remain the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Please Welcome Mid-Rivers Telephone to MICE
by Jay Hanke Feb. 15, 2021

Feb. 15, 2021

Name: Mid-Rivers Telephone Port: Main Switch (New 20G LAG) Ethernet4/12, Ethernet5/21 ASN: 11961 IPv4: 206.108.255.173/24 IPv6: 2001:504:27::2eb9:0:1/64 Peering Contact: nse(a)midrivers.coop Route Servers- YES BFD: No Please note these may be reclaimed ports. MICE uses LACP by default on all LAG ports. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

state.mn.us expired yesterday?
by Frank Bulk Feb. 7, 2021

Feb. 7, 2021

It appears that state.mn.us expired yesterday? It's possible with such kind of TLD that that doesn't matter? Frank Bulk Registry WHOIS Server : whois.nic.us Domain Name: state.mn.us Registry Domain ID: D3624286-US Registrar WHOIS Server: Registrar URL: http://www.whois.us Updated Date: 2021-01-12T14:03:55Z Creation Date: 2003-02-06T20:25:14Z Registry Expiry Date: 2021-02-06T20:25:14Z Registrar: State of Minnesota Registrar IANA ID: Registrar Abuse Contact Email: Registrar Abuse Contact Phone: Domain Status: ok https://icann.org/epp#ok Domain Status: autoRenewPeriod https://icann.org/epp#autoRenewPeriod Registry Registrant ID: C7705028-US Registrant Name: Network Operations Center Registrant Organization: State of Minnesota Registrant Street: Office of Enterprise Technology Registrant Street: 500 Centennial Office Building Registrant Street: 658 Cedar Street Registrant City: St. Paul Registrant State/Province: MN Registrant Postal Code: 55155 Registrant Country: US Registrant Phone: +1.6512971111 Registrant Phone Ext: Registrant Fax: +1.6512975368 Registrant Fax Ext: Registrant Email: nic<%>nic.state.mn.us Registrant Application Purpose: P5 Registrant Nexus Category: C21 Registry Admin ID: C7705028-US Admin Name: Network Operations Center Admin Organization: State of Minnesota Admin Street: Office of Enterprise Technology Admin Street: 500 Centennial Office Building Admin Street: 658 Cedar Street Admin City: St. Paul Admin State/Province: MN Admin Postal Code: 55155 Admin Country: US Admin Phone: +1.6512971111 Admin Phone Ext: Admin Fax: +1.6512975368 Admin Fax Ext: Admin Email: nic<%>nic.state.mn.us Admin Application Purpose: P5 Admin Nexus Category: C21 Registry Tech ID: C7705028-US Tech Name: Network Operations Center Tech Organization: State of Minnesota Tech Street: Office of Enterprise Technology Tech Street: 500 Centennial Office Building Tech Street: 658 Cedar Street Tech City: St. Paul Tech State/Province: MN Tech Postal Code: 55155 Tech Country: US Tech Phone: +1.6512971111 Tech Phone Ext: Tech Fax: +1.6512975368 Tech Fax Ext: Tech Email: nic<%>nic.state.mn.us Tech Application Purpose: P5 Tech Nexus Category: C21 Registry Domain Manager ID: C7704880-US Domain Manager Name: Network Operations Center Domain Manager Organization: State of Minnesota Domain Manager Street: Office of Enterprise Technology Domain Manager Street: 500 Centennial Office Building Domain Manager Street: 658 Cedar Street Domain Manager City: St. Paul Domain Manager State/Province: MN Domain Manager Postal Code: 55155 Domain Manager Country: US Domain Manager Phone: +1.6512971111 Domain Manager Phone Ext: Domain Manager Fax: +1.6512975368 Domain Manager Fax Ext: Domain Manager Email: nic<%>nic.state.mn.us Domain Manager Application Purpose: P5 Domain Manager Nexus Category: C21 Name Server: ns3.state.mn.us Name Server: ns2.state.mn.us Name Server: ns1.state.mn.us DNSSEC: signedDelegation URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/ >>> Last update of WHOIS database: 2021-02-07T14:25:40Z

3 2

FW: [outages] [EXT]: CCI Southern Minnesota inbound connections from other providers failing
by Frank Bulk Feb. 4, 2021

Feb. 4, 2021

FYI From: Outages <outages-bounces(a)outages.org> On Behalf Of Harris Kleyman via Outages Sent: Thursday, February 4, 2021 2:18 PM To: Nathan Engelhardt <NEngelhardt(a)yourcts.com> Cc: outages(a)outages.org Subject: Re: [outages] [EXT]: CCI Southern Minnesota inbound connections from other providers failing We are also seeing impacted clients. Harris Kleyman Manager of Data Centers & Facilities and NOC Supervisor ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2031 On Feb 4, 2021, at 2:15 PM, Nathan Engelhardt via Outages <outages(a)outages.org<mailto:outages@outages.org>> wrote: Outbound connections succeed, and inbound connections from other CCI sites succeed. Inbound connections from other providers fail. Charter/Spectrum to CCI: Tracing route to redacted [209.240.redacted] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms redacted.com [10.0.0.1] 2 * * * Request timed out. 3 17 ms 12 ms 10 ms 096-034-026-152.biz.spectrum.com [96.34.26.152] 4 14 ms 17 ms 13 ms cts01rhnlwi-tge-3-0-0.rhnl.wi.charter.com [96.34.16.147] 5 19 ms 15 ms 15 ms bbr02slidla-tge-0-1-0-6.slid.la.charter.com [96.34.1.188] 6 33 ms 21 ms 26 ms prr01mplsmn-bue-1.mpls.mn.charter.com [96.34.3.31] 7 25 ms 29 ms 21 ms AS12042.micemn.net [206.108.255.18] 8 * * * Request timed out. 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out. _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

1 0

Traffic from Minneapolis to Seattle
by Jeremy Lumby Feb. 3, 2021

Feb. 3, 2021

If anyone has large amounts of traffic that they are primarily moving from Minneapolis to Seattle please reach out to me off list. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

SFN 20G LAG Upgrade
by Jay Hanke Feb. 2, 2021

Feb. 2, 2021

Please allocate Eth14 on the SFN Remote Switch to South Front Networks Eth14 and Eth2 will be LAGed to create a 20G port channel. The new port channel interface will be Po1 on the remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

US Internet Maintenance - February 22/23
by Justin Krejci Jan. 28, 2021

Jan. 28, 2021

US Internet (AS10242) will be performing some network maintenance (primarily router reboots for updates) that will cause our IX connection and all router BGP sessions to drop. We intend to shift traffic away beforehand to be as graceful as possible. The event should be between FROM: Monday, February 22th at 11pm (CST) TO: Tuesday, February 23th at 6am (CST) We've reserved the following day (same times) as a backup date as well. FROM: Tuesday, February 23th at 11pm (CST) TO: Wednesday, February 24th at 6am (CST) Justin Krejci Network Engineer 952.253.3200 12450 Wayzata Blvd. Suite 320 | Minnetonka, MN 55305

1 0

eBGP- Google AS15169 v4+v6 peers
by Nick Twite Jan. 28, 2021

Jan. 28, 2021

Good afternoon, I am reaching out because we had a our Google peerings drop this afternoon approximately 1:45pm to 2:10 CST. They have since re-established and routes are again being shared. Just looking if anyone else experienced the same or has any details to share I would appreciate it. RP/0/RSP0/CPU0:Jan 28 13:43:59.242 CST: bgp[1020]: %ROUTING-BGP-5-ADJCHANGE_DETAIL : neighbor 206.108.255.141 Down - BGP Notification sent, hold time expired (VRF: default; AFI/SAFI: 1/1) (AS: 15169) RP/0/RSP0/CPU0:Jan 28 13:44:11.705 CST: bgp[1020]: %ROUTING-BGP-5-ADJCHANGE_DETAIL : neighbor 2001:504:27::3b41:0:1 Down - BGP Notification sent, hold time expired (VRF: default; AFI/SAFI: 2/1) (AS: 15169) RP/0/RSP0/CPU0:Jan 28 14:10:39.427 CST: bgp[1020]: %ROUTING-BGP-5-ADJCHANGE_DETAIL : neighbor 206.108.255.141 Up (VRF: default; AFI/SAFI: 1/1) (AS: 15169) RP/0/RSP0/CPU0:Jan 28 14:10:48.019 CST: bgp[1020]: %ROUTING-BGP-5-ADJCHANGE_DETAIL : neighbor 2001:504:27::3b41:0:1 Up (VRF: default; AFI/SAFI: 2/1) (AS: 15169) Thanks Nick Twite Network Operations Specialist [cid:image001.png@01D6F584.3D31CB50] 507-429-0135 Work Cell 507-896-6229 Desk ntwite(a)AcenTek.net<mailto:ntwite@AcenTek.net> www.AcenTek.net

3 3

Router12 MICE Port Move
by Jay Hanke Jan. 27, 2021

Jan. 27, 2021

Router12 has moved to port Ethernet13 on the SFN remote. All IP address information remains the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

CS Tech Upgrade 1x10G to 2x10G
by Jay Hanke Jan. 19, 2021

Jan. 19, 2021

CS Tech has completed an upgrade to 2x10G LAG on the SFN Albert Lea Remote Port Channel 3 members -Ethernet7 -Ethernet11 All IP information remains the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Fibernet Port Augment
by Jay Hanke Jan. 15, 2021

Jan. 15, 2021

Please add port Ethernet24 to the existing Fibernet LAG. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Please welcome Key Cooperative to MICE!
by Jay Hanke Jan. 13, 2021

Jan. 13, 2021

Name: Key Cooperative Port: MN VoIP 10G Ethernet25 ASN: 40350 IPv4: 206.108.255.172/24 IPv6: 2001:504:27::9d9e:0:1/64 Peering Contact: dl-peering(a)keycoop.com Route Servers- YES BFD: No Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Google MICE Issues
by Jay Hanke Jan. 12, 2021

Jan. 12, 2021

Google is currently having issues with their circuit to MICE from Chicago and has disabled traffic during troubleshooting. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Iowa Fiber Cut
by Jay Hanke Jan. 11, 2021

Jan. 11, 2021

There is a third party fiber cut in Cedar Rapids, IA affecting several members MICE connections. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Google Ports to MICE
by Dave Schwartz Jan. 11, 2021

Jan. 11, 2021

Hello MICE Members I hope this message finds you well. We have heard from a number of MICE members about congestion on Google's port to MICE so I wanted to provide a brief update to our peers on what our plans are. We are aware of the congestion on our current 100G port to MICE and started planning for a 2nd 100G port back in Q3 of 2020. For those of you not aware, the Google router that is connected to MICE is located in Chicago and we use a third party vendor to reach the MICE switch. We have been experiencing long delays with our vendor for our 2nd 100G wave and currently plan to have the 2nd 100G up at MICE in early February. This 2nd 100G link will be on a diverse Google router with a separate IP and plan to peer with the route server on this new link. While our port is congested you may see traffic inbound to your network on indirect paths. While we attempt to deliver all traffic directly, we can't offer any guarantee on where traffic will come from during congestion. If you happen to experience quality problems with any Google service over MICE during this congestion please do report these product specific issues to noc(a)google.com so we can investigate. Our CDN is normally able to handle congestion so we would like to be aware if any members are experiencing any degradation of Google services. Many Thanks, Dave -- Dave Schwartz Customer Engineering Manager Network Specialist Team Google Cloud Platform daveschwartz(a)google.com +1-424-625-9129

2 1

FW: [outages] High packet loss on Spectrum in Southern Minnesota
by Frank Bulk Jan. 6, 2021

Jan. 6, 2021

Cross-posting as this may interest some in this group. Frank From: Outages <outages-bounces(a)outages.org> On Behalf Of Abhi Devireddy via Outages Sent: Wednesday, January 6, 2021 3:46 PM To: Outages(a)outages.org; Goli, Matthew W. <Matthew.Goli(a)waupacafoundry.com> Subject: Re: [outages] High packet loss on Spectrum in Southern Minnesota Update: It's a backbone fiber between MN and WI that was impacted in Mazomanie, WI and Duluth, MN. Teams were dispatched and one's arrived on site. Still no confirmed ETR. Abhi ________________________________ From: Outages <outages-bounces(a)outages.org<mailto:outages-bounces@outages.org>> on behalf of Goli, Matthew W. via Outages <outages(a)outages.org<mailto:outages@outages.org>> Sent: Wednesday, January 6, 2021 2:36 PM To: Outages(a)outages.org<mailto:Outages@outages.org> <Outages(a)outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] High packet loss on Spectrum in Southern Minnesota Hello, We're in central Wisconsin and we're impacted by this outage. Just received communication from Spectrum stating there is a confirmed fiber cut. Fiber recovery teams have been engaged but there is no ETR at this time. From: Outages <outages-bounces(a)outages.org<mailto:outages-bounces@outages.org>> On Behalf Of John Knutson via Outages Sent: Wednesday, January 6, 2021 2:29 PM To: Abhi Devireddy <abhi(a)devireddy.com<mailto:abhi@devireddy.com>> Cc: outages(a)outages.org<mailto:outages@outages.org> Subject: Re: [outages] High packet loss on Spectrum in Southern Minnesota CAUTION!! This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I've got a coworker in Elk River, MN who has no connectivity. I am on Spectrum in Dayton, MN about 10 miles south of that and mine has been working fine so far. Both are residential cable connections. On Wed, Jan 6, 2021 at 1:36 PM Abhi Devireddy via Outages <outages(a)outages.org<mailto:outages@outages.org>> wrote: No v4/v6 connectivity in Northern MN. Looks like our peering traffic with them dropped to zero as well. Abhi ________________________________ From: Outages <outages-bounces(a)outages.org<mailto:outages-bounces@outages.org>> on behalf of Nathan Engelhardt via Outages <outages(a)outages.org<mailto:outages@outages.org>> Sent: Wednesday, January 6, 2021 12:56 PM To: outages(a)outages.org<mailto:outages@outages.org> <outages(a)outages.org<mailto:outages@outages.org>> Subject: Re: [outages] High packet loss on Spectrum in Southern Minnesota Update/additional info: Losses are only occurring over IPv4. IPv6 is performing normally, for what it's worth. From: Outages <outages-bounces(a)outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Nathan Engelhardt via Outages Sent: Wednesday, January 6, 2021 12:45 PM To: outages(a)outages.org<mailto:outages@outages.org> Subject: [outages] High packet loss on Spectrum in Southern Minnesota A few of my coworkers and I, all WFH and on Spectrum, are seeing high packet loss. South-Central Minnesota. _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages [puck.nether.net]<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailma…> CONFIDENTIALITY AND LEGAL NOTICE This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If the reader of this transmission is not the intended recipient, you are hereby notified that any distribution or copying hereof is strictly prohibited. If you have received this transmission in error, please immediately notify the sender and remove it from your system. E-mail is not necessarily a secure communication method, therefore the sender shall not be responsible for any changes that occur during its transfer. All e-mails are scanned for viruses, however, files attached to this e-mail may contain viruses that could harm the systems of the recipient. Any opinions expressed in this email must be confirmed in writing and signed by the sender to have legal validity.

1 0

config assistance?
by James Urwiller Dec. 30, 2020

Dec. 30, 2020

Greetings all, Last night I cutover to a new Arista for BGP services facing MICE and a few other peers. All my other peers are up and functioning properly, but I cannot reach any of the MICE peers. Since I am connected to the Zayo remote switch in Belle Plaine, I had Zayo bounce my port thinking it was a mac-limit issue. Even before they did this, I can see MICE peers in ARP and in the MAC-Table, but I cant ping any of them, like the traffic is filtered in one direction. Zayo’s port bouncing had no effect. Any ideas Arista/MICE guru’s? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com

4 9

Global Net (AS395466) - Upcoming Maintenance 12/28/20
by Global Net NOC Dec. 22, 2020

Dec. 22, 2020

MICEMN Members - We have a maintenance window scheduled for Monday, December 28th to replace our MICE facing router. BGP sessions will be gracefully shutdown prior to the work. In the event of a failed swap we will be reverting to our current hardware. Start: 12/28/20 1115 GMT End: 12/28/20 1300 GMT ASN: 395466 Speed: 10G IPv4: 206.108.255.97 IPv6: 2001:504:27:0:6:08ca::1 Thanks, Global Net NOC noc(a)theglobal.net Global Net Peering peering(a)theglobal.net

1 0

SFN MICE Remote Switch work
by Jay Hanke Dec. 5, 2020

Dec. 5, 2020

We will be upgrading the backhaul of the SFN remote switch to 100G at Midnight. There may be some brief packet loss when we disable the existing LAG group. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

OmniTech (AS54821) MICE Port Move
by Jay Hanke Dec. 4, 2020

Dec. 4, 2020

OmniTech has been moved from the Zayo Remote switch to SFN Remote switch port Ethernet10. All IP/ASN information remains the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

AS34962 AS Change
by Austin Gillmann Dec. 2, 2020

Dec. 2, 2020

Hi all, We recently decided to split our hosting into a subsidiary known as IP Volume to better market our hosting services. All sessions including the IX route server have been updated to the new AS and we are awaiting your connection. If you have any questions feel free to contact me. We appreciate your continued support and wish you all the best!

2 2

RS3 Route server
by Jay Hanke Dec. 1, 2020

Dec. 1, 2020

Connectivity to the MICE test route server (RS3) took a 5 minute hit about 2110 hrs. Connectivity has been restored. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

MICE Blackhole IP Allocation
by Jay Hanke Nov. 22, 2020

Nov. 22, 2020

The following IP addresses have been allocated for the upcoming blackhole project on the MICE Internet Exchange. MICE-BH IPv4: 206.108.255.228/24 IPv6: 2001:504:27::6666:dead:beef/64 We will be using 66:66:de:ad:be:ef as the mac-address and inject the arp entry from a virtual machine on the SFN-Remote until a permanent solution on the main switch is worked out. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 3