- mice-discuss - lists.micemn.net

FW: Verus AS26156 New Equipment
by Jamison Masters July 9, 2021

July 9, 2021

FYI - rejected from noc(a)veruscorp.com. Sending from jamison(a)veruscorp.com. Please see message. -----Original Message----- From: noc(a)veruscorp.com Sent: Friday, July 9, 2021 11:52 AM To: noc(a)veruscorp.com Subject: Verus AS26156 New Equipment Hello, This message is going out to multiple peers and IX's, thus a blind copy message - apologies. Verus Corporation is migrating our connections to new equipment to support 10G service. We have already verified physical connectivity and are confident. AS26156 We are planning maintenance on Saturday 7/10, between the hours of 10 AM and 4 PM CST. During this transition, our BGP session with you will be gracefully shut down on the existing router before turning the session up on the new equipment. Thank you, Jamison Masters noc(a)veruscorp.com

1 0

FW: Maintenance: ZAYO TTN-0005095553 Planned
by AnthonyAnderberg＠nuvera.net July 9, 2021

July 9, 2021

Looks like the Zayo maintenance below is affecting a number of MICE members, no updated ETR from them at the moment. Additionally, another maintenance event seems to be affecting HE’s Rochester location starting at midnight - hope everyone is doing OK. Cheers, anthony [signature_2077243253] From: MR Zayo <mr(a)zayo.com> Date: Thursday, July 8, 2021 at 10:01 PM Subject: START MAINTENANCE NOTIFICATION***Nuvera Communications***ZAYO TTN-0005095553 Planned*** Dear Zayo Customer, Please be advised that the below scheduled maintenance activity will be commencing momentarily. Maintenance Ticket #: TTN-0005095553 Maintenance Window 1st Activity Date 08-Jul-2021 22:00 to 09-Jul-2021 06:00 ( Central ) 09-Jul-2021 03:00 to 09-Jul-2021 11:00 ( GMT ) Location of Maintenance: 5200' east of Kelly Ave on County Rd 140 in Cologne, MN Reason for Maintenance: Zayo will perform maintenance to replace a box culvert on County Rd 140 in Cologne, MN. Circuit(s) Affected: Circuit Id Expected Impact A Location Address Z Location Address Legacy Circuit Id FBDK/1363902//ZFS Hard Down - Duration of maintenance window If you have any questions or need any additional information, please contact the MR group at mr(a)zayo.com or call 866-236-2824. Regards, Zayo Global Change Management Team/Équipe de Gestion du Changement Global Zayo Zayo | Our Fiber Fuels Global Innovation Toll free/No sans frais: 1.866.236.2824 United Kingdom Toll Free/No sans frais Royaume-Uni: 0800.169.1646 Email/Courriel: mr(a)zayo.com<mailto:releases@zayo.com> Website/Site Web: https://www.zayo.com<http://www.zayo.com/> Purpose<http://www.zayo.com/company/about-zayo/> | Network Map<http://www.zayo.com/solutions/global-network/> | Escalation List<https://tranzact.zayo.com/#!/escalation-lists> | LinkedIn<https://www.linkedin.com/company/530962/> | Twitter<https://twitter.com/zayogroup> | Tranzact <https://tranzact.zayo.com/#!/login> This communication is the property of Zayo and may contain confidential or privileged information. If you have received this communication in error, please promptly notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

3 2

Versus Port 10G Upgrade
by Jay Hanke July 7, 2021

July 7, 2021

Versus is upgrading to 10G New Port: MN VoIP 10g :: Ethernet21 Old Port: MICE Main 1G :: ge-0/0/18 All IP information will remain the same. The old port can be decommissioned at migration. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Zayo Belle Plaine
by Jeremy Lumby July 5, 2021

July 5, 2021

Just an FYI Zayo is having cooling issues in Belle Plaine again. In the past this has caused the MICE extension switch to power itself off as a result of over temperature protection. I have opened a ticket with Zayo. The temperature has increased 20F in the last 45 minutes. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

Please Welcome Involta to MICE!
by Jay Hanke June 24, 2021

June 24, 2021

Name: Involta Switch: Compudyne Duluth Remote Port: xe-0/0/3 ASN: 14230 IPv4: 206.108.255.180/24 IPv6: 2001:504:27::3796:0:1/64 Peering Contact: peering(a)involta.com Route Servers- Yes BFD: NO Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

EIDG
by Jay Hanke June 22, 2021

June 22, 2021

EIDG has completed the upgrade to 1x2x10G SFN Remote Switch Eth9 Eth15 Port Channel 4 Anthony, could you update graphing? -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Jumbo Frames
by Jamison Masters June 19, 2021

June 19, 2021

Hi all, We are migrating our connections to 10G, and I'm wondering if I should request jumbo frames. I'm trying for due diligence, and apologize if this is a rudimentary question. Nanog searches do not have clear recommendations, or maybe I did not search enough. My experience with jumbo frames indicates to me that I should be safe with 9000 byte frames, but there could be PMTU discovery and fragmentation issues during transit or on the receiving side. Would you or would you not enable 9K frames with IXP and ISP connections? I just do not have enough experience in this regard, so I am thankful for any thoughts here. My inclination is to go with 9K frames, yet I do not want to deal with ancillary issues that I cannot anticipate, discern, or troubleshoot - especially during a cut. Any experience from the group is appreciated. Thank you, Jamison Masters Verus Corporation jamison(a)veruscorp.com<mailto:jamison@veruscorp.com>

6 6

Akamai 2x2x100G upgrade
by Jay Hanke June 16, 2021

June 16, 2021

Akamai is upgrading from 1x1x100G LAG to 2x2x100G LAG Port Channel 1 MICE MAIN :: Ethernet3/3/1 [existing] MICE MAIN :: Ethernet3/26/1 [new] Existing IP Information AS20940 IPv4: 206.108.255.84/24 IPv6: 2001:504:27::51cc:0:1/64 Port Channel X <- needs switch ID MICE MAIN :: Ethernet6/8/1 [new] MICE MAIN :: Ethernet6/10/1 [new] New IP Information AS20940 IPv4: 206.108.255.85/24 IPv6: 2001:504:27::51cc:0:2/64 This email will serve as your LOA/CFA and has been filed with Cologix for cross connections to be ordered. We will follow up with an additional email as soon as optics have been installed and the ports have been provisioned. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Emergency Intrusive testing on SFN Albert Lea remote Transport
by Jay Hanke June 15, 2021

June 15, 2021

We are switching the SFN remote onto the backup path to perform intrusive testing. No interruption is expected. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 1

Zayo/Other?
by James Urwiller June 15, 2021

June 15, 2021

Did anyone else loose all MICE peers around 14:20 CST? I have a ticket open, but no response as of yet. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com

4 9

Re: Zayo/Other?
by Justin Krejci June 14, 2021

June 14, 2021

Looking at the MICE weathermap, it looks like most everything is up. I believe there was a fiber cut around that time a bit south of Mankato. Might that be affecting you directly or indirectly? ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Sent: Monday, June 14, 2021 6:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Zayo/Other? Did anyone else loose all MICE peers around 14:20 CST? I have a ticket open, but no response as of yet. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [cid:image001.jpg@01D76147.3F9A7B60] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=162371…>

1 0

Please welcome T-Mobile to MICE!
by Jay Hanke June 10, 2021

June 10, 2021

Name: T-Mobile USA Switch: 511 Main Switch Port: Single LAG Group ports eth3/2/1, Ethernet6/9/1 (2x100G) ASN: 21928 IPv4: 206.108.255.179/24 IPv6: 2001:504:27::55a8:0:1/64 Peering Contact: NTSO-NISS(a)T-Mobile.com Route Servers- Yes BFD: NO Willkommen an Bord! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

4 3

PCH Port Move
by Jay Hanke June 9, 2021

June 9, 2021

PCH is moving their two pending port on the former IP House remote to the MN VoIP remote switch. The ports are allocated as follows: AS42 MN VoIP 10G Switch Eth26 206.108.255.142 2001:504:27:0:0:002a::1 AS3856 MN VoIP 10G Switch Eth27 206.108.255.143 2001:504:27:0:0:0f10::1 MICE Tech- Could you verify the route server sessions are still in the configuration? If not, add them? -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

511 Issues?
by Jay Hanke June 3, 2021

June 3, 2021

Is there something going on at 511? We're seeing multiple networks on MICE down or degraded and an assortment of random issues. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

4 4

EIDG Port Channel Converstion
by Jay Hanke June 2, 2021

June 2, 2021

EIDG is upgrading to a 1x2x10G port channel on the SFN remote switch. Their new port is Ethernet15 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

HE upgrade to 1x2x100G
by Jay Hanke May 11, 2021

May 11, 2021

HE is adding one additional 100G port to their existing Port Channel 6 The new port is: MICE MAIN :: Ethernet6/5/1 This email will also serve as LOA/CFA. All IP information will remain the same. HE requested the upgrade take place ASAP. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Link to all MICE peers dropped
by Frank Bulk May 3, 2021

May 3, 2021

Our link to all MICE peers dropped at 2:47:39 pm - anyone else see an outage and/or disruption? Frank AS18883

3 5

Re: Zayo Belle Plaine - April 15th
by Justin Krejci April 30, 2021

April 30, 2021

We are not getting identical BFD timers or multiplier values from RS1 and RS2. (added blank lines for readability) usi-cr02-mpls#show bfd peers bfd interface ethernet24/1 detail | i ^Peer|^Received|^TxI Peer Addr 206.108.255.1, Intf Ethernet24/1, Type normal, State Up TxInt: 500, RxInt: 500, Multiplier: 3 Received RxInt: 500, Received Multiplier: 3 Peer Addr 206.108.255.2, Intf Ethernet24/1, Type normal, State Up TxInt: 500, RxInt: 500, Multiplier: 3 Received RxInt: 10, Received Multiplier: 5 Peer Addr 2001:504:27::d1af:0:1, Intf Ethernet24/1, Type normal, State Up TxInt: 500, RxInt: 500, Multiplier: 3 Received RxInt: 500, Received Multiplier: 3 Peer Addr 2001:504:27::d1af:0:2, Intf Ethernet24/1, Type normal, State Up TxInt: 500, RxInt: 500, Multiplier: 3 Received RxInt: 10, Received Multiplier: 5

4 4

WiscNet MICE switch maintenance 2021-04-27 7am - Noon
by Chris Wopat April 27, 2021

April 27, 2021

Hi Folks, Heads up that the WiscNet remote MICE switch will be going down for maintenance a week from today - Tuesday April 27th from 7am until approximately noon. We'll gracefully tear down BGP Sessions in advance and will chime in when complete. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

1 1

Zayo Belle Plaine - April 15th
by AnthonyAnderberg＠nuvera.net April 27, 2021

April 27, 2021

Looks like the Zayo Belle Plaine remote went down at 5:37pm, I believe Lee @ SW MN Broadband is opening a ticket now. From the MICE side we have light but no frames, and looking at other unrelated systems it appears the cut is South of Hwy 212 someplace. Best of luck everyone, Anthony

9 21

Old/Stale Infrastructure information - IXP Dashboard / AcenTek
by Jesse Reed April 26, 2021

April 26, 2021

To whom it may concern: Earlier this year, AcenTek worked w/someone to turn up a 2nd link inside a 20 Gig port-ch @ the 511 building facing our equipment. Logging into the IXP portal today, we see that our information is still only reflective of 1 single 10 Gig link. Can this information be updated accordingly inside the AcenTek portal? Thanks ahead of time, Jesse Reed

4 7

WISCNET Additional IP
by Jay Hanke April 16, 2021

April 16, 2021

WISCNET has been allocated 206.108.255.213 for testing of the ARP timeout issues on the WiscNet remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

MIDCO Port Move
by Jay Hanke April 16, 2021

April 16, 2021

Midco will be moving from port 3/25/1 (100G) to port 4/20 (10G) on the main switch. Port provisioning please! All IP information will remain the same. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

lost mice thur belle plaine
by James Urwiller April 14, 2021

April 14, 2021

anyone else about 20 min ago now?

5 9

Google's new 100G from Chicago
by Frank Bulk April 14, 2021

April 14, 2021

Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google's response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: * Clean the interface optic and your patching path to the MMR. * Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883)

5 7

MICE Issues
by Jay Hanke April 12, 2021

April 12, 2021

Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

5 7

MICE UG 39
by Richard Laager April 10, 2021

April 10, 2021

With NANOG happening this fall, it may make sense to schedule MICE UG 39 at the same time (e.g. Wednesday November 3, time TBD). That's not official yet, but tentatively "save the date". Thanks to the person who suggested this (off-list)! -- Richard

1 0

SFN Remote Uplink Change
by Jay Hanke April 1, 2021

April 1, 2021

SFN has upgraded the uplink LAG group on the SFN MICE remote switch to 12x10G from 8x10G. We were getting a small amount of tail drops where we convert from 100GE to the LAG group. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

belle plaine
by James Urwiller April 1, 2021

April 1, 2021

anyone on the belle plain remote just loose connection? we are coming in from fremont, NE.

5 7

Please welcome Xiber LLC to MICE!
by Jay Hanke March 30, 2021

March 30, 2021

Name: Xiber LLC Port: Main Ethernet 4/46 ASN: 393950 IPv4: 206.108.255.178/24 IPv6: 2001:504:27:0:6:2de:0:1/64 Peering Contact: peering(a)xiber.net Route Servers- Yes BFD: Yes This email has been filed with Cologix as your LOA/CFA. Members are no longer required to provide optics for installation in the main switch. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Nanog!
by Reid Fishler March 30, 2021

March 30, 2021

It was just announced the first in person Nanog is in MSP Nov 1-3! Reid -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178

2 1

Please Welcome Range (AS30110) to MICE!
by Jay Hanke March 24, 2021

March 24, 2021

Name: Range Port: Main Ethernet 5/38 ASN: 30110 IPv4: 206.108.255.176/24 IPv6: 2001:504:27::759e:0:1/64 Peering Contact: dave.berry(a)range.net Route Servers- Yes BFD: Yes This email has been filed with Cologix as your LOA/CFA. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Facebook Peers Drop
by Travis Fincham March 21, 2021

March 21, 2021

Did anyone else have their peering with Facebook drop this morning about 7:00 AM? Travis Fincham, CCNA ACS Network Engineer Email: Travis.Fincham(a)keycoop.com<mailto:Travis.Fincham@keycoop.com> Cell: (515)-735-5011 ________________________________ NOTICE: This message (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. § 2510-2521, is CONFIDENTIAL, and is intended only for the individual to whom it is addressed. If you are not the named addressee or an employee or agent responsible for delivering this message to the named addressee, you are hereby notified that you have received this message in error and that any review, retention, dissemination, distribution, copying or use of this communication is strictly prohibited. Please reply to the sender that you received this message in error and then delete it.

4 3

AS16842 Disconnect Notice
by Jay Hanke March 16, 2021

March 16, 2021

AS16842 has disconnected from MICE. Please remove their route server and port configurations. 206.108.255.39 will be returned to the free pool as well as ge-1/1/0 on 1G main switch. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

MICE Terabit Dreams?
by Jay Hanke March 15, 2021

March 15, 2021

When do you think we'll cross over 1 terabit? [image: image.png] Blue line is more accurate as IXP Manager isn't able to grab stats from the Zayo remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

3 2

Please welcome Les.net to MICE!
by Jay Hanke March 15, 2021

March 15, 2021

Name: Les.net Port: MN VoIP 100G Ethernet4/1 ASN: 18451 IPv4: 206.108.255.175/24 IPv6: 2001:504:27::4813:0:1/64 Peering Contact: arin(a)les.net Route Servers- Yes BFD: NO Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Supranet (AS4150) Port Move
by Jay Hanke March 13, 2021

March 13, 2021

Supra net has moved from the 1G main Switch to port Ethernet7 on the MN VoIP remote 10G switch. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

Oneneck Port Upgrade to 10G
by Jay Hanke March 13, 2021

March 13, 2021

Oneneck has been allocated the following port on the MAIN Switch MICE MAIN :: Ethernet3/7/1 This will replace their existing 1G connection. All IP/Route Server information will remain the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 1

Lost MICE VMWare server
by Jay Hanke March 12, 2021

March 12, 2021

The VMWare server that hosts RS3 and IXP Manager for MICE died. We're onsite working on restoration now. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

4 5

Please Welcome Metronet to MICE!
by Jay Hanke March 9, 2021

March 9, 2021

Name: Metronet Port: Main Ethernet6/7/1 (100G New Single Member LAG) ASN: 30600 IPv4: 206.108.255.174/24 IPv6: 2001:504:27::7788:0:1/64 Peering Contact: peering(a)metronetinc.com Route Servers- NO BFD: NO Please note this is in addition to the existing Jaguar/Metronet ports not a replacement. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Re: Packet Loss between 00:00 and 05:00 this morning
by Jeremy Lumby March 6, 2021

March 6, 2021

They do come in on one of the Zayo circuits that was impacted last night. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> Date: 3/6/21 2:45 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 14:45:37 Our Qrator monitoring noticed that BANKONIT’s BGP session going in and out many times between 11:17 pm to 5:59 am Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Keith Wirch Sent: Saturday, March 6, 2021 9:29 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

1 0

Packet Loss between 00:00 and 05:00 this morning
by Keith Wirch March 6, 2021

March 6, 2021

2 1

Re: Packet Loss between 00:00 and 05:00 this morning
by Jeremy Lumby March 6, 2021

March 6, 2021

I had some Zayo circuits leaving Minneapolis that were experiencing high packet loss during that exact same time frame Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Keith Wirch <kwirch(a)ATOMICDATA.COM> Date: 3/6/21 9:29 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Keith Wirch [kwirch(a)ATOMICDATA.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 09:29:00 Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

1 0

Hulu Live/Fastly
by Frank Bulk March 2, 2021

March 2, 2021

We've received 5 to 7 reports about Hulu Live issues ("freezing up") in the last week or so. They're not all on Roku as we've had at least one person with an Amazon Firestick. For those where we've identified the source IP it's been Fastly, which is present at MICE. Curious if anyone else has received an uptick in Hulu Live complaints? We have not (yet) been able to reproduce the issue internally. Other streaming services at these same customers appear fine. Raw throughput in speed tests to our speed test server are fine. The one customer that we know did call Hulu was told by Hulu support that their issue was with their ISP (us). Frank

2 2

MPW Port Move
by Jay Hanke Feb. 26, 2021

Feb. 26, 2021

MPW has moved from Port Ethernet 7 on the MN VoIP switch to port Ethernet 12 on the SFN remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

MICE UG 37
by Richard Laager Feb. 26, 2021

Feb. 26, 2021

Reminder: UG 37 is on Wednesday, March 10, at 3:00pm This will be on Google Meet: https://meet.google.com/zgg-eeyu-dxz In the event of serious problems, we will fall back to: 952-562-5400 passcode MICE (6423) Agenda The agenda is attached. If you have something to add, let me know. The UG 36 draft minutes are also attached. -- Richard

1 0

Sling TV
by Jeremy Lumby Feb. 23, 2021

Feb. 23, 2021

Has anyone else been experiencing issues with Sling TV's cloud DVR service, especially during peak hours within the past couple months? While they are not directly connected to MICE, some of their transit providers are. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

SFN MICE Remote Scheduled Maintenance
by Jay Hanke Feb. 23, 2021

Feb. 23, 2021

The SFN Remote switch will be undergoing maintenance and will be power cycled during our transport provider doing software upgrades. The SFN remote switch will be hard down during the part of the maintenance and be isolated during part of the maintenance. Connectivity may be intermittent during the entire window as software updates are applied. This will also affect test filtering sessions to MICE-RS3. Primary Window: FROM: Tuesday, Feb 23, 2021 0030 hours (CST) TO: Tuesday, Feb 23, 2021 0230 hours (CST) Backup Window: FROM: Tuesday, Feb 24, 2021 0030 hours (CST) TO: Tuesday, Feb 24, 2021 0230 hours (CST) Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 1

Apple links filling up?
by Frank Bulk Feb. 23, 2021

Feb. 23, 2021

The last two nights our NAGIOS-based monitoring of HTTPv6 access to http://www.apple.com has been timing out from time to time. Am I interpreting MICE's graphs correctly to suggest that Apple's two (1x10G?) LAGs are filling up? Frank

2 2

HE.net Routes Lost - Around 15:05 CST ?
by Jesse Reed Feb. 19, 2021

Feb. 19, 2021

Anyone else lose routes via HE shortly after 15:00 CST today? It was brief for us, but wondering if anyone else experienced this? Out BGP neighbor(s) remained stable but it appears like we dropped traffic during this time. Thanks ahead of time, Jesse Reed Network Operations Specialist [cid:image001.jpg@01D706D4.B26536D0] jreed(a)AcenTek.net<mailto:jreed@AcenTek.net> www.AcenTek.net

3 2