- mice-discuss - lists.micemn.net

Re: Lost MICE VMWare server

by Mike Horwath

On Wed, Mar 10, 2021 at 09:25:31PM -0600, Jay Hanke wrote: > The VMWare server that hosts RS3 and IXP Manager for MICE died. We're > onsite working on restoration now. Yuge bummer. With a tiny bit of effort I could host these on my VMware cluster located in what was the ipHouse datacenter. By effort I only mean some routing things, and an ethernet pull. While this is my 'lab' it's uptime is very high, and there would be backups, redundant networking and storae systems. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

March 12, 2021

Lost MICE VMWare server

by Jay Hanke

The VMWare server that hosts RS3 and IXP Manager for MICE died. We're onsite working on restoration now. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

March 11, 2021

Re: Please Welcome Metronet to MICE!

by Doug McIntyre

On Tue, Mar 09, 2021 at 08:00:02AM -0600, Jay Hanke wrote: >Name: Metronet >Port: Main Ethernet6/7/1 (100G New Single Member LAG) >ASN: 30600 >IPv4: 206.108.255.174/24 >IPv6: 2001:504:27::7788:0:1/64 >Peering Contact: peering(a)metronetinc.com >Route Servers- NO >BFD: NO > >Please note this is in addition to the existing Jaguar/Metronet ports >not a replacement. Setup in the switch for Port-Channel #40

March 9, 2021

Please Welcome Metronet to MICE!

by Jay Hanke

Name: Metronet Port: Main Ethernet6/7/1 (100G New Single Member LAG) ASN: 30600 IPv4: 206.108.255.174/24 IPv6: 2001:504:27::7788:0:1/64 Peering Contact: peering(a)metronetinc.com Route Servers- NO BFD: NO Please note this is in addition to the existing Jaguar/Metronet ports not a replacement. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

March 9, 2021

Re: Packet Loss between 00:00 and 05:00 this morning

by Jeremy Lumby

They do come in on one of the Zayo circuits that was impacted last night. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> Date: 3/6/21 2:45 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 14:45:37 Our Qrator monitoring noticed that BANKONIT’s BGP session going in and out many times between 11:17 pm to 5:59 am Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Keith Wirch Sent: Saturday, March 6, 2021 9:29 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 6, 2021

Re: Packet Loss between 00:00 and 05:00 this morning

by Frank Bulk

Our Qrator monitoring noticed that BANKONIT’s BGP session going in and out many times between 11:17 pm to 5:59 am Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Keith Wirch Sent: Saturday, March 6, 2021 9:29 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 6, 2021

Re: Packet Loss between 00:00 and 05:00 this morning

by Jeremy Lumby

I had some Zayo circuits leaving Minneapolis that were experiencing high packet loss during that exact same time frame Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Keith Wirch <kwirch(a)ATOMICDATA.COM> Date: 3/6/21 9:29 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Packet Loss between 00:00 and 05:00 this morning _____ >From : Keith Wirch [kwirch(a)ATOMICDATA.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Saturday, March 6 2021 09:29:00 Anyone else experience intermittent packet loss between 00:00 and 05:00 this morning? We noticed a bunch of our IPSEC tunnels were coming and going and website availability was hit and miss. We thought we saw loss in Level3 so we turned down Peering but we still were seeing loss on other providers. Didn’t seem to be MICE specific but were not able to pinpoint the issue exactly. Keith Wirch Infrastructure Engineer ---- Atomic Data® // Safe. Simple. Smart.® 250 Marquette Avenue Minneapolis, MN 55401 Main // 612.466.2000 Direct // 612.466.2106

March 6, 2021

Packet Loss between 00:00 and 05:00 this morning

by Keith Wirch

March 6, 2021

Re: Hulu Live/Fastly

by Frank Bulk

Thanks. Our helpdesk has documented 10 customers that are across a variety of access networks and prefixes and have a variety of RGs. Still haven't identified a smoking gun. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Richard Laager Sent: Monday, March 1, 2021 9:48 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Hulu Live/Fastly On 3/1/21 5:28 PM, Frank Bulk wrote: We've received 5 to 7 reports about Hulu Live issues ("freezing up") in the last week or so. They're not all on Roku as we've had at least one person with an Amazon Firestick. For those where we've identified the source IP it's been Fastly, which is present at MICE. Curious if anyone else has received an uptick in Hulu Live complaints? We had one guy with a Roku TV who said it did it more than once today. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 2, 2021

Re: Hulu Live/Fastly

by Richard Laager

On 3/1/21 5:28 PM, Frank Bulk wrote: > > We’ve received 5 to 7 reports about Hulu Live issues (“freezing up”) > in the last week or so. They’re not all on Roku as we’ve had at least > one person with an Amazon Firestick. For those where we’ve identified > the source IP it’s been Fastly, which is present at MICE. > > Curious if anyone else has received an uptick in Hulu Live complaints? > We had one guy with a Roku TV who said it did it more than once today. -- Richard

March 2, 2021

Hulu Live/Fastly

by Frank Bulk

We've received 5 to 7 reports about Hulu Live issues ("freezing up") in the last week or so. They're not all on Roku as we've had at least one person with an Amazon Firestick. For those where we've identified the source IP it's been Fastly, which is present at MICE. Curious if anyone else has received an uptick in Hulu Live complaints? We have not (yet) been able to reproduce the issue internally. Other streaming services at these same customers appear fine. Raw throughput in speed tests to our speed test server are fine. The one customer that we know did call Hulu was told by Hulu support that their issue was with their ISP (us). Frank

March 1, 2021

MPW Port Move

by Jay Hanke

MPW has moved from Port Ethernet 7 on the MN VoIP switch to port Ethernet 12 on the SFN remote. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Feb. 26, 2021

MICE UG 37

by Richard Laager

Reminder: UG 37 is on Wednesday, March 10, at 3:00pm This will be on Google Meet: https://meet.google.com/zgg-eeyu-dxz In the event of serious problems, we will fall back to: 952-562-5400 passcode MICE (6423) Agenda The agenda is attached. If you have something to add, let me know. The UG 36 draft minutes are also attached. -- Richard

Feb. 26, 2021

Sling TV

by Jeremy Lumby

Has anyone else been experiencing issues with Sling TV's cloud DVR service, especially during peak hours within the past couple months? While they are not directly connected to MICE, some of their transit providers are. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Feb. 23, 2021

Re: SFN MICE Remote Scheduled Maintenance

by Jay Hanke

This work is completed. All services restored. On Sun, Feb 14, 2021 at 11:11 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > The SFN Remote switch will be undergoing maintenance and will be power > cycled during our transport provider doing software upgrades. > > The SFN remote switch will be hard down during the part of the > maintenance and be isolated during part of the maintenance. Connectivity > may be intermittent during the entire window as software updates are > applied. > > This will also affect test filtering sessions to MICE-RS3. > > Primary Window: > FROM: Tuesday, Feb 23, 2021 0030 hours (CST) > TO: Tuesday, Feb 23, 2021 0230 hours (CST) > > Backup Window: > FROM: Tuesday, Feb 24, 2021 0030 hours (CST) > TO: Tuesday, Feb 24, 2021 0230 hours (CST) > > Thanks! > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Feb. 23, 2021

Re: Apple links filling up?

by Frank Bulk

Ok, thanks. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Tuesday, February 23, 2021 12:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Apple links filling up? they are 100G links On Mon, Feb 22, 2021 at 11:35 PM Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: The last two nights our NAGIOS-based monitoring of HTTPv6 access to http://www.apple.com has been timing out from time to time. Am I interpreting MICE's graphs correctly to suggest that Apple's two (1x10G?) LAGs are filling up? Frank -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 23, 2021

Re: Apple links filling up?

by Jay Hanke

they are 100G links On Mon, Feb 22, 2021 at 11:35 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > The last two nights our NAGIOS-based monitoring of HTTPv6 access to > http://www.apple.com has been timing out from time to time. > > Am I interpreting MICE's graphs correctly to suggest that Apple's two > (1x10G?) LAGs are filling up? > > Frank > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Feb. 23, 2021

Apple links filling up?

by Frank Bulk

The last two nights our NAGIOS-based monitoring of HTTPv6 access to http://www.apple.com has been timing out from time to time. Am I interpreting MICE's graphs correctly to suggest that Apple's two (1x10G?) LAGs are filling up? Frank

Feb. 23, 2021

Re: HE.net Routes Lost - Around 15:05 CST ?

by Darin Steffl

Yes we saw a complete drop in traffic out of the Rochester pop. It slowly came back to normal levels. On Fri, Feb 19, 2021, 3:39 PM Larry Larsen <llarsen(a)longlines.biz> wrote: > I did not notice any drop in traffic. Maybe I just got lucky? > > Thank you! > > Larry L. > > ------------------------------ > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jesse > Reed <jreed(a)ACENTEK.NET> > *Sent:* Friday, February 19, 2021 3:34:24 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* [MICE-DISCUSS] HE.net Routes Lost - Around 15:05 CST ? > > > WARNING!! This message originated from an *External Source*. Please use > proper judgment and caution when opening attachments, clicking links, or > responding to this email. > > Anyone else lose routes via HE shortly after 15:00 CST today? It was > brief for us, but wondering if anyone else experienced this? Out BGP > neighbor(s) remained stable but it appears like we dropped traffic during > this time. > > > > Thanks ahead of time, > > > > *Jesse Reed* > > Network Operations Specialist > > *jreed(a)AcenTek.net <jreed(a)AcenTek.net>* > > *www.AcenTek.net* <http://www.AcenTek.net> > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Feb. 19, 2021

Re: HE.net Routes Lost - Around 15:05 CST ?

by Larry Larsen

I did not notice any drop in traffic. Maybe I just got lucky? Thank you! Larry L. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jesse Reed <jreed(a)ACENTEK.NET> Sent: Friday, February 19, 2021 3:34:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] HE.net Routes Lost - Around 15:05 CST ? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. Anyone else lose routes via HE shortly after 15:00 CST today? It was brief for us, but wondering if anyone else experienced this? Out BGP neighbor(s) remained stable but it appears like we dropped traffic during this time. Thanks ahead of time, Jesse Reed Network Operations Specialist [cid:image001.jpg@01D706D4.B26536D0] jreed(a)AcenTek.net<mailto:jreed@AcenTek.net> www.AcenTek.net ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Feb. 19, 2021

HE.net Routes Lost - Around 15:05 CST ?

by Jesse Reed

Anyone else lose routes via HE shortly after 15:00 CST today? It was brief for us, but wondering if anyone else experienced this? Out BGP neighbor(s) remained stable but it appears like we dropped traffic during this time. Thanks ahead of time, Jesse Reed Network Operations Specialist [cid:image001.jpg@01D706D4.B26536D0] jreed(a)AcenTek.net<mailto:jreed@AcenTek.net> www.AcenTek.net

Feb. 19, 2021

WiscNet Remote Switch maintenance 2021-02-23 5pm - 10pm

by Chris Wopat

Hi Folks, The WiscNet remote switch, which currently only has active sessions for WiscNet, will be under maintenance this Tuesday, Feb 23rd at 5pm local time. We have work impacting our backbone paths to Minneapolis, so will be draining traffic from peers at this location during this time beginning at 5pm. We expect things to be fully online around 10pm. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

Feb. 19, 2021

Re: Amazon LAG Augmentations

by Jeremy Lumby

The switch has been configured, and is ready for optics/cross connects from Amazon -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Tuesday, February 16, 2021 9:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Amazon LAG Augmentations Amazon is adding 100G to both of their existing port channels. PO5: 3/12/1, 6/4/1 (New) PO24 6/1/1, 3/21/1 (New) All IP information will remain the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Feb. 16, 2021

Amazon LAG Augmentations

by Jay Hanke

Amazon is adding 100G to both of their existing port channels. PO5: 3/12/1, 6/4/1 (New) PO24 6/1/1, 3/21/1 (New) All IP information will remain the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Feb. 16, 2021

Re: Please Welcome Mid-Rivers Telephone to MICE

by Doug McIntyre

On Mon, Feb 15, 2021 at 10:48:39AM -0600, Jay Hanke wrote: >Name: Mid-Rivers Telephone >Port: Main Switch (New 20G LAG) Ethernet4/12, Ethernet5/21 >ASN: 11961 >IPv4: 206.108.255.173/24 >IPv6: 2001:504:27::2eb9:0:1/64 >Peering Contact: nse(a)midrivers.coop >Route Servers- YES >BFD: No > >Please note these may be reclaimed ports. MICE uses LACP by default on >all LAG ports. I've got the RR's setup, and the switch setup done. Port-Channel 39 has been allocated for this LACP LAG group. Welcome!

Feb. 15, 2021