- mice-discuss - lists.micemn.net

Please welcome The Constant Company to MICE!

by Jeremy Lumby

Name: The Constant Company Switch: MICE Core 511-100G 7504 Ports: E 6/12/1 ASN: 20473 IPv4: 206.108.255.15/24 IPv6: 2001:504:27:0:0:4ff9::1/64 Peering Contact: peering(a)vultr.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Aug. 19, 2026

MICE Server Updates

by Cameron Skappel

I plan to update the MICE servers on Wednesday, August 19th, at 13:00 Central and Thursday, August 20th, at 13:00 Central. I will update one route server per day. -- Cameron Skappel Wikstrom Telephone Company 218-436-5226

Aug. 17, 2026

Re: Discussion / Issues Distro List

by Michael Horwath

On Fri, Aug 07, 2026 at 11:52:26AM +0000, Jerry Spaight wrote: > Hi - Could you please add Shaun Krug (SKrug(a)atccomm.net<mailto:SKrug@atccomm.net>) to the distribution list? Or send them (you had CCed so my instructions work directly as well) to https://lists.micemn.net/postorius/lists/ and subscribe to one or both. -- Michael Horwath, reachable via drechsau(a)Geeks.ORG

Aug. 7, 2026

Discussion / Issues Distro List

by Jerry Spaight

Hi - Could you please add Shaun Krug (SKrug(a)atccomm.net<mailto:SKrug@atccomm.net>) to the distribution list? Thanks in advance for your help! Jerry Spaight General Manager [ATC-Com[C]] 85 Main Ave Atkins, IA 52206 Now Offering 2.5 & 5.0 Gigabit Fiber WWW.ATCCOMM.NET<http://www.atccomm.net/> 319-446-0500 - Direct / Desk (Preferred) 319-521-4366 - Mobile 319-446-9100 - Fax

Aug. 7, 2026

Re: 511 Power Issues

by Richard Laager

When the switch came back, there was some traffic flooding. Unknown unicast flooding, I assume. Jeremy cleared the MAC table, which made the switch happy. (I suppose that caused it to re-learn in hardware or something?) It's been stable since 3:33 PM or so. You should be able to re-enable things now. On 2026-08-06 15:33, Gartley, Dakota S via mice-discuss wrote: > We’re seeing this weird traffic spike on one of our MICE interfaces > towards our Elk River datacenter. I also see it towards other > customers when I look at the MICE cacti instance. Anyone else notice > this? We’ve shutdown our connection in Elk River for the time being. > > image.png > > image.png > > *____________________________* > > *Dakota Gartley | Optum Technology* > Director, Information Security > Data, Internet Security and Connectivity Operations (DISCO) > Network, Security and Internet Services (NSIS) > > T _+1 763-732-6405_ <tel:+17637326405> > _dakota(a)optum.com_ <mailto:dakota@optum.com> > > optum-logo-ora-rgb.png > * > * > *From: *Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> > *Date: *Thursday, August 6, 2026 at 3:28 PM > *To: *MICE Discuss <mice-discuss(a)lists.micemn.net> > *Subject: *511 Power Issues > > *Caution: External email.** Do not open attachments or click on links > if you do not recognize the sender.* > > > There are currently power issues in the 511 building affecting the > MICE infrastructure there. > > The MICE Ridgeview switch and route server are still operational. > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 507-327-2398 > https://southfront.io/ > <https://urldefense.com/v3/__https://southfront.io/path.html__;!!IqUcNYopQPk…> > > > This e-mail, including attachments, may include confidential and/or > proprietary information, and may be used only by the person or entity > to which it is addressed. If the reader of this e-mail is not the intended > recipient or intended recipient’s authorized agent, the reader is hereby > notified that any dissemination, distribution or copying of this e-mail is > prohibited. If you have received this e-mail in error, please notify the > sender by replying to this message and delete this e-mail immediately. > > _______________________________________________ > mice-discuss mailing list --mice-discuss(a)lists.micemn.net > To unsubscribe send an email tomice-discuss-leave(a)lists.micemn.net -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

Aug. 6, 2026

Re: Loss of connectivity to MICE

by Ben Wiechman

Correction MIN1-MIN4 Case #1879022: MIN1&2 Investigating Reported Power Issues Case #1879175: MIN3 Investigating Reported Power Issues Case #1879246: MIN4 Investigating Reported Power Issues First emails reporting issues arrived around 1453 Cleared notification was sent around 1518 Ben Wiechman Chief Technical Officer 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com Book a meeting with me: https://calendar.app.google/HdzQXtfWGPvgxFn58 On Thu, Aug 6, 2026 at 3:49 PM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > We had reports from Cologix of power issues in MIN3 and MIN4. They sent > notice that they believed the issues were resolved around 3:20 PM. > > > Ben Wiechman > > Chief Technical Officer > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > Book a meeting with me: https://calendar.app.google/HdzQXtfWGPvgxFn58 > > > > > > > > On Thu, Aug 6, 2026 at 3:43 PM Louis McLennan via mice-discuss < > mice-discuss(a)lists.micemn.net> wrote: > >> Our gear is online in MIN3 but on the IP side lost GTT, MICE and it also >> seems Cogent is sending no routes elsewhere in the metro. Relying on >> Chicago. >> >> On Fri, 7 Aug 2026, 6:12 am Dean Bahls, <dean.bahls(a)fiberminnesota.com> >> wrote: >> >>> We run batteries on our DC plant at 511 and they have drawn down to 38 >>> VDC. I can’t believe anything in our cage is still operating. >>> >>> >>> >>> I don’t understand why Cologix UPS and generators aren’t running down >>> there.. >>> >>> >>> >>> Dean >>> >>> >>> >>> >>> >>> *From:* Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> >>> *Sent:* Thursday, August 6, 2026 2:19 PM >>> *To:* Jeremy Schmeichel <Jeremy.Schmeichel(a)mybluepeak.com> >>> *Cc:* Josh Hoppes <Josh.Hoppes(a)cfunet.net>; >>> mice-discuss(a)lists.micemn.net >>> *Subject:* Re: Loss of connectivity to MICE >>> >>> >>> >>> Getting lots of reports about Cologix MSP. Both MICE and providers >>> >>> >>> >>> On Thu, Aug 6, 2026 at 2:13 PM Jeremy Schmeichel via mice-discuss < >>> mice-discuss(a)lists.micemn.net> wrote: >>> >>> We just lost a core router out there and all connectivity - feels like a >>> possible power issue. >>> >>> >>> >>> *From: *Josh Hoppes <Josh.Hoppes(a)cfunet.net> >>> *Date: *Thursday, August 6, 2026 at 1:05 PM >>> *To: *mice-discuss(a)lists.micemn.net <mice-discuss(a)lists.micemn.net> >>> *Subject: *Loss of connectivity to MICE >>> >>> [You don't often get email from josh.hoppes(a)cfunet.net. Learn why this >>> is important at https://aka.ms/LearnAboutSenderIdentification ] >>> >>> We just saw connectivity to MICE go down as well as our Hurricane >>> Electric transit circuit. Are there any known issues at Cologix Minneapolis? >>> >>> Josh Hoppes >>> Network Engineer II >>> Cedar Falls Utilities >>> Office 319 268 5227 >>> Cell 319 610 5332 >>> www.cfu.net >>> >>> _______________________________________________ >>> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >>> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >>> >>> _______________________________________________ >>> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >>> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >>> >>> *CAUTION:* This email originated outside the organization. Do not click >>> links or open attachments unless you recognize the sender. >>> _______________________________________________ >>> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >>> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >>> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >> >

Aug. 6, 2026

Re: Loss of connectivity to MICE

by Ben Wiechman

We had reports from Cologix of power issues in MIN3 and MIN4. They sent notice that they believed the issues were resolved around 3:20 PM. Ben Wiechman Chief Technical Officer 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com Book a meeting with me: https://calendar.app.google/HdzQXtfWGPvgxFn58 On Thu, Aug 6, 2026 at 3:43 PM Louis McLennan via mice-discuss < mice-discuss(a)lists.micemn.net> wrote: > Our gear is online in MIN3 but on the IP side lost GTT, MICE and it also > seems Cogent is sending no routes elsewhere in the metro. Relying on > Chicago. > > On Fri, 7 Aug 2026, 6:12 am Dean Bahls, <dean.bahls(a)fiberminnesota.com> > wrote: > >> We run batteries on our DC plant at 511 and they have drawn down to 38 >> VDC. I can’t believe anything in our cage is still operating. >> >> >> >> I don’t understand why Cologix UPS and generators aren’t running down >> there.. >> >> >> >> Dean >> >> >> >> >> >> *From:* Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> >> *Sent:* Thursday, August 6, 2026 2:19 PM >> *To:* Jeremy Schmeichel <Jeremy.Schmeichel(a)mybluepeak.com> >> *Cc:* Josh Hoppes <Josh.Hoppes(a)cfunet.net>; mice-discuss(a)lists.micemn.net >> *Subject:* Re: Loss of connectivity to MICE >> >> >> >> Getting lots of reports about Cologix MSP. Both MICE and providers >> >> >> >> On Thu, Aug 6, 2026 at 2:13 PM Jeremy Schmeichel via mice-discuss < >> mice-discuss(a)lists.micemn.net> wrote: >> >> We just lost a core router out there and all connectivity - feels like a >> possible power issue. >> >> >> >> *From: *Josh Hoppes <Josh.Hoppes(a)cfunet.net> >> *Date: *Thursday, August 6, 2026 at 1:05 PM >> *To: *mice-discuss(a)lists.micemn.net <mice-discuss(a)lists.micemn.net> >> *Subject: *Loss of connectivity to MICE >> >> [You don't often get email from josh.hoppes(a)cfunet.net. Learn why this >> is important at https://aka.ms/LearnAboutSenderIdentification ] >> >> We just saw connectivity to MICE go down as well as our Hurricane >> Electric transit circuit. Are there any known issues at Cologix Minneapolis? >> >> Josh Hoppes >> Network Engineer II >> Cedar Falls Utilities >> Office 319 268 5227 >> Cell 319 610 5332 >> www.cfu.net >> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >> >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >> >> *CAUTION:* This email originated outside the organization. Do not click >> links or open attachments unless you recognize the sender. >> _______________________________________________ >> mice-discuss mailing list -- mice-discuss(a)lists.micemn.net >> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >> > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

Aug. 6, 2026

Re: 511 Power Issues

by Gartley, Dakota S

We’re seeing this weird traffic spike on one of our MICE interfaces towards our Elk River datacenter. I also see it towards other customers when I look at the MICE cacti instance. Anyone else notice this? We’ve shutdown our connection in Elk River for the time being. [image.png] [image.png] ____________________________ Dakota Gartley | Optum Technology Director, Information Security Data, Internet Security and Connectivity Operations (DISCO) Network, Security and Internet Services (NSIS) T +1 763-732-6405<tel:+17637326405> dakota(a)optum.com<mailto:dakota@optum.com> [optum-logo-ora-rgb.png] From: Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> Date: Thursday, August 6, 2026 at 3:28 PM To: MICE Discuss <mice-discuss(a)lists.micemn.net> Subject: 511 Power Issues Caution: External email. Do not open attachments or click on links if you do not recognize the sender. There are currently power issues in the 511 building affecting the MICE infrastructure there. The MICE Ridgeview switch and route server are still operational. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 507-327-2398 https://southfront.io/<https://urldefense.com/v3/__https://southfront.io/path.html__;!!IqUcNYopQPk…> This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or intended recipient’s authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.

Aug. 6, 2026

Re: Loss of connectivity to MICE

by Louis McLennan

Our gear is online in MIN3 but on the IP side lost GTT, MICE and it also seems Cogent is sending no routes elsewhere in the metro. Relying on Chicago. On Fri, 7 Aug 2026, 6:12 am Dean Bahls, <dean.bahls(a)fiberminnesota.com> wrote: > We run batteries on our DC plant at 511 and they have drawn down to 38 > VDC. I can’t believe anything in our cage is still operating. > > > > I don’t understand why Cologix UPS and generators aren’t running down > there.. > > > > Dean > > > > > > *From:* Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net> > *Sent:* Thursday, August 6, 2026 2:19 PM > *To:* Jeremy Schmeichel <Jeremy.Schmeichel(a)mybluepeak.com> > *Cc:* Josh Hoppes <Josh.Hoppes(a)cfunet.net>; mice-discuss(a)lists.micemn.net > *Subject:* Re: Loss of connectivity to MICE > > > > Getting lots of reports about Cologix MSP. Both MICE and providers > > > > On Thu, Aug 6, 2026 at 2:13 PM Jeremy Schmeichel via mice-discuss < > mice-discuss(a)lists.micemn.net> wrote: > > We just lost a core router out there and all connectivity - feels like a > possible power issue. > > > > *From: *Josh Hoppes <Josh.Hoppes(a)cfunet.net> > *Date: *Thursday, August 6, 2026 at 1:05 PM > *To: *mice-discuss(a)lists.micemn.net <mice-discuss(a)lists.micemn.net> > *Subject: *Loss of connectivity to MICE > > [You don't often get email from josh.hoppes(a)cfunet.net. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > We just saw connectivity to MICE go down as well as our Hurricane Electric > transit circuit. Are there any known issues at Cologix Minneapolis? > > Josh Hoppes > Network Engineer II > Cedar Falls Utilities > Office 319 268 5227 > Cell 319 610 5332 > www.cfu.net > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net > > *CAUTION:* This email originated outside the organization. Do not click > links or open attachments unless you recognize the sender. > _______________________________________________ > mice-discuss mailing list -- mice-discuss(a)lists.micemn.net > To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net >

Aug. 6, 2026

RE: [External] Re: Loss of connectivity to MICE

by Justin Huffman

Yep, seeing 90G of traffic incoming from MICE. _________________________________ Justin Huffman Sr. Manager | Network Engineering Long Lines Broadband 712-271-5902 (W) jhuffman(a)longlines.biz<mailto:jhuffman@longlines.biz> [LL new] From: Tom Krenn <Tom.Krenn(a)hennepincounty.gov> Sent: Thursday, August 6, 2026 3:14 PM To: Jay Hanke <jayhanke(a)southfront.io>; Justin Huffman <jhuffman(a)longlines.biz> Cc: Josh Hoppes <Josh.Hoppes(a)cfunet.net>; mice-discuss(a)lists.micemn.net Subject: RE: [External] Re: Loss of connectivity to MICE You don't often get email from tom.krenn(a)hennepincounty.gov<mailto:tom.krenn@hennepincounty.gov>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. I see things starting to restore, but we are also seeing what looks like 80gbps+ of traffic from MICE toward us. That is extremely unusual. Any routing issues happening on the MICE core or route-reflectors? Tom Krenn Network Architect Enterprise Architecture - Information Technology [Hennepin County logo] From: Jay Hanke via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> Sent: Thursday, August 6, 2026 2:35 PM To: Justin Huffman <jhuffman(a)longlines.biz<mailto:jhuffman@longlines.biz>> Cc: Josh Hoppes <Josh.Hoppes(a)cfunet.net<mailto:Josh.Hoppes@cfunet.net>>; mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: [External] Re: Loss of connectivity to MICE CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments. Sounds like an issue in suite 100. No real details yet. On Thu, Aug 6, 2026 at 2:21 PM Justin Huffman via mice-discuss <mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net>> wrote: We did as well along with other connections. _________________________________ Justin Huffman Sr. Manager | Network Engineering Long Lines Broadband 712-271-5902 (W) jhuffman(a)longlines.biz<mailto:jhuffman@longlines.biz> -----Original Message----- From: Josh Hoppes <Josh.Hoppes(a)cfunet.net<mailto:Josh.Hoppes@cfunet.net>> Sent: Thursday, August 6, 2026 2:03 PM To: mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> Subject: Loss of connectivity to MICE [You don't often get email from josh.hoppes(a)cfunet.net<mailto:josh.hoppes@cfunet.net>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. We just saw connectivity to MICE go down as well as our Hurricane Electric transit circuit. Are there any known issues at Cologix Minneapolis? Josh Hoppes Network Engineer II Cedar Falls Utilities Office 319 268 5227 Cell 319 610 5332 http://www.cfu.net/ _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> _______________________________________________ mice-discuss mailing list -- mice-discuss(a)lists.micemn.net<mailto:mice-discuss@lists.micemn.net> To unsubscribe send an email to mice-discuss-leave(a)lists.micemn.net<mailto:mice-discuss-leave@lists.micemn.net> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

Aug. 6, 2026