- mice-discuss - lists.micemn.net

Re: MICE Issues

by Frank Bulk

That's unfortunate! Since this could easily happen again, what could be done to troubleshoot and/or mitigate the issue? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Sunday, April 11, 2021 8:25 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Issues When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 12, 2021

Re: MICE Issues

by Jeremy Lumby

When half of the power was taken down for maintenance, it caused a surge on the remaining side that tripped the breaker even though the circuit runs at less than 80% under failure conditions. The breaker has been reset, and things are coming back online. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Sunday, April 11, 2021 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Issues Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 11, 2021

MICE Issues

by Jay Hanke

Cologix has done power maintenance in Suite 100 that has inadvertently affected the MN VoIP remote switch as well as most of the MICE monitoring. This has isolated the SFN Remote Switch in Albert Lea. Cologix is investigating. No ETTR yet. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 11, 2021

Re: Google's new 100G from Chicago

by Frank Bulk

Thanks! Interesting how an issue like this happens late on a Friday … if it happened by itself almost surely a bug or hardware issue. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Saturday, April 10, 2021 11:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google's new 100G from Chicago Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: • Clean the interface optic and your patching path to the MMR. • Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 10, 2021

Re: Google's new 100G from Chicago

by Jay Hanke

Yes, this is not the first time we've seen issues on these links. I did reach out to them and haven't gotten a response yet. On Sat, Apr 10, 2021 at 11:02 AM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Frank, > > > > Yes, we received a notification from Google last night. > We are looking into it. We are receiving light but no Ethernet framing. I > am 99% sure the issue lies with their wave provider. > > > > Jeremy > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Frank Bulk > *Sent:* Saturday, April 10, 2021 10:52 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Google's new 100G from Chicago > > > > Our BGP session over this new link went down Friday just before 6 pm. > When I opened a ticket this was Google’s response: > > > > ======================== > > We have informed you via G-127797702 > > Hello MICE (Midwest Internet Cooperative Exchange), > > We have encountered issues with the following physical network link(s) you > have with Google, to start troubleshooting and prevent any further impact > to traffic we have removed traffic from the link(s). > Please check the below list: > > Google ASN > > Your ASN > > Google Address > > Your Address > > Your Demarc > > Metro > > Problem > > Speed > > 15169 > > PX.MICE.ORD > > 206.108.255.167 > > Peering Exchange > > AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 > > ORD > > Down > > 100 Gbps > > > Tips for troubleshooting physical link problems: > > · Clean the interface optic and your patching path to the MMR. > > · Replace the interface optic. > > > Please check your link(s) and attempt to resolve the problems described > above. > > Thanks! > ======================== > > > > Is someone from MICE performing the requested actions and/or looking into > this? > > > > Kind regards, > > > > Frank Bulk > > FiberNet (ASN18883) > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 10, 2021

Re: Google's new 100G from Chicago

by Jeremy Lumby

Frank, Yes, we received a notification from Google last night. We are looking into it. We are receiving light but no Ethernet framing. I am 99% sure the issue lies with their wave provider. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Saturday, April 10, 2021 10:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Google's new 100G from Chicago Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google’s response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASNYour ASNGoogle AddressYour AddressYour DemarcMetroProblemSpeed 15169PX.MICE.ORD206.108.255.167Peering ExchangeAS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32ORDDown100 Gbps Tips for troubleshooting physical link problems: · Clean the interface optic and your patching path to the MMR. · Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883) To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 10, 2021

Google's new 100G from Chicago

by Frank Bulk

Our BGP session over this new link went down Friday just before 6 pm. When I opened a ticket this was Google's response: ======================== We have informed you via G-127797702 Hello MICE (Midwest Internet Cooperative Exchange), We have encountered issues with the following physical network link(s) you have with Google, to start troubleshooting and prevent any further impact to traffic we have removed traffic from the link(s). Please check the below list: Google ASN Your ASN Google Address Your Address Your Demarc Metro Problem Speed 15169 PX.MICE.ORD 206.108.255.167 Peering Exchange AS53679:CH1:05:000990:PP:0204:1238777 Ports 31,32 ORD Down 100 Gbps Tips for troubleshooting physical link problems: * Clean the interface optic and your patching path to the MMR. * Replace the interface optic. Please check your link(s) and attempt to resolve the problems described above. Thanks! ======================== Is someone from MICE performing the requested actions and/or looking into this? Kind regards, Frank Bulk FiberNet (ASN18883)

April 10, 2021

MICE UG 39

by Richard Laager

With NANOG happening this fall, it may make sense to schedule MICE UG 39 at the same time (e.g. Wednesday November 3, time TBD). That's not official yet, but tentatively "save the date". Thanks to the person who suggested this (off-list)! -- Richard

April 10, 2021

SFN Remote Uplink Change

by Jay Hanke

SFN has upgraded the uplink LAG group on the SFN MICE remote switch to 12x10G from 8x10G. We were getting a small amount of tail drops where we convert from 100GE to the LAG group. Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 1, 2021

Re: belle plaine

by James Urwiller

Tried probably 8 times already.. never seem to get any notifications. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Tim Fjeld *Sent:* Thursday, April 1, 2021 08:04 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] belle plaine We received the following notice on 16MAR2021, If you guys are on legacy Neutral Path circuits you might need to have Zayo add your circuits to their notification list. *Zayo Maintenance Notification This email serves as official notification that Zayo and/or one of its providers will be performing maintenance on its network as described below. This maintenance may affect services you have with us. * *Maintenance Ticket #: *TTN-0004856934 *Urgency: *Planned *Date Notice Sent: *16-Mar-2021 *Customer: *City of Windom - Windomnet *Maintenance Window * *1st Activity Date * 31-Mar-2021 22:00 to 01-Apr-2021 05:00 ( Central ) 01-Apr-2021 03:00 to 01-Apr-2021 10:00 ( GMT ) *Location of Maintenance: *5th Ave & Excelesior Blvd Hopkins, MN *Reason for Maintenance: *Zayo will perform of Neutral Path fiber for new storm sewer construction. *Expected Impact: *Service Affecting Activity: Any Maintenance Activity directly impacting the service(s) of customers. Service(s) are expected to go down as a result of these activities. *Circuit(s) Affected: * *Circuit Id* *Expected Impact* *A Location CLLI* *Z Location CLLI* *Legacy Circuit Id* Thanks, Tim Fjeld Network Operations City of Windom Telecom 443 10th Street, P.O. Box 38 Windom, MN 56101 Phone: 507-832-8676 *From:* Tim Fjeld *Sent:* Thursday, April 1, 2021 6:48 AM *To:* MICE Discuss <MICE-DISCUSS(a)lists.iphouse.net> *Subject:* Re: [MICE-DISCUSS] belle plaine Zayo had scheduled maintenance last night at 22:00 to move a fiber for a sewer project in Hopkins. Sent from my iPhone On Apr 1, 2021, at 12:08 AM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: SDN confirmed that there was fiber cut with the “affiliate”. They presumably mean Zayo, but I don’t know for sure. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Jay Hanke *Sent:* Wednesday, March 31, 2021 10:44 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] belle plaine Looks like a bigger zayo transport issue. Other members with zayo transport have also dropped. If a member has the master ticket number please post it up. It will save some time for the other affected members. On Wed, Mar 31, 2021, 10:27 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: Yes, both FiberComm and FiberNet did at 10:02:31 pm U.S. Central. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James Urwiller *Sent:* Wednesday, March 31, 2021 10:15 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] belle plaine anyone on the belle plain remote just loose connection? we are coming in from fremont, NE. ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 1, 2021