November 2019 - mice-discuss - lists.micemn.net

Re: Google

by Andrew Hoyos

The latest update we had from Level3 (hey, look, a non Zayo related fiber issue, haha!) *** CASCADED EXTERNAL NOTES 18-Nov-2019 20:39:54 GMT From CASE: 17573224 - Event The Transport NOC reports that light level testing indicates a potential fault 13.8 kilometers from the testing site. The Outside Plant Engineer (OSPE) was engaged, provided the test results, and requested to assist with locating the exact point of failure. The OSPE has dispatched Field Operations personnel to the approximate location of the fault to begin searching for the physical damage point. As a precaution, construction teams have been engaged and dispatched to the approximate location of the damage. An ETA of approximately 23:00 GMT has been provided for the construction teams. > On Nov 18, 2019, at 3:11 PM, Jeremy Schmeichel <Jeremy.Schmeichel(a)VASTBROADBAND.COM> wrote: > > I’m glad someone asked, because I was wondering why our peer went down. Makes sense. > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Andrew Hoyos > Sent: Monday, November 18, 2019 2:04 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Google > > We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). > I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago. > > > On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls(a)COOPERATIVE-NETWORKS.COM <mailto:dean.bahls@COOPERATIVE-NETWORKS.COM>> wrote: > > Any word on what’s going on at Google? I’m guessing someone had a bad day. > > Dean > > > Dean Bahls > Network Services Manager | Cooperative Network Services, LLC > 14 Main St SW | PO Box J | Menahga, MN 56464 > 218.346.3635 (Direct) | 218.564.3000 (Main Office) > dean.bahls(a)cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com> > > <image003.jpg> > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. > CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

Nov. 18, 2019

Re: Google

by Ben Wiechman

Did you first see issues around 12:30? Access to some Google services dropped for me around then. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Mon, Nov 18, 2019 at 3:11 PM Jeremy Schmeichel < Jeremy.Schmeichel(a)vastbroadband.com> wrote: > I’m glad someone asked, because I was wondering why our peer went down. > Makes sense. > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Andrew > Hoyos > *Sent:* Monday, November 18, 2019 2:04 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Google > > > > We lost some Level3 circuits at the same time as their peer dropped, due > to a fiber cut in Wisconsin (near Dousman, WI). > > I’m 99.99% sure Google optical network rides that fiber from Chicago -> > MSP, which implies they seem to be single homed, and backhauled to Chicago. > > > > On Nov 18, 2019, at 3:01 PM, Dean Bahls < > dean.bahls(a)COOPERATIVE-NETWORKS.COM> wrote: > > > > Any word on what’s going on at Google? I’m guessing someone had a bad day. > > > > Dean > > > > > > *Dean Bahls* > > Network Services Manager | Cooperative Network Services, LLC > > 14 Main St SW | PO Box J | Menahga, MN 56464 > > 218.346.3635 (Direct) | 218.564.3000 (Main Office) > > dean.bahls(a)cooperative-networks.com > > > > <image003.jpg> > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > CONFIDENTIAL: This email and any files transmitted with it are > confidential and intended solely for the use of Vast Broadband/NTS and to > whom they are addressed. If you have received this email in error, please > notify the system manager. This message contains confidential information > and is intended only for the individual named. If you are not the named > addressee, you should not disseminate, distribute or copy this email. > Please notify the sender immediately by email if you have received this > email by mistake and delete this email from your system. If you are not the > intended recipient, you are notified that disclosing, copying, distributing > or taking any action in reliance on the contents of this information is > strictly prohibited. > CONFIDENTIAL: This email and any files transmitted with it are > confidential and intended solely for the use of Vast Broadband/NTS and to > whom they are addressed. If you have received this email in error, please > notify the system manager. This message contains confidential information > and is intended only for the individual named. If you are not the named > addressee, you should not disseminate, distribute or copy this email. > Please notify the sender immediately by email if you have received this > email by mistake and delete this email from your system. If you are not the > intended recipient, you are notified that disclosing, copying, distributing > or taking any action in reliance on the contents of this information is > strictly prohibited. > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Nov. 18, 2019

Re: Google

by Jeremy Schmeichel

I’m glad someone asked, because I was wondering why our peer went down. Makes sense. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Monday, November 18, 2019 2:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago. On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls(a)COOPERATIVE-NETWORKS.COM<mailto:dean.bahls@COOPERATIVE-NETWORKS.COM>> wrote: Any word on what’s going on at Google? I’m guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com<mailto:dean.bahls@cooperative-networks.com> <image003.jpg> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. CONFIDENTIAL: This email and any files transmitted with it are confidential and intended solely for the use of Vast Broadband/NTS and to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

Nov. 18, 2019

Re: Google

by Andrew Hoyos

We lost some Level3 circuits at the same time as their peer dropped, due to a fiber cut in Wisconsin (near Dousman, WI). I’m 99.99% sure Google optical network rides that fiber from Chicago -> MSP, which implies they seem to be single homed, and backhauled to Chicago. > On Nov 18, 2019, at 3:01 PM, Dean Bahls <dean.bahls(a)COOPERATIVE-NETWORKS.COM> wrote: > > Any word on what’s going on at Google? I’m guessing someone had a bad day. > > Dean > > > Dean Bahls > Network Services Manager | Cooperative Network Services, LLC > 14 Main St SW | PO Box J | Menahga, MN 56464 > 218.346.3635 (Direct) | 218.564.3000 (Main Office) > dean.bahls(a)cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com> > > <image003.jpg> > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

Nov. 18, 2019

Google

by Dean Bahls

Any word on what's going on at Google? I'm guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>

Nov. 18, 2019

Re: Please Welcome SBT to MICE!

by Doug McIntyre

On Fri, Nov 15, 2019 at 08:54:58AM -0600, Jay Hanke wrote: >Name: SBT, Int >Port: Ethernet 4 MNVoip Remote Switch >ASN: 11255 >IPv4: 206.108.255.149/24 >IPv6: 2001:504:27:0:0:2bf7::1/64 >Peering Contact: peering(a)sbt.net >Route Servers- YES >BFD: YES Added to the route servers. Welcome! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Nov. 18, 2019

Please Welcome SBT to MICE!

by Jay Hanke

Name: SBT, Int Port: Ethernet 4 MNVoip Remote Switch ASN: 11255 IPv4: 206.108.255.149/24 IPv6: 2001:504:27:0:0:2bf7::1/64 Peering Contact: peering(a)sbt.net Route Servers- YES BFD: YES Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Nov. 15, 2019

Re: random packet loss since yesterday afternoon anyone?

by Jeremy Lumby

I saw it, and it appears related to: GENERAL INFORMATION Case Number: TTN-0003669752 Customer Ticket Number: Customer Circuit Id: Case Subject: Minnesota VoIP / ETYX/176681//ZYO / Packet loss Case Impact: Impaired Case Status: Service Restored Case Sub Status: Monitor Initial Down Time: Sun Nov 03 20:41:12 GMT 2019 Restore Date Time: Wed Nov 06 15:11:24 GMT 2019 Thu Nov 07 13:35:09 GMT 2019 The replacement has been completed please notify Zayo if you have further issues Wed Nov 06 18:05:19 GMT 2019 Current update: Correction. A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 AM CDT on 11/07/2019 We will provide updates as they become available. Wed Nov 06 18:05:08 GMT 2019 Current update: Correction. A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 AM CDT on 11/07/2019 We will provide updates as they become available. Wed Nov 06 17:05:06 GMT 2019 Current update: A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 PM CDT. We will provide updates as they become available. Mon Nov 04 20:00:14 GMT 2019 Here's how the ports in question trace out after our dispatch today: NP_BLLPMN08YB1 1-3-4 = NNP-LS0000129 (MICE) = RR 001.100.05 Juniper 4500 MICE XE 38 NP_BLLPMN08YB1 1-3-6 = NNP-LB0000170 (MPLS) = RR 001.100.05 Juniper 4500 MSP1 XE 37 NP_BLLPMN08YB1 1-3-7 = NNP-LS0000186 (Nebraskalink) = RR 001.100.05 Brocade Netiron Port 2/4 NP_BLLPMN08YB1 1-3-8 = NNP-LB0000231 (MPLS) = RR 001.100.05 Juniper 4500 MSP1 XE 38 Per Ciena ( C10237538 ), since there is no common shelf/equipment, we are going to move forward with scheduling an MR in order to swap out the CYAN card and the optic to have absolute certainty that its not the optical gear. I'll be setting this up for us and letting everyone know when the day/time is for MR. Thanks for your patience throughout this process. Eric P. Reed - Zayo ONCC Mon Nov 04 19:36:04 GMT 2019 Hello, Our transport team is dispatching back out to further isolate the device causing the backbone flapping. -Zach Mon Nov 04 15:40:37 GMT 2019 CURRENT STATUS: I'm picking this tkt up for review and will get back to you once I have an opportunity to work with our TAC support and get some clear direction. Eric P. Reed - Zayo ONCC. Mon Nov 04 13:10:07 GMT 2019 Dear Customer, We are working with the equipment vendor and transport team to resolve the issue More updates to follow as they become available Mon Nov 04 12:00:19 GMT 2019 Dear Customer, We continue to escalate with the equipment vendor to get this issue resolved. More updates to follow as they become available. Mon Nov 04 10:05:25 GMT 2019 Dear Customer, We continue to work with the equipment vendor. More updates to follow as they become available Mon Nov 04 08:55:06 GMT 2019 Dear Customer, The transport team is currently working with the vendor. More updates to follow as they become available. Mon Nov 04 05:15:39 GMT 2019 Dear Customer, We have engaged our transport team to check that portion of the circuit. More updates to follow as they become available. Mon Nov 04 03:20:08 GMT 2019 Dear Customer, We are in the process of dispatching to the POP site to trouble. More updates to follow as they become available. If you are need of immediate assistance please contact the NOC at 1.866.236.2824 Opt 1, 3 Mon Nov 04 00:44:46 GMT 2019 <pre>Hello Jerey, We have identified a backbone flap which might be impacting your circuit. We are currently investigating the issue. Regards, Akramul H Zayo US:1.866.236.2824 Zayo Canada:1.888.404.9296 Cloud:+855.528.4797 zColo:+877.926.5687 Website: https://www.zayo.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, November 07, 2019 7:29 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I saw some disruption to MICE and all of my private peering sessions between 2 and 3 am this (11/7/2019) morning – anyone else see that? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 6, 2019 12:33 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I just got an update stating that the last update was incorrect, and the card will be replaced tonight at midnight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 11:08 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Even though I never got the notice of when Zayo was going to replace the card, they did replace it around the same time of last night’s damaged fiber replacement. I just now got notice that it completed, and they closed the ticket. I will watch for more loss tonight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 8:29 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Unfortunately the maintenance was unrelated, it was to replaced a damaged 288 count fiber. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Wednesday, November 06, 2019 8:23 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Apparently they did the maintenance last night?? I was not notified, nor have I checked to see if its any better yet. But I did lose that link last night for about an hour from midnight to 1 AM CST. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Tuesday, November 5, 2019 09:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I got the same update. I did see loss last night at Peak utilization, however it was nowhere near as bad as the previous night. I am also waiting on notification for them to swap the cards in the cyan gear Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Date: 11/5/19 9:26 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? >From : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, November 5 2019 09:26:57 Last ticket I received they traced out some other customers path (not mine) and told me there was no common path, but they wanted to swap a cyan transport card anyway, and would schedule the maintenance “soon” However I think its better-ish now?? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Tuesday, November 5, 2019 09:17 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We also opened up a case with Zayo however they put a loopback on the circuit and did not see any errors apparently. Are you all still seeing loss? If so, what did Zayo come up with? Thanks in advance, Justin From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 11:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. The last few updates I had were that they dispatched a tech to Belle Plaine, then they moved the ticket back to the backbone group who engaged the hardware vendor. I still do not have an ETR, however yesterday I did see a little bit of loss at 8AM, however it got much worse as the traffic to MICE increased in the evening. Currently I am seeing loss, however it is much less than Sunday evening. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 10:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Only updates I have on my ticket is that a tech is investigating.. it has certainly been sporadic for us. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Monday, November 4, 2019 10:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We have had a few customers complain about packet loss to google.com today as well and was able to see packet loss from one of our routers up to 511 building. However, I don’t see packet loss anymore, but maybe it is really intermittent. Did you hear any update from Zayo? Justin Long Lines Broadband From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 8:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. I am seeing it on the Zayo path between Belle Plaine, and Minneapolis. I have a ticket open since yesterday afternoon, and they appear to be struggling to nail it down. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 8:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Greetings all, Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Nov. 7, 2019

Re: random packet loss since yesterday afternoon anyone?

by Frank Bulk

I saw some disruption to MICE and all of my private peering sessions between 2 and 3 am this (11/7/2019) morning – anyone else see that? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, November 6, 2019 12:33 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I just got an update stating that the last update was incorrect, and the card will be replaced tonight at midnight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 11:08 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Even though I never got the notice of when Zayo was going to replace the card, they did replace it around the same time of last night’s damaged fiber replacement. I just now got notice that it completed, and they closed the ticket. I will watch for more loss tonight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 8:29 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Unfortunately the maintenance was unrelated, it was to replaced a damaged 288 count fiber. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Wednesday, November 06, 2019 8:23 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Apparently they did the maintenance last night?? I was not notified, nor have I checked to see if its any better yet. But I did lose that link last night for about an hour from midnight to 1 AM CST. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Tuesday, November 5, 2019 09:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I got the same update. I did see loss last night at Peak utilization, however it was nowhere near as bad as the previous night. I am also waiting on notification for them to swap the cards in the cyan gear Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> -------- Original message -------- From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM<mailto:james.urwiller@TURNKEY-TELECOM.COM>> Date: 11/5/19 9:26 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? ________________________________ From : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM<mailto:james.urwiller@TURNKEY-TELECOM.COM>] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> [MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] Date : Tuesday, November 5 2019 09:26:57 Last ticket I received they traced out some other customers path (not mine) and told me there was no common path, but they wanted to swap a cyan transport card anyway, and would schedule the maintenance “soon” However I think its better-ish now?? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Huffman Sent: Tuesday, November 5, 2019 09:17 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We also opened up a case with Zayo however they put a loopback on the circuit and did not see any errors apparently. Are you all still seeing loss? If so, what did Zayo come up with? Thanks in advance, Justin From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 11:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. The last few updates I had were that they dispatched a tech to Belle Plaine, then they moved the ticket back to the backbone group who engaged the hardware vendor. I still do not have an ETR, however yesterday I did see a little bit of loss at 8AM, however it got much worse as the traffic to MICE increased in the evening. Currently I am seeing loss, however it is much less than Sunday evening. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 10:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Only updates I have on my ticket is that a tech is investigating.. it has certainly been sporadic for us. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Huffman Sent: Monday, November 4, 2019 10:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We have had a few customers complain about packet loss to google.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__google.com&d=DwMFaQ&c=e…> today as well and was able to see packet loss from one of our routers up to 511 building. However, I don’t see packet loss anymore, but maybe it is really intermittent. Did you hear any update from Zayo? Justin Long Lines Broadband From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 8:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. I am seeing it on the Zayo path between Belle Plaine, and Minneapolis. I have a ticket open since yesterday afternoon, and they appear to be struggling to nail it down. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 8:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Greetings all, Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Nov. 7, 2019

Re: random packet loss since yesterday afternoon anyone?

by Jeremy Lumby

I just got an update stating that the last update was incorrect, and the card will be replaced tonight at midnight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 11:08 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Even though I never got the notice of when Zayo was going to replace the card, they did replace it around the same time of last night’s damaged fiber replacement. I just now got notice that it completed, and they closed the ticket. I will watch for more loss tonight. From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, November 06, 2019 8:29 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Unfortunately the maintenance was unrelated, it was to replaced a damaged 288 count fiber. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Wednesday, November 06, 2019 8:23 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Apparently they did the maintenance last night?? I was not notified, nor have I checked to see if its any better yet. But I did lose that link last night for about an hour from midnight to 1 AM CST. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Tuesday, November 5, 2019 09:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? I got the same update. I did see loss last night at Peak utilization, however it was nowhere near as bad as the previous night. I am also waiting on notification for them to swap the cards in the cyan gear Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Date: 11/5/19 9:26 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? >From : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, November 5 2019 09:26:57 Last ticket I received they traced out some other customers path (not mine) and told me there was no common path, but they wanted to swap a cyan transport card anyway, and would schedule the maintenance “soon” However I think its better-ish now?? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Tuesday, November 5, 2019 09:17 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We also opened up a case with Zayo however they put a loopback on the circuit and did not see any errors apparently. Are you all still seeing loss? If so, what did Zayo come up with? Thanks in advance, Justin From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 11:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. The last few updates I had were that they dispatched a tech to Belle Plaine, then they moved the ticket back to the backbone group who engaged the hardware vendor. I still do not have an ETR, however yesterday I did see a little bit of loss at 8AM, however it got much worse as the traffic to MICE increased in the evening. Currently I am seeing loss, however it is much less than Sunday evening. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 10:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Only updates I have on my ticket is that a tech is investigating.. it has certainly been sporadic for us. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Monday, November 4, 2019 10:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We have had a few customers complain about packet loss to google.com today as well and was able to see packet loss from one of our routers up to 511 building. However, I don’t see packet loss anymore, but maybe it is really intermittent. Did you hear any update from Zayo? Justin Long Lines Broadband From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 8:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. I am seeing it on the Zayo path between Belle Plaine, and Minneapolis. I have a ticket open since yesterday afternoon, and they appear to be struggling to nail it down. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 8:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Greetings all, Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Nov. 6, 2019