September 2019 - mice-discuss

MN VoIP Remote Upgrade

by Jeremy Lumby

In accordance with the MICE remote switch operator guidelines, MN VoIP will be upgrading their remote switch and uplink to the MICE core on the afternoon of October 11th at approx 2PM to keep uplink utilization under 66% of the link capacity. Prior to the maintenance, BGP will be blocked so traffic can gracefully reroute. During the maintenance connectivity will be down for approximately 15 minutes. A 100G switch will be added between the existing MN VoIP extension and the MICE core. This switch will not only increase the MN VoIP uplink from 40G to 100G, it will also make 32 100G ports available to anyone wishing to connect to MICE via MN VoIP @ 100G. I will send out a notification to the discuss list right before I commence with the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

Sept. 30, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Frank Bulk

12:41 am Current Update: The crews have completed splicing all fibers and we are checking for Service Restoration now. 2:20 am Current Update: We are showing all services restored. If you are seeing any circuits down please contact the NOC at 866-236-2824 option 1 option 5 4:25 am Reason for Outage Summary: A fiber cut was reported in Bigelow MN. Zayo crews were dispatched out to 16 miles out of Bigelow and repaired the damaged fiber which was caused by an underground break. All Repairs were completed at 2:18 AM Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Thursday, September 26, 2019 12:14 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? I see MICE links are coming back up, and we got this from Zayo a bit ago along with a direct note from their NOC stating that they still had the South splice case open and so weren't hands-off yet. 11:30PM: "Technicians are splicing fibers and have already spliced a total of 36 fibers. We should start seeing restoration of services soon. Thank you so much for your patience with us during this event." -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, September 25, 2019 8:49 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? 7:41 pm Current update, Crews have both sides dug up and are splicing the fiber back up 6:41 pm Current update, Zayo has dug up one side of the 288ct and have it in the trailer with the splice team, Constitution teams are starting on the other side. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Dahna Sent: Wednesday, September 25, 2019 4:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Frank Bulk

9:10 pm Current Update: Crews have the Fiber on the north side all spliced back up adn are now heading to the South side to start getting things prepped. We still have no ETR at this time but we will continue to keep you updated. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Wednesday, September 25, 2019 8:49 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? 7:41 pm Current update, Crews have both sides dug up and are splicing the fiber back up 6:41 pm Current update, Zayo has dug up one side of the 288ct and have it in the trailer with the splice team, Constitution teams are starting on the other side. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Dahna Sent: Wednesday, September 25, 2019 4:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Dean Bahls

Dang, I missed it by 4 hours. Instead of 12, it was 16... -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)NUVERA.NET Sent: Thursday, September 26, 2019 12:14 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? I see MICE links are coming back up, and we got this from Zayo a bit ago along with a direct note from their NOC stating that they still had the South splice case open and so weren't hands-off yet. 11:30PM: "Technicians are splicing fibers and have already spliced a total of 36 fibers. We should start seeing restoration of services soon. Thank you so much for your patience with us during this event." -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, September 25, 2019 8:49 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? 7:41 pm Current update, Crews have both sides dug up and are splicing the fiber back up 6:41 pm Current update, Zayo has dug up one side of the 288ct and have it in the trailer with the splice team, Constitution teams are starting on the other side. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Dahna Sent: Wednesday, September 25, 2019 4:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by AnthonyAnderberg＠nuvera.net

I see MICE links are coming back up, and we got this from Zayo a bit ago along with a direct note from their NOC stating that they still had the South splice case open and so weren't hands-off yet. 11:30PM: "Technicians are splicing fibers and have already spliced a total of 36 fibers. We should start seeing restoration of services soon. Thank you so much for your patience with us during this event." -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, September 25, 2019 8:49 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? 7:41 pm Current update, Crews have both sides dug up and are splicing the fiber back up 6:41 pm Current update, Zayo has dug up one side of the 288ct and have it in the trailer with the splice team, Constitution teams are starting on the other side. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Dahna Sent: Wednesday, September 25, 2019 4:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Jeff Dahna

Current update 9:51 PM: ETR 2-3 hours

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Frank Bulk

7:41 pm Current update, Crews have both sides dug up and are splicing the fiber back up 6:41 pm Current update, Zayo has dug up one side of the 288ct and have it in the trailer with the splice team, Constitution teams are starting on the other side. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Dahna Sent: Wednesday, September 25, 2019 4:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 26, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Jeff Dahna

latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

Sept. 25, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by Jeff Dahna

Zayo is responding to our ticket about every hour with the standard general response "Issues are still currently being investigated by the field teams on site"

Sept. 25, 2019

Re: Anyone else see their Zayo-based link to MICE go down?

by James Urwiller

The problem is they outsource their locating to a sub-par contractor and zayo themselves (along the old neutral path part) don’t seem to know where any of it truly is. One fiber cut earlier in the year, they actually located the opposite side of the highway in which their fiber resided… James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Justin Krejci *Sent:* Wednesday, September 25, 2019 15:10 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? Does Zayo just not mark their fibers? This seems like an overly excessive amount of cut Zayo fibers over the last 24 months. ------------------------------ *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jeff Dahna <jeff.dahna(a)WINDOMMN.COM> *Sent:* Wednesday, September 25, 2019 1:52 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else see their Zayo-based link to MICE go down? Zayo ticket update: Wed 9/25/2019 1:31 PM Current update, Zayo OSP is on site searching for the damage, Splicers will be on site in 1 hour, Underground construction team is on site with dig equipment and Trailer with conduit and fiber expected in 90 minutes We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 25, 2019