June 2019 - mice-discuss - lists.micemn.net

Re: Google ASN Migration/Port Move

by Dave Schwartz

Hi MICE Members, Google AS15169 can deliver all Google traffic. Once we are closer to turn up of the new link I'll share some details with the members on the next steps to peer directly with us. We plan also to peer the route server. Thanks, Dave -- Dave Schwartz Peering & Content Distribution Google AS15169 daveschwartz(a)google.com +1-650-253-7615 +44 20 3820 9224 On Fri, Jun 14, 2019, 07:27 Jay Hanke <jayhanke(a)southfront.io> wrote: > Google is connecting MICE directly to the Google backbone AS so more > of the services will be available following the upgrade at MICE. > > On Fri, Jun 14, 2019 at 9:20 AM Jeff Wilde <jeff.wilde(a)parkregion.com> > wrote: > > > > I was under the impression that they only basically do youtube/etc > through there. I asked about bilateral and that is what they said. > > > > Jeff Wilde > > IT Manager > > Park Region Telephone Company > > P: 218-826-8330 > > E: jeff.wilde(a)parkregion.com > > > > -----Original Message----- > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > > Sent: Friday, June 14, 2019 7:31 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > > > I think this is a great move for Google/MICE, and I can easily foresee > Google outgrowing the 100G quickly. On a side note I would update the > website to state "Partial" for the route server. Google had recently > changed their policy for AS15169 where they still utilize the route > servers, however not all content is advertized through it. If you want to > maximize your Google backbone traffic, you need to setup a bilateral with > them. > > > > Jeremy Lumby > > Minnesota VoIP > > 9217 17th Ave S #216 > > Bloomington, MN 55425 > > M: 612-355-7740 > > D: 612-392-6814 > > F: 952-873-7425 > > jlumby(a)mnvoip.com > > > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Richard Laager > > Sent: Friday, June 14, 2019 2:21 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > > > On 6/13/19 11:53 AM, Jay Hanke wrote: > > > Google is converting their existing MICE connection to their backbone > > > AS (15169). > > > > > > Immediately following the migration, Google will disconnect their > > > existing 100G port (Ethernet3/2/1). > > > > > > The new addressing will be: > > > ASN: 15169 > > > IPv4: 206.108.255.141/24 > > > IPv6: 2001:504:27:0:0:3b41::1/64 > > > > > > The new 100G port will be Ethernet3/13/1. > > > > > > Google has requested the port be set up as a new single channel LAG > group. > > > > I've updated the website. > > > > -- > > Richard > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 >

June 14, 2019

Re: Google ASN Migration/Port Move

by Jay Hanke

Google is connecting MICE directly to the Google backbone AS so more of the services will be available following the upgrade at MICE. On Fri, Jun 14, 2019 at 9:20 AM Jeff Wilde <jeff.wilde(a)parkregion.com> wrote: > > I was under the impression that they only basically do youtube/etc through there. I asked about bilateral and that is what they said. > > Jeff Wilde > IT Manager > Park Region Telephone Company > P: 218-826-8330 > E: jeff.wilde(a)parkregion.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby > Sent: Friday, June 14, 2019 7:31 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager > Sent: Friday, June 14, 2019 2:21 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > On 6/13/19 11:53 AM, Jay Hanke wrote: > > Google is converting their existing MICE connection to their backbone > > AS (15169). > > > > Immediately following the migration, Google will disconnect their > > existing 100G port (Ethernet3/2/1). > > > > The new addressing will be: > > ASN: 15169 > > IPv4: 206.108.255.141/24 > > IPv6: 2001:504:27:0:0:3b41::1/64 > > > > The new 100G port will be Ethernet3/13/1. > > > > Google has requested the port be set up as a new single channel LAG group. > > I've updated the website. > > -- > Richard -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 14, 2019

Re: Google ASN Migration/Port Move

by Jeff Wilde

I was under the impression that they only basically do youtube/etc through there. I asked about bilateral and that is what they said. Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, June 14, 2019 7:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, June 14, 2019 2:21 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Re: Google ASN Migration/Port Move

by Jeremy Lumby

I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, June 14, 2019 2:21 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Re: Google ASN Migration/Port Move

by Richard Laager

On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Google ASN Migration/Port Move

by Jay Hanke

Google is converting their existing MICE connection to their backbone AS (15169). Immediately following the migration, Google will disconnect their existing 100G port (Ethernet3/2/1). The new addressing will be: ASN: 15169 IPv4: 206.108.255.141/24 IPv6: 2001:504:27:0:0:3b41::1/64 The new 100G port will be Ethernet3/13/1. Google has requested the port be set up as a new single channel LAG group. We'll need new route server connections as well. Welcome aboard (Again)! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 13, 2019

Re: lost MICE

by David Farmer

A bit of trivia, the fiber cut is right near the highest point in Iowa, Hawkeye Point, just east of the intersection of 130th St and Hy60. It's all downhill from there. :) https://en.wikipedia.org/wiki/Hawkeye_Point On Tue, Jun 11, 2019 at 1:40 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > And RFO: > > Reason for Outage Summary: A fiber cut was reported in Sibley IA. Zayo > crews were dispatched out to > > 130th and 140th along highway 60 and repaired the damaged fiber which was > caused by a Drain tile contractor trenched through the cable. All > > Repairs were completed at 1:00 PM CDT. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 11:38 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 11:30 am: OSP reported that splicing is complete. Please verify that your > Services have restored. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 10:44 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 10:33 am: OSP has reported that splicing on the north case is complete. > Splicers working on South case. You should start to see service restore as > splicing continues. A rough ETR is 1 hour. More updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 10:05 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. > > > > From what I heard from our field staff, the issue wasn’t that bad – > looking through the update, it just took Zayo’s staff and/or contractors a > long time to get on site and perform the work. It seems like they had just > one fiber splicing crew, so they didn’t prep and splice on both sides at > the same time. We’ve had similar cuts many times before and we would likely > have this resolved in 6 to 8 hours. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James > Urwiller > *Sent:* Tuesday, June 11, 2019 9:43 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > I would like to Thank Frank for reporting all this info. He is keeping up > with the Zayo ticket responses very well! > > You mentioned this was cut near your area.. was it really that bad to > warrant 24+ hrs? seems like they are pretty slow with these repairs? > > > > James Urwiller > > Network Architect > > Stealth Broadband > > james.urwiller(a)stealthbroadband.com > > [image: stealth300] > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 07:58 > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 11:04 pm: Excavation is currently underway at this time. Techs are > trenching out the 200 feet of cable required to make repairs. > > 12:58 am: Underground crews have completed the trench on one side of the > break. OSP splicers have the cable exposed on that side and are prepping > the old cable and the new for repairs. The Underground crew is working to > excavate on the opposite end at this time. No ETTR at this time. > > 2:22 am: OSP crews are still continuing to work at this time. Excavation > crews are continuing to work to expose the remaining fiber to prep for > splicing. No ETTR at this time. > > 3:20 am: At this time the underground crew has completed excavation and > the cable is fully exposed for repairs. Prep work is still ongoing on both > sides. Splicers expect to be splicing on one end of the cable in about 1 > hour. The other side will be spliced once prep work is complete. No ETR at > this time. > > 5:04 am: OSP splicers are continuing the splice at this time. Techs are > prioritizing the active fibers to speed up restoration. No ETTR at this > time. > > 5:57 am: OSP splicers are continue to finish up the splicing on the North > side of the cut. Once complete crews will move to the South side to > complete prep work and begin splicing. We have a tentative ETTR of 10 CST > today. > > 7:07 am: OSP splicers onsite advised they have been a bit delayed on the > splicing at the North side of the cut. Splicers are nearly complete on the > North side, and will be moving to the South to complete prep work and > repairs. Crews have revised the tentative ETTR for 12 CST. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:47 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 9:26 pm: Emergency locates are nearly complete so the underground > construction crew will be able to start major excavations soon. The crew > will need to trench along 200 feet of the cable path to add in a new > section of fiber. No ETTR at this time. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 9:09 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 8:19 pm: OSP has reported that Underground crew has arrived on site. They > will begin excavation within the next few minutes. There is no ETTR at this > time. More updates to follow. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 7:15 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 6:58 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route and are trying to get around flooded roads. ETA is 30 mins. > More updates to follow as we receive them. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 6:00 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 4:47 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive > them. > > > > 5:51 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route with ETA of 30-45 mins. More updates to follow as we receive > them. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 2:56 PM > *To:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 > mins. More updates to follow. > > > > Note that the location listed in the ticket is incorrect – I’ve > communicated the correct address (130th and Hwy 60) to Zayo’s NCC and > they’ve acknowledged receipt of that. > > > > If they do get onsite by 3:15 pm it will have been about 10 hours since > the initial cut. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 1:24 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 12:35 pm: OSP reported they are trying to get resources to the damage > location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA > or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More > updates to follow. > > 12:43 pm: OSP reported Splicer is in route to the damage location right > off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More > updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:35 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Another update from Zayo: > > > > 10:30 OSP reported they are trying to get resources to the damage > location. Damage appears to be right off of I60 and 170th in Sibley IA. > Currently there is no ETA for techs. More updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:27 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > First two updates from Zayo: > > > > 9:48 am: Your case has been associated with a higher level outage ticket, > your ticket number will remain the same. The latest update is as follows: > Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN > affecting multiple customers. The appropriate resources have been engaged > for repairs. At this time, OSP is en route to the suspected trouble > location to begin investigating. No ETA available as of yet. > > > > 10:21 am: Our OSP crews are en route to begin investigations. More updates > to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 8:53 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy > 60 ( > https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) > in Sibley, Iowa. The tiler called one of our techs and the farmer called > another one of our techs, inquiring if it was our fiber or not (it’s not). > The tiler reported hitting a line below the tape. > > > > I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and > haven’t yet received an acknowledgement, but once I do I will inform them > of the location. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 8:06 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I > assume it’s an issue between Sioux City and Belle Plaine. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James > Urwiller > *Sent:* Monday, June 10, 2019 7:58 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] lost MICE > > > > I lost MICE again over ZAYO about 30 minutes ago.. anyone else? > > > > James Urwiller > > Network Architect > > Stealth Broadband > > james.urwiller(a)stealthbroadband.com > > [image: stealth300] > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

June 11, 2019

Re: lost MICE

by Frank Bulk

And RFO: Reason for Outage Summary: A fiber cut was reported in Sibley IA. Zayo crews were dispatched out to 130th and 140th along highway 60 and repaired the damaged fiber which was caused by a Drain tile contractor trenched through the cable. All Repairs were completed at 1:00 PM CDT. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 11:38 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 11:30 am: OSP reported that splicing is complete. Please verify that your Services have restored. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:44 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

11:30 am: OSP reported that splicing is complete. Please verify that your Services have restored. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:44 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019