- mice-discuss - lists.micemn.net

Re: SFN Remote Outage

by Jay Hanke

Yes, that lines up with our timing/vendor as well. On Mon, Jul 22, 2019 at 3:29 PM Darin Steffl <darin.steffl(a)mnwifi.com> wrote: > > We lost an arvig circuit at 2:45 so it must be the same cut. > > On Mon, Jul 22, 2019, 3:23 PM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> Our transport to Albert Lea is currently down. >> >> -- >> Jay Hanke, President >> South Front Networks >> jayhanke(a)southfront.io >> Phone 612-204-0000 > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 22, 2019

Re: SFN Remote Outage

by Darin Steffl

We lost an arvig circuit at 2:45 so it must be the same cut. On Mon, Jul 22, 2019, 3:23 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > Our transport to Albert Lea is currently down. > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 >

July 22, 2019

SFN Remote Outage

by Jay Hanke

Our transport to Albert Lea is currently down. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 22, 2019

Parking/Meeting Information for MICE UG32

by Jay Hanke

The meeting is at: July 17, 2019, 3 PM Central Time Cedar Falls Utilities 1 Utility Pkwy Cedar Falls, IA 50613 Remote Conferencing: 952-562-5400 passcode MICE (6423) Please park in the visitor parking area adjacent to the power plant and enter at the main entrance under the canopy. When you enter, turn left and sign in at reception. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 16, 2019

Re: MICE UG 32

by Richard Laager

It looks like I forgot to attach the UG 31 minutes. Here they are. -- Richard

July 16, 2019

MICE UG 32

by Richard Laager

Reminder: UG 32 is on Wednesday of next week (July 17) at 3:00pm Remote Conferencing: 952-562-5400 passcode MICE (6423) Location Details: Cedar Falls Utilities 1 Utility Pkwy Cedar Falls, IA Agenda The agenda is attached. I'm behind on getting this out, so I didn't send a separate call for agenda. If you have something to add, let me know ASAP and I'll send an updated agenda. As this is an "away" meeting, I intend to skip the repeat old business items. We can pick those up in October. As far as I know, we have no other technical committee report this meeting; if anyone is planning on giving a technical committee report, please let me know. I've also taken the liberty to reorder things a bit. The idea here is to group business (official reports and things that could involve a motion/vote) before open discussion. This way, people who are not interested in the discussion can leave earlier if they wish, confident that they won't miss an official motion/vote. By shortening the "mandatory" part of the meeting, this may make the meetings more accessible. I'd like to give this a try for a couple meetings and see what people think. That said, with this being the summer outreach meeting, we traditionally cover "What is MICE?" first. I kept that at the top, rather than putting it after adjournment. -- Richard

July 9, 2019

Re: Blackholing DoS traffic

by Steve Howard

July 5, 2019

Off Topic Cisco L2TPv3

by Jeremy Lumby

If anyone on MICE has experience with a L2TPv3 tunnel between Cisco routers, could you please contact me directly. I have a situation where the tunnel comes up, and broadcast traffic makes it across, however not unicast traffic. I have tried different routers, and settings, and IOS versions to no avail. Any help on this subject would be greatly appreciated. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

July 2, 2019

SFN MN-AlbertLea Remote Switch Notification #72

by Jay Hanke

An upstream vendor will be performing service affecting work on the circuit between Albert Lea and Minneapolis on the following dates. This will potentially affect the connectivity between the SFN MICE Switch and the MICE core. Estimated Start: NIGHT 1 2019-07-09 12:00 AM Estimated End: NIGHT 1 2019-07-09 05:00 AM Estimated Start: NIGHT 2 2019-07-10 12:00 AM Estimated End: NIGHT 2 2019-07-10 05:00 AM -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 2, 2019

Re: Please Welcome FDL Communications Corp to MICE!

by AnthonyAnderberg＠nuvera.net

Added FDL to the route servers, switch config, and graph system. Welcome! [signature_21095059] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jay Hanke <jayhanke(a)SOUTHFRONT.IO> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Friday, June 21, 2019 at 2:29 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Please Welcome FDL Communications Corp to MICE! Name: FDL Communications Corp Port: Ethernet5/27 (Main Switch) 10GE ASN: 396939 IPv4: 206.108.255.144/24<http://206.108.255.144/24> IPv6: 2001:504:27:0:6:0e8b::1/64 Peering Contact: peering(a)aaniin.net<mailto:peering@aaniin.net> Route Servers- YES BFD: NO Welcome Aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

June 22, 2019

Please Welcome FDL Communications Corp to MICE!

by Jay Hanke

Name: FDL Communications Corp Port: Ethernet5/27 (Main Switch) 10GE ASN: 396939 IPv4: 206.108.255.144/24 IPv6: 2001:504:27:0:6:0e8b::1/64 Peering Contact: peering(a)aaniin.net Route Servers- YES BFD: NO Welcome Aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 21, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Darin Steffl

Yes we saw issues from 6:50 to about 7:20 as well. None of our BGP sessions dropped to cogent, HE, or MICE but our customers noticed it and I did see a drop in traffic during that time.. It's stumped me and I'm glad I'm not the only one since I thought it was internal to our network but it appears not. On Thu, Jun 20, 2019, 12:30 PM Jeff Wilde <jeff.wilde(a)parkregion.com> wrote: > On a separate note, did anyone experience very intermittent Internet > issues last night for about 30 minutes or so? > > > > Jeff Wilde > > IT Manager > > *Park Region Telephone Company* > > P: 218-826-8330 > > E: jeff.wilde(a)parkregion.com > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Andrew > Hoyos > *Sent:* Thursday, June 20, 2019 11:48 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic > > > > On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > > On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote: > > https://www.seattleix.net/blackholing > > Does MICE have an blackholng functionality equivalent to SIX? > > I was visiting with a DDoS mitigation vendor this morning and was curious > if there was a way we could automatically mitigate DoS attack traffic > coming from a MICE peer. > > > > You can adjust the routing with communities, ie. in the MICE communities > aera of > http://micemn.net/technical.html > > you could block-hole the AS that is sending you that traffic. > > > > unfortunately, that just has the effect of traffic going elsewhere - not a > blackhole effect. the communities in place would just cause the route not > to be advertised to said peer, and the traffic would just ingress your > network via a different path. In the case of a DDOS, it’s likely you have > multiple ASN’s targeting you. > > > > https://www.seattleix.net/blackholing > > > > SIX, as an example, has a blackhole IP address, and the route servers > matching a blackhole community to set next hop to this to sink the traffic > on the switch fabric. > > Perhaps something we should look into for MICE. > > > > > > — > > Andrew Hoyos > > hoyosa(a)gmail.com > > > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Dave Williams

We had issues reported last night... it was 'fixed' by the time I looked at things Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Mike Horwath <drechsau(a)GEEKS.ORG> Date: 6/20/19 12:32 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote: > On a separate note, did anyone experience very intermittent Internet > issues last night for about 30 minutes or so? You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Jay Hanke

I was in a hotel in waterloo ia on centruylink and had a similar problem about the same time to fb and google. On Thu, Jun 20, 2019, 1:00 PM Brian Riggleman <briggleman(a)jagcom.net> wrote: > We also lost google related sites from 6:50 PM until 7:18 PM. I also was > unable to explain this. > > > Brian Riggleman > > Network Administrator > > Jaguar Communications > > 507-214-0258 briggleman(a)jagcom.net > > > On Thu, Jun 20, 2019 at 12:33 PM Jeff Wilde <jeff.wilde(a)parkregion.com> > wrote: > >> No lost icmp to google and others allegedly but I'm not finding anything. >> >> Jeff Wilde >> IT Manager >> Park Region Telephone Company >> P: 218-826-8330 >> E: jeff.wilde(a)parkregion.com >> >> -----Original Message----- >> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Mike >> Horwath >> Sent: Thursday, June 20, 2019 12:32 PM >> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic >> >> On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote: >> > On a separate note, did anyone experience very intermittent Internet >> > issues last night for about 30 minutes or so? >> >> You mean for Hulu and Netflix going down? :) >> >> -- >> Mike Horwath, reachable via drechsau(a)Geeks.ORG >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Andrew Hoyos

On Jun 20, 2019, at 12:31 PM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: > > On Thu, Jun 20, 2019 at 11:48:13AM -0500, Andrew Hoyos wrote: >> SIX, as an example, has a blackhole IP address, and the route >> servers matching a blackhole community to set next hop to this to >> sink the traffic on the switch fabric. Perhaps something we should >> look into for MICE. > > A good idea - perahps use a specific community advertisement with > correct next-hop into the ether. > Yes, exactly. Rough logic = route server accepts /32’s from you tagged with 65535:666 route server sets next hop no matching prefixes to 206.108.255.0 206.108.255.0 is a blackhole/null destination on switching fabric, or at least translates to dst-mac-addr that is layer2 acl’d to drop. I’d think a prerequisite for this, however, is getting to a point on the route-servers where we are validating/strict filtering prefixes that folks are sending (via IRR), otherwise, we run the risk of people essentially being able to blackhole others. > You can even automate a few things as well from known blacklists, then > another system to accept new networks to be reviewed then blackholed > for <period of time>. Sure, essentially what many folks do already internally for RTBH, and signaling to upstream. We utilize Kentik to flip back community tagged routes via BGP to for us for dst ip’s under attack matching certain parameters. We blackhole these on our core routers via the same logic above (match community, set next hop to a IP w/ a discard/null route). In turn, you can community match these and send upstream if your upstreams support a RTBH community as well. — Andrew Hoyos hoyosa(a)gmail.com

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Brian Riggleman

We also lost google related sites from 6:50 PM until 7:18 PM. I also was unable to explain this. Brian Riggleman Network Administrator Jaguar Communications 507-214-0258 briggleman(a)jagcom.net On Thu, Jun 20, 2019 at 12:33 PM Jeff Wilde <jeff.wilde(a)parkregion.com> wrote: > No lost icmp to google and others allegedly but I'm not finding anything. > > Jeff Wilde > IT Manager > Park Region Telephone Company > P: 218-826-8330 > E: jeff.wilde(a)parkregion.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Mike > Horwath > Sent: Thursday, June 20, 2019 12:32 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic > > On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote: > > On a separate note, did anyone experience very intermittent Internet > > issues last night for about 30 minutes or so? > > You mean for Hulu and Netflix going down? :) > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG >

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Jeff Wilde

No lost icmp to google and others allegedly but I'm not finding anything. Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Mike Horwath Sent: Thursday, June 20, 2019 12:32 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote: > On a separate note, did anyone experience very intermittent Internet > issues last night for about 30 minutes or so? You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Mike Horwath

On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote: > On a separate note, did anyone experience very intermittent Internet > issues last night for about 30 minutes or so? You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Mike Horwath

On Thu, Jun 20, 2019 at 11:48:13AM -0500, Andrew Hoyos wrote: > SIX, as an example, has a blackhole IP address, and the route > servers matching a blackhole community to set next hop to this to > sink the traffic on the switch fabric. Perhaps something we should > look into for MICE. A good idea - perahps use a specific community advertisement with correct next-hop into the ether. Then folks can subscribe as needed. You can even automate a few things as well from known blacklists, then another system to accept new networks to be reviewed then blackholed for <period of time>. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Jeff Wilde

On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so? Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com<mailto:jeff.wilde@parkregion.com> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Thursday, June 20, 2019 11:48 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn(a)IPHOUSE.NET<mailto:merlyn@IPHOUSE.NET>> wrote: On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote: https://www.seattleix.net/blackholing Does MICE have an blackholng functionality equivalent to SIX? I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html you could block-hole the AS that is sending you that traffic. unfortunately, that just has the effect of traffic going elsewhere - not a blackhole effect. the communities in place would just cause the route not to be advertised to said peer, and the traffic would just ingress your network via a different path. In the case of a DDOS, it’s likely you have multiple ASN’s targeting you. https://www.seattleix.net/blackholing SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE. — Andrew Hoyos hoyosa(a)gmail.com<mailto:hoyosa@gmail.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Andrew Hoyos

On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote: >> https://www.seattleix.net/blackholing >> >> Does MICE have an blackholng functionality equivalent to SIX? >> >> I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. > > > You can adjust the routing with communities, ie. in the MICE communities aera of > http://micemn.net/technical.html > > you could block-hole the AS that is sending you that traffic. unfortunately, that just has the effect of traffic going elsewhere - not a blackhole effect. the communities in place would just cause the route not to be advertised to said peer, and the traffic would just ingress your network via a different path. In the case of a DDOS, it’s likely you have multiple ASN’s targeting you. https://www.seattleix.net/blackholing <https://www.seattleix.net/blackholing> SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE. — Andrew Hoyos hoyosa(a)gmail.com

June 20, 2019

Re: ADV: [MICE-DISCUSS] Blackholing DoS traffic

by Doug McIntyre

On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote: >https://www.seattleix.net/blackholing > >Does MICE have an blackholng functionality equivalent to SIX? > >I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html you could block-hole the AS that is sending you that traffic. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

June 20, 2019

Blackholing DoS traffic

by Frank Bulk

https://www.seattleix.net/blackholing Does MICE have an blackholng functionality equivalent to SIX? I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. Regards, Frank

June 20, 2019

Re: Bi-lateral peering with Yahoo!

by Frank Bulk

Thanks, Jay. I saw service return Monday evening. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Monday, June 17, 2019 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Bi-lateral peering with Yahoo! I opened ticket # PTKT1583794 with Yahoo/Oath regarding the LoS. Jay On Fri, Jun 14, 2019 at 6:01 PM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: Looks like 1 of 2 Oath links are down. On Fri, Jun 14, 2019 at 5:47 PM Andrew Hoyos <hoyosa(a)gmail.com<mailto:hoyosa@gmail.com>> wrote: Same here: Jun 14 20:12:13 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: NOTIFICATION sent to 2001:504:27::2846:0:1 (External AS 10310): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 2001:504:27::2846:0:1 (External AS 10310), socket buffer sndcc: 38 rcvcc: 0 TCP state: 4, snd_una: 715538896 snd_nxt: 715538934 snd_wnd: 16384 rcv_nxt: 937051995 rcv_adv: 937068379, hold timer 90s, hold timer remain 0s, last sent 26s, TCP port (local 60544, remote 179) Jun 14 20:12:30 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: NOTIFICATION sent to 206.108.255.58 (External AS 10310): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 206.108.255.58 (External AS 10310), socket buffer sndcc: 57 rcvcc: 0 TCP state: 4, snd_una: 3794713179 snd_nxt: 3794713236 snd_wnd: 16384 rcv_nxt: 58606928 rcv_adv: 58623312, hold timer 90s, hold timer remain 0s, last sent 22s, TCP port (local 179, remote 65426) — Andrew Hoyos hoyosa(a)gmail.com<mailto:hoyosa@gmail.com> > On Jun 14, 2019, at 5:44 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM<mailto:fbulk@MYPREMIERONLINE.COM>> wrote: > > I assume some others are seeing the same, that peering with one of Yahoo!'s IPs has been down since just before 3:45 pm? > > Host premier.sxct-north.fbnt.netins.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__premier.sxct-2Dnorth.fb…> [207.32.15.28] with interface to BGP peer AS10310-1.micemn.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__AS10310-2D1.micemn.net&…> [206.108.255.58/AS<https://urldefense.proofpoint.com/v2/url?u=http-3A__206.108.255.58_AS&d=DwM…> 10310 (Oath Holdings Inc., US)]: the BGP peer state is in state connect > > Frank -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 19, 2019

Re: Bi-lateral peering with Yahoo!

by Jay Hanke

I opened ticket # PTKT1583794 with Yahoo/Oath regarding the LoS. Jay On Fri, Jun 14, 2019 at 6:01 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > Looks like 1 of 2 Oath links are down. > > On Fri, Jun 14, 2019 at 5:47 PM Andrew Hoyos <hoyosa(a)gmail.com> wrote: > >> Same here: >> >> Jun 14 20:12:13 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: >> NOTIFICATION sent to 2001:504:27::2846:0:1 (External AS 10310): code 4 >> (Hold Timer Expired Error), Reason: holdtime expired for >> 2001:504:27::2846:0:1 (External AS 10310), socket buffer sndcc: 38 rcvcc: 0 >> TCP state: 4, snd_una: 715538896 snd_nxt: 715538934 snd_wnd: 16384 rcv_nxt: >> 937051995 rcv_adv: 937068379, hold timer 90s, hold timer remain 0s, last >> sent 26s, TCP port (local 60544, remote 179) >> >> Jun 14 20:12:30 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: >> NOTIFICATION sent to 206.108.255.58 (External AS 10310): code 4 (Hold Timer >> Expired Error), Reason: holdtime expired for 206.108.255.58 (External AS >> 10310), socket buffer sndcc: 57 rcvcc: 0 TCP state: 4, snd_una: 3794713179 >> snd_nxt: 3794713236 snd_wnd: 16384 rcv_nxt: 58606928 rcv_adv: 58623312, >> hold timer 90s, hold timer remain 0s, last sent 22s, TCP port (local 179, >> remote 65426) >> >> — >> Andrew Hoyos >> hoyosa(a)gmail.com >> >> >> >> > On Jun 14, 2019, at 5:44 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> >> wrote: >> > >> > I assume some others are seeing the same, that peering with one of >> Yahoo!'s IPs has been down since just before 3:45 pm? >> > >> > Host premier.sxct-north.fbnt.netins.net [207.32.15.28] with interface >> to BGP peer AS10310-1.micemn.net [206.108.255.58/AS 10310 (Oath Holdings >> Inc., US)]: the BGP peer state is in state connect >> > >> > Frank >> > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 17, 2019

Re: Google ASN Migration/Port Move

by Doug McIntyre

On Thu, Jun 13, 2019 at 11:53:21AM -0500, Jay Hanke wrote: >Google is converting their existing MICE connection to their backbone >AS (15169). > >Immediately following the migration, Google will disconnect their >existing 100G port (Ethernet3/2/1). > >The new addressing will be: >ASN: 15169 >IPv4: 206.108.255.141/24 >IPv6: 2001:504:27:0:0:3b41::1/64 I've added AS 15169 to the route servers to prep this and ready them up. >The new 100G port will be Ethernet3/13/1. > >Google has requested the port be set up as a new single channel LAG group. I've setup Port-Channel 29 for this LAG group, and have the Arista core switch config'd up on Port Eth3/13/1. They should be all set to bring up this link as needed. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

June 17, 2019

Re: Bi-lateral peering with Yahoo!

by Jay Hanke

Looks like 1 of 2 Oath links are down. On Fri, Jun 14, 2019 at 5:47 PM Andrew Hoyos <hoyosa(a)gmail.com> wrote: > Same here: > > Jun 14 20:12:13 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: > NOTIFICATION sent to 2001:504:27::2846:0:1 (External AS 10310): code 4 > (Hold Timer Expired Error), Reason: holdtime expired for > 2001:504:27::2846:0:1 (External AS 10310), socket buffer sndcc: 38 rcvcc: 0 > TCP state: 4, snd_una: 715538896 snd_nxt: 715538934 snd_wnd: 16384 rcv_nxt: > 937051995 rcv_adv: 937068379, hold timer 90s, hold timer remain 0s, last > sent 26s, TCP port (local 60544, remote 179) > > Jun 14 20:12:30 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: > NOTIFICATION sent to 206.108.255.58 (External AS 10310): code 4 (Hold Timer > Expired Error), Reason: holdtime expired for 206.108.255.58 (External AS > 10310), socket buffer sndcc: 57 rcvcc: 0 TCP state: 4, snd_una: 3794713179 > snd_nxt: 3794713236 snd_wnd: 16384 rcv_nxt: 58606928 rcv_adv: 58623312, > hold timer 90s, hold timer remain 0s, last sent 22s, TCP port (local 179, > remote 65426) > > — > Andrew Hoyos > hoyosa(a)gmail.com > > > > > On Jun 14, 2019, at 5:44 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> > wrote: > > > > I assume some others are seeing the same, that peering with one of > Yahoo!'s IPs has been down since just before 3:45 pm? > > > > Host premier.sxct-north.fbnt.netins.net [207.32.15.28] with interface > to BGP peer AS10310-1.micemn.net [206.108.255.58/AS 10310 (Oath Holdings > Inc., US)]: the BGP peer state is in state connect > > > > Frank > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 14, 2019

Re: Bi-lateral peering with Yahoo!

by Andrew Hoyos

Same here: Jun 14 20:12:13 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: NOTIFICATION sent to 2001:504:27::2846:0:1 (External AS 10310): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 2001:504:27::2846:0:1 (External AS 10310), socket buffer sndcc: 38 rcvcc: 0 TCP state: 4, snd_una: 715538896 snd_nxt: 715538934 snd_wnd: 16384 rcv_nxt: 937051995 rcv_adv: 937068379, hold timer 90s, hold timer remain 0s, last sent 26s, TCP port (local 60544, remote 179) Jun 14 20:12:30 er1.511.msp rpd[1832]: bgp_hold_timeout:4677: NOTIFICATION sent to 206.108.255.58 (External AS 10310): code 4 (Hold Timer Expired Error), Reason: holdtime expired for 206.108.255.58 (External AS 10310), socket buffer sndcc: 57 rcvcc: 0 TCP state: 4, snd_una: 3794713179 snd_nxt: 3794713236 snd_wnd: 16384 rcv_nxt: 58606928 rcv_adv: 58623312, hold timer 90s, hold timer remain 0s, last sent 22s, TCP port (local 179, remote 65426) — Andrew Hoyos hoyosa(a)gmail.com > On Jun 14, 2019, at 5:44 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: > > I assume some others are seeing the same, that peering with one of Yahoo!'s IPs has been down since just before 3:45 pm? > > Host premier.sxct-north.fbnt.netins.net [207.32.15.28] with interface to BGP peer AS10310-1.micemn.net [206.108.255.58/AS 10310 (Oath Holdings Inc., US)]: the BGP peer state is in state connect > > Frank

June 14, 2019

Bi-lateral peering with Yahoo!

by Frank Bulk

I assume some others are seeing the same, that peering with one of Yahoo!'s IPs has been down since just before 3:45 pm? Host premier.sxct-north.fbnt.netins.net [207.32.15.28] with interface to BGP peer AS10310-1.micemn.net [206.108.255.58/AS 10310 (Oath Holdings Inc., US)]: the BGP peer state is in state connect Frank

June 14, 2019

Re: Google ASN Migration/Port Move

by Dave Schwartz

Hi MICE Members, Google AS15169 can deliver all Google traffic. Once we are closer to turn up of the new link I'll share some details with the members on the next steps to peer directly with us. We plan also to peer the route server. Thanks, Dave -- Dave Schwartz Peering & Content Distribution Google AS15169 daveschwartz(a)google.com +1-650-253-7615 +44 20 3820 9224 On Fri, Jun 14, 2019, 07:27 Jay Hanke <jayhanke(a)southfront.io> wrote: > Google is connecting MICE directly to the Google backbone AS so more > of the services will be available following the upgrade at MICE. > > On Fri, Jun 14, 2019 at 9:20 AM Jeff Wilde <jeff.wilde(a)parkregion.com> > wrote: > > > > I was under the impression that they only basically do youtube/etc > through there. I asked about bilateral and that is what they said. > > > > Jeff Wilde > > IT Manager > > Park Region Telephone Company > > P: 218-826-8330 > > E: jeff.wilde(a)parkregion.com > > > > -----Original Message----- > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > > Sent: Friday, June 14, 2019 7:31 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > > > I think this is a great move for Google/MICE, and I can easily foresee > Google outgrowing the 100G quickly. On a side note I would update the > website to state "Partial" for the route server. Google had recently > changed their policy for AS15169 where they still utilize the route > servers, however not all content is advertized through it. If you want to > maximize your Google backbone traffic, you need to setup a bilateral with > them. > > > > Jeremy Lumby > > Minnesota VoIP > > 9217 17th Ave S #216 > > Bloomington, MN 55425 > > M: 612-355-7740 > > D: 612-392-6814 > > F: 952-873-7425 > > jlumby(a)mnvoip.com > > > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Richard Laager > > Sent: Friday, June 14, 2019 2:21 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > > > On 6/13/19 11:53 AM, Jay Hanke wrote: > > > Google is converting their existing MICE connection to their backbone > > > AS (15169). > > > > > > Immediately following the migration, Google will disconnect their > > > existing 100G port (Ethernet3/2/1). > > > > > > The new addressing will be: > > > ASN: 15169 > > > IPv4: 206.108.255.141/24 > > > IPv6: 2001:504:27:0:0:3b41::1/64 > > > > > > The new 100G port will be Ethernet3/13/1. > > > > > > Google has requested the port be set up as a new single channel LAG > group. > > > > I've updated the website. > > > > -- > > Richard > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 >

June 14, 2019

Re: Google ASN Migration/Port Move

by Jay Hanke

Google is connecting MICE directly to the Google backbone AS so more of the services will be available following the upgrade at MICE. On Fri, Jun 14, 2019 at 9:20 AM Jeff Wilde <jeff.wilde(a)parkregion.com> wrote: > > I was under the impression that they only basically do youtube/etc through there. I asked about bilateral and that is what they said. > > Jeff Wilde > IT Manager > Park Region Telephone Company > P: 218-826-8330 > E: jeff.wilde(a)parkregion.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby > Sent: Friday, June 14, 2019 7:31 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager > Sent: Friday, June 14, 2019 2:21 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move > > On 6/13/19 11:53 AM, Jay Hanke wrote: > > Google is converting their existing MICE connection to their backbone > > AS (15169). > > > > Immediately following the migration, Google will disconnect their > > existing 100G port (Ethernet3/2/1). > > > > The new addressing will be: > > ASN: 15169 > > IPv4: 206.108.255.141/24 > > IPv6: 2001:504:27:0:0:3b41::1/64 > > > > The new 100G port will be Ethernet3/13/1. > > > > Google has requested the port be set up as a new single channel LAG group. > > I've updated the website. > > -- > Richard -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 14, 2019

Re: Google ASN Migration/Port Move

by Jeff Wilde

I was under the impression that they only basically do youtube/etc through there. I asked about bilateral and that is what they said. Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, June 14, 2019 7:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, June 14, 2019 2:21 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Re: Google ASN Migration/Port Move

by Jeremy Lumby

I think this is a great move for Google/MICE, and I can easily foresee Google outgrowing the 100G quickly. On a side note I would update the website to state "Partial" for the route server. Google had recently changed their policy for AS15169 where they still utilize the route servers, however not all content is advertized through it. If you want to maximize your Google backbone traffic, you need to setup a bilateral with them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, June 14, 2019 2:21 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google ASN Migration/Port Move On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Re: Google ASN Migration/Port Move

by Richard Laager

On 6/13/19 11:53 AM, Jay Hanke wrote: > Google is converting their existing MICE connection to their backbone > AS (15169). > > Immediately following the migration, Google will disconnect their > existing 100G port (Ethernet3/2/1). > > The new addressing will be: > ASN: 15169 > IPv4: 206.108.255.141/24 > IPv6: 2001:504:27:0:0:3b41::1/64 > > The new 100G port will be Ethernet3/13/1. > > Google has requested the port be set up as a new single channel LAG group. I've updated the website. -- Richard

June 14, 2019

Google ASN Migration/Port Move

by Jay Hanke

Google is converting their existing MICE connection to their backbone AS (15169). Immediately following the migration, Google will disconnect their existing 100G port (Ethernet3/2/1). The new addressing will be: ASN: 15169 IPv4: 206.108.255.141/24 IPv6: 2001:504:27:0:0:3b41::1/64 The new 100G port will be Ethernet3/13/1. Google has requested the port be set up as a new single channel LAG group. We'll need new route server connections as well. Welcome aboard (Again)! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

June 13, 2019

Re: lost MICE

by David Farmer

A bit of trivia, the fiber cut is right near the highest point in Iowa, Hawkeye Point, just east of the intersection of 130th St and Hy60. It's all downhill from there. :) https://en.wikipedia.org/wiki/Hawkeye_Point On Tue, Jun 11, 2019 at 1:40 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > And RFO: > > Reason for Outage Summary: A fiber cut was reported in Sibley IA. Zayo > crews were dispatched out to > > 130th and 140th along highway 60 and repaired the damaged fiber which was > caused by a Drain tile contractor trenched through the cable. All > > Repairs were completed at 1:00 PM CDT. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 11:38 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 11:30 am: OSP reported that splicing is complete. Please verify that your > Services have restored. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 10:44 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 10:33 am: OSP has reported that splicing on the north case is complete. > Splicers working on South case. You should start to see service restore as > splicing continues. A rough ETR is 1 hour. More updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 10:05 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. > > > > From what I heard from our field staff, the issue wasn’t that bad – > looking through the update, it just took Zayo’s staff and/or contractors a > long time to get on site and perform the work. It seems like they had just > one fiber splicing crew, so they didn’t prep and splice on both sides at > the same time. We’ve had similar cuts many times before and we would likely > have this resolved in 6 to 8 hours. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James > Urwiller > *Sent:* Tuesday, June 11, 2019 9:43 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > I would like to Thank Frank for reporting all this info. He is keeping up > with the Zayo ticket responses very well! > > You mentioned this was cut near your area.. was it really that bad to > warrant 24+ hrs? seems like they are pretty slow with these repairs? > > > > James Urwiller > > Network Architect > > Stealth Broadband > > james.urwiller(a)stealthbroadband.com > > [image: stealth300] > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Tuesday, June 11, 2019 07:58 > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 11:04 pm: Excavation is currently underway at this time. Techs are > trenching out the 200 feet of cable required to make repairs. > > 12:58 am: Underground crews have completed the trench on one side of the > break. OSP splicers have the cable exposed on that side and are prepping > the old cable and the new for repairs. The Underground crew is working to > excavate on the opposite end at this time. No ETTR at this time. > > 2:22 am: OSP crews are still continuing to work at this time. Excavation > crews are continuing to work to expose the remaining fiber to prep for > splicing. No ETTR at this time. > > 3:20 am: At this time the underground crew has completed excavation and > the cable is fully exposed for repairs. Prep work is still ongoing on both > sides. Splicers expect to be splicing on one end of the cable in about 1 > hour. The other side will be spliced once prep work is complete. No ETR at > this time. > > 5:04 am: OSP splicers are continuing the splice at this time. Techs are > prioritizing the active fibers to speed up restoration. No ETTR at this > time. > > 5:57 am: OSP splicers are continue to finish up the splicing on the North > side of the cut. Once complete crews will move to the South side to > complete prep work and begin splicing. We have a tentative ETTR of 10 CST > today. > > 7:07 am: OSP splicers onsite advised they have been a bit delayed on the > splicing at the North side of the cut. Splicers are nearly complete on the > North side, and will be moving to the South to complete prep work and > repairs. Crews have revised the tentative ETTR for 12 CST. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:47 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 9:26 pm: Emergency locates are nearly complete so the underground > construction crew will be able to start major excavations soon. The crew > will need to trench along 200 feet of the cable path to add in a new > section of fiber. No ETTR at this time. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 9:09 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 8:19 pm: OSP has reported that Underground crew has arrived on site. They > will begin excavation within the next few minutes. There is no ETTR at this > time. More updates to follow. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 7:15 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 6:58 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route and are trying to get around flooded roads. ETA is 30 mins. > More updates to follow as we receive them. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 6:00 PM > *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 4:47 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive > them. > > > > 5:51 pm: No Change in Update. OSP has located the damage between 130th and > 140th along highway 60 in Sibley IA. Looks like land owner or contractor > trenched through the cable. Cable is completely severed. Underground crew > is in route with ETA of 30-45 mins. More updates to follow as we receive > them. > > > > Frank > > > > *From:* Frank Bulk > *Sent:* Monday, June 10, 2019 2:56 PM > *To:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject:* RE: [MICE-DISCUSS] lost MICE > > > > 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 > mins. More updates to follow. > > > > Note that the location listed in the ticket is incorrect – I’ve > communicated the correct address (130th and Hwy 60) to Zayo’s NCC and > they’ve acknowledged receipt of that. > > > > If they do get onsite by 3:15 pm it will have been about 10 hours since > the initial cut. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 1:24 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > 12:35 pm: OSP reported they are trying to get resources to the damage > location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA > or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More > updates to follow. > > 12:43 pm: OSP reported Splicer is in route to the damage location right > off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More > updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:35 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Another update from Zayo: > > > > 10:30 OSP reported they are trying to get resources to the damage > location. Damage appears to be right off of I60 and 170th in Sibley IA. > Currently there is no ETA for techs. More updates to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 10:27 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > First two updates from Zayo: > > > > 9:48 am: Your case has been associated with a higher level outage ticket, > your ticket number will remain the same. The latest update is as follows: > Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN > affecting multiple customers. The appropriate resources have been engaged > for repairs. At this time, OSP is en route to the suspected trouble > location to begin investigating. No ETA available as of yet. > > > > 10:21 am: Our OSP crews are en route to begin investigations. More updates > to follow. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 8:53 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy > 60 ( > https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) > in Sibley, Iowa. The tiler called one of our techs and the farmer called > another one of our techs, inquiring if it was our fiber or not (it’s not). > The tiler reported hitting a line below the tape. > > > > I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and > haven’t yet received an acknowledgement, but once I do I will inform them > of the location. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank > Bulk > *Sent:* Monday, June 10, 2019 8:06 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] lost MICE > > > > Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I > assume it’s an issue between Sioux City and Belle Plaine. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James > Urwiller > *Sent:* Monday, June 10, 2019 7:58 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] lost MICE > > > > I lost MICE again over ZAYO about 30 minutes ago.. anyone else? > > > > James Urwiller > > Network Architect > > Stealth Broadband > > james.urwiller(a)stealthbroadband.com > > [image: stealth300] > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

June 11, 2019

Re: lost MICE

by Frank Bulk

And RFO: Reason for Outage Summary: A fiber cut was reported in Sibley IA. Zayo crews were dispatched out to 130th and 140th along highway 60 and repaired the damaged fiber which was caused by a Drain tile contractor trenched through the cable. All Repairs were completed at 1:00 PM CDT. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 11:38 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 11:30 am: OSP reported that splicing is complete. Please verify that your Services have restored. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:44 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

11:30 am: OSP reported that splicing is complete. Please verify that your Services have restored. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:44 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

10:33 am: OSP has reported that splicing on the north case is complete. Splicers working on South case. You should start to see service restore as splicing continues. A rough ETR is 1 hour. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 10:05 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Aaron Clark

We have experienced extremely poor responsiveness and very uninformed call center reps on recent fiber outages with Zayo as well. Our circuits were previously part of Neutral Path so I don’t have any previous experience with Zayo to compare to. Thank you Aaron Clark | Vice President – COO *Future Technologies* *www.futuretech.net <www.futuretech.net>* *Office: 402.817.2682x121* *Mobile: 402.770.9251* [image: cid:image001.png@01D36053.05503150] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Tuesday, June 11, 2019 10:05 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. >From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James Urwiller *Sent:* Tuesday, June 11, 2019 9:43 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Tuesday, June 11, 2019 07:58 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:47 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 9:09 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 7:15 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 6:00 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 2:56 PM *To:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 1:24 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:35 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:27 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 8:53 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 ( https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U… <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 8:06 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James Urwiller *Sent:* Monday, June 10, 2019 7:58 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

June 11, 2019

Re: lost MICE

by Frank Bulk

My own link alarm cleared at 9:52 am, so just a hair under 26.5 hours. From what I heard from our field staff, the issue wasn’t that bad – looking through the update, it just took Zayo’s staff and/or contractors a long time to get on site and perform the work. It seems like they had just one fiber splicing crew, so they didn’t prep and splice on both sides at the same time. We’ve had similar cuts many times before and we would likely have this resolved in 6 to 8 hours. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Urwiller Sent: Tuesday, June 11, 2019 9:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Tuesday, June 11, 2019 07:58 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by James Urwiller

I would like to Thank Frank for reporting all this info. He is keeping up with the Zayo ticket responses very well! You mentioned this was cut near your area.. was it really that bad to warrant 24+ hrs? seems like they are pretty slow with these repairs? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Tuesday, June 11, 2019 07:58 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:47 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 9:09 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 7:15 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 6:00 PM *To:* 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank *From:* Frank Bulk *Sent:* Monday, June 10, 2019 2:56 PM *To:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject:* RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 1:24 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:35 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 10:27 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 8:53 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 ( https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U… <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, June 10, 2019 8:06 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *James Urwiller *Sent:* Monday, June 10, 2019 7:58 AM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

June 11, 2019

Re: lost MICE

by Frank Bulk

11:04 pm: Excavation is currently underway at this time. Techs are trenching out the 200 feet of cable required to make repairs. 12:58 am: Underground crews have completed the trench on one side of the break. OSP splicers have the cable exposed on that side and are prepping the old cable and the new for repairs. The Underground crew is working to excavate on the opposite end at this time. No ETTR at this time. 2:22 am: OSP crews are still continuing to work at this time. Excavation crews are continuing to work to expose the remaining fiber to prep for splicing. No ETTR at this time. 3:20 am: At this time the underground crew has completed excavation and the cable is fully exposed for repairs. Prep work is still ongoing on both sides. Splicers expect to be splicing on one end of the cable in about 1 hour. The other side will be spliced once prep work is complete. No ETR at this time. 5:04 am: OSP splicers are continuing the splice at this time. Techs are prioritizing the active fibers to speed up restoration. No ETTR at this time. 5:57 am: OSP splicers are continue to finish up the splicing on the North side of the cut. Once complete crews will move to the South side to complete prep work and begin splicing. We have a tentative ETTR of 10 CST today. 7:07 am: OSP splicers onsite advised they have been a bit delayed on the splicing at the North side of the cut. Splicers are nearly complete on the North side, and will be moving to the South to complete prep work and repairs. Crews have revised the tentative ETTR for 12 CST. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

9:26 pm: Emergency locates are nearly complete so the underground construction crew will be able to start major excavations soon. The crew will need to trench along 200 feet of the cable path to add in a new section of fiber. No ETTR at this time. Frank From: Frank Bulk Sent: Monday, June 10, 2019 9:09 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: [MICE-DISCUSS] lost MICE 8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

8:19 pm: OSP has reported that Underground crew has arrived on site. They will begin excavation within the next few minutes. There is no ETTR at this time. More updates to follow. Frank From: Frank Bulk Sent: Monday, June 10, 2019 7:15 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: [MICE-DISCUSS] lost MICE 6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

6:58 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route and are trying to get around flooded roads. ETA is 30 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 6:00 PM To: 'MICE Discuss' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: [MICE-DISCUSS] lost MICE 4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 11, 2019

Re: lost MICE

by Frank Bulk

4:47 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 2 - 2.5 hours. More updates to follow as we receive them. 5:51 pm: No Change in Update. OSP has located the damage between 130th and 140th along highway 60 in Sibley IA. Looks like land owner or contractor trenched through the cable. Cable is completely severed. Underground crew is in route with ETA of 30-45 mins. More updates to follow as we receive them. Frank From: Frank Bulk Sent: Monday, June 10, 2019 2:56 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: [MICE-DISCUSS] lost MICE 2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 10, 2019

Re: lost MICE

by Frank Bulk

2:42 pm: OSP has reported that splicer is in route and has a ETA of 30 mins. More updates to follow. Note that the location listed in the ticket is incorrect – I’ve communicated the correct address (130th and Hwy 60) to Zayo’s NCC and they’ve acknowledged receipt of that. If they do get onsite by 3:15 pm it will have been about 10 hours since the initial cut. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 1:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE 12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 10, 2019

Re: lost MICE

by Frank Bulk

12:35 pm: OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. ETA or 2.5 hours at 10:58 AM CDT, so splicers should be on-site shortly. More updates to follow. 12:43 pm: OSP reported Splicer is in route to the damage location right off of I60 and 170th in Sibley IA. ETA is approximately 1.5 hrs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:35 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 10, 2019

Re: lost MICE

by Frank Bulk

Another update from Zayo: 10:30 OSP reported they are trying to get resources to the damage location. Damage appears to be right off of I60 and 170th in Sibley IA. Currently there is no ETA for techs. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 10:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] lost MICE First two updates from Zayo: 9:48 am: Your case has been associated with a higher level outage ticket, your ticket number will remain the same. The latest update is as follows: Zayo is experiencing a network event between SIOUX CITY, IA and MANKATO MN affecting multiple customers. The appropriate resources have been engaged for repairs. At this time, OSP is en route to the suspected trouble location to begin investigating. No ETA available as of yet. 10:21 am: Our OSP crews are en route to begin investigations. More updates to follow. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Turns out that the fiber got hit by a tiler in our area, by 130th and Hwy 60 (https://www.google.com/maps?q=hwy+60+and+130th,+sibley&rlz=1C1GCEU_enUS814U…<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps-3F…>) in Sibley, Iowa. The tiler called one of our techs and the farmer called another one of our techs, inquiring if it was our fiber or not (it’s not). The tiler reported hitting a line below the tape. I opened a ticket with Zayo’s NCC email address about 35 minutes ago, and haven’t yet received an acknowledgement, but once I do I will inform them of the location. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, June 10, 2019 8:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] lost MICE Yes, same thing, between 7:25 and 7:26 am our 10G link went down. I assume it’s an issue between Sioux City and Belle Plaine. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller Sent: Monday, June 10, 2019 7:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] lost MICE I lost MICE again over ZAYO about 30 minutes ago.. anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

June 10, 2019