- mice-discuss - lists.micemn.net

Google CDN at MICE

by AnthonyAnderberg＠nuvera.net

Does anyone have insight into what happened with Google at MICE last night? [cid:image001.png@01D4B7AA.040189C0] [signature_1049503188]

Jan. 29, 2019

Re: Exchange Stable

by James Urwiller

same Frank!! same... i got a call back from Zayo noc at 19:07 tonight, with no real assurance that anyone knew what was going on. they were just asking if I was still experiencing issues. :( Sent from my iPhone 7 > On Jan 22, 2019, at 18:38, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > Zayo had scheduled maintenance for the 14th, rescheduled it to the 16th and then canceled maintenance (on the 16th)… but then our link to MICE, through their network, was down today (22nd) from ~12:07 to ~1:10 am with no warning at all. > > Even the maintenance window notifications were unclear – no real clarity from Zayo change maintenance. > > We also saw the BGP service interruption at 3:32 pm, and then a link down at 3:33 pm (presumably when the 100G wave over which our 10G rides went down). =( > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Schmeichel > Sent: Tuesday, January 22, 2019 5:08 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Exchange Stable > > Zayo has had a fiber crew out working today from what we’ve been told. It’s been impacting our traffic for sure. > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andy Koch > Sent: Tuesday, January 22, 2019 2:58 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Exchange Stable > > Hi James, > > My sessions are showing uptimes of ~4days on the MICE, which correlates to the main switch maintenance last week. You may want to check in with Zayo as it looks that they lost all traffic on the remote switch in Belle Plaine at 1AM. Traffic graphs are not quite as clear this afternoon, but there was an upward spike on several ports. > > http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… > > > > Andy Koch > Hoyos Consulting LLC > ofc: +1 608 616 9950 > andyk(a)hoyosconsulting.com > http://www.hoyosconsulting.com > > On 1/22/19 3:41 PM, James Urwiller wrote: > Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? > Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! > > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 23, 2019

Re: Exchange Stable

by Frank Bulk

Zayo had scheduled maintenance for the 14th, rescheduled it to the 16th and then canceled maintenance (on the 16th)… but then our link to MICE, through their network, was down today (22nd) from ~12:07 to ~1:10 am with no warning at all. Even the maintenance window notifications were unclear – no real clarity from Zayo change maintenance. We also saw the BGP service interruption at 3:32 pm, and then a link down at 3:33 pm (presumably when the 100G wave over which our 10G rides went down). =( Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Schmeichel Sent: Tuesday, January 22, 2019 5:08 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Exchange Stable Zayo has had a fiber crew out working today from what we’ve been told. It’s been impacting our traffic for sure. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Andy Koch Sent: Tuesday, January 22, 2019 2:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Exchange Stable Hi James, My sessions are showing uptimes of ~4days on the MICE, which correlates to the main switch maintenance last week. You may want to check in with Zayo as it looks that they lost all traffic on the remote switch in Belle Plaine at 1AM. Traffic graphs are not quite as clear this afternoon, but there was an upward spike on several ports. http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… Andy Koch Hoyos Consulting LLC ofc: +1 608 616 9950 andyk(a)hoyosconsulting.com<mailto:andyk@hoyosconsulting.com> http://www.hoyosconsulting.com On 1/22/19 3:41 PM, James Urwiller wrote: Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 23, 2019

Re: Exchange Stable

by Justin Krejci

We were told Zayo had scheduled maintenance but did not send notifications to all parties that were to be affected (eg US Internet). We have several circuits still affected (ie are down) and we are continuing to claw through their support channels and are hearing some disturbing things. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Jeremy Schmeichel <Jeremy.Schmeichel(a)VASTBROADBAND.COM> Sent: Tuesday, January 22, 2019 5:07 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Exchange Stable Zayo has had a fiber crew out working today from what we've been told. It's been impacting our traffic for sure. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andy Koch Sent: Tuesday, January 22, 2019 2:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Exchange Stable Hi James, My sessions are showing uptimes of ~4days on the MICE, which correlates to the main switch maintenance last week. You may want to check in with Zayo as it looks that they lost all traffic on the remote switch in Belle Plaine at 1AM. Traffic graphs are not quite as clear this afternoon, but there was an upward spike on several ports. http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… Andy Koch Hoyos Consulting LLC ofc: +1 608 616 9950 andyk(a)hoyosconsulting.com<mailto:andyk@hoyosconsulting.com> http://www.hoyosconsulting.com On 1/22/19 3:41 PM, James Urwiller wrote: Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 23, 2019

Re: Exchange Stable

by Jeremy Schmeichel

Zayo has had a fiber crew out working today from what we’ve been told. It’s been impacting our traffic for sure. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andy Koch Sent: Tuesday, January 22, 2019 2:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Exchange Stable Hi James, My sessions are showing uptimes of ~4days on the MICE, which correlates to the main switch maintenance last week. You may want to check in with Zayo as it looks that they lost all traffic on the remote switch in Belle Plaine at 1AM. Traffic graphs are not quite as clear this afternoon, but there was an upward spike on several ports. http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… Andy Koch Hoyos Consulting LLC ofc: +1 608 616 9950 andyk(a)hoyosconsulting.com<mailto:andyk@hoyosconsulting.com> http://www.hoyosconsulting.com On 1/22/19 3:41 PM, James Urwiller wrote: Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 22, 2019

Re: Exchange Stable

by Andy Koch

Hi James, My sessions are showing uptimes of ~4days on the MICE, which correlates to the main switch maintenance last week. You may want to check in with Zayo as it looks that they lost all traffic on the remote switch in Belle Plaine at 1AM. Traffic graphs are not quite as clear this afternoon, but there was an upward spike on several ports. http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… Andy Koch Hoyos Consulting LLC ofc: +1 608 616 9950 andyk(a)hoyosconsulting.com http://www.hoyosconsulting.com On 1/22/19 3:41 PM, James Urwiller wrote: > > Any one else notice all their peers bounce last night around 1AM CST, > and a few again just a few minutes ago? > > Just checking to see If I should talk to my transport provider or if > this is something @ MICE. Let me know! > > James Urwiller > > Network Architect > > Stealth Broadband > > james.urwiller(a)stealthbroadband.com > <mailto:james.urwiller@stealthbroadband.com> > > stealth300 > > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Jan. 22, 2019

Re: Exchange Stable

by AnthonyAnderberg＠nuvera.net

Looks like it went down last night too, or at least traffic on that LAG dropped to zero – I see a little drip from the silliness a bit ago too: [cid:image001.png@01D4B26A.E02ED4B0] [signature_521369910] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Tuesday, January 22, 2019 at 3:52 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Exchange Stable Zayo is my transport.. I will bring it up with them. Thank you! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET> Sent: Tuesday, January 22, 2019 15:52 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Exchange Stable Looks like the ports making up the Zayo-Remote bounced down and back up at 15:31:19. I’d thought perhaps Zayo was adding the 3rd link to that LAG but it is still showing two active members. [signature_198002810] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM<mailto:james.urwiller@TURNKEY-TELECOM.COM>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Date: Tuesday, January 22, 2019 at 3:41 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> Subject: [MICE-DISCUSS] Exchange Stable Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 22, 2019

Re: Exchange Stable

by James Urwiller

Zayo is my transport.. I will bring it up with them. Thank you! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of * AnthonyAnderberg(a)NUVERA.NET *Sent:* Tuesday, January 22, 2019 15:52 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Exchange Stable Looks like the ports making up the Zayo-Remote bounced down and back up at 15:31:19. I’d thought perhaps Zayo was adding the 3rd link to that LAG but it is still showing two active members. [image: signature_198002810] *From: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> *Reply-To: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Date: *Tuesday, January 22, 2019 at 3:41 PM *To: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *Subject: *[MICE-DISCUSS] Exchange Stable Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 22, 2019

Re: Exchange Stable

by AnthonyAnderberg＠nuvera.net

Looks like the ports making up the Zayo-Remote bounced down and back up at 15:31:19. I’d thought perhaps Zayo was adding the 3rd link to that LAG but it is still showing two active members. [signature_198002810] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Tuesday, January 22, 2019 at 3:41 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Exchange Stable Any one else notice all their peers bounce last night around 1AM CST, and a few again just a few minutes ago? Just checking to see If I should talk to my transport provider or if this is something @ MICE. Let me know! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Jan. 22, 2019

Re: ADV: [MICE-DISCUSS] Exchange Stable

by Doug McIntyre

On Tue, Jan 22, 2019 at 03:41:54PM -0600, James Urwiller wrote: >Any one else notice all their peers bounce last night around 1AM CST, and a >few again just a few minutes ago? > >Just checking to see If I should talk to my transport provider or if this >is something @ MICE. Let me know! No changes here. Looking at the route servers, there's a few BGP sessions changes in the last 24h, but nothing like what you've reported. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Jan. 22, 2019