- mice-discuss - lists.micemn.net

Re: Anyone else down to Belle Plain since 3:45 pm?

by Frank Bulk

Another update at 10:09 pm: Current Update: Our OSP reported that the repair crew are running a little behind the 2 hour ETA provided earlier and have noted the are still about 50 minutes from the damage area. So they will arrive around 11 pm, about 7h 15m from the start of the outage. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 8:30 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? More detail in Zayo’s last update at 8:08 pm We apologize for the delayed update. We've spoken with our OSP PM and have been advised that the splice crew's ETA to the site is still about 1 to 1.5 hours and the ETTR is 4.5 to 5 hours. The damage was caused by a construction crew and the damage area is south of Hinton, IA around County Rd 60 and Iowa State Rd 75. Additional information to follow once the splice crew is on site. So it does seem related to the Hwy 75 re-construction project. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Krejci Sent: Monday, September 9, 2019 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM<mailto:fbulk@MYPREMIERONLINE.COM>> Sent: Monday, September 9, 2019 4:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig…<https://urldefense.proofpoint.com/v2/url?u=https-3A__siouxcityjournal.com_n…> Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET%20on%20behalf%20of%20james.urwiller@TURNKEY-TELECOM.COM>> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Sept. 10, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by Andrew Hoyos

Do we need to create a ‘zayo-outages’ list for this sort of thing? :) — Andrew Hoyos hoyosa(a)gmail.com > On Sep 9, 2019, at 9:49 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: > > I’ve asked our OSP if he had any local knowledge, but he’s not aware. > > I will alert our Hinton field tech just in case he hears anything about it. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of James Urwiller > Sent: Monday, September 9, 2019 8:33 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > So is it common for IA/MN construction to hit things all willy nilly without locates? Or is this another botched locate job? > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com <mailto:james.urwiller@stealthbroadband.com> > <image001.jpg> > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk > Sent: Monday, September 9, 2019 20:30 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > More detail in Zayo’s last update at 8:08 pm > We apologize for the delayed update. We've spoken with our OSP PM and have been advised that the splice crew's ETA to the site is still about 1 to 1.5 hours and the ETTR is 4.5 to 5 hours. The damage was caused by a construction crew and the damage area is south of Hinton, IA around County Rd 60 and Iowa State Rd 75. Additional information to follow once the splice crew is on site. > > So it does seem related to the Hwy 75 re-construction project. > > Frank > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Krejci > Sent: Monday, September 9, 2019 6:13 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. > > > > We don't have growing pains we have Zayo pains. > > > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM <mailto:fbulk@MYPREMIERONLINE.COM>> > Sent: Monday, September 9, 2019 4:37 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > Could be part of the Hwy 75 project: > https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig… <https://urldefense.proofpoint.com/v2/url?u=https-3A__siouxcityjournal.com_n…> > Our OSP staff said that Zayo was the laggard on the required moves. > > Frank > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net <mailto:AnthonyAnderberg@nuvera.net> > Sent: Monday, September 09, 2019 4:21 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. > > > > On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET%20on%20behalf%20of%20james.urwiller@TURNKEY-TELECOM.COM>> wrote: > > YES SIR! > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com <mailto:james.urwiller@stealthbroadband.com> > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank > Bulk > Sent: Monday, September 9, 2019 16:02 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? > > Bounce and then down. > > Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical > (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to > Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet > Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, > 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 > 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link > (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via > WIT for MICE > > Frank > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

Sept. 10, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by Frank Bulk

I’ve asked our OSP if he had any local knowledge, but he’s not aware. I will alert our Hinton field tech just in case he hears anything about it. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Urwiller Sent: Monday, September 9, 2019 8:33 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? So is it common for IA/MN construction to hit things all willy nilly without locates? Or is this another botched locate job? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> [stealth300] From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 20:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? More detail in Zayo’s last update at 8:08 pm We apologize for the delayed update. We've spoken with our OSP PM and have been advised that the splice crew's ETA to the site is still about 1 to 1.5 hours and the ETTR is 4.5 to 5 hours. The damage was caused by a construction crew and the damage area is south of Hinton, IA around County Rd 60 and Iowa State Rd 75. Additional information to follow once the splice crew is on site. So it does seem related to the Hwy 75 re-construction project. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Krejci Sent: Monday, September 9, 2019 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM<mailto:fbulk@MYPREMIERONLINE.COM>> Sent: Monday, September 9, 2019 4:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig…<https://urldefense.proofpoint.com/v2/url?u=https-3A__siouxcityjournal.com_n…> Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET%20on%20behalf%20of%20james.urwiller@TURNKEY-TELECOM.COM>> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Sept. 10, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by James Urwiller

So is it common for IA/MN construction to hit things all willy nilly without locates? Or is this another botched locate job? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Frank Bulk *Sent:* Monday, September 9, 2019 20:30 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? More detail in Zayo’s last update at 8:08 pm *We apologize for the delayed update. We've spoken with our OSP PM and have been advised that the splice crew's ETA to the site is still about 1 to 1.5 hours and the ETTR is 4.5 to 5 hours. The damage was caused by a construction crew and the damage area is south of Hinton, IA around County Rd 60 and Iowa State Rd 75. Additional information to follow once the splice crew is on site.* So it does seem related to the Hwy 75 re-construction project. Frank *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Justin Krejci *Sent:* Monday, September 9, 2019 6:13 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ------------------------------ *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> *Sent:* Monday, September 9, 2019 4:37 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig… <https://urldefense.proofpoint.com/v2/url?u=https-3A__siouxcityjournal.com_n…> Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 10, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by Frank Bulk

More detail in Zayo’s last update at 8:08 pm We apologize for the delayed update. We've spoken with our OSP PM and have been advised that the splice crew's ETA to the site is still about 1 to 1.5 hours and the ETTR is 4.5 to 5 hours. The damage was caused by a construction crew and the damage area is south of Hinton, IA around County Rd 60 and Iowa State Rd 75. Additional information to follow once the splice crew is on site. So it does seem related to the Hwy 75 re-construction project. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Monday, September 9, 2019 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM<mailto:fbulk@MYPREMIERONLINE.COM>> Sent: Monday, September 9, 2019 4:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig…<https://urldefense.proofpoint.com/v2/url?u=https-3A__siouxcityjournal.com_n…> Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of AnthonyAnderberg(a)nuvera.net<mailto:AnthonyAnderberg@nuvera.net> Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET%20on%20behalf%20of%20james.urwiller@TURNKEY-TELECOM.COM>> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Sept. 10, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by James Urwiller

Anyone that can provide transport from Fremont to Omaha NOT on Zayo? I’m ready to jump ship James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300] *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Justin Krejci *Sent:* Monday, September 9, 2019 18:13 *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ------------------------------ *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> *Sent:* Monday, September 9, 2019 4:37 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig… Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 9, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by Justin Krejci

I heard it is out somewhere between Sioux City and Alton, so it definitely could be that IDOT construction zone. We don't have growing pains we have Zayo pains. ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> Sent: Monday, September 9, 2019 4:37 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig… Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. ?On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

Sept. 9, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by Frank Bulk

Could be part of the Hwy 75 project: https://siouxcityjournal.com/news/local/idot-to-close-us-highway-during-nig… Our OSP staff said that Zayo was the laggard on the required moves. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of AnthonyAnderberg(a)nuvera.net Sent: Monday, September 09, 2019 4:21 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

Sept. 9, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by AnthonyAnderberg＠nuvera.net

We've got a link in NW Iowa down as well, we're on the same (former Neutral Path) link as you guys to Sioux City - so it's someplace between Sioux City, IA and Windom, MN. On 9/9/19, 4:02 PM, "MICE Discuss on behalf of James Urwiller" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET on behalf of james.urwiller(a)TURNKEY-TELECOM.COM> wrote: YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

Sept. 9, 2019

Re: Anyone else down to Belle Plain since 3:45 pm?

by James Urwiller

YES SIR! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 9, 2019 16:02 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Anyone else down to Belle Plain since 3:45 pm? Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

Sept. 9, 2019

Anyone else down to Belle Plain since 3:45 pm?

by Frank Bulk

Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

Sept. 9, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Frank Bulk

Good news -- Jeremy identified an issue in his transport that looks very similar to a Zayo issue he worked through in another link a few months ago, so he's opened up a ticket with Zayo. Thanks, Doug and Jeremy, for your help! Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Monday, September 09, 2019 8:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Thanks, I will try to turn one of the two route-reflector sessions up this morning. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Sunday, September 08, 2019 11:32 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote: >Doug, > >Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre >Sent: Saturday, September 7, 2019 4:15 PM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? > >On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote: >>For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. >> >>All our other peers with MICE are fine. >> >>I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. > > >We've been getting a lot of these.. > >Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution > > >And these > >2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 > >As well as I suppose the expected > >2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown > >If you have 18883 link shutdown. > >But with the first and second errors, I see you have two AS's, and two >IPs coming across Eth10 on the switch. The middle error message seems >to imply that the configuration we have setup for AS 53347 on >206.108.255.67 is receiving a peer attempt for 18883 which is >configured to talk on 206.108.255.133 on our side. > >Could these two be mixed up at all? > > > > >-- >Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 9, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Frank Bulk

Thanks, I will try to turn one of the two route-reflector sessions up this morning. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Sunday, September 08, 2019 11:32 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote: >Doug, > >Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre >Sent: Saturday, September 7, 2019 4:15 PM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? > >On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote: >>For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. >> >>All our other peers with MICE are fine. >> >>I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. > > >We've been getting a lot of these.. > >Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution > > >And these > >2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 > >As well as I suppose the expected > >2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown > >If you have 18883 link shutdown. > >But with the first and second errors, I see you have two AS's, and two >IPs coming across Eth10 on the switch. The middle error message seems >to imply that the configuration we have setup for AS 53347 on >206.108.255.67 is receiving a peer attempt for 18883 which is >configured to talk on 206.108.255.133 on our side. > >Could these two be mixed up at all? > > > > >-- >Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 9, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Doug McIntyre

Sure, I have commented out AS53347 for you now. Both IPv4 and IPv6 sessions. On Mon, Sep 09, 2019 at 02:15:18AM +0000, Frank Bulk wrote: >Doug, > >Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great. > >Frank > >-----Original Message----- >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre >Sent: Saturday, September 7, 2019 4:15 PM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? > >On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote: >>For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. >> >>All our other peers with MICE are fine. >> >>I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. > > >We've been getting a lot of these.. > >Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution >Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution > > >And these > >2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 > >As well as I suppose the expected > >2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown >2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown > >If you have 18883 link shutdown. > >But with the first and second errors, I see you have two AS's, and two >IPs coming across Eth10 on the switch. The middle error message seems >to imply that the configuration we have setup for AS 53347 on >206.108.255.67 is receiving a peer attempt for 18883 which is >configured to talk on 206.108.255.133 on our side. > >Could these two be mixed up at all? > > > > >-- >Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 9, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Frank Bulk

Doug, Thanks. We have been transitioning from AS53347 to AS18883 .... so if you can remove the AS55347 configuration on MICE's RR's, that would be great. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Saturday, September 7, 2019 4:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote: >For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. > >All our other peers with MICE are fine. > >I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. We've been getting a lot of these.. Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution And these 2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 As well as I suppose the expected 2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown If you have 18883 link shutdown. But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side. Could these two be mixed up at all? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 9, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Doug McIntyre

On Fri, Sep 06, 2019 at 11:51:41PM +0000, Frank Bulk wrote: >For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. > >All our other peers with MICE are fine. > >I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. We've been getting a lot of these.. Sep 3 18:02:14 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:18 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:23 micemn-01 bird6: NASN_18883: Received: Connection collision resolution Sep 3 18:02:28 micemn-01 bird6: NASN_18883: Received: Connection collision resolution And these 2019-09-07 16:08:15 <RMT> NASN_53347: Error: Bad peer AS: 18883 As well as I suppose the expected 2019-09-07 16:08:16 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:20 <RMT> NASN_18883: Received: Administrative shutdown 2019-09-07 16:08:24 <RMT> NASN_18883: Received: Administrative shutdown If you have 18883 link shutdown. But with the first and second errors, I see you have two AS's, and two IPs coming across Eth10 on the switch. The middle error message seems to imply that the configuration we have setup for AS 53347 on 206.108.255.67 is receiving a peer attempt for 18883 which is configured to talk on 206.108.255.133 on our side. Could these two be mixed up at all? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 7, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Frank Bulk

It did not -- still waiting to hear from MICE admins on what the RR saw on its side. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke Sent: Saturday, September 7, 2019 10:53 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Could I have a MICE admin look at the route reflectors? Did this get resolved? On Fri, Sep 6, 2019 at 6:52 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. > > All our other peers with MICE are fine. > > I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. > > Any hints would be welcomed. > > I've shutdown those six sessions for now. > > Frank > > Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > ... > > Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Sept. 7, 2019

Re: Could I have a MICE admin look at the route reflectors?

by Jay Hanke

Did this get resolved? On Fri, Sep 6, 2019 at 6:52 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. > > All our other peers with MICE are fine. > > I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. > > Any hints would be welcomed. > > I've shutdown those six sessions for now. > > Frank > > Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 > Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle > Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 > Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established > ... > > Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established > Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle > Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Sept. 7, 2019

Re: zayo again?

by Frank Bulk

Sorry for the delayed response to this thread. I did get the maintenance ... see attached. Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Mike Johnston Sent: Friday, August 30, 2019 10:03 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] zayo again? On 2019-08-30 05:00, James Urwiller wrote: > they also asked me how to gain access to the location and provide a > local contact. I tried explaining that it is their hut, not mine and I > don't have access...because its not my hut. Classic. That, unfortunately, sounds normal for Zayo.

Sept. 6, 2019

Could I have a MICE admin look at the route reflectors?

by Frank Bulk

For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. All our other peers with MICE are fine. I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. Any hints would be welcomed. I've shutdown those six sessions for now. Frank Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established ... Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established

Sept. 6, 2019

US Internet - AS 10242 - Router Replacement

by Justin Krejci

MICE, US Internet is shutting down BGP sessions for a few minutes to migrate the MICE connection onto a new router. BGP sessions will be re-established within a few minutes. This will be happening in the next few minutes after this email. Justin Krejci 952-253-3200 ? 1-800-US-INTERNET 12450 Wayzata Blvd. Suite 315 ? Minnetonka, MN 55305 [http://www.usinternet.com/img/usi-signature.png]

Sept. 6, 2019

Re: zayo again?

by Mike Johnston

On 2019-08-30 05:00, James Urwiller wrote: > they also asked me how to gain access to the location and provide a > local contact. I tried explaining that it is their hut, not mine and I > don't have access...because its not my hut. Classic. That, unfortunately, sounds normal for Zayo.

Aug. 30, 2019

Re: zayo again?

by James Urwiller

Yes that is what I am learning - we were not notified of the maintenance and no one at zayo can find the maintenance ticket either.. we are back up and running for now. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Lubitz, Jake (MNIT) Sent: Friday, August 30, 2019 08:46 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] zayo again? For what's it worth, it sounds like Zayo had a maintenance event for a fiber relocate project at TH 13 and County 21 in Prior Lake last night starting at 12:01AM. We had some network appliances in Prior Lake go down at 12:01AM and come back up at 1:50AM. I'm not sure if it's related this issue but the timestamps match what you're seeing. Many thanks. Jake -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Urwiller Sent: Friday, August 30, 2019 5:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] zayo again? This message may be from an external email source. Do not select links or open attachments unless verified. Report all suspicious emails to Minnesota IT Services Security Operations Center. I have been told they cannot reach a cyan shelf in belle plaine. they also asked me how to gain access to the location and provide a local contact. I tried explaining that it is their hut, not mine and I don't have access...because its not my hut. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 02:16 To: 'MICE-DISCUSS(a)LISTS.IPHOUSE.NET' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: zayo again? My initial down was at 00:14 CST, and came back 01:04 CST. Then back down again 01:20 CST, and then back around 01:43CST. Seems to be stable at the moment. Anyone else experience this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 01:37 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: zayo again? been seeing BGP seesion bounces sine around midnight. anyone else zayo2belleplaine??? Sent from my iPhone 7

Aug. 30, 2019

Re: zayo again?

by Lubitz, Jake (MNIT)

For what's it worth, it sounds like Zayo had a maintenance event for a fiber relocate project at TH 13 and County 21 in Prior Lake last night starting at 12:01AM. We had some network appliances in Prior Lake go down at 12:01AM and come back up at 1:50AM. I'm not sure if it's related this issue but the timestamps match what you're seeing. Many thanks. Jake -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Urwiller Sent: Friday, August 30, 2019 5:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] zayo again? This message may be from an external email source. Do not select links or open attachments unless verified. Report all suspicious emails to Minnesota IT Services Security Operations Center. I have been told they cannot reach a cyan shelf in belle plaine. they also asked me how to gain access to the location and provide a local contact. I tried explaining that it is their hut, not mine and I don't have access...because its not my hut. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 02:16 To: 'MICE-DISCUSS(a)LISTS.IPHOUSE.NET' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: zayo again? My initial down was at 00:14 CST, and came back 01:04 CST. Then back down again 01:20 CST, and then back around 01:43CST. Seems to be stable at the moment. Anyone else experience this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 01:37 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: zayo again? been seeing BGP seesion bounces sine around midnight. anyone else zayo2belleplaine??? Sent from my iPhone 7

Aug. 30, 2019

Re: zayo again?

by James Urwiller

I have been told they cannot reach a cyan shelf in belle plaine. they also asked me how to gain access to the location and provide a local contact. I tried explaining that it is their hut, not mine and I don't have access...because its not my hut. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 02:16 To: 'MICE-DISCUSS(a)LISTS.IPHOUSE.NET' <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: RE: zayo again? My initial down was at 00:14 CST, and came back 01:04 CST. Then back down again 01:20 CST, and then back around 01:43CST. Seems to be stable at the moment. Anyone else experience this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 01:37 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: zayo again? been seeing BGP seesion bounces sine around midnight. anyone else zayo2belleplaine??? Sent from my iPhone 7

Aug. 30, 2019

Re: zayo again?

by James Urwiller

My initial down was at 00:14 CST, and came back 01:04 CST. Then back down again 01:20 CST, and then back around 01:43CST. Seems to be stable at the moment. Anyone else experience this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: James Urwiller <james.urwiller(a)turnkey-telecom.com> Sent: Friday, August 30, 2019 01:37 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: zayo again? been seeing BGP seesion bounces sine around midnight. anyone else zayo2belleplaine??? Sent from my iPhone 7

Aug. 30, 2019

zayo again?

by James Urwiller

been seeing BGP seesion bounces sine around midnight. anyone else zayo2belleplaine??? Sent from my iPhone 7

Aug. 30, 2019

Re: Of minor note

by Bahls, Dean

Yeah, I forgot to notify the group when we did a software update this morning. Sorry about that. Dean On Mon, Aug 26, 2019 at 8:33 PM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > FYI, saw it again this morning: > > Aug 26 05:42:02 207.32.15.28 Rib: mpbgp_recv_mpreach: peer > 2001:504:27::d1af:0:1 (AS 53679) link-local next hop > fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > Aug 26 05:42:02 207.32.15.28 Rib: mpbgp_recv_mpreach: peer > 2001:504:27::d1af:0:2 (AS 53679) link-local next hop > fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > > Frank > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank > Bulk > Sent: Wednesday, August 21, 2019 8:00 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] Of minor note > > Our route logged these two items Monday morning with MICE's AS: > > Aug 19 10:35:41 207.32.15.28 Rib: mpbgp_recv_mpreach: peer > 2001:504:27::d1af:0:2 (AS 53679) link-local next hop > fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > Aug 19 11:08:50 207.32.15.28 Rib: mpbgp_recv_mpreach: peer > 2001:504:27::d1af:0:2 (AS 53679) link-local next hop > fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > > Not sure what that was about. > > Frank > -- *Dean Bahls* Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | P.O. Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com

Aug. 27, 2019

Re: Of minor note

by Frank Bulk

FYI, saw it again this morning: Aug 26 05:42:02 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:1 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Aug 26 05:42:02 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Wednesday, August 21, 2019 8:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Of minor note Our route logged these two items Monday morning with MICE's AS: Aug 19 10:35:41 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Aug 19 11:08:50 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Not sure what that was about. Frank

Aug. 27, 2019

Re: ADV: [MICE-DISCUSS] Can a PTR be created for Google IP?

by Doug McIntyre

On Fri, Aug 23, 2019 at 04:45:35AM +0000, Frank Bulk wrote: >Can a PTR be created for this IP: 206.108.255.141 (AS 15169: Google LLC, US) Sure, I've setup the newest batch of IPv4/IPv6 reverses for the MICE block. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 23, 2019

Can a PTR be created for Google IP?

by Frank Bulk

Can a PTR be created for this IP: 206.108.255.141 (AS 15169: Google LLC, US) Frank

Aug. 23, 2019

Re: Saturated ports

by Jay Hanke

> If someone's port is full on a regular basis, having someone from the Board contact the member and ask if they are aware of the issue seems approprate and reasonable. Basically asking, "have you noticed your port is full quite a bit lately?" But its the member's responsibility to manage their network as they see fit, including their ports to the exchange. I'd tweak this a little to encourage the member reach out directly to the other peer whenever there might be a problem. If you need help getting attention of a non-responsive peer we can help with some "peer pressure". :) -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 22, 2019

Re: Saturated ports

by David Farmer

On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > > To be clear, I was not advocating for punitive action, and we can go and > pull the bandwidth reports periodically. I was just posing the question > regarding whether a more proactive stance regarding visibility would be > appropriate. > > What would you propose? > First, let's not make a federal case of this, links fill up it happens, get over it. In this case it occurred right after big changes by a big content provider. I'm actually surprised more participants' links didn't fill up. Second, you should send a polite note to any another member, if you notice anything that isn't quite right, their link is full, they are leaking routes, or something just looks weird. We are a community, talk to your neighbors, be their early warning system. Hopefully, they already know about what ever it is, but we are all busy and things slip through the cracks all the time. So if you notice something that doesn't seem right, let your neighbors know about it. But be polite about it, next time it might be you who has something weird happening. Third, the IX is an optimization, if you are having issues sending traffic to another member you have other routes, use them. We have had to do this, not because the other member ignored us or wouldn't upgrade their link, but sometimes upgrades can take a little time. A few weeks or even a couple months to complete an upgrade isn't that unrealistic and not uncommon depending on the circumstances. Not everyone has extra 10G or 100G ports just sitting around, and if you have to backhaul the traffic, arranging that upgrade can easily take several months. The policy we have for remote switches makes sense, but members need to manage their own capacity. Even if we were to forced them to add port capacity to the exchange the bottleneck could just move to the next hop. None of us has sufficient visibility into the other guys network or business to be making those kinds of judgements for them. If someone's port is full on a regular basis, having someone from the Board contact the member and ask if they are aware of the issue seems approprate and reasonable. Basically asking, "have you noticed your port is full quite a bit lately?" But its the member's responsibility to manage their network as they see fit, including their ports to the exchange. Thanks -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 22, 2019

Re: Saturated ports

by Corey Hauer

Indeed I did! 😮 On Wed, Aug 21, 2019 at 2:00 PM Matthew Beckwell <mbeckwell(a)evolveip.net> wrote: > I feel like we’re mixing up “SDN” and “DCN” here? > > > > > > *From: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Corey > Hauer <coreyhauer(a)GMAIL.COM> > *Reply-To: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Date: *Wednesday, August 21, 2019 at 3:10 PM > *To: *"MICE-DISCUSS(a)LISTS.IPHOUSE.NET" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject: *Re: [MICE-DISCUSS] Saturated ports > > > > Honestly, as an eyeball network, I would fix saturating that pipe that by > filtering prefixes or adding capacity. My customers would vote with their > feet if we started having packet loss or adverse latency because we filled > our IX facing pipes. I can't imagine SDN and their members choosing to > endure thousands of customer complaints. > > > > SDN is a well run, well capitalized network - they'll fix it one way or > another. I am sure once this lands on the right person's desk - it gets > fixed. > > > > > > On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com> > wrote: > > Is it possible to send a weekly/monthly reporting showing a high water > mark for utilization for each member? Is that appropriate? Or do we just > ask each member to monitor the utilization graphs if this is of interest to > them? > > > > For my part it is easier to ingest and handle a periodic review if we are > pushed the information versus having to have someone remember to go and > pull the information. I'm interested in what others would find useful. > > > > *Ben Wiechman* > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 > <https://www.google.com/maps/search/224+East+Main+Street+%7C+Melrose,+MN+563…> > | arvig.com > > > > > > > > On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > > > To be clear, I was not advocating for punitive action, and we can go and > pull the bandwidth reports periodically. I was just posing the question > regarding whether a more proactive stance regarding visibility would be > appropriate. > > What would you propose? > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Saturated ports

by Darin Steffl

My original thread is about DCN, not SDN. On Wed, Aug 21, 2019, 4:00 PM Matthew Beckwell <mbeckwell(a)evolveip.net> wrote: > I feel like we’re mixing up “SDN” and “DCN” here? > > > > > > *From: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Corey > Hauer <coreyhauer(a)GMAIL.COM> > *Reply-To: *MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Date: *Wednesday, August 21, 2019 at 3:10 PM > *To: *"MICE-DISCUSS(a)LISTS.IPHOUSE.NET" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > *Subject: *Re: [MICE-DISCUSS] Saturated ports > > > > Honestly, as an eyeball network, I would fix saturating that pipe that by > filtering prefixes or adding capacity. My customers would vote with their > feet if we started having packet loss or adverse latency because we filled > our IX facing pipes. I can't imagine SDN and their members choosing to > endure thousands of customer complaints. > > > > SDN is a well run, well capitalized network - they'll fix it one way or > another. I am sure once this lands on the right person's desk - it gets > fixed. > > > > > > On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com> > wrote: > > Is it possible to send a weekly/monthly reporting showing a high water > mark for utilization for each member? Is that appropriate? Or do we just > ask each member to monitor the utilization graphs if this is of interest to > them? > > > > For my part it is easier to ingest and handle a periodic review if we are > pushed the information versus having to have someone remember to go and > pull the information. I'm interested in what others would find useful. > > > > *Ben Wiechman* > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > > > > On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > > > To be clear, I was not advocating for punitive action, and we can go and > pull the bandwidth reports periodically. I was just posing the question > regarding whether a more proactive stance regarding visibility would be > appropriate. > > What would you propose? > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Saturated ports

by AnthonyAnderberg＠nuvera.net

I think there is some confusion – the original thread was talking about DCN, not SDN. SDN is a long-time MICE member and hasn’t ever had any issues that I recall. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Gary Glissendorf <Gary.Glissendorf(a)SDNCOMMUNICATIONS.COM> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Wednesday, August 21, 2019 at 3:45 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Saturated ports I noticed this thread today but wasn’t sure what topic started the thread. SDN like Corey said our customers report latency, packet loss quickly if we have saturation. We monitor and alert on network saturation at 60% and if needed we adjust prefix advertisements until we can find the proper balance and then increase bandwidth when needed. We have not developed a cool automatic was to do this yet but we rarely have issues with Peaks or DDoS Attacks. It’s the occasional fiber cuts that can cause us issues and the need to adjust bandwidth. What was topic that started the Thread about BW and reports. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D55838.820116E0] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D55838.820116E0] <http://twitter.com/sdncomm> [cid:image003.jpg@01D55838.820116E0] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D55838.820116E0] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D55838.820116E0] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Corey Hauer Sent: Wednesday, August 21, 2019 3:10 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Saturated ports Honestly, as an eyeball network, I would fix saturating that pipe that by filtering prefixes or adding capacity. My customers would vote with their feet if we started having packet loss or adverse latency because we filled our IX facing pipes. I can't imagine SDN and their members choosing to endure thousands of customer complaints. SDN is a well run, well capitalized network - they'll fix it one way or another. I am sure once this lands on the right person's desk - it gets fixed. On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com>> wrote: Is it possible to send a weekly/monthly reporting showing a high water mark for utilization for each member? Is that appropriate? Or do we just ask each member to monitor the utilization graphs if this is of interest to them? For my part it is easier to ingest and handle a periodic review if we are pushed the information versus having to have someone remember to go and pull the information. I'm interested in what others would find useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<http://arvig.com> On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: > To be clear, I was not advocating for punitive action, and we can go and pull the bandwidth reports periodically. I was just posing the question regarding whether a more proactive stance regarding visibility would be appropriate. What would you propose? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2019

Re: Saturated ports

by Matthew Beckwell

I feel like we’re mixing up “SDN” and “DCN” here? From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Corey Hauer <coreyhauer(a)GMAIL.COM> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Wednesday, August 21, 2019 at 3:10 PM To: "MICE-DISCUSS(a)LISTS.IPHOUSE.NET" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Saturated ports Honestly, as an eyeball network, I would fix saturating that pipe that by filtering prefixes or adding capacity. My customers would vote with their feet if we started having packet loss or adverse latency because we filled our IX facing pipes. I can't imagine SDN and their members choosing to endure thousands of customer complaints. SDN is a well run, well capitalized network - they'll fix it one way or another. I am sure once this lands on the right person's desk - it gets fixed. On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com>> wrote: Is it possible to send a weekly/monthly reporting showing a high water mark for utilization for each member? Is that appropriate? Or do we just ask each member to monitor the utilization graphs if this is of interest to them? For my part it is easier to ingest and handle a periodic review if we are pushed the information versus having to have someone remember to go and pull the information. I'm interested in what others would find useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<http://arvig.com> On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: > To be clear, I was not advocating for punitive action, and we can go and pull the bandwidth reports periodically. I was just posing the question regarding whether a more proactive stance regarding visibility would be appropriate. What would you propose? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2019

Re: Saturated ports

by Gary Glissendorf

I noticed this thread today but wasn’t sure what topic started the thread. SDN like Corey said our customers report latency, packet loss quickly if we have saturation. We monitor and alert on network saturation at 60% and if needed we adjust prefix advertisements until we can find the proper balance and then increase bandwidth when needed. We have not developed a cool automatic was to do this yet but we rarely have issues with Peaks or DDoS Attacks. It’s the occasional fiber cuts that can cause us issues and the need to adjust bandwidth. What was topic that started the Thread about BW and reports. Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com 2900 W. 10th St. | Sioux Falls, SD 57104 (w) 605.978.3558 | (c) 605.359-3737 | (tf) 800.247.1442 SDN NOC 877.287.8023 NOC Support email: sdnsupport(a)sdncommunications.com “Ausgezeichnet Zueinander Sein” [cid:image001.jpg@01D55837.7362FB90] <http://www.facebook.com/sdncommunications> [cid:image002.jpg@01D55837.7362FB90] <http://twitter.com/sdncomm> [cid:image003.jpg@01D55837.7362FB90] <http://plus.google.com/107192460129695945716/posts> [cid:image004.jpg@01D55837.7362FB90] <http://www.linkedin.com/company/sdn-communications> [cid:image005.jpg@01D55837.7362FB90] <http://www.youtube.com/user/sdncomm> From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Corey Hauer Sent: Wednesday, August 21, 2019 3:10 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Saturated ports Honestly, as an eyeball network, I would fix saturating that pipe that by filtering prefixes or adding capacity. My customers would vote with their feet if we started having packet loss or adverse latency because we filled our IX facing pipes. I can't imagine SDN and their members choosing to endure thousands of customer complaints. SDN is a well run, well capitalized network - they'll fix it one way or another. I am sure once this lands on the right person's desk - it gets fixed. On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com>> wrote: Is it possible to send a weekly/monthly reporting showing a high water mark for utilization for each member? Is that appropriate? Or do we just ask each member to monitor the utilization graphs if this is of interest to them? For my part it is easier to ingest and handle a periodic review if we are pushed the information versus having to have someone remember to go and pull the information. I'm interested in what others would find useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<http://arvig.com> On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: > To be clear, I was not advocating for punitive action, and we can go and pull the bandwidth reports periodically. I was just posing the question regarding whether a more proactive stance regarding visibility would be appropriate. What would you propose? ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

Aug. 21, 2019

Re: Saturated ports

by Corey Hauer

Honestly, as an eyeball network, I would fix saturating that pipe that by filtering prefixes or adding capacity. My customers would vote with their feet if we started having packet loss or adverse latency because we filled our IX facing pipes. I can't imagine SDN and their members choosing to endure thousands of customer complaints. SDN is a well run, well capitalized network - they'll fix it one way or another. I am sure once this lands on the right person's desk - it gets fixed. On Wed, Aug 21, 2019 at 12:17 PM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > Is it possible to send a weekly/monthly reporting showing a high water > mark for utilization for each member? Is that appropriate? Or do we just > ask each member to monitor the utilization graphs if this is of interest to > them? > > For my part it is easier to ingest and handle a periodic review if we are > pushed the information versus having to have someone remember to go and > pull the information. I'm interested in what others would find useful. > > Ben Wiechman > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > >> > To be clear, I was not advocating for punitive action, and we can go >> and pull the bandwidth reports periodically. I was just posing the question >> regarding whether a more proactive stance regarding visibility would be >> appropriate. >> >> What would you propose? >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Saturated ports

by Andy Koch

On 8/21/19 2:17 PM, Ben Wiechman wrote: > Is it possible to send a weekly/monthly reporting showing a high water > mark for utilization for each member? Is that appropriate? Or do we > just ask each member to monitor the utilization graphs if this is of > interest to them? > > For my part it is easier to ingest and handle a periodic review if we > are pushed the information versus having to have someone remember to > go and pull the information. I'm interested in what others would find > useful. I don't mind if this is an opt-in item on a per-participant basis, but I am not in-favor of having yet more to my inbox. I feel that any good participant should have their own monitoring and be properly alerted based on that - possibly directly into a capacity planning workflow. If you want to have a periodic email, I am sure a curl request to the MICE monitoring server on a cron would be fairly simple to setup if you have cleared it with US Internet who graciously hosts that for us. If the recommendation is for a mass-email to the mailing list with all participant data, that I am even less in support of. We have several participants that do not publicly share their capacity data and we should respect that - not blast it out weekly/monthly, even if the recipient list is opt-in. Regards, Andy Andy Koch Hoyos Consulting LLC ofc: +1 608 616 9950 andyk(a)hoyosconsulting.com http://www.hoyosconsulting.com

Aug. 21, 2019

Re: Saturated ports

by Ben Wiechman

Is it possible to send a weekly/monthly reporting showing a high water mark for utilization for each member? Is that appropriate? Or do we just ask each member to monitor the utilization graphs if this is of interest to them? For my part it is easier to ingest and handle a periodic review if we are pushed the information versus having to have someone remember to go and pull the information. I'm interested in what others would find useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Wed, Aug 21, 2019 at 1:59 PM Jay Hanke <jayhanke(a)southfront.io> wrote: > > To be clear, I was not advocating for punitive action, and we can go and > pull the bandwidth reports periodically. I was just posing the question > regarding whether a more proactive stance regarding visibility would be > appropriate. > > What would you propose? >

Aug. 21, 2019

Re: Saturated ports

by Jay Hanke

> To be clear, I was not advocating for punitive action, and we can go and pull the bandwidth reports periodically. I was just posing the question regarding whether a more proactive stance regarding visibility would be appropriate. What would you propose?

Aug. 21, 2019

Re: Saturated ports

by Ben Wiechman

To be clear, I was not advocating for punitive action, and we can go and pull the bandwidth reports periodically. I was just posing the question regarding whether a more proactive stance regarding visibility would be appropriate. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Wed, Aug 21, 2019 at 11:55 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > > I don't think blocking a port is a good idea but I think proactive > notifications is a good first step then more harshly worded emails if that > doesn't work. > > We're already doing this. Just not automated. > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 >

Aug. 21, 2019

Re: Of minor note

by Steve Howard

Aug. 21, 2019

Re: Saturated ports

by Jay Hanke

> I don't think blocking a port is a good idea but I think proactive notifications is a good first step then more harshly worded emails if that doesn't work. We're already doing this. Just not automated. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 21, 2019

Re: Saturated ports

by Jay Hanke

The graphing is public, there's currently no alerting. From the IX, perspective the only leverage is removal from the fabric or total depeering. Not a very good options. Members can always filter or preference their routes using the MICE communities or policies on their own gear. When we start tail dropping (or getting full) I reach out to the peer directly and make them aware of the situation and offer up MICE upgrade options. Sometimes multiple times. MICE is not advocating members congest their ports, it defeats the purpose of the IX. MICE is not disconnecting member connected congested ports. On Wed, Aug 21, 2019 at 11:28 AM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > > I would dispute the "only their network" statement. We've troubleshot various issues in the past with latency and loss, especially for internet based services used by businesses when that traffic traverses MICE. While the peering policy of a service provider is their decision, it is potentially appropriate to provide proactive notice in some fashion that would alert other operators that they may want to adjust their peering strategy to minimize customer complaints. > > Is some kind of alarming or reporting viable and useful? From my perspective it would be useful. > > Ben Wiechman > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> The board talked about this back in the day. The thought process was that remotes affect multiple members so the congestion policy should be enforced. >> >> For a participant it's only their network (and their customers). Also, a disconnect might make the overall internet worse as their transit may fill. >> >> On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net <AnthonyAnderberg(a)nuvera.net> wrote: >>> >>> I know in the past several of us have reached out privately to folks at DCN about their port saturation, which seems to come and go. >>> >>> We do not have a policy about member-port saturation, and my recollection is similar to Richard's - we didn't want to be bossy about people's peering. >>> >>> Cheers, >>> anthony >>> >>> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >>> >>> On 8/20/19 8:48 PM, Darin Steffl wrote: >>> > Is there any policy in place for peers that let their ports saturate at >>> > 100% for an extended period of time? DCN looks like they could use an >>> > upgrade to their 10G port and not sure if anyone proactively reaches out >>> > to members when saturation occurs. >>> >>> I've forwarded your message to DCN. >>> >>> For remote switch ports, we have a policy of requiring upgrades before >>> saturation. >>> >>> For regular participants, I'm not sure that we have a policy. >>> >>> I'm also not sure if we want a policy there, as that might be considered >>> dictating peering policy. I'm not personally opposed, but this is >>> something that would need some thought. >>> >>> -- >>> Richard >>> >>> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 21, 2019

Re: Saturated ports

by Darin Steffl

I agree this can affect multiple parties. Say we let our port get saturated. Now all the content providers start receiving trouble calls. Netflix, akamai, YouTube, traffic between us and any MICE peer is congested and our customers will probably blame us but they may also blame the other peer who they receive traffic from when it's not their fault. I don't think blocking a port is a good idea but I think proactive notifications is a good first step then more harshly worded emails if that doesn't work. We don't want to let anyone think running a port at 100% is a good idea. It puts a burden on more than just the member with a congested port in my opinion like Ben said. On Wed, Aug 21, 2019, 11:28 AM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > I would dispute the "only their network" statement. We've troubleshot > various issues in the past with latency and loss, especially for internet > based services used by businesses when that traffic traverses MICE. While > the peering policy of a service provider is their decision, it is > potentially appropriate to provide proactive notice in some fashion that > would alert other operators that they may want to adjust their peering > strategy to minimize customer complaints. > > Is some kind of alarming or reporting viable and useful? From my > perspective it would be useful. > > Ben Wiechman > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > >> The board talked about this back in the day. The thought process was that >> remotes affect multiple members so the congestion policy should be >> enforced. >> >> For a participant it's only their network (and their customers). Also, a >> disconnect might make the overall internet worse as their transit may fill. >> >> On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net < >> AnthonyAnderberg(a)nuvera.net> wrote: >> >>> I know in the past several of us have reached out privately to folks at >>> DCN about their port saturation, which seems to come and go. >>> >>> We do not have a policy about member-port saturation, and my >>> recollection is similar to Richard's - we didn't want to be bossy about >>> people's peering. >>> >>> Cheers, >>> anthony >>> >>> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >>> >>> On 8/20/19 8:48 PM, Darin Steffl wrote: >>> > Is there any policy in place for peers that let their ports >>> saturate at >>> > 100% for an extended period of time? DCN looks like they could use >>> an >>> > upgrade to their 10G port and not sure if anyone proactively >>> reaches out >>> > to members when saturation occurs. >>> >>> I've forwarded your message to DCN. >>> >>> For remote switch ports, we have a policy of requiring upgrades >>> before >>> saturation. >>> >>> For regular participants, I'm not sure that we have a policy. >>> >>> I'm also not sure if we want a policy there, as that might be >>> considered >>> dictating peering policy. I'm not personally opposed, but this is >>> something that would need some thought. >>> >>> -- >>> Richard >>> >>> >>> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Saturated ports

by Ben Wiechman

I would dispute the "only their network" statement. We've troubleshot various issues in the past with latency and loss, especially for internet based services used by businesses when that traffic traverses MICE. While the peering policy of a service provider is their decision, it is potentially appropriate to provide proactive notice in some fashion that would alert other operators that they may want to adjust their peering strategy to minimize customer complaints. Is some kind of alarming or reporting viable and useful? From my perspective it would be useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > The board talked about this back in the day. The thought process was that > remotes affect multiple members so the congestion policy should be > enforced. > > For a participant it's only their network (and their customers). Also, a > disconnect might make the overall internet worse as their transit may fill. > > On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net < > AnthonyAnderberg(a)nuvera.net> wrote: > >> I know in the past several of us have reached out privately to folks at >> DCN about their port saturation, which seems to come and go. >> >> We do not have a policy about member-port saturation, and my recollection >> is similar to Richard's - we didn't want to be bossy about people's peering. >> >> Cheers, >> anthony >> >> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >> >> On 8/20/19 8:48 PM, Darin Steffl wrote: >> > Is there any policy in place for peers that let their ports >> saturate at >> > 100% for an extended period of time? DCN looks like they could use >> an >> > upgrade to their 10G port and not sure if anyone proactively >> reaches out >> > to members when saturation occurs. >> >> I've forwarded your message to DCN. >> >> For remote switch ports, we have a policy of requiring upgrades before >> saturation. >> >> For regular participants, I'm not sure that we have a policy. >> >> I'm also not sure if we want a policy there, as that might be >> considered >> dictating peering policy. I'm not personally opposed, but this is >> something that would need some thought. >> >> -- >> Richard >> >> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Of minor note

by Doug McIntyre

On Wed, Aug 21, 2019 at 01:00:21PM +0000, Frank Bulk wrote: >Our route logged these two items Monday morning with MICE's AS: > >Aug 19 10:35:41 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop >Aug 19 11:08:50 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > >Not sure what that was about. Odd, the 2nd RR's link local address is fe80::f21f:afff:fedb:a5d5 I don't find the one listed above known by either of the RR's ND process. Something freaked out for a minute? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 21, 2019

Re: 10G to MICE down

by Corey Hauer

Zayo personnel are on site - they were alerted at 4am. So the good news is that appears they do monitor that site for temp at least. They have a large industrial fan in the door trying to push out heat while they await HVAC techs. On Wed, Aug 21, 2019 at 6:41 AM Darin Steffl <darin.steffl(a)mnwifi.com> wrote: > Corey, > > Did you open a ticket with them? Our gear is still running but with high > fan speeds (knock on wood) but hopefully they can figure it out soon. > > On Wed, Aug 21, 2019 at 8:39 AM Corey Hauer <coreyhauer(a)gmail.com> wrote: > >> The universe clearly hates Zayo. Their air conditioning in Rochester has >> failed this morning - feels like 110 degrees in our colo. Our core CCR1072 >> overheated as a result. >> >> On Wed, Aug 21, 2019 at 5:15 AM Frank Bulk <fbulk(a)mypremieronline.com> >> wrote: >> >>> Likely canceled due to Belle Plain hut overheating. They were going to >>> do a software upgrade in Bigelow, MN. >>> >>> >>> >>> Frank >>> >>> >>> >>> *From:* MR Zayo <mr(a)zayo.com> >>> *Sent:* Wednesday, August 21, 2019 12:21 AM >>> *Subject:* Maintenance Status***ZAYO TTN-0003483970 MAINTENANCE >>> NOTIFICATION*** >>> >>> >>> >>> Dear Zayo Customer, >>> >>> >>> >>> Due to unforeseen circumstances this maintenance will not take place >>> tonight. We will send you a reschedule notification soon as new dates are >>> confirmed. >>> >>> *Zay**o **Global Change Management Team/Équipe de Gestion du Changement >>> Global Zay* >>> *o **Zay**o **Group/Le Groupe Zay**o* >>> Toll free/*N° sans frais*: 1.866.236.2824 <1-866-236-2824> >>> United Kingdom Toll Free/*N° sans frais Royaume-Uni*: 0800.169.1646 >>> <0800-169-1646> >>> Email/*Courriel*: mr(a)zayo.com >>> Website/*Site Web*: https://www.zayo.com >>> >>> >>> >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank >>> Bulk >>> *Sent:* Tuesday, August 20, 2019 11:54 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> My BGP sessions restored at 11:23 pm. >>> >>> >>> >>> Frank >>> >>> >>> >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Darin >>> Steffl >>> *Sent:* Tuesday, August 20, 2019 9:32 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> Same thing happened last summer in Rochester at Neutral Path suite after >>> Zayo took over. I was visiting our rack and it was over 90 degrees in there >>> and Zayo didn't know about it yet since they didn't monitor temps. Once I >>> notified them, it took 8 hours for them to fix the AC. They have since >>> added temperature probes in a few spots but who knows if they actually >>> monitor or get alerted since they're Zayo after all. >>> >>> >>> >>> On Tue, Aug 20, 2019 at 9:27 PM Bahls, Dean < >>> dean.bahls(a)cooperative-networks.com> wrote: >>> >>> They don’t have redundancy at that hut? Or is it just a cabinet? >>> >>> >>> >>> Dean >>> >>> >>> >>> On Tue, Aug 20, 2019 at 9:25 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >>> >>> Since I was not getting straight answers from Zayo, I drove out to the >>> Belle Plaine. The air conditioning has failed, and the temperature was >>> over 110 degrees inside. There is a technician out here who is waiting on >>> a HVAC repair company to come and fix the issue, and doing the best he can >>> to ventilate until then >>> >>> >>> >>> >>> >>> Jeremy Lumby >>> >>> Minnesota VoIP >>> >>> 9217 17th Ave S >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Suite 216 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Bloomington MN 55425 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Main 612-355-7740 >>> >>> Direct 612-392-6814 >>> >>> EFax 952-873-7425 >>> >>> jlumby(a)mnvoip.com >>> >>> >>> >>> >>> >>> -------- Original message -------- >>> >>> From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>> >>> Date: 8/20/19 8:37 PM (GMT-06:00) >>> >>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> >>> Subject: Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> ------------------------------ >>> >>> *From* : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] >>> >>> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >>> >>> *Date* : Tuesday, August 20 2019 20:37:47 >>> >>> my circuit just went down 7 minutes ago, was fine prior to 20:30 cst >>> >>> Sent from my iPhone 7 >>> >>> >>> On Aug 20, 2019, at 20:01, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >>> >>> It is looking like it is something specific to Belle Plaine. I have a >>> diverse back door in to there that is down as well. >>> >>> >>> >>> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET >>> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Corey Hauer >>> *Sent:* Tuesday, August 20, 2019 7:58 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> Oddly, our 511 to Mankato 10 gig transit is on that route - and it up >>> and running. >>> >>> >>> >>> Possibly Zayo restored the protect path we had been getting from Neutral >>> Path after I complained they took away our protect when they re-groomed us >>> onto Zayo's native network? >>> >>> >>> >>> Corey Hauer >>> LTD Broadband >>> >>> >>> >>> On Tue, Aug 20, 2019 at 5:21 PM Frank Bulk <fbulk(a)mypremieronline.com> >>> wrote: >>> >>> Thanks, Jeremy. >>> >>> Frank >>> >>> >>> >>> On Tue, Aug 20, 2019 at 6:21 PM -0500, "Jeremy Lumby" <jlumby(a)MNVOIP.COM> >>> wrote: >>> >>> I lost all of my connectivity between 511 and and Belle Plaine, I have >>> opened a ticket with zayo >>> >>> >>> >>> >>> >>> >>> >>> Jeremy Lumby >>> >>> Minnesota VoIP >>> >>> 9217 17th Ave S >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Suite 216 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Bloomington MN 55425 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Main 612-355-7740 >>> >>> Direct 612-392-6814 >>> >>> EFax 952-873-7425 >>> >>> jlumby(a)mnvoip.com >>> >>> >>> >>> >>> >>> -------- Original message -------- >>> >>> From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> >>> >>> Date: 8/20/19 6:13 PM (GMT-06:00) >>> >>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> >>> Subject: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> ------------------------------ >>> >>> *From* : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] >>> >>> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >>> >>> *Date* : Tuesday, August 20 2019 18:13:03 >>> >>> We saw our 10 gig to MICE go down around 6:00 p.m. my guess is somewhere >>> between northwest Iowa and Belle Plaine? >>> >>> Frank >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> -- >>> >>> *Dean Bahls* >>> >>> Network Services Manager | Cooperative Network Services, LLC >>> >>> 14 Main St SW | P.O. Box J | Menahga, MN 56464 >>> >>> 218.346.3635 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.346.3635&d=DwMFaQ&c…> >>> (Direct) | 218.564.3000 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.564.3000&d=DwMFaQ&c…> >>> (Main Office) >>> >>> dean.bahls(a)cooperative-networks.com >>> >>> >>> >>> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> >>> >>> -- >>> >>> Darin Steffl >>> >>> Minnesota WiFi >>> >>> www.mnwifi.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mnwifi.com_&d=DwMFa…> >>> >>> 507-634-WiFi >>> >>> >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >>> Like us on Facebook >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > <http://www.facebook.com/minnesotawifi> Like us on Facebook > <http://www.facebook.com/minnesotawifi> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019