- mice-discuss - lists.micemn.net

Re: Saturated ports

by Jay Hanke

> I don't think blocking a port is a good idea but I think proactive notifications is a good first step then more harshly worded emails if that doesn't work. We're already doing this. Just not automated. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 21, 2019

Re: Saturated ports

by Jay Hanke

The graphing is public, there's currently no alerting. From the IX, perspective the only leverage is removal from the fabric or total depeering. Not a very good options. Members can always filter or preference their routes using the MICE communities or policies on their own gear. When we start tail dropping (or getting full) I reach out to the peer directly and make them aware of the situation and offer up MICE upgrade options. Sometimes multiple times. MICE is not advocating members congest their ports, it defeats the purpose of the IX. MICE is not disconnecting member connected congested ports. On Wed, Aug 21, 2019 at 11:28 AM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > > I would dispute the "only their network" statement. We've troubleshot various issues in the past with latency and loss, especially for internet based services used by businesses when that traffic traverses MICE. While the peering policy of a service provider is their decision, it is potentially appropriate to provide proactive notice in some fashion that would alert other operators that they may want to adjust their peering strategy to minimize customer complaints. > > Is some kind of alarming or reporting viable and useful? From my perspective it would be useful. > > Ben Wiechman > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: >> >> The board talked about this back in the day. The thought process was that remotes affect multiple members so the congestion policy should be enforced. >> >> For a participant it's only their network (and their customers). Also, a disconnect might make the overall internet worse as their transit may fill. >> >> On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net <AnthonyAnderberg(a)nuvera.net> wrote: >>> >>> I know in the past several of us have reached out privately to folks at DCN about their port saturation, which seems to come and go. >>> >>> We do not have a policy about member-port saturation, and my recollection is similar to Richard's - we didn't want to be bossy about people's peering. >>> >>> Cheers, >>> anthony >>> >>> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >>> >>> On 8/20/19 8:48 PM, Darin Steffl wrote: >>> > Is there any policy in place for peers that let their ports saturate at >>> > 100% for an extended period of time? DCN looks like they could use an >>> > upgrade to their 10G port and not sure if anyone proactively reaches out >>> > to members when saturation occurs. >>> >>> I've forwarded your message to DCN. >>> >>> For remote switch ports, we have a policy of requiring upgrades before >>> saturation. >>> >>> For regular participants, I'm not sure that we have a policy. >>> >>> I'm also not sure if we want a policy there, as that might be considered >>> dictating peering policy. I'm not personally opposed, but this is >>> something that would need some thought. >>> >>> -- >>> Richard >>> >>> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 21, 2019

Re: Saturated ports

by Darin Steffl

I agree this can affect multiple parties. Say we let our port get saturated. Now all the content providers start receiving trouble calls. Netflix, akamai, YouTube, traffic between us and any MICE peer is congested and our customers will probably blame us but they may also blame the other peer who they receive traffic from when it's not their fault. I don't think blocking a port is a good idea but I think proactive notifications is a good first step then more harshly worded emails if that doesn't work. We don't want to let anyone think running a port at 100% is a good idea. It puts a burden on more than just the member with a congested port in my opinion like Ben said. On Wed, Aug 21, 2019, 11:28 AM Ben Wiechman <ben.wiechman(a)arvig.com> wrote: > I would dispute the "only their network" statement. We've troubleshot > various issues in the past with latency and loss, especially for internet > based services used by businesses when that traffic traverses MICE. While > the peering policy of a service provider is their decision, it is > potentially appropriate to provide proactive notice in some fashion that > would alert other operators that they may want to adjust their peering > strategy to minimize customer complaints. > > Is some kind of alarming or reporting viable and useful? From my > perspective it would be useful. > > Ben Wiechman > > Director of IP Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > > > > > On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > >> The board talked about this back in the day. The thought process was that >> remotes affect multiple members so the congestion policy should be >> enforced. >> >> For a participant it's only their network (and their customers). Also, a >> disconnect might make the overall internet worse as their transit may fill. >> >> On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net < >> AnthonyAnderberg(a)nuvera.net> wrote: >> >>> I know in the past several of us have reached out privately to folks at >>> DCN about their port saturation, which seems to come and go. >>> >>> We do not have a policy about member-port saturation, and my >>> recollection is similar to Richard's - we didn't want to be bossy about >>> people's peering. >>> >>> Cheers, >>> anthony >>> >>> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >>> >>> On 8/20/19 8:48 PM, Darin Steffl wrote: >>> > Is there any policy in place for peers that let their ports >>> saturate at >>> > 100% for an extended period of time? DCN looks like they could use >>> an >>> > upgrade to their 10G port and not sure if anyone proactively >>> reaches out >>> > to members when saturation occurs. >>> >>> I've forwarded your message to DCN. >>> >>> For remote switch ports, we have a policy of requiring upgrades >>> before >>> saturation. >>> >>> For regular participants, I'm not sure that we have a policy. >>> >>> I'm also not sure if we want a policy there, as that might be >>> considered >>> dictating peering policy. I'm not personally opposed, but this is >>> something that would need some thought. >>> >>> -- >>> Richard >>> >>> >>> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Saturated ports

by Ben Wiechman

I would dispute the "only their network" statement. We've troubleshot various issues in the past with latency and loss, especially for internet based services used by businesses when that traffic traverses MICE. While the peering policy of a service provider is their decision, it is potentially appropriate to provide proactive notice in some fashion that would alert other operators that they may want to adjust their peering strategy to minimize customer complaints. Is some kind of alarming or reporting viable and useful? From my perspective it would be useful. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Wed, Aug 21, 2019 at 7:39 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > The board talked about this back in the day. The thought process was that > remotes affect multiple members so the congestion policy should be > enforced. > > For a participant it's only their network (and their customers). Also, a > disconnect might make the overall internet worse as their transit may fill. > > On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net < > AnthonyAnderberg(a)nuvera.net> wrote: > >> I know in the past several of us have reached out privately to folks at >> DCN about their port saturation, which seems to come and go. >> >> We do not have a policy about member-port saturation, and my recollection >> is similar to Richard's - we didn't want to be bossy about people's peering. >> >> Cheers, >> anthony >> >> On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: >> >> On 8/20/19 8:48 PM, Darin Steffl wrote: >> > Is there any policy in place for peers that let their ports >> saturate at >> > 100% for an extended period of time? DCN looks like they could use >> an >> > upgrade to their 10G port and not sure if anyone proactively >> reaches out >> > to members when saturation occurs. >> >> I've forwarded your message to DCN. >> >> For remote switch ports, we have a policy of requiring upgrades before >> saturation. >> >> For regular participants, I'm not sure that we have a policy. >> >> I'm also not sure if we want a policy there, as that might be >> considered >> dictating peering policy. I'm not personally opposed, but this is >> something that would need some thought. >> >> -- >> Richard >> >> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: Of minor note

by Doug McIntyre

On Wed, Aug 21, 2019 at 01:00:21PM +0000, Frank Bulk wrote: >Our route logged these two items Monday morning with MICE's AS: > >Aug 19 10:35:41 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop >Aug 19 11:08:50 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop > >Not sure what that was about. Odd, the 2nd RR's link local address is fe80::f21f:afff:fedb:a5d5 I don't find the one listed above known by either of the RR's ND process. Something freaked out for a minute? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 21, 2019

Re: 10G to MICE down

by Corey Hauer

Zayo personnel are on site - they were alerted at 4am. So the good news is that appears they do monitor that site for temp at least. They have a large industrial fan in the door trying to push out heat while they await HVAC techs. On Wed, Aug 21, 2019 at 6:41 AM Darin Steffl <darin.steffl(a)mnwifi.com> wrote: > Corey, > > Did you open a ticket with them? Our gear is still running but with high > fan speeds (knock on wood) but hopefully they can figure it out soon. > > On Wed, Aug 21, 2019 at 8:39 AM Corey Hauer <coreyhauer(a)gmail.com> wrote: > >> The universe clearly hates Zayo. Their air conditioning in Rochester has >> failed this morning - feels like 110 degrees in our colo. Our core CCR1072 >> overheated as a result. >> >> On Wed, Aug 21, 2019 at 5:15 AM Frank Bulk <fbulk(a)mypremieronline.com> >> wrote: >> >>> Likely canceled due to Belle Plain hut overheating. They were going to >>> do a software upgrade in Bigelow, MN. >>> >>> >>> >>> Frank >>> >>> >>> >>> *From:* MR Zayo <mr(a)zayo.com> >>> *Sent:* Wednesday, August 21, 2019 12:21 AM >>> *Subject:* Maintenance Status***ZAYO TTN-0003483970 MAINTENANCE >>> NOTIFICATION*** >>> >>> >>> >>> Dear Zayo Customer, >>> >>> >>> >>> Due to unforeseen circumstances this maintenance will not take place >>> tonight. We will send you a reschedule notification soon as new dates are >>> confirmed. >>> >>> *Zay**o **Global Change Management Team/Équipe de Gestion du Changement >>> Global Zay* >>> *o **Zay**o **Group/Le Groupe Zay**o* >>> Toll free/*N° sans frais*: 1.866.236.2824 <1-866-236-2824> >>> United Kingdom Toll Free/*N° sans frais Royaume-Uni*: 0800.169.1646 >>> <0800-169-1646> >>> Email/*Courriel*: mr(a)zayo.com >>> Website/*Site Web*: https://www.zayo.com >>> >>> >>> >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank >>> Bulk >>> *Sent:* Tuesday, August 20, 2019 11:54 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> My BGP sessions restored at 11:23 pm. >>> >>> >>> >>> Frank >>> >>> >>> >>> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Darin >>> Steffl >>> *Sent:* Tuesday, August 20, 2019 9:32 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> Same thing happened last summer in Rochester at Neutral Path suite after >>> Zayo took over. I was visiting our rack and it was over 90 degrees in there >>> and Zayo didn't know about it yet since they didn't monitor temps. Once I >>> notified them, it took 8 hours for them to fix the AC. They have since >>> added temperature probes in a few spots but who knows if they actually >>> monitor or get alerted since they're Zayo after all. >>> >>> >>> >>> On Tue, Aug 20, 2019 at 9:27 PM Bahls, Dean < >>> dean.bahls(a)cooperative-networks.com> wrote: >>> >>> They don’t have redundancy at that hut? Or is it just a cabinet? >>> >>> >>> >>> Dean >>> >>> >>> >>> On Tue, Aug 20, 2019 at 9:25 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >>> >>> Since I was not getting straight answers from Zayo, I drove out to the >>> Belle Plaine. The air conditioning has failed, and the temperature was >>> over 110 degrees inside. There is a technician out here who is waiting on >>> a HVAC repair company to come and fix the issue, and doing the best he can >>> to ventilate until then >>> >>> >>> >>> >>> >>> Jeremy Lumby >>> >>> Minnesota VoIP >>> >>> 9217 17th Ave S >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Suite 216 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Bloomington MN 55425 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Main 612-355-7740 >>> >>> Direct 612-392-6814 >>> >>> EFax 952-873-7425 >>> >>> jlumby(a)mnvoip.com >>> >>> >>> >>> >>> >>> -------- Original message -------- >>> >>> From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >>> >>> Date: 8/20/19 8:37 PM (GMT-06:00) >>> >>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> >>> Subject: Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> ------------------------------ >>> >>> *From* : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] >>> >>> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >>> >>> *Date* : Tuesday, August 20 2019 20:37:47 >>> >>> my circuit just went down 7 minutes ago, was fine prior to 20:30 cst >>> >>> Sent from my iPhone 7 >>> >>> >>> On Aug 20, 2019, at 20:01, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >>> >>> It is looking like it is something specific to Belle Plaine. I have a >>> diverse back door in to there that is down as well. >>> >>> >>> >>> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET >>> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Corey Hauer >>> *Sent:* Tuesday, August 20, 2019 7:58 PM >>> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> >>> Oddly, our 511 to Mankato 10 gig transit is on that route - and it up >>> and running. >>> >>> >>> >>> Possibly Zayo restored the protect path we had been getting from Neutral >>> Path after I complained they took away our protect when they re-groomed us >>> onto Zayo's native network? >>> >>> >>> >>> Corey Hauer >>> LTD Broadband >>> >>> >>> >>> On Tue, Aug 20, 2019 at 5:21 PM Frank Bulk <fbulk(a)mypremieronline.com> >>> wrote: >>> >>> Thanks, Jeremy. >>> >>> Frank >>> >>> >>> >>> On Tue, Aug 20, 2019 at 6:21 PM -0500, "Jeremy Lumby" <jlumby(a)MNVOIP.COM> >>> wrote: >>> >>> I lost all of my connectivity between 511 and and Belle Plaine, I have >>> opened a ticket with zayo >>> >>> >>> >>> >>> >>> >>> >>> Jeremy Lumby >>> >>> Minnesota VoIP >>> >>> 9217 17th Ave S >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Suite 216 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Bloomington MN 55425 >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >>> >>> Main 612-355-7740 >>> >>> Direct 612-392-6814 >>> >>> EFax 952-873-7425 >>> >>> jlumby(a)mnvoip.com >>> >>> >>> >>> >>> >>> -------- Original message -------- >>> >>> From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> >>> >>> Date: 8/20/19 6:13 PM (GMT-06:00) >>> >>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>> >>> Subject: [MICE-DISCUSS] 10G to MICE down >>> >>> >>> ------------------------------ >>> >>> *From* : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] >>> >>> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >>> >>> *Date* : Tuesday, August 20 2019 18:13:03 >>> >>> We saw our 10 gig to MICE go down around 6:00 p.m. my guess is somewhere >>> between northwest Iowa and Belle Plaine? >>> >>> Frank >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> -- >>> >>> *Dean Bahls* >>> >>> Network Services Manager | Cooperative Network Services, LLC >>> >>> 14 Main St SW | P.O. Box J | Menahga, MN 56464 >>> >>> 218.346.3635 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.346.3635&d=DwMFaQ&c…> >>> (Direct) | 218.564.3000 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.564.3000&d=DwMFaQ&c…> >>> (Main Office) >>> >>> dean.bahls(a)cooperative-networks.com >>> >>> >>> >>> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> >>> >>> -- >>> >>> Darin Steffl >>> >>> Minnesota WiFi >>> >>> www.mnwifi.com >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mnwifi.com_&d=DwMFa…> >>> >>> 507-634-WiFi >>> >>> >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >>> Like us on Facebook >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >>> >>> ------------------------------ >>> >>> To unsubscribe from the MICE-DISCUSS list, click the following link: >>> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >>> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > <http://www.facebook.com/minnesotawifi> Like us on Facebook > <http://www.facebook.com/minnesotawifi> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Re: 10G to MICE down

by Darin Steffl

Corey, Did you open a ticket with them? Our gear is still running but with high fan speeds (knock on wood) but hopefully they can figure it out soon. On Wed, Aug 21, 2019 at 8:39 AM Corey Hauer <coreyhauer(a)gmail.com> wrote: > The universe clearly hates Zayo. Their air conditioning in Rochester has > failed this morning - feels like 110 degrees in our colo. Our core CCR1072 > overheated as a result. > > On Wed, Aug 21, 2019 at 5:15 AM Frank Bulk <fbulk(a)mypremieronline.com> > wrote: > >> Likely canceled due to Belle Plain hut overheating. They were going to >> do a software upgrade in Bigelow, MN. >> >> >> >> Frank >> >> >> >> *From:* MR Zayo <mr(a)zayo.com> >> *Sent:* Wednesday, August 21, 2019 12:21 AM >> *Subject:* Maintenance Status***ZAYO TTN-0003483970 MAINTENANCE >> NOTIFICATION*** >> >> >> >> Dear Zayo Customer, >> >> >> >> Due to unforeseen circumstances this maintenance will not take place >> tonight. We will send you a reschedule notification soon as new dates are >> confirmed. >> >> *Zay**o **Global Change Management Team/Équipe de Gestion du Changement >> Global Zay* >> *o **Zay**o **Group/Le Groupe Zay**o* >> Toll free/*N° sans frais*: 1.866.236.2824 <1-866-236-2824> >> United Kingdom Toll Free/*N° sans frais Royaume-Uni*: 0800.169.1646 >> <0800-169-1646> >> Email/*Courriel*: mr(a)zayo.com >> Website/*Site Web*: https://www.zayo.com >> >> >> >> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank >> Bulk >> *Sent:* Tuesday, August 20, 2019 11:54 PM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >> >> >> >> My BGP sessions restored at 11:23 pm. >> >> >> >> Frank >> >> >> >> *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Darin >> Steffl >> *Sent:* Tuesday, August 20, 2019 9:32 PM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >> >> >> >> Same thing happened last summer in Rochester at Neutral Path suite after >> Zayo took over. I was visiting our rack and it was over 90 degrees in there >> and Zayo didn't know about it yet since they didn't monitor temps. Once I >> notified them, it took 8 hours for them to fix the AC. They have since >> added temperature probes in a few spots but who knows if they actually >> monitor or get alerted since they're Zayo after all. >> >> >> >> On Tue, Aug 20, 2019 at 9:27 PM Bahls, Dean < >> dean.bahls(a)cooperative-networks.com> wrote: >> >> They don’t have redundancy at that hut? Or is it just a cabinet? >> >> >> >> Dean >> >> >> >> On Tue, Aug 20, 2019 at 9:25 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >> >> Since I was not getting straight answers from Zayo, I drove out to the >> Belle Plaine. The air conditioning has failed, and the temperature was >> over 110 degrees inside. There is a technician out here who is waiting on >> a HVAC repair company to come and fix the issue, and doing the best he can >> to ventilate until then >> >> >> >> >> >> Jeremy Lumby >> >> Minnesota VoIP >> >> 9217 17th Ave S >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Suite 216 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Bloomington MN 55425 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Main 612-355-7740 >> >> Direct 612-392-6814 >> >> EFax 952-873-7425 >> >> jlumby(a)mnvoip.com >> >> >> >> >> >> -------- Original message -------- >> >> From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> >> >> Date: 8/20/19 8:37 PM (GMT-06:00) >> >> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> >> Subject: Re: [MICE-DISCUSS] 10G to MICE down >> >> >> ------------------------------ >> >> *From* : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] >> >> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >> >> *Date* : Tuesday, August 20 2019 20:37:47 >> >> my circuit just went down 7 minutes ago, was fine prior to 20:30 cst >> >> Sent from my iPhone 7 >> >> >> On Aug 20, 2019, at 20:01, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: >> >> It is looking like it is something specific to Belle Plaine. I have a >> diverse back door in to there that is down as well. >> >> >> >> *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET >> <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Corey Hauer >> *Sent:* Tuesday, August 20, 2019 7:58 PM >> *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> *Subject:* Re: [MICE-DISCUSS] 10G to MICE down >> >> >> >> Oddly, our 511 to Mankato 10 gig transit is on that route - and it up and >> running. >> >> >> >> Possibly Zayo restored the protect path we had been getting from Neutral >> Path after I complained they took away our protect when they re-groomed us >> onto Zayo's native network? >> >> >> >> Corey Hauer >> LTD Broadband >> >> >> >> On Tue, Aug 20, 2019 at 5:21 PM Frank Bulk <fbulk(a)mypremieronline.com> >> wrote: >> >> Thanks, Jeremy. >> >> Frank >> >> >> >> On Tue, Aug 20, 2019 at 6:21 PM -0500, "Jeremy Lumby" <jlumby(a)MNVOIP.COM> >> wrote: >> >> I lost all of my connectivity between 511 and and Belle Plaine, I have >> opened a ticket with zayo >> >> >> >> >> >> >> >> Jeremy Lumby >> >> Minnesota VoIP >> >> 9217 17th Ave S >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Suite 216 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Bloomington MN 55425 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> >> >> Main 612-355-7740 >> >> Direct 612-392-6814 >> >> EFax 952-873-7425 >> >> jlumby(a)mnvoip.com >> >> >> >> >> >> -------- Original message -------- >> >> From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> >> >> Date: 8/20/19 6:13 PM (GMT-06:00) >> >> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> >> Subject: [MICE-DISCUSS] 10G to MICE down >> >> >> ------------------------------ >> >> *From* : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] >> >> *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] >> >> *Date* : Tuesday, August 20 2019 18:13:03 >> >> We saw our 10 gig to MICE go down around 6:00 p.m. my guess is somewhere >> between northwest Iowa and Belle Plaine? >> >> Frank >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> -- >> >> *Dean Bahls* >> >> Network Services Manager | Cooperative Network Services, LLC >> >> 14 Main St SW | P.O. Box J | Menahga, MN 56464 >> >> 218.346.3635 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.346.3635&d=DwMFaQ&c…> >> (Direct) | 218.564.3000 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.564.3000&d=DwMFaQ&c…> >> (Main Office) >> >> dean.bahls(a)cooperative-networks.com >> >> >> >> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> >> >> -- >> >> Darin Steffl >> >> Minnesota WiFi >> >> www.mnwifi.com >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mnwifi.com_&d=DwMFa…> >> >> 507-634-WiFi >> >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >> Like us on Facebook >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> >> >> ------------------------------ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi <http://www.facebook.com/minnesotawifi> Like us on Facebook <http://www.facebook.com/minnesotawifi>

Aug. 21, 2019

Re: 10G to MICE down

by Corey Hauer

The universe clearly hates Zayo. Their air conditioning in Rochester has failed this morning - feels like 110 degrees in our colo. Our core CCR1072 overheated as a result. On Wed, Aug 21, 2019 at 5:15 AM Frank Bulk <fbulk(a)mypremieronline.com> wrote: > Likely canceled due to Belle Plain hut overheating. They were going to do > a software upgrade in Bigelow, MN. > > > > Frank > > > > *From:* MR Zayo <mr(a)zayo.com> > *Sent:* Wednesday, August 21, 2019 12:21 AM > *Subject:* Maintenance Status***ZAYO TTN-0003483970 MAINTENANCE > NOTIFICATION*** > > > > Dear Zayo Customer, > > > > Due to unforeseen circumstances this maintenance will not take place > tonight. We will send you a reschedule notification soon as new dates are > confirmed. > > *Zay**o **Global Change Management Team/Équipe de Gestion du Changement > Global Zay* > *o **Zay**o **Group/Le Groupe Zay**o* > Toll free/*N° sans frais*: 1.866.236.2824 <1-866-236-2824> > United Kingdom Toll Free/*N° sans frais Royaume-Uni*: 0800.169.1646 > <0800-169-1646> > Email/*Courriel*: mr(a)zayo.com > Website/*Site Web*: https://www.zayo.com > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Frank > Bulk > *Sent:* Tuesday, August 20, 2019 11:54 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] 10G to MICE down > > > > My BGP sessions restored at 11:23 pm. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Darin > Steffl > *Sent:* Tuesday, August 20, 2019 9:32 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] 10G to MICE down > > > > Same thing happened last summer in Rochester at Neutral Path suite after > Zayo took over. I was visiting our rack and it was over 90 degrees in there > and Zayo didn't know about it yet since they didn't monitor temps. Once I > notified them, it took 8 hours for them to fix the AC. They have since > added temperature probes in a few spots but who knows if they actually > monitor or get alerted since they're Zayo after all. > > > > On Tue, Aug 20, 2019 at 9:27 PM Bahls, Dean < > dean.bahls(a)cooperative-networks.com> wrote: > > They don’t have redundancy at that hut? Or is it just a cabinet? > > > > Dean > > > > On Tue, Aug 20, 2019 at 9:25 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > Since I was not getting straight answers from Zayo, I drove out to the > Belle Plaine. The air conditioning has failed, and the temperature was > over 110 degrees inside. There is a technician out here who is waiting on > a HVAC repair company to come and fix the issue, and doing the best he can > to ventilate until then > > > > > > Jeremy Lumby > > Minnesota VoIP > > 9217 17th Ave S > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Suite 216 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Bloomington MN 55425 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Main 612-355-7740 > > Direct 612-392-6814 > > EFax 952-873-7425 > > jlumby(a)mnvoip.com > > > > > > -------- Original message -------- > > From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> > > Date: 8/20/19 8:37 PM (GMT-06:00) > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] 10G to MICE down > > > ------------------------------ > > *From* : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] > > *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] > > *Date* : Tuesday, August 20 2019 20:37:47 > > my circuit just went down 7 minutes ago, was fine prior to 20:30 cst > > Sent from my iPhone 7 > > > On Aug 20, 2019, at 20:01, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > It is looking like it is something specific to Belle Plaine. I have a > diverse back door in to there that is down as well. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Corey Hauer > *Sent:* Tuesday, August 20, 2019 7:58 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] 10G to MICE down > > > > Oddly, our 511 to Mankato 10 gig transit is on that route - and it up and > running. > > > > Possibly Zayo restored the protect path we had been getting from Neutral > Path after I complained they took away our protect when they re-groomed us > onto Zayo's native network? > > > > Corey Hauer > LTD Broadband > > > > On Tue, Aug 20, 2019 at 5:21 PM Frank Bulk <fbulk(a)mypremieronline.com> > wrote: > > Thanks, Jeremy. > > Frank > > > > On Tue, Aug 20, 2019 at 6:21 PM -0500, "Jeremy Lumby" <jlumby(a)MNVOIP.COM> > wrote: > > I lost all of my connectivity between 511 and and Belle Plaine, I have > opened a ticket with zayo > > > > > > > > Jeremy Lumby > > Minnesota VoIP > > 9217 17th Ave S > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Suite 216 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Bloomington MN 55425 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.google.com_maps_se…> > > Main 612-355-7740 > > Direct 612-392-6814 > > EFax 952-873-7425 > > jlumby(a)mnvoip.com > > > > > > -------- Original message -------- > > From: Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> > > Date: 8/20/19 6:13 PM (GMT-06:00) > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: [MICE-DISCUSS] 10G to MICE down > > > ------------------------------ > > *From* : Frank Bulk [fbulk(a)MYPREMIERONLINE.COM] > > *To* : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] > > *Date* : Tuesday, August 20 2019 18:13:03 > > We saw our 10 gig to MICE go down around 6:00 p.m. my guess is somewhere > between northwest Iowa and Belle Plaine? > > Frank > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > -- > > *Dean Bahls* > > Network Services Manager | Cooperative Network Services, LLC > > 14 Main St SW | P.O. Box J | Menahga, MN 56464 > > 218.346.3635 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.346.3635&d=DwMFaQ&c…> > (Direct) | 218.564.3000 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__218.564.3000&d=DwMFaQ&c…> > (Main Office) > > dean.bahls(a)cooperative-networks.com > > > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > > > -- > > Darin Steffl > > Minnesota WiFi > > www.mnwifi.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mnwifi.com_&d=DwMFa…> > > 507-634-WiFi > > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> > Like us on Facebook > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_minnes…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 21, 2019

Of minor note

by Frank Bulk

Our route logged these two items Monday morning with MICE's AS: Aug 19 10:35:41 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Aug 19 11:08:50 207.32.15.28 Rib: mpbgp_recv_mpreach: peer 2001:504:27::d1af:0:2 (AS 53679) link-local next hop fe80::e2ac:f1ef:fe4b:d93b%10000 improper, ignoring this next-hop Not sure what that was about. Frank

Aug. 21, 2019

Re: Saturated ports

by Jay Hanke

The board talked about this back in the day. The thought process was that remotes affect multiple members so the congestion policy should be enforced. For a participant it's only their network (and their customers). Also, a disconnect might make the overall internet worse as their transit may fill. On Tue, Aug 20, 2019, 11:57 PM AnthonyAnderberg(a)nuvera.net < AnthonyAnderberg(a)nuvera.net> wrote: > I know in the past several of us have reached out privately to folks at > DCN about their port saturation, which seems to come and go. > > We do not have a policy about member-port saturation, and my recollection > is similar to Richard's - we didn't want to be bossy about people's peering. > > Cheers, > anthony > > On 8/20/19, 9:05 PM, "Richard Laager" <rlaager(a)WIKTEL.COM> wrote: > > On 8/20/19 8:48 PM, Darin Steffl wrote: > > Is there any policy in place for peers that let their ports saturate > at > > 100% for an extended period of time? DCN looks like they could use an > > upgrade to their 10G port and not sure if anyone proactively reaches > out > > to members when saturation occurs. > > I've forwarded your message to DCN. > > For remote switch ports, we have a policy of requiring upgrades before > saturation. > > For regular participants, I'm not sure that we have a policy. > > I'm also not sure if we want a policy there, as that might be > considered > dictating peering policy. I'm not personally opposed, but this is > something that would need some thought. > > -- > Richard > > >

Aug. 21, 2019