April 2017 - mice-discuss - lists.micemn.net

Re: Arista Round 3 Moves

by Jeremy Lumby

John, There are several factors as to why this is done during the day. It all goes back to your 35 years of experience in the industry. The fact is that over 99% of the industry is for profit. Not only has MICE always aimed for nonprofit status, they are also 100% volunteer based. I have been volunteering for MICE since they came to be 7 years ago. Also up until this last year MICE did not charge any fees, and now the fee is so minimal, and not even completely mandatory, it is just more than enough to cover the maintenance contract on the switch. When everything is 100% volunteer, it is difficult enough to get volunteers, let alone ones willing to show up in the middle of the night. Cologix was kind enough to offer to move all cross connects for free as long as it was done during normal business hours. Members always had the option to pay Cologix to move their cross connect in the middle of the night. As far as I know no one took Cologix up on that. I feel the main reason is since we are all multi-homed running BGP, as long as enough notice is given, BGP can be gracefully shutdown prior to the maintenance, and there are no real issues. I am sorry that your equipment did not perform as expected. The remote switch you are connected to was part of the maintenance, and the maintenance notification did state that all members connected to the remote switches listed should shutdown BGP ahead of time as well. As for a refund, whenever I run into a potential member looking for a service level agreement, I point out how all service agreements in the industry offer you a portion of what you paid depending on the duration of the outage. Since MICE has been free for the last 7 years, any portion would still be 0. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of John Unger Sent: Wednesday, April 26, 2017 6:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not make sense to me? One factor was that Cologix needs to be involved in the fiber moves from the old cabinet to the new cabinet big enough to hold the new switch, and if done at night, MICE would have had to pay for at least 3 full nights of overtime tech worker to be there coordinating the fiber moves. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Arista Round 4 Moves - FINAL

by Jeremy Lumby

I am planning on doing the final round of moves to the Arista switch on the afternoon of 5/9/2017 beginning at 1PM. The second route server will be moved to the new cabinet, as well as the following connections to the main switch will also be moved: Great Plains Acentek Consolidated Communications Long Lines HBC Savage UW System Radio Link LTD Broadband Integra Inteliquent Hoyos Consulting Visionary (Mammoth) Code42 Emergent HCMC Unggoy I suggest that all members listed above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Maintenance Complete

by Jeremy Lumby

The maintenance just completed. All ports are showing link, however I have not gone through a detailed check yet. Please shoot me an email if you see an issue before I do. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: US Internet to move to 100gig interface

by John McLaughlin

Welcome to the future fellas! On Tue, Apr 18, 2017 at 6:56 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Looks nice and clean on the MICE switch side as well. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Justin Krejci > *Sent:* Tuesday, April 18, 2017 6:13 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] US Internet to move to 100gig interface > > > > Cutover is complete. Everything looks good on my side and error free. > Please let me know if anyone runs into any problems to USI. > > I think this marks the first 100gig connection (member or otherwise) on > MICE, so that is cool! > > > ------------------------------ > > *From:* MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin > Krejci [JKrejci(a)USINTERNET.COM] > *Sent:* Tuesday, April 18, 2017 4:57 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] US Internet to move to 100gig interface > > I am going to be shutting down our BGP sessions on MICE in the next few > minutes so I can move my traffic over to the 100gig interface. I will > re-enable BGP afterwards, assuming all goes well. > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- *John McLaughlin* Major Accounts Manager States: MN, IA, SD, ND Mobile: 612-816-9024 jmclaughlin(a)arista.com 2016 Gartner Report <https://www.gartner.com/doc/reprints?id=1-36UFTCZ&ct=160517&st=sb>

April 18, 2017

Re: US Internet to move to 100gig interface

by Jeremy Lumby

Looks nice and clean on the MICE switch side as well. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Justin Krejci Sent: Tuesday, April 18, 2017 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] US Internet to move to 100gig interface Cutover is complete. Everything looks good on my side and error free. Please let me know if anyone runs into any problems to USI. I think this marks the first 100gig connection (member or otherwise) on MICE, so that is cool! From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin Krejci [JKrejci(a)USINTERNET.COM] Sent: Tuesday, April 18, 2017 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] US Internet to move to 100gig interface I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 18, 2017

Re: US Internet to move to 100gig interface

by Justin Krejci

Cutover is complete. Everything looks good on my side and error free. Please let me know if anyone runs into any problems to USI. I think this marks the first 100gig connection (member or otherwise) on MICE, so that is cool! ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin Krejci [JKrejci(a)USINTERNET.COM] Sent: Tuesday, April 18, 2017 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] US Internet to move to 100gig interface I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 18, 2017