- mice-discuss - lists.micemn.net

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years > this is just crazy? Nobody does maintenance during the day? Which industry? Because the Internet industry isn't 35 years old. Many people do maintenance during the day. So many. It's a 24x7 world now and for many businesses the time of day for work doesn't necessarily follow the sun. > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? Are you direct connected or are you via remote switch? If you are via remote switch then did your switch operator notify you? > Is MICE issuing credits for outages?? Planned don't count. And no SLA means .. no 'credit'. If you are via remote switch then you should take it up with them and not the community as a whole. But beyond that snark: this is volunteer work and as such Jeremy is already taking time out of his business day to do things for this group. Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, I surely did not mean to cause any feelings of un-appreciation for what you are doing. Just the timing of it and now I do understand the reasoning. Thanks and appreciate your time and efforts. John From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dean Bahls Sent: Wednesday, April 26, 2017 8:39 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com<mailto:hoyosa@gmail.com>> wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 26, 2017

Re: Arista Round 3 Moves

by Dean Bahls

Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

April 26, 2017

Re: Arista Round 3 Moves

by Colin Baker

On Wed, Apr 26, 2017 at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years this > is just crazy? Nobody does maintenance during the day? > > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? > > Is MICE issuing credits for outages?? I'll add that most MICE members are eyeballs whose peak usage occurs at night. Daytime is probably preferable for these networks. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net

April 26, 2017

Re: Arista Round 3 Moves

by Mike Johnston

On 2017-04-26 08:28, Ben Wiechman wrote: > > On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com > <mailto:hoyosa@gmail.com>> wrote: > > As an aside, thanks Jeremy for doing all this move work, it's not > a small effort and definitely appreciated. > > > I concur. This is greatly appreciated by us as well. Thank you. > I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

I do agree that filtering BGP prior to the move is a great thing to do. I know we attempted it in the past, and we had some issues getting it to work properly. Now that we are on the Arista, I have tested it and it seems to work well. I plan on utilizing it on all future maintenance on the Arista switch. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Wednesday, April 26, 2017 8:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Ben Wiechman

On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > As an aside, thanks Jeremy for doing all this move work, it's not a small > effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

John, There are several factors as to why this is done during the day. It all goes back to your 35 years of experience in the industry. The fact is that over 99% of the industry is for profit. Not only has MICE always aimed for nonprofit status, they are also 100% volunteer based. I have been volunteering for MICE since they came to be 7 years ago. Also up until this last year MICE did not charge any fees, and now the fee is so minimal, and not even completely mandatory, it is just more than enough to cover the maintenance contract on the switch. When everything is 100% volunteer, it is difficult enough to get volunteers, let alone ones willing to show up in the middle of the night. Cologix was kind enough to offer to move all cross connects for free as long as it was done during normal business hours. Members always had the option to pay Cologix to move their cross connect in the middle of the night. As far as I know no one took Cologix up on that. I feel the main reason is since we are all multi-homed running BGP, as long as enough notice is given, BGP can be gracefully shutdown prior to the maintenance, and there are no real issues. I am sorry that your equipment did not perform as expected. The remote switch you are connected to was part of the maintenance, and the maintenance notification did state that all members connected to the remote switches listed should shutdown BGP ahead of time as well. As for a refund, whenever I run into a potential member looking for a service level agreement, I point out how all service agreements in the industry offer you a portion of what you paid depending on the duration of the outage. Since MICE has been free for the last 7 years, any portion would still be 0. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of John Unger Sent: Wednesday, April 26, 2017 6:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not make sense to me? One factor was that Cologix needs to be involved in the fiber moves from the old cabinet to the new cabinet big enough to hold the new switch, and if done at night, MICE would have had to pay for at least 3 full nights of overtime tech worker to be there coordinating the fiber moves. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017