- mice-discuss - lists.micemn.net

Re: Arista Round 4 Moves - FINAL

by Jeremy Lumby

I am no longer seeing any of the packet loss that I was previously seeing between the Juniper 4500, and 4550. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Justin Krejci Sent: Wednesday, May 10, 2017 10:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 4 Moves - FINAL Yes this is awesome!! Thanks for all of the efforts!! Jeremy, I recall before there was some discussions around some small amount of packet loss when traffic moved across a certain path from one member to another and it seemed one of the Juniper components was most likely the involved: stacking cables, a particular switch, or something. Without checking the list archives, I believe you were involved in identifying some of this, some of which was improved by re-working a bit of the fabric topology. Since the Arista moves are basically all done, I am curious to hear if any of that previous packet loss is still happening now that many of the previous fabric components are decommissioned. Also, not urgent but I think it would be great if the technical page details could be updated at some point: config details, exchange topology diagram, etc. ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Mike Horwath [drechsau(a)GEEKS.ORG] Sent: Tuesday, May 09, 2017 6:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 4 Moves - FINAL On Tue, May 09, 2017 at 04:11:00PM -0500, Jeremy Lumby wrote: > The maintenance has completed, please let me know if you are > experiencing any errors. THANKS! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

May 10, 2017

Re: Arista Round 4 Moves - FINAL

by Justin Krejci

Yes this is awesome!! Thanks for all of the efforts!! Jeremy, I recall before there was some discussions around some small amount of packet loss when traffic moved across a certain path from one member to another and it seemed one of the Juniper components was most likely the involved: stacking cables, a particular switch, or something. Without checking the list archives, I believe you were involved in identifying some of this, some of which was improved by re-working a bit of the fabric topology. Since the Arista moves are basically all done, I am curious to hear if any of that previous packet loss is still happening now that many of the previous fabric components are decommissioned. Also, not urgent but I think it would be great if the technical page details could be updated at some point: config details, exchange topology diagram, etc. ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Mike Horwath [drechsau(a)GEEKS.ORG] Sent: Tuesday, May 09, 2017 6:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 4 Moves - FINAL On Tue, May 09, 2017 at 04:11:00PM -0500, Jeremy Lumby wrote: > The maintenance has completed, please let me know if you are > experiencing any errors. THANKS! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

May 10, 2017

Re: Arista Round 4 Moves - FINAL

by Mike Horwath

On Tue, May 09, 2017 at 04:11:00PM -0500, Jeremy Lumby wrote: > The maintenance has completed, please let me know if you are > experiencing any errors. THANKS! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

May 9, 2017

Re: Arista Round 4 Moves - FINAL

by Jeremy Lumby

The maintenance has completed, please let me know if you are experiencing any errors. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Tuesday, May 09, 2017 12:51 PM To: MICE Discuss Subject: RE: Arista Round 4 Moves - FINAL Just a quick reminder that maintenance will begin in 10 Minutes -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Tuesday, April 25, 2017 8:20 PM To: MICE Discuss Subject: Arista Round 4 Moves - FINAL I am planning on doing the final round of moves to the Arista switch on the afternoon of 5/9/2017 beginning at 1PM. The second route server will be moved to the new cabinet, as well as the following connections to the main switch will also be moved: Great Plains Acentek Consolidated Communications Long Lines HBC Savage UW System Radio Link LTD Broadband Integra Inteliquent Hoyos Consulting Visionary (Mammoth) Code42 Emergent HCMC Unggoy I suggest that all members listed above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

May 9, 2017

Re: Outage on EX4500

by Jeremy Lumby

Today is the final round of moves. The second route server is being moved as well. The 4500, and 4550 will be out of service as of today. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Tuesday, May 09, 2017 2:36 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Outage on EX4500 Anyone else seeing an outage on the EX4500? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

May 9, 2017

Outage on EX4500

by Jason Hanke

Anyone else seeing an outage on the EX4500? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

May 9, 2017

Re: Arista Round 4 Moves - FINAL

by Jeremy Lumby

Just a quick reminder that maintenance will begin in 10 Minutes -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Tuesday, April 25, 2017 8:20 PM To: MICE Discuss Subject: Arista Round 4 Moves - FINAL I am planning on doing the final round of moves to the Arista switch on the afternoon of 5/9/2017 beginning at 1PM. The second route server will be moved to the new cabinet, as well as the following connections to the main switch will also be moved: Great Plains Acentek Consolidated Communications Long Lines HBC Savage UW System Radio Link LTD Broadband Integra Inteliquent Hoyos Consulting Visionary (Mammoth) Code42 Emergent HCMC Unggoy I suggest that all members listed above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

May 9, 2017

MinnState using it own ASN now

by David Farmer

This is just a heads-up to let you know that MinnState(fKA MnSCU), as of around 07:00 this morning, is now using its own ASN, AS675, for originating 199.17.0.0/16, 134.29.0.0/16, and 204.77.32.0/19. Also at the same time, MinnState now has it's own pair of 2x10G connections to Northern Lights GigaPoP at the 511 Building and Centennial Office Building (State Capitol Campus), that are separated from the State of Minnesota's (MN.IT <http://mn.it/>) connections to the GigaPoP at the same locations. I can't imagine this should have caused any issues for anyone, but some of you may have noticed the change, and I thought I would send out an explanation. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

May 7, 2017

Arista/HPE & Docker Containerized data Center Webinar

by John McLaughlin

Hi MICE Team: Please join this upcoming webinar if you can and pass on to others. See registration link below: <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*W9fxlYL8RW…> <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*W98m9706Rf…> <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*W1btf4s2Pm…> <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*W5lr-rc3pL…> <mailto:?subject=Check%20out%20https%3A%2F%2Fsolutions.arista.com%2F-temporary-slug-d376ded9-c8ae-4778-95fd-6f33ca5bb410%3Futm_medium%3Dsocial%26utm_source%3Demail%20&body=Check%20out%20https%3A%2F%2Fsolutions.arista.com%2F-temporary-slug-d376ded9-c8ae-4778-95fd-6f33ca5bb410%3Futm_medium%3Dsocial%26utm_source%3Demail> <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*W8VT5Mv2Wg…> Learn about Arista's integration with HPE and Docker Arista and HPE partner with Docker to discuss the future of data center architectures combining Arista/HPE powered infrastructure with Docker's container ecosystem. Fred Hsu, Arista Technical Marketing Engineer, Chris Young, HPE Technical Marketing Engineer, FlexManagement, and Brent Salisbury, Docker Software Alliance Engineer, will lead a deep dive into the following topics: Converged infrastructure with Arista EOS, HPE servers/storage, and Docker Arista Container Tracer for visibility ContainerOps with Docker containers on servers and switches Network tools and workflows using containers The presentation will conclude with an interactive Q&A session. We look forward to seeing you there. <https://solutions.arista.com/e1t/c/*W6-QG1v2PfN4_W4WXmF-2HR2WW0/*VYBNgz7YBn…> Fred Hsu Technical Marketing Engineer, Arista Networks Brent Salisbury Software Alliance Engineer, Docker Chris Young Technical Marketing Engineer, FlexManagement, HPE Tel. 302.123.4567 Fax. 302.123.4567 5453 Great America Pkwy Santa Clara, CA 95054 United States You received this email because you are subscribed to Marketing Information from Arista Networks, Inc. John McLaughlin 612-816-9024 Major Account Manager jmclaughlin(a)arista.com

May 2, 2017

Re: Please welcome Yomura Corporation to MICE!

by Doug McIntyre

On Fri, Apr 28, 2017 at 03:52:31PM -0500, Jason Hanke wrote: > Name: Yomura Corporation > ASN: 7363 > Port: Ethernet 5/4 (Main Switch) 10G > IPv4: 206.108.255.77/24 > IPv6: 2001:504:27:0:0:1cc3::1/64 > Peering Contact Email: peering(a)yomura.com > Route Server: Yes > BFD Support: Yes The route servers are all setup and ready to go on this. Welcome! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 1, 2017

MICE hits 300 gbps

by Justin Krejci

Clearly MICE has been trending upwards but hitting another 100gbps multiple is a pretty nice milestone I think. (yes this is using marketing math). Maybe who ever manages to the social media presence for MICE wants to post an update about this milestone? Good work to all, especially those lately working on the migrations to the new switch fabric, it is definitely appreciated!

May 1, 2017

Re: Netflix Cleanup

by Jason Hanke

Ok, I had them allocated on the new switch just in case. I'll clear them out. On Fri, Apr 28, 2017 at 3:57 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > The Netflix 10G ports were all on the Junipers that are being taken out of > service. Netflix has only ever been 100G on the Arista. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Jason Hanke > *Sent:* Friday, April 28, 2017 3:53 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Netflix Cleanup > > > > Are the Netflix 10G ports ready for reclamation following the 100G move? > > > > -- > > Jay Hanke > > CTO > > Neutral Path Communications > > 3 Civic Center Plaza, Suite 204 > > Mankato, MN 56001 > > (507) 327-2398 mobile > > jayhanke(a)neutralpath.net > > www.neutralpath.net > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

April 28, 2017

Re: Netflix Cleanup

by Jeremy Lumby

The Netflix 10G ports were all on the Junipers that are being taken out of service. Netflix has only ever been 100G on the Arista. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Friday, April 28, 2017 3:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Netflix Cleanup Are the Netflix 10G ports ready for reclamation following the 100G move? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 28, 2017

Netflix Cleanup

by Jason Hanke

Are the Netflix 10G ports ready for reclamation following the 100G move? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

April 28, 2017

Please welcome Yomura Corporation to MICE!

by Jason Hanke

Name: Yomura Corporation ASN: 7363 Port: Ethernet 5/4 (Main Switch) 10G IPv4: 206.108.255.77/24 IPv6: 2001:504:27:0:0:1cc3::1/64 Peering Contact Email: peering(a)yomura.com Route Server: Yes BFD Support: Yes Billing contact information: Yomura Corp 55 Marietta Street Atlanta, GA 30303 Email: payables(a)yomura.com Welcome Aboard! -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

April 28, 2017

Re: MICE Graphs

by David Farmer

Thanks, it looks much better now. By the way, all I really wanted to know was that some one was working on it, and that it wasn't forgotten about. It's completely fine that it took a couple days to make happen. On Thu, Apr 27, 2017 at 3:35 PM, anthonyanderberg(a)nu-telecom.net < anthonyanderberg(a)nu-telecom.net> wrote: > Since so much changed I took the opportunity to audit the aggregate graph > - obviously I want to preserve our history without double-counting any > bits. My actual job also cut into my MICE efforts yesterday some too... > anyway I’ve got all the links updated on the Weathermap too - will slide > the icons around to reflect the current architecture next. > > > > Cheers, > > anthony > > > > *Anthony Anderberg* > > Sr. Systems Analyst > > [image: NUtel_email_logo_1] > > 320-234-5239 <(320)%20234-5239> > > anthonyanderberg(a)nu-telecom.net > > *www.nutelecom.net <http://www.nutelecom.net>* > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *David Farmer > *Sent:* Thursday, April 27, 2017 11:36 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] MICE Graphs > > > > I see the data there on the individual graphs, but the aggregate graph > doesn't seem to be including the new switch. > > > > On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> > wrote: > > On 04/26/2017 11:32 AM, David Farmer wrote: > > According to the graphs on the web page there was a network holocaust on > > MICE yesterday. Could someone please fix the graphs. > > Anthony is actively working on this now. The RRDs were copied last > night, so the data is preserved. > > -- > Richard > > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 <(612)%20626-0815> > Minneapolis, MN 55414-3029 Cell: 612-812-9952 <(612)%20812-9952> > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 28, 2017

Re: MICE Graphs

by anthonyanderberg＠nu-telecom.net

Since so much changed I took the opportunity to audit the aggregate graph - obviously I want to preserve our history without double-counting any bits. My actual job also cut into my MICE efforts yesterday some too... anyway I’ve got all the links updated on the Weathermap too - will slide the icons around to reflect the current architecture next. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Thursday, April 27, 2017 11:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Graphs I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com<mailto:rlaager@wiktel.com>> wrote: On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 27, 2017

Re: MICE Graphs

by Richard Laager

Anthony finished the new aggregate graph. I updated the website link. Our glory is restored! -- Richard

April 27, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Thu, Apr 27, 2017 at 12:00 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > Thanks for the great explanation. > > I wanted to reiterate a couple things: > > This ACL approach is standard practice at SIX. We know it works on the Arista switch. The plan is to use it there moving forward. Can confirm, method as outlined is sound and reasonable for MADIX and MKEIX, both 'semi distributed' and resemble MICE (conceptually, obv not in bits/sec scaling). -Tk

April 27, 2017

Re: Arista Round 3 Moves

by Richard Laager

Thanks for the great explanation. I wanted to reiterate a couple things: This ACL approach is standard practice at SIX. We know it works on the Arista switch. The plan is to use it there moving forward. -- Richard -- Richard

April 27, 2017

Re: MICE Graphs

by Richard Laager

Correct. Anthony is rebuilding the aggregate graph. -- Richard > On Apr 27, 2017, at 11:36, David Farmer <farmer(a)UMN.EDU> wrote: > > I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. > >> On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> wrote: >> On 04/26/2017 11:32 AM, David Farmer wrote: >> > According to the graphs on the web page there was a network holocaust on >> > MICE yesterday. Could someone please fix the graphs. >> >> Anthony is actively working on this now. The RRDs were copied last >> night, so the data is preserved. >> >> -- >> Richard > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 27, 2017

Re: MICE Graphs

by David Farmer

I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > On 04/26/2017 11:32 AM, David Farmer wrote: > > According to the graphs on the web page there was a network holocaust on > > MICE yesterday. Could someone please fix the graphs. > > Anthony is actively working on this now. The RRDs were copied last > night, so the data is preserved. > > -- > Richard > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 27, 2017

Re: Arista Round 3 Moves

by Justin Krejci

Forcing an issue is going to happen when the port goes down. The point is the ACL can make it more graceful for those directly affected and also those indirectly affected by the port going down by ensuring all BGP connections (MLPA and bilateral) are terminated with the directly affected members before the link goes down for the directly affected members. In this context: directly affected member = those whose path to the main fabric are going to be physically moved (includes downstream members on an extension switch that is being moved) indirectly affected member = everyone else who are not going to have their own path to the fabric go down I am not sure who would intentionally prefer a less graceful option of losing all/most MICE connectivity. The addition of very short lived ACL's on the exchange fabric ports for directly affected members does not preclude members from implementing their own proactive measures; people are certainly free to heed the announcement that they are going to be directly affected and pre-emptively shutdown all of their own BGP sessions at whatever time they want in advance and re-activate them anytime they choose afterwards be it minutes, hours or days later. The addition of the exchange enacted ACLs will help minimize the impact to the indirectly affected members who have bilateral peering with directly affected members. ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Mike Horwath [drechsau(a)GEEKS.ORG] Sent: Wednesday, April 26, 2017 6:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 10:51:34AM -0500, Doug McIntyre wrote: > On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > > Bilateral are unaffected and they can arrange things anyway with their > > peers. > > I think there is a lot more bilateral BGP going on across MICE than > you think. While I probably go talk to anybody, I have 11 > unilateral peers on MICE besides the two route servers. Yep, I am assuming a lot of bilateral BGP peers. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: MICE Graphs

by Richard Laager

On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard

April 26, 2017

MICE Graphs

by David Farmer

According to the graphs on the web page there was a network holocaust on MICE yesterday. Could someone please fix the graphs. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 26, 2017

Re: Arista Round 3 Moves

by Richard Laager

Doug, Please do go ahead and expand passive BFD. Everyone on one route server followed by the second route server a few days later should be fine. -- Richard > On Apr 26, 2017, at 10:51, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I think there is a lot more bilateral BGP going on across MICE than you think. > While I probably go talk to anybody, I have 11 unilateral peers on MICE besides > the two route servers. > > I think BFD for BGP would do a lot more good than BGP filtering if people would > implement it. > > Speaking of that, I have passive BFD testing on route server #1 for quite some time. > I've heard of zero problems. > > I would like to go more widespread than the test people (ie. offer it > to every member connection on route server #1) > > Since it is passive, the member end drives it, if they do it, the > route server will do it, if they don't, nothing else happens. I'd also like > to roll out passive BFD as default for new members. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > Bilateral are unaffected and they can arrange things anyway with their > peers. I think there is a lot more bilateral BGP going on across MICE than you think. While I probably go talk to anybody, I have 11 unilateral peers on MICE besides the two route servers. I think BFD for BGP would do a lot more good than BGP filtering if people would implement it. Speaking of that, I have passive BFD testing on route server #1 for quite some time. I've heard of zero problems. I would like to go more widespread than the test people (ie. offer it to every member connection on route server #1) Since it is passive, the member end drives it, if they do it, the route server will do it, if they don't, nothing else happens. I'd also like to roll out passive BFD as default for new members. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Wed, Apr 26, 2017 at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. Sadly, I can't thumbs-up this post as I can on FB... -Tk

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

I’ll also add, this is currently a draft with the IETF grow WG: https://tools.ietf.org/html/draft-ietf-grow-bgp-session-culling-01 (and one which I fully support) -- Andrew Hoyos hoyosa(a)gmail.com > On Apr 26, 2017, at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: >> >> On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >>> I would suggest that perhaps we look into filtering BGP (tcp/179) >>> with an ACL prior to maintenance start on those specific ports being >>> moved. Many other IXs are doing this for maintenance as a way to >>> gracefully take things down, and let bilateral and RS sessions time >>> out without killing active traffic. As we've noticed, not all >>> members being moved are bothering to shut down sessions prior, which >>> causes impact to/from those members. (i.e.: >>> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) >> >> Don't even need ACLs. >> >> Just take down the route servers for the 2 hour period. >> >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. > > Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. > I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > >> Adding another step to the process creates more complications as well, >> and another point of failure if you screw up along the way. > > Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. > Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. > > >

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: > > On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >> I would suggest that perhaps we look into filtering BGP (tcp/179) >> with an ACL prior to maintenance start on those specific ports being >> moved. Many other IXs are doing this for maintenance as a way to >> gracefully take things down, and let bilateral and RS sessions time >> out without killing active traffic. As we've noticed, not all >> members being moved are bothering to shut down sessions prior, which >> causes impact to/from those members. (i.e.: >> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) > > Don't even need ACLs. > > Just take down the route servers for the 2 hour period. > > Bilateral are unaffected and they can arrange things anyway with their > peers. I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > Adding another step to the process creates more complications as well, > and another point of failure if you screw up along the way. Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > Clean shutdown of bird is easier, quicker, and will for sure make the > multilateral peering not be further affected by bouncing repeatedly. Yes, great for MLPA, but not for bilateral. Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole.

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

> Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. It is worth pointing out that while I am on site doing the physical work, I interact with Doug, Anthony, Jay, and Richard to make sure things go smoothly, since each of them has their area of expertise. Jeremy

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: > I would suggest that perhaps we look into filtering BGP (tcp/179) > with an ACL prior to maintenance start on those specific ports being > moved. Many other IXs are doing this for maintenance as a way to > gracefully take things down, and let bilateral and RS sessions time > out without killing active traffic. As we've noticed, not all > members being moved are bothering to shut down sessions prior, which > causes impact to/from those members. (i.e.: > https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) Don't even need ACLs. Just take down the route servers for the 2 hour period. Bilateral are unaffected and they can arrange things anyway with their peers. Adding another step to the process creates more complications as well, and another point of failure if you screw up along the way. Clean shutdown of bird is easier, quicker, and will for sure make the multilateral peering not be further affected by bouncing repeatedly. > As an aside, thanks Jeremy for doing all this move work, it's not a > small effort and definitely appreciated. +1 -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years > this is just crazy? Nobody does maintenance during the day? Which industry? Because the Internet industry isn't 35 years old. Many people do maintenance during the day. So many. It's a 24x7 world now and for many businesses the time of day for work doesn't necessarily follow the sun. > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? Are you direct connected or are you via remote switch? If you are via remote switch then did your switch operator notify you? > Is MICE issuing credits for outages?? Planned don't count. And no SLA means .. no 'credit'. If you are via remote switch then you should take it up with them and not the community as a whole. But beyond that snark: this is volunteer work and as such Jeremy is already taking time out of his business day to do things for this group. Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, I surely did not mean to cause any feelings of un-appreciation for what you are doing. Just the timing of it and now I do understand the reasoning. Thanks and appreciate your time and efforts. John From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dean Bahls Sent: Wednesday, April 26, 2017 8:39 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com<mailto:hoyosa@gmail.com>> wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 26, 2017

Re: Arista Round 3 Moves

by Dean Bahls

Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

April 26, 2017

Re: Arista Round 3 Moves

by Colin Baker

On Wed, Apr 26, 2017 at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years this > is just crazy? Nobody does maintenance during the day? > > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? > > Is MICE issuing credits for outages?? I'll add that most MICE members are eyeballs whose peak usage occurs at night. Daytime is probably preferable for these networks. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net

April 26, 2017

Re: Arista Round 3 Moves

by Mike Johnston

On 2017-04-26 08:28, Ben Wiechman wrote: > > On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com > <mailto:hoyosa@gmail.com>> wrote: > > As an aside, thanks Jeremy for doing all this move work, it's not > a small effort and definitely appreciated. > > > I concur. This is greatly appreciated by us as well. Thank you. > I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

I do agree that filtering BGP prior to the move is a great thing to do. I know we attempted it in the past, and we had some issues getting it to work properly. Now that we are on the Arista, I have tested it and it seems to work well. I plan on utilizing it on all future maintenance on the Arista switch. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Wednesday, April 26, 2017 8:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Ben Wiechman

On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > As an aside, thanks Jeremy for doing all this move work, it's not a small > effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

John, There are several factors as to why this is done during the day. It all goes back to your 35 years of experience in the industry. The fact is that over 99% of the industry is for profit. Not only has MICE always aimed for nonprofit status, they are also 100% volunteer based. I have been volunteering for MICE since they came to be 7 years ago. Also up until this last year MICE did not charge any fees, and now the fee is so minimal, and not even completely mandatory, it is just more than enough to cover the maintenance contract on the switch. When everything is 100% volunteer, it is difficult enough to get volunteers, let alone ones willing to show up in the middle of the night. Cologix was kind enough to offer to move all cross connects for free as long as it was done during normal business hours. Members always had the option to pay Cologix to move their cross connect in the middle of the night. As far as I know no one took Cologix up on that. I feel the main reason is since we are all multi-homed running BGP, as long as enough notice is given, BGP can be gracefully shutdown prior to the maintenance, and there are no real issues. I am sorry that your equipment did not perform as expected. The remote switch you are connected to was part of the maintenance, and the maintenance notification did state that all members connected to the remote switches listed should shutdown BGP ahead of time as well. As for a refund, whenever I run into a potential member looking for a service level agreement, I point out how all service agreements in the industry offer you a portion of what you paid depending on the duration of the outage. Since MICE has been free for the last 7 years, any portion would still be 0. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of John Unger Sent: Wednesday, April 26, 2017 6:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not make sense to me? One factor was that Cologix needs to be involved in the fiber moves from the old cabinet to the new cabinet big enough to hold the new switch, and if done at night, MICE would have had to pay for at least 3 full nights of overtime tech worker to be there coordinating the fiber moves. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Arista Round 4 Moves - FINAL

by Jeremy Lumby

I am planning on doing the final round of moves to the Arista switch on the afternoon of 5/9/2017 beginning at 1PM. The second route server will be moved to the new cabinet, as well as the following connections to the main switch will also be moved: Great Plains Acentek Consolidated Communications Long Lines HBC Savage UW System Radio Link LTD Broadband Integra Inteliquent Hoyos Consulting Visionary (Mammoth) Code42 Emergent HCMC Unggoy I suggest that all members listed above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Maintenance Complete

by Jeremy Lumby

The maintenance just completed. All ports are showing link, however I have not gone through a detailed check yet. Please shoot me an email if you see an issue before I do. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: US Internet to move to 100gig interface

by John McLaughlin

Welcome to the future fellas! On Tue, Apr 18, 2017 at 6:56 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Looks nice and clean on the MICE switch side as well. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Justin Krejci > *Sent:* Tuesday, April 18, 2017 6:13 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] US Internet to move to 100gig interface > > > > Cutover is complete. Everything looks good on my side and error free. > Please let me know if anyone runs into any problems to USI. > > I think this marks the first 100gig connection (member or otherwise) on > MICE, so that is cool! > > > ------------------------------ > > *From:* MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin > Krejci [JKrejci(a)USINTERNET.COM] > *Sent:* Tuesday, April 18, 2017 4:57 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] US Internet to move to 100gig interface > > I am going to be shutting down our BGP sessions on MICE in the next few > minutes so I can move my traffic over to the 100gig interface. I will > re-enable BGP afterwards, assuming all goes well. > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- *John McLaughlin* Major Accounts Manager States: MN, IA, SD, ND Mobile: 612-816-9024 jmclaughlin(a)arista.com 2016 Gartner Report <https://www.gartner.com/doc/reprints?id=1-36UFTCZ&ct=160517&st=sb>

April 18, 2017

Re: US Internet to move to 100gig interface

by Jeremy Lumby

Looks nice and clean on the MICE switch side as well. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Justin Krejci Sent: Tuesday, April 18, 2017 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] US Internet to move to 100gig interface Cutover is complete. Everything looks good on my side and error free. Please let me know if anyone runs into any problems to USI. I think this marks the first 100gig connection (member or otherwise) on MICE, so that is cool! From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin Krejci [JKrejci(a)USINTERNET.COM] Sent: Tuesday, April 18, 2017 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] US Internet to move to 100gig interface I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 18, 2017