- mice-discuss - lists.micemn.net

Re: MICE Graphs

by David Farmer

Thanks, it looks much better now. By the way, all I really wanted to know was that some one was working on it, and that it wasn't forgotten about. It's completely fine that it took a couple days to make happen. On Thu, Apr 27, 2017 at 3:35 PM, anthonyanderberg(a)nu-telecom.net < anthonyanderberg(a)nu-telecom.net> wrote: > Since so much changed I took the opportunity to audit the aggregate graph > - obviously I want to preserve our history without double-counting any > bits. My actual job also cut into my MICE efforts yesterday some too... > anyway I’ve got all the links updated on the Weathermap too - will slide > the icons around to reflect the current architecture next. > > > > Cheers, > > anthony > > > > *Anthony Anderberg* > > Sr. Systems Analyst > > [image: NUtel_email_logo_1] > > 320-234-5239 <(320)%20234-5239> > > anthonyanderberg(a)nu-telecom.net > > *www.nutelecom.net <http://www.nutelecom.net>* > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *David Farmer > *Sent:* Thursday, April 27, 2017 11:36 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] MICE Graphs > > > > I see the data there on the individual graphs, but the aggregate graph > doesn't seem to be including the new switch. > > > > On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> > wrote: > > On 04/26/2017 11:32 AM, David Farmer wrote: > > According to the graphs on the web page there was a network holocaust on > > MICE yesterday. Could someone please fix the graphs. > > Anthony is actively working on this now. The RRDs were copied last > night, so the data is preserved. > > -- > Richard > > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 <(612)%20626-0815> > Minneapolis, MN 55414-3029 Cell: 612-812-9952 <(612)%20812-9952> > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 28, 2017

Re: MICE Graphs

by anthonyanderberg＠nu-telecom.net

Since so much changed I took the opportunity to audit the aggregate graph - obviously I want to preserve our history without double-counting any bits. My actual job also cut into my MICE efforts yesterday some too... anyway I’ve got all the links updated on the Weathermap too - will slide the icons around to reflect the current architecture next. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of David Farmer Sent: Thursday, April 27, 2017 11:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Graphs I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com<mailto:rlaager@wiktel.com>> wrote: On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 27, 2017

Re: MICE Graphs

by Richard Laager

Anthony finished the new aggregate graph. I updated the website link. Our glory is restored! -- Richard

April 27, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Thu, Apr 27, 2017 at 12:00 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > Thanks for the great explanation. > > I wanted to reiterate a couple things: > > This ACL approach is standard practice at SIX. We know it works on the Arista switch. The plan is to use it there moving forward. Can confirm, method as outlined is sound and reasonable for MADIX and MKEIX, both 'semi distributed' and resemble MICE (conceptually, obv not in bits/sec scaling). -Tk

April 27, 2017

Re: Arista Round 3 Moves

by Richard Laager

Thanks for the great explanation. I wanted to reiterate a couple things: This ACL approach is standard practice at SIX. We know it works on the Arista switch. The plan is to use it there moving forward. -- Richard -- Richard

April 27, 2017

Re: MICE Graphs

by Richard Laager

Correct. Anthony is rebuilding the aggregate graph. -- Richard > On Apr 27, 2017, at 11:36, David Farmer <farmer(a)UMN.EDU> wrote: > > I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. > >> On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> wrote: >> On 04/26/2017 11:32 AM, David Farmer wrote: >> > According to the graphs on the web page there was a network holocaust on >> > MICE yesterday. Could someone please fix the graphs. >> >> Anthony is actively working on this now. The RRDs were copied last >> night, so the data is preserved. >> >> -- >> Richard > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 27, 2017

Re: MICE Graphs

by David Farmer

I see the data there on the individual graphs, but the aggregate graph doesn't seem to be including the new switch. On Wed, Apr 26, 2017 at 4:39 PM, Richard Laager <rlaager(a)wiktel.com> wrote: > On 04/26/2017 11:32 AM, David Farmer wrote: > > According to the graphs on the web page there was a network holocaust on > > MICE yesterday. Could someone please fix the graphs. > > Anthony is actively working on this now. The RRDs were copied last > night, so the data is preserved. > > -- > Richard > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 27, 2017

Re: Arista Round 3 Moves

by Justin Krejci

Forcing an issue is going to happen when the port goes down. The point is the ACL can make it more graceful for those directly affected and also those indirectly affected by the port going down by ensuring all BGP connections (MLPA and bilateral) are terminated with the directly affected members before the link goes down for the directly affected members. In this context: directly affected member = those whose path to the main fabric are going to be physically moved (includes downstream members on an extension switch that is being moved) indirectly affected member = everyone else who are not going to have their own path to the fabric go down I am not sure who would intentionally prefer a less graceful option of losing all/most MICE connectivity. The addition of very short lived ACL's on the exchange fabric ports for directly affected members does not preclude members from implementing their own proactive measures; people are certainly free to heed the announcement that they are going to be directly affected and pre-emptively shutdown all of their own BGP sessions at whatever time they want in advance and re-activate them anytime they choose afterwards be it minutes, hours or days later. The addition of the exchange enacted ACLs will help minimize the impact to the indirectly affected members who have bilateral peering with directly affected members. ________________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Mike Horwath [drechsau(a)GEEKS.ORG] Sent: Wednesday, April 26, 2017 6:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 10:51:34AM -0500, Doug McIntyre wrote: > On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > > Bilateral are unaffected and they can arrange things anyway with their > > peers. > > I think there is a lot more bilateral BGP going on across MICE than > you think. While I probably go talk to anybody, I have 11 > unilateral peers on MICE besides the two route servers. Yep, I am assuming a lot of bilateral BGP peers. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 27, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 09:34:21AM -0500, Andrew Hoyos wrote: > I???d disagree. The maintenance currently taking place affects more > than just the route servers. Plenty of people are doing bi-lateral > peering on MICE, and that *IS* affected by maintenance events like > these. Right - but that's for the bilateral peers to work on. Forcing the issue by adding an ACL that they may not have asked for seems kind of .. intruding. > Adding an ACL to the port ensures graceful shutdown/end of traffic, > rather than an abrupt drop and hold timer fun. I???d much rather > that someone running the maintenance and in control of the ultimate > link up/down events be the one deciding when things are > starting/ending and re-enabling traffic gracefully. I don't know if everyone accepts that. I'm playing devils advocate only. > > Clean shutdown of bird is easier, quicker, and will for sure make the > > multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. Bilateral users are already adding to their complication by having many more peers than just going multilateral. > Why break everyone and cause a total route server outage, when > it???s not necessary at all? Yesterday???s maintenance only > affected a portion of members. ACL???s on member ports would be the > cleanest way to minimize outage duration for all members with the > least impact to the IX as a whole. That member would have still been affected and would not have helped reduce the noise. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: MICE Graphs

by Richard Laager

On 04/26/2017 11:32 AM, David Farmer wrote: > According to the graphs on the web page there was a network holocaust on > MICE yesterday. Could someone please fix the graphs. Anthony is actively working on this now. The RRDs were copied last night, so the data is preserved. -- Richard

April 26, 2017

MICE Graphs

by David Farmer

According to the graphs on the web page there was a network holocaust on MICE yesterday. Could someone please fix the graphs. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 26, 2017

Re: Arista Round 3 Moves

by Richard Laager

Doug, Please do go ahead and expand passive BFD. Everyone on one route server followed by the second route server a few days later should be fine. -- Richard > On Apr 26, 2017, at 10:51, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I think there is a lot more bilateral BGP going on across MICE than you think. > While I probably go talk to anybody, I have 11 unilateral peers on MICE besides > the two route servers. > > I think BFD for BGP would do a lot more good than BGP filtering if people would > implement it. > > Speaking of that, I have passive BFD testing on route server #1 for quite some time. > I've heard of zero problems. > > I would like to go more widespread than the test people (ie. offer it > to every member connection on route server #1) > > Since it is passive, the member end drives it, if they do it, the > route server will do it, if they don't, nothing else happens. I'd also like > to roll out passive BFD as default for new members. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 09:19:39AM -0500, Mike Horwath wrote: > Bilateral are unaffected and they can arrange things anyway with their > peers. I think there is a lot more bilateral BGP going on across MICE than you think. While I probably go talk to anybody, I have 11 unilateral peers on MICE besides the two route servers. I think BFD for BGP would do a lot more good than BGP filtering if people would implement it. Speaking of that, I have passive BFD testing on route server #1 for quite some time. I've heard of zero problems. I would like to go more widespread than the test people (ie. offer it to every member connection on route server #1) Since it is passive, the member end drives it, if they do it, the route server will do it, if they don't, nothing else happens. I'd also like to roll out passive BFD as default for new members. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by Anton Kapela

On Wed, Apr 26, 2017 at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. Sadly, I can't thumbs-up this post as I can on FB... -Tk

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

I’ll also add, this is currently a draft with the IETF grow WG: https://tools.ietf.org/html/draft-ietf-grow-bgp-session-culling-01 (and one which I fully support) -- Andrew Hoyos hoyosa(a)gmail.com > On Apr 26, 2017, at 9:34 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: >> >> On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >>> I would suggest that perhaps we look into filtering BGP (tcp/179) >>> with an ACL prior to maintenance start on those specific ports being >>> moved. Many other IXs are doing this for maintenance as a way to >>> gracefully take things down, and let bilateral and RS sessions time >>> out without killing active traffic. As we've noticed, not all >>> members being moved are bothering to shut down sessions prior, which >>> causes impact to/from those members. (i.e.: >>> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) >> >> Don't even need ACLs. >> >> Just take down the route servers for the 2 hour period. >> >> Bilateral are unaffected and they can arrange things anyway with their >> peers. > > I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. > > Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. > I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > >> Adding another step to the process creates more complications as well, >> and another point of failure if you screw up along the way. > > Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. > Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > >> Clean shutdown of bird is easier, quicker, and will for sure make the >> multilateral peering not be further affected by bouncing repeatedly. > > Yes, great for MLPA, but not for bilateral. > > Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole. > > >

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 9:19 AM, Mike Horwath <drechsau(a)Geeks.ORG> wrote: > > On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: >> I would suggest that perhaps we look into filtering BGP (tcp/179) >> with an ACL prior to maintenance start on those specific ports being >> moved. Many other IXs are doing this for maintenance as a way to >> gracefully take things down, and let bilateral and RS sessions time >> out without killing active traffic. As we've noticed, not all >> members being moved are bothering to shut down sessions prior, which >> causes impact to/from those members. (i.e.: >> https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) > > Don't even need ACLs. > > Just take down the route servers for the 2 hour period. > > Bilateral are unaffected and they can arrange things anyway with their > peers. I’d disagree. The maintenance currently taking place affects more than just the route servers. Plenty of people are doing bi-lateral peering on MICE, and that *IS* affected by maintenance events like these. Adding an ACL to the port ensures graceful shutdown/end of traffic, rather than an abrupt drop and hold timer fun. I’d much rather that someone running the maintenance and in control of the ultimate link up/down events be the one deciding when things are starting/ending and re-enabling traffic gracefully. > Adding another step to the process creates more complications as well, > and another point of failure if you screw up along the way. Disagree, adding an ACL to a port is pretty trivial. Add (pre-existing) ACL to port 10 minutes before maintenance starts. Remove when complete. Script up into copy/paste thing with port numbers for bonus points and less changes of failure. > Clean shutdown of bird is easier, quicker, and will for sure make the > multilateral peering not be further affected by bouncing repeatedly. Yes, great for MLPA, but not for bilateral. Lastly, In this *specific* case, this presents issues with other members ports who are *NOT* affected by the maintenance and a loss of traffic for them if they are doing MLPA. Why break everyone and cause a total route server outage, when it’s not necessary at all? Yesterday’s maintenance only affected a portion of members. ACL’s on member ports would be the cleanest way to minimize outage duration for all members with the least impact to the IX as a whole.

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

> Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. It is worth pointing out that while I am on site doing the physical work, I interact with Doug, Anthony, Jay, and Richard to make sure things go smoothly, since each of them has their area of expertise. Jeremy

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 08:07:45AM -0500, Andrew Hoyos wrote: > I would suggest that perhaps we look into filtering BGP (tcp/179) > with an ACL prior to maintenance start on those specific ports being > moved. Many other IXs are doing this for maintenance as a way to > gracefully take things down, and let bilateral and RS sessions time > out without killing active traffic. As we've noticed, not all > members being moved are bothering to shut down sessions prior, which > causes impact to/from those members. (i.e.: > https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) Don't even need ACLs. Just take down the route servers for the 2 hour period. Bilateral are unaffected and they can arrange things anyway with their peers. Adding another step to the process creates more complications as well, and another point of failure if you screw up along the way. Clean shutdown of bird is easier, quicker, and will for sure make the multilateral peering not be further affected by bouncing repeatedly. > As an aside, thanks Jeremy for doing all this move work, it's not a > small effort and definitely appreciated. +1 -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by Mike Horwath

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years > this is just crazy? Nobody does maintenance during the day? Which industry? Because the Internet industry isn't 35 years old. Many people do maintenance during the day. So many. It's a 24x7 world now and for many businesses the time of day for work doesn't necessarily follow the sun. > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? Are you direct connected or are you via remote switch? If you are via remote switch then did your switch operator notify you? > Is MICE issuing credits for outages?? Planned don't count. And no SLA means .. no 'credit'. If you are via remote switch then you should take it up with them and not the community as a whole. But beyond that snark: this is volunteer work and as such Jeremy is already taking time out of his business day to do things for this group. Jeremy - you have my heartfelt thanks. You're doing this alone and you're doing this instead of taking the time to build your business further. I'd buy you a beer if we ever saw each other IRL. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, I surely did not mean to cause any feelings of un-appreciation for what you are doing. Just the timing of it and now I do understand the reasoning. Thanks and appreciate your time and efforts. John From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Dean Bahls Sent: Wednesday, April 26, 2017 8:39 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com<mailto:hoyosa@gmail.com>> wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 26, 2017

Re: Arista Round 3 Moves

by Dean Bahls

Along with the others who have chimed in so far, CNS also appreciates what Jeremy (and the rest of the volunteer team) has done on this migration! Thanks! Dean From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Johnston Sent: Wednesday, April 26, 2017 8:34 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On 2017-04-26 08:28, Ben Wiechman wrote: On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > wrote: As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

April 26, 2017

Re: Arista Round 3 Moves

by Colin Baker

On Wed, Apr 26, 2017 at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > Jeremy, why are we doing these moves during the day? This does not > make sense to me? Having been in this industry for over 35 years this > is just crazy? Nobody does maintenance during the day? > > Yesterday's maintenance caused our switch to have problems and we > were not even part of the move?? > > Is MICE issuing credits for outages?? I'll add that most MICE members are eyeballs whose peak usage occurs at night. Daytime is probably preferable for these networks. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net

April 26, 2017

Re: Arista Round 3 Moves

by Mike Johnston

On 2017-04-26 08:28, Ben Wiechman wrote: > > On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com > <mailto:hoyosa@gmail.com>> wrote: > > As an aside, thanks Jeremy for doing all this move work, it's not > a small effort and definitely appreciated. > > > I concur. This is greatly appreciated by us as well. Thank you. > I will also concur. Thank you Jeremy for your hard work! It is MUCH appreciated. Mike Johnston Wikstrom Telephone Company

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

I do agree that filtering BGP prior to the move is a great thing to do. I know we attempted it in the past, and we had some issues getting it to work properly. Now that we are on the Arista, I have tested it and it seems to work well. I plan on utilizing it on all future maintenance on the Arista switch. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Wednesday, April 26, 2017 8:08 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Ben Wiechman

On Wed, Apr 26, 2017 at 8:07 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > As an aside, thanks Jeremy for doing all this move work, it's not a small > effort and definitely appreciated. I concur. This is greatly appreciated by us as well. Thank you. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com

April 26, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

John, There are several factors as to why this is done during the day. It all goes back to your 35 years of experience in the industry. The fact is that over 99% of the industry is for profit. Not only has MICE always aimed for nonprofit status, they are also 100% volunteer based. I have been volunteering for MICE since they came to be 7 years ago. Also up until this last year MICE did not charge any fees, and now the fee is so minimal, and not even completely mandatory, it is just more than enough to cover the maintenance contract on the switch. When everything is 100% volunteer, it is difficult enough to get volunteers, let alone ones willing to show up in the middle of the night. Cologix was kind enough to offer to move all cross connects for free as long as it was done during normal business hours. Members always had the option to pay Cologix to move their cross connect in the middle of the night. As far as I know no one took Cologix up on that. I feel the main reason is since we are all multi-homed running BGP, as long as enough notice is given, BGP can be gracefully shutdown prior to the maintenance, and there are no real issues. I am sorry that your equipment did not perform as expected. The remote switch you are connected to was part of the maintenance, and the maintenance notification did state that all members connected to the remote switches listed should shutdown BGP ahead of time as well. As for a refund, whenever I run into a potential member looking for a service level agreement, I point out how all service agreements in the industry offer you a portion of what you paid depending on the duration of the outage. Since MICE has been free for the last 7 years, any portion would still be 0. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of John Unger Sent: Wednesday, April 26, 2017 6:52 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Re: Arista Round 3 Moves

by Andrew Hoyos

On Apr 26, 2017, at 6:52 AM, John Unger <john(a)VAULTAS.COM> wrote: > > Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? If you are worried about impact, shut down *your* MICE peering sessions to affected parties before the maintenance starts. I would suggest that perhaps we look into filtering BGP (tcp/179) with an ACL prior to maintenance start on those specific ports being moved. Many other IXs are doing this for maintenance as a way to gracefully take things down, and let bilateral and RS sessions time out without killing active traffic. As we've noticed, not all members being moved are bothering to shut down sessions prior, which causes impact to/from those members. (i.e.: https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf) A small minority of members are also doing BFD which can lessen impact here. If your gear supports it, I'd suggest looking into as a way to minimize impact of things like this. > Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? You're on the Mankato remote switch. That was listed in the maintenance notification as being affected. > Is MICE issuing credits for outages?? Surely, you're joking, right? -- As an aside, thanks Jeremy for doing all this move work, it's not a small effort and definitely appreciated. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby > Sent: Tuesday, April 25, 2017 1:20 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves > > The maintenance is now starting > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 15, 2017 6:03 PM > To: MICE Discuss > Subject: Arista Round 3 Moves > > I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: > > Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) > > I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington, MN 55425 > Main: 612-355-7740 x211 > Direct: 612-392-6814 > EFax: 952-873-7425 > jlumby(a)mnvoip.com -- Andrew Hoyos hoyosa(a)gmail.com

April 26, 2017

Re: Arista Round 3 Moves

by Doug McIntyre

On Wed, Apr 26, 2017 at 11:52:16AM +0000, John Unger wrote: > Jeremy, why are we doing these moves during the day? This does not make sense to me? One factor was that Cologix needs to be involved in the fiber moves from the old cabinet to the new cabinet big enough to hold the new switch, and if done at night, MICE would have had to pay for at least 3 full nights of overtime tech worker to be there coordinating the fiber moves. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 26, 2017

Re: Arista Round 3 Moves

by John Unger

Jeremy, why are we doing these moves during the day? This does not make sense to me? Having been in this industry for over 35 years this is just crazy? Nobody does maintenance during the day? Yesterday's maintenance caused our switch to have problems and we were not even part of the move?? Is MICE issuing credits for outages?? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Tuesday, April 25, 2017 1:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Arista Round 3 Moves The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Arista Round 4 Moves - FINAL

by Jeremy Lumby

I am planning on doing the final round of moves to the Arista switch on the afternoon of 5/9/2017 beginning at 1PM. The second route server will be moved to the new cabinet, as well as the following connections to the main switch will also be moved: Great Plains Acentek Consolidated Communications Long Lines HBC Savage UW System Radio Link LTD Broadband Integra Inteliquent Hoyos Consulting Visionary (Mammoth) Code42 Emergent HCMC Unggoy I suggest that all members listed above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 26, 2017

Maintenance Complete

by Jeremy Lumby

The maintenance just completed. All ports are showing link, however I have not gone through a detailed check yet. Please shoot me an email if you see an issue before I do. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: Arista Round 3 Moves

by Jeremy Lumby

The maintenance is now starting -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 15, 2017 6:03 PM To: MICE Discuss Subject: Arista Round 3 Moves I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 25, 2017

Re: US Internet to move to 100gig interface

by John McLaughlin

Welcome to the future fellas! On Tue, Apr 18, 2017 at 6:56 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Looks nice and clean on the MICE switch side as well. > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Justin Krejci > *Sent:* Tuesday, April 18, 2017 6:13 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] US Internet to move to 100gig interface > > > > Cutover is complete. Everything looks good on my side and error free. > Please let me know if anyone runs into any problems to USI. > > I think this marks the first 100gig connection (member or otherwise) on > MICE, so that is cool! > > > ------------------------------ > > *From:* MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin > Krejci [JKrejci(a)USINTERNET.COM] > *Sent:* Tuesday, April 18, 2017 4:57 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] US Internet to move to 100gig interface > > I am going to be shutting down our BGP sessions on MICE in the next few > minutes so I can move my traffic over to the 100gig interface. I will > re-enable BGP afterwards, assuming all goes well. > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- *John McLaughlin* Major Accounts Manager States: MN, IA, SD, ND Mobile: 612-816-9024 jmclaughlin(a)arista.com 2016 Gartner Report <https://www.gartner.com/doc/reprints?id=1-36UFTCZ&ct=160517&st=sb>

April 18, 2017

Re: US Internet to move to 100gig interface

by Jeremy Lumby

Looks nice and clean on the MICE switch side as well. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Justin Krejci Sent: Tuesday, April 18, 2017 6:13 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] US Internet to move to 100gig interface Cutover is complete. Everything looks good on my side and error free. Please let me know if anyone runs into any problems to USI. I think this marks the first 100gig connection (member or otherwise) on MICE, so that is cool! From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin Krejci [JKrejci(a)USINTERNET.COM] Sent: Tuesday, April 18, 2017 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] US Internet to move to 100gig interface I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 18, 2017

Re: US Internet to move to 100gig interface

by Justin Krejci

Cutover is complete. Everything looks good on my side and error free. Please let me know if anyone runs into any problems to USI. I think this marks the first 100gig connection (member or otherwise) on MICE, so that is cool! ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Justin Krejci [JKrejci(a)USINTERNET.COM] Sent: Tuesday, April 18, 2017 4:57 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] US Internet to move to 100gig interface I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well. ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 18, 2017

US Internet to move to 100gig interface

by Justin Krejci

I am going to be shutting down our BGP sessions on MICE in the next few minutes so I can move my traffic over to the 100gig interface. I will re-enable BGP afterwards, assuming all goes well.

April 18, 2017

Re: Northern Lights Gigapop AS57 Maintenance Notice for 4/22/17

by Colin Murphy

Hi Richard, We will be doing hardware upgrades on 3 routers starting at 6am until we're finished. I reserved the whole day just in case. The specific router that connects to mice will down approximately 6 am to 8 am. I wasn't planning moving to the Arista switch during this window, but I'd be open to it if we can make it work logistically. Thanks, Colin Sent from my mobile phone On Apr 17, 2017 11:42 AM, "Richard Laager" <rlaager(a)wiktel.com> wrote: On 04/17/2017 08:56 AM, Colin Murphy wrote: > The Northern Lights Gigapop will be performing router maintenance on > 4/22/17 in a window from 6 am to 11 PM. A couple questions: 1) Is this all day window correct, or should one of the AM/PM be switched? 2) Is the GigaPoP also moving to the MICE Arista switch during this window? -- Richard

April 17, 2017

Re: Northern Lights Gigapop AS57 Maintenance Notice for 4/22/17

by Richard Laager

On 04/17/2017 08:56 AM, Colin Murphy wrote: > The Northern Lights Gigapop will be performing router maintenance on > 4/22/17 in a window from 6 am to 11 PM. A couple questions: 1) Is this all day window correct, or should one of the AM/PM be switched? 2) Is the GigaPoP also moving to the MICE Arista switch during this window? -- Richard

April 17, 2017

Northern Lights Gigapop AS57 Maintenance Notice for 4/22/17

by Colin Murphy

The Northern Lights Gigapop will be performing router maintenance on 4/22/17 in a window from 6 am to 11 PM. During this maintenance window you can expect one 2 to 3 hour service outage if you are not redundantly connected to the Gigapop. If your service to the Northern Lights Gigapop has not restored by the end of the maintenance window please send an email to help(a)umn.edu with your BGP AS number and a description on the issue and will address it as soon as possible. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Help Desk: help(a)umn.edu, 612-301-4357

April 17, 2017

Arista Round 3 Moves

by Jeremy Lumby

I am planning on doing round 3 of moves to the Arista switch on the afternoon of 4/25/17 beginning at 1PM. The following connections to the main switch will be moved: Netflix (Will simultaneously upgrade to 2x100G) Google Hurricane Electric CNS (remote switch feed) Neutral Path (remote switch feed) Vast Broadband DCN Charter Broadband Visions Jaguar Yahoo SDN TDS CloudFlare Fastly Mankato Networks (remote switch feed) MN VoIP (remote switch feed) I suggest that all members listed above, as well as the members connected to the remote switches above shutdown BGP by 12:00PM CST to avoid dropped packets while being moved. Please leave the interface up so that we can test link, as well as the ability to ping the interface IP. Additional notice will be sent immediately before and after the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com

April 15, 2017

Re: Comcast NOC

by Ben Wiechman

Thanks. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Thu, Apr 13, 2017 at 6:17 PM, Ryan Goldberg <RGoldberg(a)compudyne.com> wrote: > 877-543-3961 <(877)%20543-3961> > > > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Ben Wiechman > *Sent:* Thursday, April 13, 2017 6:12 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Comcast NOC > > > > Does someone have a contact number for the Comcast NOC? The one we have > recorded puts us into a circular queue of insanity. > > > > Ben Wiechman > > Network Engineer IV > Direct: 320.256.0184 <(320)%20256-0184> > Cell: 320.247.3224 <(320)%20247-3224> > > ben.wiechman(a)arvig.com > > > > 150 Second Street SW | Perham, MN 56573 | arvig.com > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 13, 2017

Re: Comcast NOC

by Ryan Goldberg

877-543-3961 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Ben Wiechman Sent: Thursday, April 13, 2017 6:12 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Comcast NOC Does someone have a contact number for the Comcast NOC? The one we have recorded puts us into a circular queue of insanity. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> 150 Second Street SW | Perham, MN 56573 | arvig.com<http://arvig.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 13, 2017

Comcast NOC

by Ben Wiechman

Does someone have a contact number for the Comcast NOC? The one we have recorded puts us into a circular queue of insanity. Ben Wiechman Network Engineer IV Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman(a)arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com

April 13, 2017

Akamai at MICE

by DeLong, Owen

I have turned Akamai back on at MICE. A few sessions did not come up. If you are having difficulty with your Akamai peering via MICE, please feel free to reach out to me directly and I will be happy to work with you to resolve any issues. Owen

April 12, 2017

Re: Arista Round 2 Moves

by Jeremy Lumby

The maintenance is complete. I am still checking some details, however if you see an issue before I do please email it to me ASAP. >From : Jeremy Lumby [jlumby(a)mnvoip.com] To : MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, April 11 2017 09:01:00 Just a reminder that the moves described below will begin today after 1PM Central time. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 01, 2017 1:03 PM To: MICE Discuss Subject: Arista Round 2 Moves Everyone, Everything has been working smoothly with the test networks on the Arista switch for the last couple weeks. I feel confident in the switch, and am ready for a larger round 2 of moving members. I am planning on doing the round 2 moves on Tuesday April 11 in the afternoon. I would request that all members being moved shutdown BGP prior to 12:00 PM CST, however please leave the interface up so we can make sure link returns, and we can ping the interface IP. In addition to moving members, I plan on moving both copper Juniper 4200 switches, as well as the looking glass, and Route Server #1. The following members will be moved: -All 1G connections to the main switch (both copper, and fiber) ipHouse Implex OneNeck Supranet Nextera North Star Fiber Atomic Data Advanced Integrated Tech IVDesk OneNetUSA Verus Corporation 5 Nines CDW/Berbee Velocity Telephone Paragon -Arvig (will simultaneously upgrade to a 2x10G LAG) -Akamai -Amazon Once the moves are complete, I will email the list to let everyone know. Jeremy

April 11, 2017

Re: Arista Round 2 Moves

by Jeremy Lumby

The maintenance described below will be beginning in the next 5 minutes Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 612.355.7740 Main 612.392.6814 Direct 952.873.7425 EFax jlumby(a)mnvoip.com -------- Original message -------- From: Jeremy Lumby <jlumby(a)mnvoip.com> Date: 4/11/17 9:01 AM (GMT-06:00) To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Arista Round 2 Moves _____ >From : Jeremy Lumby [jlumby(a)mnvoip.com] To : MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, April 11 2017 09:01:00 Just a reminder that the moves described below will begin today after 1PM Central time. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Saturday, April 01, 2017 1:03 PM To: MICE Discuss Subject: Arista Round 2 Moves Everyone, Everything has been working smoothly with the test networks on the Arista switch for the last couple weeks. I feel confident in the switch, and am ready for a larger round 2 of moving members. I am planning on doing the round 2 moves on Tuesday April 11 in the afternoon. I would request that all members being moved shutdown BGP prior to 12:00 PM CST, however please leave the interface up so we can make sure link returns, and we can ping the interface IP. In addition to moving members, I plan on moving both copper Juniper 4200 switches, as well as the looking glass, and Route Server #1. The following members will be moved: -All 1G connections to the main switch (both copper, and fiber) ipHouse Implex OneNeck Supranet Nextera North Star Fiber Atomic Data Advanced Integrated Tech IVDesk OneNetUSA Verus Corporation 5 Nines CDW/Berbee Velocity Telephone Paragon -Arvig (will simultaneously upgrade to a 2x10G LAG) -Akamai -Amazon Once the moves are complete, I will email the list to let everyone know. Jeremy

April 11, 2017

Re: Akamai peer

by DeLong, Owen

Really?… I’ll look into this. Unfortunately, I will have to wait until the move is complete as currently, the number of prefixes we are advertising to MICE is 0. Owen > On Apr 11, 2017, at 05:00 , Frank Bulk <fbulk(a)mypremieronline.com> wrote: > > Does anyone happen to know why the number of prefixes that Akamai is advertising in its direct peer is continually bouncing back and forth between 16 and 17 prefixes since Monday at 4:40 pm or so? > > Frank

April 11, 2017

Re: Arista Round 2 Moves

by DeLong, Owen

I have shut down all MICE BGP sessions for Akamai in preparation for this move. The interface remains up and the router should remain ping-able if the circuit is live. Please feel free to contact me at 408-890-7992 should any issues arise. Also, please contact me at that number when you are ready for Akamai to restore service to the IX. Owen > On Apr 11, 2017, at 07:01 , Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Just a reminder that the moves described below will begin today after 1PM Central time. > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Saturday, April 01, 2017 1:03 PM > To: MICE Discuss > Subject: Arista Round 2 Moves > > Everyone, > > Everything has been working smoothly with the test networks on the Arista switch for the last couple weeks. I feel confident in the switch, and am ready for a larger round 2 of moving members. I am planning on doing the round 2 moves on Tuesday April 11 in the afternoon. I would request that all members being moved shutdown BGP prior to 12:00 PM CST, however please leave the interface up so we can make sure link returns, and we can ping the interface IP. In addition to moving members, I plan on moving both copper Juniper 4200 switches, as well as the looking glass, and Route Server #1. The following members will be moved: > > -All 1G connections to the main switch (both copper, and fiber) > ipHouse > Implex > OneNeck > Supranet > Nextera > North Star Fiber > Atomic Data > Advanced Integrated Tech > IVDesk > OneNetUSA > Verus Corporation > 5 Nines > CDW/Berbee > Velocity Telephone > Paragon > -Arvig (will simultaneously upgrade to a 2x10G LAG) > -Akamai > -Amazon > > Once the moves are complete, I will email the list to let everyone know. > > Jeremy

April 11, 2017

Re: To the person(s) in charge of the PTRs

by Doug McIntyre

On Tue, Apr 11, 2017 at 11:58:35AM +0000, Frank Bulk wrote: > Note that 206.108.255.84 currently has a PTR of AS29040.micemn.net, but it should probably be AS20940.micemn.net. Sorry for the typo, should be all fixed up. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 11, 2017