- mice-discuss - lists.micemn.net

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Fri, Oct 28, 2016 at 09:45:00AM -0500, Jeremy Lumby wrote: > I do believe the result of that discussion is that we will not offer > service level agreements, and that whenever a new member agreement > was drafted, it would essentially say that we cannot guarantee > uptime, or resolution response time, and the solution would be that > the member would responsible to disconnect themselves if > quality/reliability issues arose that the member found unacceptable. That's also just fine. You can have some defined parameters about scheduled work (time, announcements, etc) yet still skip an SLA. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Thu, Oct 27, 2016 at 02:07:39PM -0500, Doug McIntyre wrote: > On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote: > > > EX4550 is 32 ports base, 48 ports with addon cards. > > It is 32 + 8 ports if you want to use the back expansion slot to house > a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) > (technically the VC card is slotted into the 4500, but I've never seen one > bought without it there). So the EX4500 can do more ports than a EX4550. Ooh I didn't know that! Thanks! > The QFX5200-32C can get into newer technologies like adding 25G/50G > and 100G capable ports. Again using breakout cables to breakout ports > into the "slower" speeds we are running, and can get into the range of > 128 x 10GB ports with breakout cables in 1U. > > The QFX10002-36Q is much like the above, and can get into 144 x 10G > ports with breakout cables, but I know the price on this switch pushes > now into 6 figures. I have to assume that regardless they are expensive. If only that non-profit thing were in place *today*. A lot of work and I don't envy anyone having to do it. But I, and I don't think I am alone, am not donating money until there is a formal organization wrapped up in pretty bows. I have faith, though. This is coming and there will be much celebration. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Tue, Oct 25, 2016 at 01:56:32PM -0500, Richard Laager wrote: > 3) asking for a MICE-TECH and/or MICE-OPS list be created (depending > on whether we have distinct groups and what those people want). I > expect at least one list will be closed to members only. Give the word and it can be done. Hell both could be done though they seem a little redundant in naming. :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Jeremy Lumby

I do believe the result of that discussion is that we will not offer service level agreements, and that whenever a new member agreement was drafted, it would essentially say that we cannot guarantee uptime, or resolution response time, and the solution would be that the member would responsible to disconnect themselves if quality/reliability issues arose that the member found unacceptable. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Horwath Sent: Friday, October 28, 2016 9:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT On Tue, Oct 25, 2016 at 09:58:06AM -0500, Colin Baker wrote: > I remember some discussion in a MICE UG meeting a few years back that > some larger networks may have strict requirements that must be met in > order to connect to the exchange. We've added some big content > providers recently, which I'm thrilled to see, but I think establishing > some defined and documented standards can only help continue the growth. Confirm and agree. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Tue, Oct 25, 2016 at 10:10:00AM -0500, Jeremy Lumby wrote: > I agree standards would help, however I know that my personal > donations to MICE would probably have been far less if I was forced > to do it on a corporate standard schedule, and lets remember > donations like that got us here, and I would prefer not to lose > donors over standards, and I do not feel membership will be impacted > because of lack of standards since all of our current members have > gotten along quite well all this time without them. These aren't corporate standard schedules. While I am now on the outside of the service provider world, and no longer a direct member of MICE in any capacity (mailing list doens't count), I do feel that we should have something defined and adhered to for the benefit of the group as a whole. We didn't get here all willy nilly, using 1994 ISP style operations. Should there be defined announcement windows for scheduled maintenance? I think most everyone would agree that would be a yes. Should there be notifications of unscheduled outages, maintenance, lunch changes? Yes but this would be looser for timeframes. In the heat of the moment would you rather see work done to further forward the repair or would you like notifications every 15 minutes that things are still broken? Tough stuff.. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Tue, Oct 25, 2016 at 09:58:06AM -0500, Colin Baker wrote: > I remember some discussion in a MICE UG meeting a few years back that > some larger networks may have strict requirements that must be met in > order to connect to the exchange. We've added some big content > providers recently, which I'm thrilled to see, but I think establishing > some defined and documented standards can only help continue the growth. Confirm and agree. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 28, 2016

BGP with BFD support

by Doug McIntyre

We had been testing BFD support for BGP in the last few days, and I specificly watched it during the maintenance last night. I currently have route server 1 configured for passive BFD for a couple members, and that has been working just fine as far as I can tell for myself. I'm ready to open it up to any member that wants to test, just let me know your ASN and that you want passive BFD setup, and I can get that going on route-server 1. I'll do IPv4 and IPv6 sessions both. I assume everybody wants more of a gradual test. Otherwise I could enable it on route-server 2 as well. Finally, since this is a passive setting until triggered.. Should this get applied for every member by default? As long as the member side doesn't trigger it, it sits dormant. I could see that being part of the default template rollout for everybody, as well as every member retroactively, so it is totally controlled by what the member wants to do. If they want it, they do it. If they don't care to, it just sits passive not doing it. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by David Farmer

On Thu, Oct 27, 2016 at 2:07 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote: > > On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote: > > > There aren't enough ports on the 4550 to swap it in we would need > > > two. We are due for a fabric upgrade. > > Correct, while one could be brought in with additional cards in both, > that would barely cover us with zero future growth without adding > in a 2nd one. The U of MN expressed interest in a 40Gbps connection, > and doing just one switch would knock that out of contention, (or > we keep the EX4500 and run it knowing that it is EOL, and software will > probably stop being available within 14 months). > Don't worry about a 40G for us I'm looking past that now to 100G. I think the priority is to stabilize what we have and add a 100G/50G/40G/25G capability soon next two or three months. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Levi Pederson

Doug, We've just finished some research into 10Gig + 40 to 100Gig Density mapping in Juniper. I'll collect my notes and send them to the list with my thoughts. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Thu, Oct 27, 2016 at 2:07 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote: > > On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote: > > > There aren't enough ports on the 4550 to swap it in we would need > > > two. We are due for a fabric upgrade. > > Correct, while one could be brought in with additional cards in both, > that would barely cover us with zero future growth without adding > in a 2nd one. The U of MN expressed interest in a 40Gbps connection, > and doing just one switch would knock that out of contention, (or > we keep the EX4500 and run it knowing that it is EOL, and software will > probably stop being available within 14 months). > > > EX4550 is 32 ports base, 48 ports with addon cards. > > It is 32 + 8 ports if you want to use the back expansion slot to house > a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) > (technically the VC card is slotted into the 4500, but I've never seen one > bought without it there). So the EX4500 can do more ports than a EX4550. > > So, the back slot on a 4550 could be another 8x10G, but that loses the > VC port, which means we then have to take multiples of 10G ports on > both switches to devote to VC instead, probably at least 4 x 10GB on > both switches, which is a losing of 8 ports either way. > > Effectively making the stacked EX4550 only 40 ports instead of the 48 ports > on the 4500. > > > Is the current EX4500 running with 40 ports or 48? > > It has 48 ports, with 47 ports plugged in. (didn't count how many as lit, > but we just did an inventory and I know how few empty ports we have). > > The EX4550 is the direct replacement for the EX4500. I've run multiple > stacks of them myself, and they are solid boxes. > > Juniper also has newer switches at higher price points (and not as > easy upgrade path). The EX4600 can do 72 x 10GB in 1U. It has 40 fixed > 10G, 4 x 40G. BUT no VC ports. You use the 40G ports for VC > functions. To get the 72 port density, you'd add in a 14 x 40G card, > and any and each 40G port can be broken out to 4 x 10G port with breakout > cables. So, stacking the EX4600's typically takes out the fixed 4x40G > ports with the QSFP+ DAC cabling up and down, but adding on the 14 x 40G > card > brings back many ports and capabilities. > The EX4600 won't VC with the EX4550 so it becomes another stack connected > in. > > The QFX5200-32C can get into newer technologies like adding 25G/50G > and 100G capable ports. Again using breakout cables to breakout ports > into the "slower" speeds we are running, and can get into the range of > 128 x 10GB ports with breakout cables in 1U. > > The QFX10002-36Q is much like the above, and can get into 144 x 10G > ports with breakout cables, but I know the price on this switch pushes > now into 6 figures. > > > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote: > On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote: > > There aren't enough ports on the 4550 to swap it in we would need > > two. We are due for a fabric upgrade. Correct, while one could be brought in with additional cards in both, that would barely cover us with zero future growth without adding in a 2nd one. The U of MN expressed interest in a 40Gbps connection, and doing just one switch would knock that out of contention, (or we keep the EX4500 and run it knowing that it is EOL, and software will probably stop being available within 14 months). > EX4550 is 32 ports base, 48 ports with addon cards. It is 32 + 8 ports if you want to use the back expansion slot to house a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) (technically the VC card is slotted into the 4500, but I've never seen one bought without it there). So the EX4500 can do more ports than a EX4550. So, the back slot on a 4550 could be another 8x10G, but that loses the VC port, which means we then have to take multiples of 10G ports on both switches to devote to VC instead, probably at least 4 x 10GB on both switches, which is a losing of 8 ports either way. Effectively making the stacked EX4550 only 40 ports instead of the 48 ports on the 4500. > Is the current EX4500 running with 40 ports or 48? It has 48 ports, with 47 ports plugged in. (didn't count how many as lit, but we just did an inventory and I know how few empty ports we have). The EX4550 is the direct replacement for the EX4500. I've run multiple stacks of them myself, and they are solid boxes. Juniper also has newer switches at higher price points (and not as easy upgrade path). The EX4600 can do 72 x 10GB in 1U. It has 40 fixed 10G, 4 x 40G. BUT no VC ports. You use the 40G ports for VC functions. To get the 72 port density, you'd add in a 14 x 40G card, and any and each 40G port can be broken out to 4 x 10G port with breakout cables. So, stacking the EX4600's typically takes out the fixed 4x40G ports with the QSFP+ DAC cabling up and down, but adding on the 14 x 40G card brings back many ports and capabilities. The EX4600 won't VC with the EX4550 so it becomes another stack connected in. The QFX5200-32C can get into newer technologies like adding 25G/50G and 100G capable ports. Again using breakout cables to breakout ports into the "slower" speeds we are running, and can get into the range of 128 x 10GB ports with breakout cables in 1U. The QFX10002-36Q is much like the above, and can get into 144 x 10G ports with breakout cables, but I know the price on this switch pushes now into 6 figures. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Reid Fishler

Thanks, team! Reid On Oct 27, 2016 2:29 PM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote: > > i think we owe you one, Doug > > > > thanks! > > No problem. Anthony was there too checking out things and helping, > so thanks out to him too. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Levi Pederson

Kudos to Anthony as well. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Thu, Oct 27, 2016 at 1:29 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote: > > i think we owe you one, Doug > > > > thanks! > > No problem. Anthony was there too checking out things and helping, > so thanks out to him too. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote: > i think we owe you one, Doug > > thanks! No problem. Anthony was there too checking out things and helping, so thanks out to him too. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: Peering with research orgs

by DeLong, Owen

I’m not entirely sure this is true in all cases. Owen On Oct 26, 2016, at 9:28 PM, Reid Fishler <rfishler(a)he.net<mailto:rfishler@he.net>> wrote: Just keep in mind we don't want to give away data that people are going to sell...unless we get something from it. Reid On Thu, Oct 27, 2016 at 12:15 AM, Richard Laager <rlaager(a)wiktel.com<mailto:rlaager@wiktel.com>> wrote: On 10/26/2016 11:07 PM, Frank Bulk wrote: > Does MICE have a policy regarding peering its router reflectors with research/data gathering orgs like Cymru, Dyn, Norse, ThousandEyes, QRator, etc. AFAIK, no policy. Somewhat related: I peer with BGPmon.net<http://bgpmon.net>, which is not on your list. Apparently they have been acquired by OpenDNS since I last looked at their website. I'm not aware of Cymru having a data gathering thing. Do you have a link? I peer with their bogons, UTRS, and IXP services. The IXP one was just added today actually: https://www.team-cymru.org/IXP/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.team-2Dcymru.org_I…> When I turned it up, I saw they were missing the IP ranges used by MICE and several others, so I let them know. -- Richard -- Reid Fishler Director Hurricane Electric +1-510-580-4178 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Ryan Goldberg

Having been through exact same scenario with one of our datacenter stacks, I feel yah. Thank you On Oct 27, 2016, at 8:53 AM, Levi Pederson <levipederson(a)MANKATONETWORKS.NET<mailto:levipederson@MANKATONETWORKS.NET>> wrote: Doug, Thanks Doug for all the work and effort. Levi Pederson Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net<mailto:levipederson@mankatonetworks.net> [http://www.mankatonetworks.com/images/mn_logo_email.png] On Thu, Oct 27, 2016 at 8:42 AM, David Farmer <farmer(a)umn.edu<mailto:farmer@umn.edu>> wrote: +1 Thanks for the good work On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com<mailto:mike@haven2.com>> wrote: i think we owe you one, Doug thanks! m > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net<mailto:merlyn@iphouse.net>> wrote: > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >>> We're going to upgrade the core MICE 10G switches coming up this week >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> support has run out again, and JTAC will not help us (even at a pay >> per-incidence event). In fact, since the EX4500 has reached EOL, (although >> not EOM), and we let the contract expire, Juniper says we will be unable to >> renew maintenance support on the EX4500 at all. >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> If it comes down to it, I should have non-production hardware that could >> temporarily go in. > > Okay, I'm going to have to drive in. > > Right now, the state of things. > > The primary boot partition on the EX4500 took a digger, so we have bad > hardware. I was able to boot off the backup boot partition on the > EX4500, but this is still the older software version since I was in > the middle of the upgrade and the upgrade process didn't get to the > 2nd partition to upgrade it yet. > > The virtual chassis is now split version, which means VC won't work. > > Furthmore, networking is not working on either unit, even to the OOB. > > I am going to have to drive in to be physically in front of the box. > > I have two plans of attack. I will put the install image on a USB > drive and load it off USB onto the unit. IIRC, having done this > before long ago, this takes *ages* to do. > > If I can get both units happy with the same software versions, VC can > reestablish, and things can go along. > > I also have two EX4550's not-in-production right now, I'll bring them > along just in case. If things are really to the point of no return on > the EX4500, I can put in one or two and bring them up, restoring the > saved configs onto them. Of course, things will have to shuffle a little > then. This will delay me a bit in unmounting them from where they are, > but I think it is a safe backup plan. > > Another option would be to make the EX4550 standalone, but I have to > unhook the VC cables before this can happen, and it only helps along > a portion of the members. I don't think this is worthwhile to pursue. > > So, it'll be another hour before I can get downtown to my place, get > hardware, and get over to 511. > > -- > Doug McIntyre <merlyn(a)iphouse.net<mailto:merlyn@iphouse.net>> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades WI:(715) 598-4284<tel:%28715%29%20598-4284> MN:(651) 647-6109<tel:%28651%29%20647-6109> Toll Free:(800) 896-0907<tel:%28800%29%20896-0907> Fax:(866)-280-2356<tel:%28866%29-280-2356> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815<tel:612-626-0815> Minneapolis, MN 55414-3029 Cell: 612-812-9952<tel:612-812-9952> =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Levi Pederson

Doug, Thanks Doug for all the work and effort. *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Thu, Oct 27, 2016 at 8:42 AM, David Farmer <farmer(a)umn.edu> wrote: > +1 Thanks for the good work > > On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > >> i think we owe you one, Doug >> >> thanks! >> >> m >> >> > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> > >> > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >> >>> We're going to upgrade the core MICE 10G switches coming up this week >> >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 >> October. >> >> >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> >> support has run out again, and JTAC will not help us (even at a pay >> >> per-incidence event). In fact, since the EX4500 has reached EOL, >> (although >> >> not EOM), and we let the contract expire, Juniper says we will be >> unable to >> >> renew maintenance support on the EX4500 at all. >> >> >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> >> >> If it comes down to it, I should have non-production hardware that >> could >> >> temporarily go in. >> > >> > Okay, I'm going to have to drive in. >> > >> > Right now, the state of things. >> > >> > The primary boot partition on the EX4500 took a digger, so we have bad >> > hardware. I was able to boot off the backup boot partition on the >> > EX4500, but this is still the older software version since I was in >> > the middle of the upgrade and the upgrade process didn't get to the >> > 2nd partition to upgrade it yet. >> > >> > The virtual chassis is now split version, which means VC won't work. >> > >> > Furthmore, networking is not working on either unit, even to the OOB. >> > >> > I am going to have to drive in to be physically in front of the box. >> > >> > I have two plans of attack. I will put the install image on a USB >> > drive and load it off USB onto the unit. IIRC, having done this >> > before long ago, this takes *ages* to do. >> > >> > If I can get both units happy with the same software versions, VC can >> > reestablish, and things can go along. >> > >> > I also have two EX4550's not-in-production right now, I'll bring them >> > along just in case. If things are really to the point of no return on >> > the EX4500, I can put in one or two and bring them up, restoring the >> > saved configs onto them. Of course, things will have to shuffle a little >> > then. This will delay me a bit in unmounting them from where they are, >> > but I think it is a safe backup plan. >> > >> > Another option would be to make the EX4550 standalone, but I have to >> > unhook the VC cables before this can happen, and it only helps along >> > a portion of the members. I don't think this is worthwhile to pursue. >> > >> > So, it'll be another hour before I can get downtown to my place, get >> > hardware, and get over to 511. >> > >> > -- >> > Doug McIntyre <merlyn(a)iphouse.net> >> > ~.~ ipHouse ~.~ >> > Network Engineer/Provisioning/Jack of all Trades >> >> WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: >> (866)-280-2356 >> > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by David Farmer

+1 Thanks for the good work On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > i think we owe you one, Doug > > thanks! > > m > > > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > >>> We're going to upgrade the core MICE 10G switches coming up this week > >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 > October. > >> > >> The upgrade is NOT going as planned, and we discovered our maintenance > >> support has run out again, and JTAC will not help us (even at a pay > >> per-incidence event). In fact, since the EX4500 has reached EOL, > (although > >> not EOM), and we let the contract expire, Juniper says we will be > unable to > >> renew maintenance support on the EX4500 at all. > >> > >> I'm looking at ways to bring the VC cluster fully back online now. > >> > >> If it comes down to it, I should have non-production hardware that could > >> temporarily go in. > > > > Okay, I'm going to have to drive in. > > > > Right now, the state of things. > > > > The primary boot partition on the EX4500 took a digger, so we have bad > > hardware. I was able to boot off the backup boot partition on the > > EX4500, but this is still the older software version since I was in > > the middle of the upgrade and the upgrade process didn't get to the > > 2nd partition to upgrade it yet. > > > > The virtual chassis is now split version, which means VC won't work. > > > > Furthmore, networking is not working on either unit, even to the OOB. > > > > I am going to have to drive in to be physically in front of the box. > > > > I have two plans of attack. I will put the install image on a USB > > drive and load it off USB onto the unit. IIRC, having done this > > before long ago, this takes *ages* to do. > > > > If I can get both units happy with the same software versions, VC can > > reestablish, and things can go along. > > > > I also have two EX4550's not-in-production right now, I'll bring them > > along just in case. If things are really to the point of no return on > > the EX4500, I can put in one or two and bring them up, restoring the > > saved configs onto them. Of course, things will have to shuffle a little > > then. This will delay me a bit in unmounting them from where they are, > > but I think it is a safe backup plan. > > > > Another option would be to make the EX4550 standalone, but I have to > > unhook the VC cables before this can happen, and it only helps along > > a portion of the members. I don't think this is worthwhile to pursue. > > > > So, it'll be another hour before I can get downtown to my place, get > > hardware, and get over to 511. > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: > (866)-280-2356 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote: > There aren't enough ports on the 4550 to swap it in we would need > two. We are due for a fabric upgrade. EX4500 is 40 ports base, 48 ports with addon cards. EX4550 is 32 ports base, 48 ports with addon cards. Is the current EX4500 running with 40 ports or 48? Could a EX4550 be brought into the fabric with addon cards to match up? (not disagreeing about fabric upgrade though will be much easier once non-profit status is done and some of us can donate cleanly) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > If it comes down to it, I should have non-production hardware that > could temporarily go in. Haven't read the rest of the thread yet. Thanks for your work, Doug. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Jason Hanke

There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade. On Oct 27, 2016 7:23 AM, "Ben Wiechman" <ben.wiechman(a)arvig.com> wrote: > What would be the proposed solution? Replace the failing EX4500 with a > 4550? If so, what does one of those run? > > > Ben Wiechman > Network Engineer IV | Arvig > Direct: 320.256.0184 > Cell: 320.247.3224 > Office: 320.256.7471 > ben.wiechman(a)arvig.com > > On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > >> i think we owe you one, Doug >> >> thanks! >> >> m >> >> > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> > >> > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >> >>> We're going to upgrade the core MICE 10G switches coming up this week >> >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 >> October. >> >> >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> >> support has run out again, and JTAC will not help us (even at a pay >> >> per-incidence event). In fact, since the EX4500 has reached EOL, >> (although >> >> not EOM), and we let the contract expire, Juniper says we will be >> unable to >> >> renew maintenance support on the EX4500 at all. >> >> >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> >> >> If it comes down to it, I should have non-production hardware that >> could >> >> temporarily go in. >> > >> > Okay, I'm going to have to drive in. >> > >> > Right now, the state of things. >> > >> > The primary boot partition on the EX4500 took a digger, so we have bad >> > hardware. I was able to boot off the backup boot partition on the >> > EX4500, but this is still the older software version since I was in >> > the middle of the upgrade and the upgrade process didn't get to the >> > 2nd partition to upgrade it yet. >> > >> > The virtual chassis is now split version, which means VC won't work. >> > >> > Furthmore, networking is not working on either unit, even to the OOB. >> > >> > I am going to have to drive in to be physically in front of the box. >> > >> > I have two plans of attack. I will put the install image on a USB >> > drive and load it off USB onto the unit. IIRC, having done this >> > before long ago, this takes *ages* to do. >> > >> > If I can get both units happy with the same software versions, VC can >> > reestablish, and things can go along. >> > >> > I also have two EX4550's not-in-production right now, I'll bring them >> > along just in case. If things are really to the point of no return on >> > the EX4500, I can put in one or two and bring them up, restoring the >> > saved configs onto them. Of course, things will have to shuffle a little >> > then. This will delay me a bit in unmounting them from where they are, >> > but I think it is a safe backup plan. >> > >> > Another option would be to make the EX4550 standalone, but I have to >> > unhook the VC cables before this can happen, and it only helps along >> > a portion of the members. I don't think this is worthwhile to pursue. >> > >> > So, it'll be another hour before I can get downtown to my place, get >> > hardware, and get over to 511. >> > >> > -- >> > Doug McIntyre <merlyn(a)iphouse.net> >> > ~.~ ipHouse ~.~ >> > Network Engineer/Provisioning/Jack of all Trades >> >> WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: >> (866)-280-2356 >> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Ben Wiechman

What would be the proposed solution? Replace the failing EX4500 with a 4550? If so, what does one of those run? Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman(a)arvig.com On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > i think we owe you one, Doug > > thanks! > > m > > > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > >>> We're going to upgrade the core MICE 10G switches coming up this week > >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 > October. > >> > >> The upgrade is NOT going as planned, and we discovered our maintenance > >> support has run out again, and JTAC will not help us (even at a pay > >> per-incidence event). In fact, since the EX4500 has reached EOL, > (although > >> not EOM), and we let the contract expire, Juniper says we will be > unable to > >> renew maintenance support on the EX4500 at all. > >> > >> I'm looking at ways to bring the VC cluster fully back online now. > >> > >> If it comes down to it, I should have non-production hardware that could > >> temporarily go in. > > > > Okay, I'm going to have to drive in. > > > > Right now, the state of things. > > > > The primary boot partition on the EX4500 took a digger, so we have bad > > hardware. I was able to boot off the backup boot partition on the > > EX4500, but this is still the older software version since I was in > > the middle of the upgrade and the upgrade process didn't get to the > > 2nd partition to upgrade it yet. > > > > The virtual chassis is now split version, which means VC won't work. > > > > Furthmore, networking is not working on either unit, even to the OOB. > > > > I am going to have to drive in to be physically in front of the box. > > > > I have two plans of attack. I will put the install image on a USB > > drive and load it off USB onto the unit. IIRC, having done this > > before long ago, this takes *ages* to do. > > > > If I can get both units happy with the same software versions, VC can > > reestablish, and things can go along. > > > > I also have two EX4550's not-in-production right now, I'll bring them > > along just in case. If things are really to the point of no return on > > the EX4500, I can put in one or two and bring them up, restoring the > > saved configs onto them. Of course, things will have to shuffle a little > > then. This will delay me a bit in unmounting them from where they are, > > but I think it is a safe backup plan. > > > > Another option would be to make the EX4550 standalone, but I have to > > unhook the VC cables before this can happen, and it only helps along > > a portion of the members. I don't think this is worthwhile to pursue. > > > > So, it'll be another hour before I can get downtown to my place, get > > hardware, and get over to 511. > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: > (866)-280-2356 >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike O'Connor

i think we owe you one, Doug thanks! m > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >>> We're going to upgrade the core MICE 10G switches coming up this week >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> support has run out again, and JTAC will not help us (even at a pay >> per-incidence event). In fact, since the EX4500 has reached EOL, (although >> not EOM), and we let the contract expire, Juniper says we will be unable to >> renew maintenance support on the EX4500 at all. >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> If it comes down to it, I should have non-production hardware that could >> temporarily go in. > > Okay, I'm going to have to drive in. > > Right now, the state of things. > > The primary boot partition on the EX4500 took a digger, so we have bad > hardware. I was able to boot off the backup boot partition on the > EX4500, but this is still the older software version since I was in > the middle of the upgrade and the upgrade process didn't get to the > 2nd partition to upgrade it yet. > > The virtual chassis is now split version, which means VC won't work. > > Furthmore, networking is not working on either unit, even to the OOB. > > I am going to have to drive in to be physically in front of the box. > > I have two plans of attack. I will put the install image on a USB > drive and load it off USB onto the unit. IIRC, having done this > before long ago, this takes *ages* to do. > > If I can get both units happy with the same software versions, VC can > reestablish, and things can go along. > > I also have two EX4550's not-in-production right now, I'll bring them > along just in case. If things are really to the point of no return on > the EX4500, I can put in one or two and bring them up, restoring the > saved configs onto them. Of course, things will have to shuffle a little > then. This will delay me a bit in unmounting them from where they are, > but I think it is a safe backup plan. > > Another option would be to make the EX4550 standalone, but I have to > unhook the VC cables before this can happen, and it only helps along > a portion of the members. I don't think this is worthwhile to pursue. > > So, it'll be another hour before I can get downtown to my place, get > hardware, and get over to 511. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax:(866)-280-2356

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 02:07:46AM -0500, Doug McIntyre wrote: > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > > On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > > > We're going to upgrade the core MICE 10G switches coming up this week > > > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. > > > > The upgrade is NOT going as planned.. The USB recovery did go as planned, so the 4500 is back up, the 4500 and 4550 are VC'd together, and everything is looking good pingwise, trafficwise, so I brought up BGP for everybody again as well. Traffic is picking up across the Exchange. I see lots of routes, especially from HE, so things should be picking up for everybody. Anthony and I have checked out a lot of members traffic levels and they look like they are returning to normal ranges. We're calling it a night. End result, we are up to date on software, everything should be normalled up, and now we have to determine what to do about end of life, semi-broken hardware that is the 4500 box. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > > We're going to upgrade the core MICE 10G switches coming up this week > > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. > > The upgrade is NOT going as planned, and we discovered our maintenance > support has run out again, and JTAC will not help us (even at a pay > per-incidence event). In fact, since the EX4500 has reached EOL, (although > not EOM), and we let the contract expire, Juniper says we will be unable to > renew maintenance support on the EX4500 at all. > > I'm looking at ways to bring the VC cluster fully back online now. > > If it comes down to it, I should have non-production hardware that could > temporarily go in. Okay, I'm going to have to drive in. Right now, the state of things. The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet. The virtual chassis is now split version, which means VC won't work. Furthmore, networking is not working on either unit, even to the OOB. I am going to have to drive in to be physically in front of the box. I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do. If I can get both units happy with the same software versions, VC can reestablish, and things can go along. I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan. Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue. So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > We're going to upgrade the core MICE 10G switches coming up this week > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all. I'm looking at ways to bring the VC cluster fully back online now. If it comes down to it, I should have non-production hardware that could temporarily go in. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016