- mice-discuss - lists.micemn.net

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by David Farmer

+1 Thanks for the good work On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > i think we owe you one, Doug > > thanks! > > m > > > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > >>> We're going to upgrade the core MICE 10G switches coming up this week > >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 > October. > >> > >> The upgrade is NOT going as planned, and we discovered our maintenance > >> support has run out again, and JTAC will not help us (even at a pay > >> per-incidence event). In fact, since the EX4500 has reached EOL, > (although > >> not EOM), and we let the contract expire, Juniper says we will be > unable to > >> renew maintenance support on the EX4500 at all. > >> > >> I'm looking at ways to bring the VC cluster fully back online now. > >> > >> If it comes down to it, I should have non-production hardware that could > >> temporarily go in. > > > > Okay, I'm going to have to drive in. > > > > Right now, the state of things. > > > > The primary boot partition on the EX4500 took a digger, so we have bad > > hardware. I was able to boot off the backup boot partition on the > > EX4500, but this is still the older software version since I was in > > the middle of the upgrade and the upgrade process didn't get to the > > 2nd partition to upgrade it yet. > > > > The virtual chassis is now split version, which means VC won't work. > > > > Furthmore, networking is not working on either unit, even to the OOB. > > > > I am going to have to drive in to be physically in front of the box. > > > > I have two plans of attack. I will put the install image on a USB > > drive and load it off USB onto the unit. IIRC, having done this > > before long ago, this takes *ages* to do. > > > > If I can get both units happy with the same software versions, VC can > > reestablish, and things can go along. > > > > I also have two EX4550's not-in-production right now, I'll bring them > > along just in case. If things are really to the point of no return on > > the EX4500, I can put in one or two and bring them up, restoring the > > saved configs onto them. Of course, things will have to shuffle a little > > then. This will delay me a bit in unmounting them from where they are, > > but I think it is a safe backup plan. > > > > Another option would be to make the EX4550 standalone, but I have to > > unhook the VC cables before this can happen, and it only helps along > > a portion of the members. I don't think this is worthwhile to pursue. > > > > So, it'll be another hour before I can get downtown to my place, get > > hardware, and get over to 511. > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: > (866)-280-2356 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote: > There aren't enough ports on the 4550 to swap it in we would need > two. We are due for a fabric upgrade. EX4500 is 40 ports base, 48 ports with addon cards. EX4550 is 32 ports base, 48 ports with addon cards. Is the current EX4500 running with 40 ports or 48? Could a EX4550 be brought into the fabric with addon cards to match up? (not disagreeing about fabric upgrade though will be much easier once non-profit status is done and some of us can donate cleanly) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike Horwath

On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > If it comes down to it, I should have non-production hardware that > could temporarily go in. Haven't read the rest of the thread yet. Thanks for your work, Doug. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Jason Hanke

There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade. On Oct 27, 2016 7:23 AM, "Ben Wiechman" <ben.wiechman(a)arvig.com> wrote: > What would be the proposed solution? Replace the failing EX4500 with a > 4550? If so, what does one of those run? > > > Ben Wiechman > Network Engineer IV | Arvig > Direct: 320.256.0184 > Cell: 320.247.3224 > Office: 320.256.7471 > ben.wiechman(a)arvig.com > > On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > >> i think we owe you one, Doug >> >> thanks! >> >> m >> >> > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> > >> > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >> >>> We're going to upgrade the core MICE 10G switches coming up this week >> >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 >> October. >> >> >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> >> support has run out again, and JTAC will not help us (even at a pay >> >> per-incidence event). In fact, since the EX4500 has reached EOL, >> (although >> >> not EOM), and we let the contract expire, Juniper says we will be >> unable to >> >> renew maintenance support on the EX4500 at all. >> >> >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> >> >> If it comes down to it, I should have non-production hardware that >> could >> >> temporarily go in. >> > >> > Okay, I'm going to have to drive in. >> > >> > Right now, the state of things. >> > >> > The primary boot partition on the EX4500 took a digger, so we have bad >> > hardware. I was able to boot off the backup boot partition on the >> > EX4500, but this is still the older software version since I was in >> > the middle of the upgrade and the upgrade process didn't get to the >> > 2nd partition to upgrade it yet. >> > >> > The virtual chassis is now split version, which means VC won't work. >> > >> > Furthmore, networking is not working on either unit, even to the OOB. >> > >> > I am going to have to drive in to be physically in front of the box. >> > >> > I have two plans of attack. I will put the install image on a USB >> > drive and load it off USB onto the unit. IIRC, having done this >> > before long ago, this takes *ages* to do. >> > >> > If I can get both units happy with the same software versions, VC can >> > reestablish, and things can go along. >> > >> > I also have two EX4550's not-in-production right now, I'll bring them >> > along just in case. If things are really to the point of no return on >> > the EX4500, I can put in one or two and bring them up, restoring the >> > saved configs onto them. Of course, things will have to shuffle a little >> > then. This will delay me a bit in unmounting them from where they are, >> > but I think it is a safe backup plan. >> > >> > Another option would be to make the EX4550 standalone, but I have to >> > unhook the VC cables before this can happen, and it only helps along >> > a portion of the members. I don't think this is worthwhile to pursue. >> > >> > So, it'll be another hour before I can get downtown to my place, get >> > hardware, and get over to 511. >> > >> > -- >> > Doug McIntyre <merlyn(a)iphouse.net> >> > ~.~ ipHouse ~.~ >> > Network Engineer/Provisioning/Jack of all Trades >> >> WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: >> (866)-280-2356 >> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Ben Wiechman

What would be the proposed solution? Replace the failing EX4500 with a 4550? If so, what does one of those run? Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman(a)arvig.com On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike(a)haven2.com> wrote: > i think we owe you one, Doug > > thanks! > > m > > > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > >>> We're going to upgrade the core MICE 10G switches coming up this week > >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 > October. > >> > >> The upgrade is NOT going as planned, and we discovered our maintenance > >> support has run out again, and JTAC will not help us (even at a pay > >> per-incidence event). In fact, since the EX4500 has reached EOL, > (although > >> not EOM), and we let the contract expire, Juniper says we will be > unable to > >> renew maintenance support on the EX4500 at all. > >> > >> I'm looking at ways to bring the VC cluster fully back online now. > >> > >> If it comes down to it, I should have non-production hardware that could > >> temporarily go in. > > > > Okay, I'm going to have to drive in. > > > > Right now, the state of things. > > > > The primary boot partition on the EX4500 took a digger, so we have bad > > hardware. I was able to boot off the backup boot partition on the > > EX4500, but this is still the older software version since I was in > > the middle of the upgrade and the upgrade process didn't get to the > > 2nd partition to upgrade it yet. > > > > The virtual chassis is now split version, which means VC won't work. > > > > Furthmore, networking is not working on either unit, even to the OOB. > > > > I am going to have to drive in to be physically in front of the box. > > > > I have two plans of attack. I will put the install image on a USB > > drive and load it off USB onto the unit. IIRC, having done this > > before long ago, this takes *ages* to do. > > > > If I can get both units happy with the same software versions, VC can > > reestablish, and things can go along. > > > > I also have two EX4550's not-in-production right now, I'll bring them > > along just in case. If things are really to the point of no return on > > the EX4500, I can put in one or two and bring them up, restoring the > > saved configs onto them. Of course, things will have to shuffle a little > > then. This will delay me a bit in unmounting them from where they are, > > but I think it is a safe backup plan. > > > > Another option would be to make the EX4550 standalone, but I have to > > unhook the VC cables before this can happen, and it only helps along > > a portion of the members. I don't think this is worthwhile to pursue. > > > > So, it'll be another hour before I can get downtown to my place, get > > hardware, and get over to 511. > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: > (866)-280-2356 >

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Mike O'Connor

i think we owe you one, Doug thanks! m > On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: >> On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: >>> We're going to upgrade the core MICE 10G switches coming up this week >>> on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. >> >> The upgrade is NOT going as planned, and we discovered our maintenance >> support has run out again, and JTAC will not help us (even at a pay >> per-incidence event). In fact, since the EX4500 has reached EOL, (although >> not EOM), and we let the contract expire, Juniper says we will be unable to >> renew maintenance support on the EX4500 at all. >> >> I'm looking at ways to bring the VC cluster fully back online now. >> >> If it comes down to it, I should have non-production hardware that could >> temporarily go in. > > Okay, I'm going to have to drive in. > > Right now, the state of things. > > The primary boot partition on the EX4500 took a digger, so we have bad > hardware. I was able to boot off the backup boot partition on the > EX4500, but this is still the older software version since I was in > the middle of the upgrade and the upgrade process didn't get to the > 2nd partition to upgrade it yet. > > The virtual chassis is now split version, which means VC won't work. > > Furthmore, networking is not working on either unit, even to the OOB. > > I am going to have to drive in to be physically in front of the box. > > I have two plans of attack. I will put the install image on a USB > drive and load it off USB onto the unit. IIRC, having done this > before long ago, this takes *ages* to do. > > If I can get both units happy with the same software versions, VC can > reestablish, and things can go along. > > I also have two EX4550's not-in-production right now, I'll bring them > along just in case. If things are really to the point of no return on > the EX4500, I can put in one or two and bring them up, restoring the > saved configs onto them. Of course, things will have to shuffle a little > then. This will delay me a bit in unmounting them from where they are, > but I think it is a safe backup plan. > > Another option would be to make the EX4550 standalone, but I have to > unhook the VC cables before this can happen, and it only helps along > a portion of the members. I don't think this is worthwhile to pursue. > > So, it'll be another hour before I can get downtown to my place, get > hardware, and get over to 511. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax:(866)-280-2356

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 02:07:46AM -0500, Doug McIntyre wrote: > On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > > On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > > > We're going to upgrade the core MICE 10G switches coming up this week > > > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. > > > > The upgrade is NOT going as planned.. The USB recovery did go as planned, so the 4500 is back up, the 4500 and 4550 are VC'd together, and everything is looking good pingwise, trafficwise, so I brought up BGP for everybody again as well. Traffic is picking up across the Exchange. I see lots of routes, especially from HE, so things should be picking up for everybody. Anthony and I have checked out a lot of members traffic levels and they look like they are returning to normal ranges. We're calling it a night. End result, we are up to date on software, everything should be normalled up, and now we have to determine what to do about end of life, semi-broken hardware that is the 4500 box. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote: > On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > > We're going to upgrade the core MICE 10G switches coming up this week > > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. > > The upgrade is NOT going as planned, and we discovered our maintenance > support has run out again, and JTAC will not help us (even at a pay > per-incidence event). In fact, since the EX4500 has reached EOL, (although > not EOM), and we let the contract expire, Juniper says we will be unable to > renew maintenance support on the EX4500 at all. > > I'm looking at ways to bring the VC cluster fully back online now. > > If it comes down to it, I should have non-production hardware that could > temporarily go in. Okay, I'm going to have to drive in. Right now, the state of things. The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet. The virtual chassis is now split version, which means VC won't work. Furthmore, networking is not working on either unit, even to the OOB. I am going to have to drive in to be physically in front of the box. I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do. If I can get both units happy with the same software versions, VC can reestablish, and things can go along. I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan. Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue. So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT

by Doug McIntyre

On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote: > We're going to upgrade the core MICE 10G switches coming up this week > on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all. I'm looking at ways to bring the VC cluster fully back online now. If it comes down to it, I should have non-production hardware that could temporarily go in. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2016

Re: Peering with research orgs

by David Farmer

On Wed, Oct 26, 2016 at 11:07 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > Does MICE have a policy regarding peering its router reflectors with > research/data gathering orgs like Cymru, Dyn, Norse, ThousandEyes, QRator, > etc. A number of MICE participants peer with various of these and other services from their own networks, exposing some or all of the routes available through MICE to many of those services, at least indirectly. Do you see a particular advantage to MICE's participants, or the Internet, by making MICE's routes more directly available to such projects? Thanks -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Oct. 27, 2016