MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October. This will take us up the latest JTAC recommended software release, and also get a reboot in as further testing the packet loss that Jeremy is tracking down. The 1G switch stack is already at the version we are moving to, so 1G only members on it should still have connectivity to one route-server while the 10G stack does its' upgrade on both stack boxes. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On 10/24/2016 01:20 PM, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches
Has it been MICE practice to block BGP using a L4 ACL before the maintenance? Something like this: https://www.nanog.org/sites/default/files/wednesday.general-lt.hargrave.ixpm... I know that in saying this, I'm basically volunteering others to do more work, but... We move a lot of bits now, and I think it's important we do what we can to gracefully move the traffic. -- Richard
On Oct 24, 2016, at 8:14 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
Has it been MICE practice to block BGP using a L4 ACL before the maintenance?
Something like this: https://www.nanog.org/sites/default/files/wednesday.general-lt.hargrave.ixpm...
I know that in saying this, I'm basically volunteering others to do more work, but... We move a lot of bits now, and I think it's important we do what we can to gracefully move the traffic.
I was actually in the process of penning a similar note (or suggesting at a minimum, a mass-shutdown of sessions on the route servers pre-work). Would go along way in “doing no harm” as things are rebooting and flapping up/down. Richard - I’d happily help with this in coming up with an ACL that works, and any testing. Also, given how big MICE has become and some of the players involved, I’d think we can do better than 3 days advance notice on service affecting work that will take down the the majority of the IX? Do we have standards there as far as this sort of work goes (and along those lines, peer/tech committee review of work/plan, change approval, etc)? If not, perhaps a topic for next meeting. My $0.02, ymmv... -- Andrew Hoyos hoyosa@gmail.com
On Mon, Oct 24, 2016 at 08:22:46PM -0500, Andrew Hoyos wrote:
Also, given how big MICE has become and some of the players involved, I’d think we can do better than 3 days advance notice on service affecting work that will take down the the majority of the IX?
OOTH, if you get out past 7 days you have the problem that people forget about the window, and are blindsided by it actually happening. I get more complaints about further out windows than shorter. I would typically aim for 6 days, but didn't want to do it during MEA, and now Holloween is coming up as well. Given 2.5 solid days (not weekend), went with a window that fit around my and other's schedule. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Oct 24, 2016, at 11:29 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Mon, Oct 24, 2016 at 08:22:46PM -0500, Andrew Hoyos wrote:
Also, given how big MICE has become and some of the players involved, I’d think we can do better than 3 days advance notice on service affecting work that will take down the the majority of the IX?
OOTH, if you get out past 7 days you have the problem that people forget about the window, and are blindsided by it actually happening. I get more complaints about further out windows than shorter.
Hrm, 7-10 days seems to be industry standard for this sort of thing, and in practice, I personally have never had anyone complain about providing *more* advance notice for service affecting work. Calendars and all…..
I would typically aim for 6 days, but didn't want to do it during MEA, and now Holloween is coming up as well. Given 2.5 solid days (not weekend), went with a window that fit around my and other's schedule.
Understood, and truly appreciate you volunteering to do the work, however, what about the input of the other 60+ members of MICE into this? Unless I've completely missed something, this all seems very unstructured and ad-hoc. Board folks and community - I’d encourage some more discussion here on governance of the technical side of the IX, including how and when changes are scheduled/approved, and perhaps some formalization of a technical committee. I think we’ve reached a point where continuing to do things ad-hoc paints a shaky picture. -- Andrew Hoyos hoyosa@gmail.com
As one of the other 60+ members, I am offering my $0.02. Honestly I think Doug has gone above, and beyond with his notification window, as well as willingness to do it during a low traffic period. If I was the one doing the work, it would be in the middle of the day, and would be thrown together at the last minute when I happened to know I was going past Cologix for some other reason. Since there have never been port fees, and even when they do come about, there is a lot of opposition to making them mandatory, or recurring, I think the membership needs to judge the exchange on the value it has been, and will be providing, and not some formalized service level agreement that in the industry is typically just a document that says if we screw up here is how much we will refund. I feel it is far more important to allow Doug to do it on his time since people like him have made the donations that are the only reason that MICE has made it as far and done as well as it has. If MICE had been built on a surplus of cash, and not donated time/hardware, then I would feel much different. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, October 25, 2016 12:00 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT On Oct 24, 2016, at 11:29 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Mon, Oct 24, 2016 at 08:22:46PM -0500, Andrew Hoyos wrote:
Also, given how big MICE has become and some of the players involved, I’d think we can do better than 3 days advance notice on service affecting work that will take down the the majority of the IX?
OOTH, if you get out past 7 days you have the problem that people forget about the window, and are blindsided by it actually happening. I get more complaints about further out windows than shorter.
Hrm, 7-10 days seems to be industry standard for this sort of thing, and in practice, I personally have never had anyone complain about providing *more* advance notice for service affecting work. Calendars and all…..
I would typically aim for 6 days, but didn't want to do it during MEA, and now Holloween is coming up as well. Given 2.5 solid days (not weekend), went with a window that fit around my and other's schedule.
Understood, and truly appreciate you volunteering to do the work, however, what about the input of the other 60+ members of MICE into this? Unless I've completely missed something, this all seems very unstructured and ad-hoc. Board folks and community - I’d encourage some more discussion here on governance of the technical side of the IX, including how and when changes are scheduled/approved, and perhaps some formalization of a technical committee. I think we’ve reached a point where continuing to do things ad-hoc paints a shaky picture. -- Andrew Hoyos hoyosa@gmail.com
On Tue, Oct 25, 2016 at 07:34:00AM -0500, Jeremy Lumby wrote:
If MICE had been built on a surplus of cash, and not donated time/hardware, then I would feel much different.
Hear hear! -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Oct 25, 2016, at 7:34 AM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
If MICE had been built on a surplus of cash, and not donated time/hardware, then I would feel much different.
Lack of a surplus of cash or charging fees seems to have little if nothing to do with having some operational norms and processes/timelines that we could all agree to follow. I’m not discounting Doug’s willingness to do the upgrade at all, it’s the process by which it’s happened, and I think we can do better as a group to bring some sane operational due process and some standards we can all agree to follow as a community when it comes to things like this (upgrades, changes, etc) to minimize impact to the IX and it’s members. If folks feel differently as a group, I’ll bite my tongue and turn the other way — but we’re big enough now where folks are paying attention to this sort of thing. My suggestion would be for the formal creation of a technical committee by the board, which can work on and implement these operational details in a way that works for everyone. -- Andrew Hoyos hoyosa@gmail.com
On 25.10.2016 09:40, Andrew Hoyos wrote:
Lack of a surplus of cash or charging fees seems to have little if nothing to do with having some operational norms and processes/timelines that we could all agree to follow.
I’m not discounting Doug’s willingness to do the upgrade at all, it’s the process by which it’s happened, and I think we can do better as a group to bring some sane operational due process and some standards we can all agree to follow as a community when it comes to things like this (upgrades, changes, etc) to minimize impact to the IX and it’s members.
If folks feel differently as a group, I’ll bite my tongue and turn the other way — but we’re big enough now where folks are paying attention to this sort of thing.
My suggestion would be for the formal creation of a technical committee by the board, which can work on and implement these operational details in a way that works for everyone.
I remember some discussion in a MICE UG meeting a few years back that some larger networks may have strict requirements that must be met in order to connect to the exchange. We've added some big content providers recently, which I'm thrilled to see, but I think establishing some defined and documented standards can only help continue the growth. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
I agree standards would help, however I know that my personal donations to MICE would probably have been far less if I was forced to do it on a corporate standard schedule, and lets remember donations like that got us here, and I would prefer not to lose donors over standards, and I do not feel membership will be impacted because of lack of standards since all of our current members have gotten along quite well all this time without them. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Baker Sent: Tuesday, October 25, 2016 9:58 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT On 25.10.2016 09:40, Andrew Hoyos wrote:
Lack of a surplus of cash or charging fees seems to have little if nothing to do with having some operational norms and processes/timelines that we could all agree to follow.
I’m not discounting Doug’s willingness to do the upgrade at all, it’s the process by which it’s happened, and I think we can do better as a group to bring some sane operational due process and some standards we can all agree to follow as a community when it comes to things like this (upgrades, changes, etc) to minimize impact to the IX and it’s members.
If folks feel differently as a group, I’ll bite my tongue and turn the other way — but we’re big enough now where folks are paying attention to this sort of thing.
My suggestion would be for the formal creation of a technical committee by the board, which can work on and implement these operational details in a way that works for everyone.
I remember some discussion in a MICE UG meeting a few years back that some larger networks may have strict requirements that must be met in order to connect to the exchange. We've added some big content providers recently, which I'm thrilled to see, but I think establishing some defined and documented standards can only help continue the growth. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
On Tue, Oct 25, 2016 at 10:10:00AM -0500, Jeremy Lumby wrote:
I agree standards would help, however I know that my personal donations to MICE would probably have been far less if I was forced to do it on a corporate standard schedule, and lets remember donations like that got us here, and I would prefer not to lose donors over standards, and I do not feel membership will be impacted because of lack of standards since all of our current members have gotten along quite well all this time without them.
These aren't corporate standard schedules. While I am now on the outside of the service provider world, and no longer a direct member of MICE in any capacity (mailing list doens't count), I do feel that we should have something defined and adhered to for the benefit of the group as a whole. We didn't get here all willy nilly, using 1994 ISP style operations. Should there be defined announcement windows for scheduled maintenance? I think most everyone would agree that would be a yes. Should there be notifications of unscheduled outages, maintenance, lunch changes? Yes but this would be looser for timeframes. In the heat of the moment would you rather see work done to further forward the repair or would you like notifications every 15 minutes that things are still broken? Tough stuff.. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Tue, Oct 25, 2016 at 09:58:06AM -0500, Colin Baker wrote:
I remember some discussion in a MICE UG meeting a few years back that some larger networks may have strict requirements that must be met in order to connect to the exchange. We've added some big content providers recently, which I'm thrilled to see, but I think establishing some defined and documented standards can only help continue the growth.
Confirm and agree. -- Mike Horwath, reachable via drechsau@Geeks.ORG
I do believe the result of that discussion is that we will not offer service level agreements, and that whenever a new member agreement was drafted, it would essentially say that we cannot guarantee uptime, or resolution response time, and the solution would be that the member would responsible to disconnect themselves if quality/reliability issues arose that the member found unacceptable. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Horwath Sent: Friday, October 28, 2016 9:40 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE core switch maintenance window - 10/27/2016 12:30AM-1:00AM CDT On Tue, Oct 25, 2016 at 09:58:06AM -0500, Colin Baker wrote:
I remember some discussion in a MICE UG meeting a few years back that some larger networks may have strict requirements that must be met in order to connect to the exchange. We've added some big content providers recently, which I'm thrilled to see, but I think establishing some defined and documented standards can only help continue the growth.
Confirm and agree. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Fri, Oct 28, 2016 at 09:45:00AM -0500, Jeremy Lumby wrote:
I do believe the result of that discussion is that we will not offer service level agreements, and that whenever a new member agreement was drafted, it would essentially say that we cannot guarantee uptime, or resolution response time, and the solution would be that the member would responsible to disconnect themselves if quality/reliability issues arose that the member found unacceptable.
That's also just fine. You can have some defined parameters about scheduled work (time, announcements, etc) yet still skip an SLA. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On 10/25/2016 09:40 AM, Andrew Hoyos wrote:
My suggestion would be for the formal creation of a technical committee by the board
This is something I intend to work on soon, but the tax-exempt thing is higher priority. I'm envisioning: 1) making a page on the MICE website like the following page: https://www.seattleix.net/whos-who 2) asking the board to ACK its contents 3) asking for a MICE-TECH and/or MICE-OPS list be created (depending on whether we have distinct groups and what those people want). I expect at least one list will be closed to members only. -- Richard
On Tue, Oct 25, 2016 at 01:56:32PM -0500, Richard Laager wrote:
3) asking for a MICE-TECH and/or MICE-OPS list be created (depending on whether we have distinct groups and what those people want). I expect at least one list will be closed to members only.
Give the word and it can be done. Hell both could be done though they seem a little redundant in naming. :) -- Mike Horwath, reachable via drechsau@Geeks.ORG
On 10/28/2016 09:45 AM, Mike Horwath wrote:
On Tue, Oct 25, 2016 at 01:56:32PM -0500, Richard Laager wrote:
3) asking for a MICE-TECH and/or MICE-OPS list be created (depending on whether we have distinct groups and what those people want). I expect at least one list will be closed to members only.
Give the word and it can be done.
Hell both could be done though they seem a little redundant in naming. :)
There may be a need to have one list for people with actual access to the equipment and a separate list for another set of users. But before we go there, I need to have more discussions with people who might be on those lists. -- Richard
On Fri, Oct 28, 2016 at 09:57:19AM -0500, Richard Laager wrote:
There may be a need to have one list for people with actual access to the equipment and a separate list for another set of users. But before we go there, I need to have more discussions with people who might be on those lists.
Oh that idea was bandied about back in month-one of MICE. Once you have an idea just reach out to me. I maintain the MICE-* lists still (and Doug recently helped with something that had been missed. ewps) -- Mike Horwath, reachable via drechsau@Geeks.ORG
While I like the idea of having some sort of tech list for deep discussions amongst those with specific access, I think we also need to keep the open nature of the exchange in mind to ensure we're not accidentally excluding someone or creating barriers to communication and collaboration in the process. S Shaun Carlson Director of Information Technology | Arvig ph: (218) 346.8673 | em: shaun.carlson@arvig.com On Fri, Oct 28, 2016 at 9:57 AM -0500, "Richard Laager" <rlaager@wiktel.com> wrote: On 10/28/2016 09:45 AM, Mike Horwath wrote:
On Tue, Oct 25, 2016 at 01:56:32PM -0500, Richard Laager wrote:
3) asking for a MICE-TECH and/or MICE-OPS list be created (depending on whether we have distinct groups and what those people want). I expect at least one list will be closed to members only.
Give the word and it can be done.
Hell both could be done though they seem a little redundant in naming. :)
There may be a need to have one list for people with actual access to the equipment and a separate list for another set of users. But before we go there, I need to have more discussions with people who might be on those lists. -- Richard
On Fri, Oct 28, 2016 at 04:10:35PM +0000, Shaun Carlson wrote:
While I like the idea of having some sort of tech list for deep discussions amongst those with specific access, I think we also need to keep the open nature of the exchange in mind to ensure we're not accidentally excluding someone or creating barriers to communication and collaboration in the process.
My question in return: why can't those discussions happen on -DISCUSS? Just playing *advocate here. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On 10/28/2016 11:10 AM, Shaun Carlson wrote:
While I like the idea of having some sort of tech list for deep discussions amongst those with specific access, I think we also need to keep the open nature of the exchange in mind to ensure we're not accidentally excluding someone or creating barriers to communication and collaboration in the process.
The alternative is off-list emails, which is what already happens. A mailing list brings an archive. -- Richard
On Fri, Oct 28, 2016 at 04:10:35PM +0000, Shaun Carlson wrote:
While I like the idea of having some sort of tech list for deep discussions amongst those with specific access, I think we also need to keep the open nature of the exchange in mind to ensure we're not accidentally excluding someone or creating barriers to communication and collaboration in the process.
I actually have a MICE-TECH list already setup, and I feed the RANCID config diffs into it. I've restricted its' membership to just the tech committee and have generally pushed anybody that found it to go talk on MICE-DISCUSS anyway. There's hardly anything that happens on it other than config diffs. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On 10/25/2016 01:56 PM, Richard Laager wrote:
On 10/25/2016 09:40 AM, Andrew Hoyos wrote:
My suggestion would be for the formal creation of a technical committee by the board
This is something I intend to work on soon, but the tax-exempt thing is higher priority.
I'm envisioning:
1) making a page on the MICE website like the following page: https://www.seattleix.net/whos-who
This is now done, and was reviewed by the people mentioned: http://micemn.net/people.html
2) asking the board to ACK its contents
The officers were previously appointed by board vote. We just now ACK'ed the committees. It's definitely all official now.
3) asking for a MICE-TECH and/or MICE-OPS list be created (depending on whether we have distinct groups and what those people want). I expect at least one list will be closed to members only.
This was discussed previously and there wasn't a lot of interest. -- Richard
On Mar 1, 2017, at 5:49 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
On 10/25/2016 01:56 PM, Richard Laager wrote:
On 10/25/2016 09:40 AM, Andrew Hoyos wrote:
My suggestion would be for the formal creation of a technical committee by the board
This is something I intend to work on soon, but the tax-exempt thing is higher priority.
I'm envisioning:
1) making a page on the MICE website like the following page: https://www.seattleix.net/whos-who
This is now done, and was reviewed by the people mentioned: http://micemn.net/people.html
2) asking the board to ACK its contents
The officers were previously appointed by board vote. We just now ACK'ed the committees. It's definitely all official now.
Thanks for your work on this, Richard! -- Andrew Hoyos hoyosa@gmail.com
On Mon, Oct 24, 2016 at 08:14:17PM -0500, Richard Laager wrote:
On 10/24/2016 01:20 PM, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches
Has it been MICE practice to block BGP using a L4 ACL before the maintenance?
There hasn't been that many maintenance periods, like 3 events, one was unintended AFAIK. But we could, it wouldn't be too difficult.
Something like this: https://www.nanog.org/sites/default/files/wednesday.general-lt.hargrave.ixpm...
I can implement this 5 minutes before the upgrade. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Oct 24, 2016, at 1:20 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
This should probably be sent to MICE-ANNOUNCE as well. -- Andrew Hoyos hoyosa@gmail.com
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all. I'm looking at ways to bring the VC cluster fully back online now. If it comes down to it, I should have non-production hardware that could temporarily go in. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in. Right now, the state of things. The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet. The virtual chassis is now split version, which means VC won't work. Furthmore, networking is not working on either unit, even to the OOB. I am going to have to drive in to be physically in front of the box. I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do. If I can get both units happy with the same software versions, VC can reestablish, and things can go along. I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan. Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue. So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Thu, Oct 27, 2016 at 02:07:46AM -0500, Doug McIntyre wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned..
The USB recovery did go as planned, so the 4500 is back up, the 4500 and 4550 are VC'd together, and everything is looking good pingwise, trafficwise, so I brought up BGP for everybody again as well. Traffic is picking up across the Exchange. I see lots of routes, especially from HE, so things should be picking up for everybody. Anthony and I have checked out a lot of members traffic levels and they look like they are returning to normal ranges. We're calling it a night. End result, we are up to date on software, everything should be normalled up, and now we have to determine what to do about end of life, semi-broken hardware that is the 4500 box. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
i think we owe you one, Doug thanks! m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax:(866)-280-2356
What would be the proposed solution? Replace the failing EX4500 with a 4550? If so, what does one of those run? Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman@arvig.com On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike@haven2.com> wrote:
i think we owe you one, Doug
thanks!
m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: (866)-280-2356
There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade. On Oct 27, 2016 7:23 AM, "Ben Wiechman" <ben.wiechman@arvig.com> wrote:
What would be the proposed solution? Replace the failing EX4500 with a 4550? If so, what does one of those run?
Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman@arvig.com
On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike@haven2.com> wrote:
i think we owe you one, Doug
thanks!
m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: (866)-280-2356
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote:
There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade.
EX4500 is 40 ports base, 48 ports with addon cards. EX4550 is 32 ports base, 48 ports with addon cards. Is the current EX4500 running with 40 ports or 48? Could a EX4550 be brought into the fabric with addon cards to match up? (not disagreeing about fabric upgrade though will be much easier once non-profit status is done and some of us can donate cleanly) -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote:
On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote:
There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade.
Correct, while one could be brought in with additional cards in both, that would barely cover us with zero future growth without adding in a 2nd one. The U of MN expressed interest in a 40Gbps connection, and doing just one switch would knock that out of contention, (or we keep the EX4500 and run it knowing that it is EOL, and software will probably stop being available within 14 months).
EX4550 is 32 ports base, 48 ports with addon cards.
It is 32 + 8 ports if you want to use the back expansion slot to house a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) (technically the VC card is slotted into the 4500, but I've never seen one bought without it there). So the EX4500 can do more ports than a EX4550. So, the back slot on a 4550 could be another 8x10G, but that loses the VC port, which means we then have to take multiples of 10G ports on both switches to devote to VC instead, probably at least 4 x 10GB on both switches, which is a losing of 8 ports either way. Effectively making the stacked EX4550 only 40 ports instead of the 48 ports on the 4500.
Is the current EX4500 running with 40 ports or 48?
It has 48 ports, with 47 ports plugged in. (didn't count how many as lit, but we just did an inventory and I know how few empty ports we have). The EX4550 is the direct replacement for the EX4500. I've run multiple stacks of them myself, and they are solid boxes. Juniper also has newer switches at higher price points (and not as easy upgrade path). The EX4600 can do 72 x 10GB in 1U. It has 40 fixed 10G, 4 x 40G. BUT no VC ports. You use the 40G ports for VC functions. To get the 72 port density, you'd add in a 14 x 40G card, and any and each 40G port can be broken out to 4 x 10G port with breakout cables. So, stacking the EX4600's typically takes out the fixed 4x40G ports with the QSFP+ DAC cabling up and down, but adding on the 14 x 40G card brings back many ports and capabilities. The EX4600 won't VC with the EX4550 so it becomes another stack connected in. The QFX5200-32C can get into newer technologies like adding 25G/50G and 100G capable ports. Again using breakout cables to breakout ports into the "slower" speeds we are running, and can get into the range of 128 x 10GB ports with breakout cables in 1U. The QFX10002-36Q is much like the above, and can get into 144 x 10G ports with breakout cables, but I know the price on this switch pushes now into 6 figures. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Doug, We've just finished some research into 10Gig + 40 to 100Gig Density mapping in Juniper. I'll collect my notes and send them to the list with my thoughts. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net On Thu, Oct 27, 2016 at 2:07 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote:
On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote:
There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade.
Correct, while one could be brought in with additional cards in both, that would barely cover us with zero future growth without adding in a 2nd one. The U of MN expressed interest in a 40Gbps connection, and doing just one switch would knock that out of contention, (or we keep the EX4500 and run it knowing that it is EOL, and software will probably stop being available within 14 months).
EX4550 is 32 ports base, 48 ports with addon cards.
It is 32 + 8 ports if you want to use the back expansion slot to house a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) (technically the VC card is slotted into the 4500, but I've never seen one bought without it there). So the EX4500 can do more ports than a EX4550.
So, the back slot on a 4550 could be another 8x10G, but that loses the VC port, which means we then have to take multiples of 10G ports on both switches to devote to VC instead, probably at least 4 x 10GB on both switches, which is a losing of 8 ports either way.
Effectively making the stacked EX4550 only 40 ports instead of the 48 ports on the 4500.
Is the current EX4500 running with 40 ports or 48?
It has 48 ports, with 47 ports plugged in. (didn't count how many as lit, but we just did an inventory and I know how few empty ports we have).
The EX4550 is the direct replacement for the EX4500. I've run multiple stacks of them myself, and they are solid boxes.
Juniper also has newer switches at higher price points (and not as easy upgrade path). The EX4600 can do 72 x 10GB in 1U. It has 40 fixed 10G, 4 x 40G. BUT no VC ports. You use the 40G ports for VC functions. To get the 72 port density, you'd add in a 14 x 40G card, and any and each 40G port can be broken out to 4 x 10G port with breakout cables. So, stacking the EX4600's typically takes out the fixed 4x40G ports with the QSFP+ DAC cabling up and down, but adding on the 14 x 40G card brings back many ports and capabilities. The EX4600 won't VC with the EX4550 so it becomes another stack connected in.
The QFX5200-32C can get into newer technologies like adding 25G/50G and 100G capable ports. Again using breakout cables to breakout ports into the "slower" speeds we are running, and can get into the range of 128 x 10GB ports with breakout cables in 1U.
The QFX10002-36Q is much like the above, and can get into 144 x 10G ports with breakout cables, but I know the price on this switch pushes now into 6 figures.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Thu, Oct 27, 2016 at 2:07 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote:
On Thu, Oct 27, 2016 at 08:06:45AM -0500, Jason Hanke wrote:
There aren't enough ports on the 4550 to swap it in we would need two. We are due for a fabric upgrade.
Correct, while one could be brought in with additional cards in both, that would barely cover us with zero future growth without adding in a 2nd one. The U of MN expressed interest in a 40Gbps connection, and doing just one switch would knock that out of contention, (or we keep the EX4500 and run it knowing that it is EOL, and software will probably stop being available within 14 months).
Don't worry about a 40G for us I'm looking past that now to 100G. I think the priority is to stabilize what we have and add a 100G/50G/40G/25G capability soon next two or three months. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
On Thu, Oct 27, 2016 at 02:07:39PM -0500, Doug McIntyre wrote:
On Thu, Oct 27, 2016 at 08:21:54AM -0500, Mike Horwath wrote:
EX4550 is 32 ports base, 48 ports with addon cards.
It is 32 + 8 ports if you want to use the back expansion slot to house a VC card instead. The 4550 doesn't have built-in VC (unlike the 4200/4500) (technically the VC card is slotted into the 4500, but I've never seen one bought without it there). So the EX4500 can do more ports than a EX4550.
Ooh I didn't know that! Thanks!
The QFX5200-32C can get into newer technologies like adding 25G/50G and 100G capable ports. Again using breakout cables to breakout ports into the "slower" speeds we are running, and can get into the range of 128 x 10GB ports with breakout cables in 1U.
The QFX10002-36Q is much like the above, and can get into 144 x 10G ports with breakout cables, but I know the price on this switch pushes now into 6 figures.
I have to assume that regardless they are expensive. If only that non-profit thing were in place *today*. A lot of work and I don't envy anyone having to do it. But I, and I don't think I am alone, am not donating money until there is a formal organization wrapped up in pretty bows. I have faith, though. This is coming and there will be much celebration. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Fri, Oct 28, 2016 at 09:53:02AM -0500, Mike Horwath wrote:
On Thu, Oct 27, 2016 at 02:07:39PM -0500, Doug McIntyre wrote:
The QFX5200-32C can get into newer technologies like adding 25G/50G and 100G capable ports. Again using breakout cables to breakout ports into the "slower" speeds we are running, and can get into the range of 128 x 10GB ports with breakout cables in 1U. .. I have to assume that regardless they are expensive.
Actually, I think the QFX5200-32C is pretty reasonable price, and I think one to shoot for, although watch out pricing it as hardware & software are seperate line items. But in some ways, that is good, as this application doesn't need VxLAN and EVPN or all the fancy new things. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Oct 28, 2016, at 10:30 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
I have to assume that regardless they are expensive.
Actually, I think the QFX5200-32C is pretty reasonable price, and I think one to shoot for, although watch out pricing it as hardware & software are seperate line items. But in some ways, that is good, as this application doesn't need VxLAN and EVPN or all the fancy new things.
QFX5100-48 can be had for 48 port 10g density as well for like $6k NFR with the right Juniper partner (hint hint). Depending on direction the group wants to go as far as a new fabric and timing, I may even have one we would be willing to donate to the cause. -- Andrew Hoyos hoyosa@gmail.com
A Brocade ICX7750 with a similar port density is also potentially a similar cost. Havent compared feature set between those. The recent Broadcom announcement muddies that water a bit as well. On Oct 28, 2016 10:33 AM, "Andrew Hoyos" <hoyosa@gmail.com> wrote:
On Oct 28, 2016, at 10:30 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
I have to assume that regardless they are expensive.
Actually, I think the QFX5200-32C is pretty reasonable price, and I think one to shoot for, although watch out pricing it as hardware & software are seperate line items. But in some ways, that is good, as this application doesn't need VxLAN and EVPN or all the fancy new things.
QFX5100-48 can be had for 48 port 10g density as well for like $6k NFR with the right Juniper partner (hint hint). Depending on direction the group wants to go as far as a new fabric and timing, I may even have one we would be willing to donate to the cause.
-- Andrew Hoyos hoyosa@gmail.com
+1 Thanks for the good work On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike@haven2.com> wrote:
i think we owe you one, Doug
thanks!
m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: (866)-280-2356
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
Doug, Thanks Doug for all the work and effort. *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net On Thu, Oct 27, 2016 at 8:42 AM, David Farmer <farmer@umn.edu> wrote:
+1 Thanks for the good work
On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike@haven2.com> wrote:
i think we owe you one, Doug
thanks!
m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax: (866)-280-2356
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Having been through exact same scenario with one of our datacenter stacks, I feel yah. Thank you On Oct 27, 2016, at 8:53 AM, Levi Pederson <levipederson@MANKATONETWORKS.NET<mailto:levipederson@MANKATONETWORKS.NET>> wrote: Doug, Thanks Doug for all the work and effort. Levi Pederson Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net<mailto:levipederson@mankatonetworks.net> [http://www.mankatonetworks.com/images/mn_logo_email.png] On Thu, Oct 27, 2016 at 8:42 AM, David Farmer <farmer@umn.edu<mailto:farmer@umn.edu>> wrote: +1 Thanks for the good work On Thu, Oct 27, 2016 at 7:03 AM, Mike O'Connor <mike@haven2.com<mailto:mike@haven2.com>> wrote: i think we owe you one, Doug thanks! m
On Oct 27, 2016, at 2:07 AM, Doug McIntyre <merlyn@iphouse.net<mailto:merlyn@iphouse.net>> wrote:
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
On Mon, Oct 24, 2016 at 01:20:45PM -0500, Doug McIntyre wrote:
We're going to upgrade the core MICE 10G switches coming up this week on Wednesday night/ Thursday morning at 12:30AM-1:00AM CDT on 27 October.
The upgrade is NOT going as planned, and we discovered our maintenance support has run out again, and JTAC will not help us (even at a pay per-incidence event). In fact, since the EX4500 has reached EOL, (although not EOM), and we let the contract expire, Juniper says we will be unable to renew maintenance support on the EX4500 at all.
I'm looking at ways to bring the VC cluster fully back online now.
If it comes down to it, I should have non-production hardware that could temporarily go in.
Okay, I'm going to have to drive in.
Right now, the state of things.
The primary boot partition on the EX4500 took a digger, so we have bad hardware. I was able to boot off the backup boot partition on the EX4500, but this is still the older software version since I was in the middle of the upgrade and the upgrade process didn't get to the 2nd partition to upgrade it yet.
The virtual chassis is now split version, which means VC won't work.
Furthmore, networking is not working on either unit, even to the OOB.
I am going to have to drive in to be physically in front of the box.
I have two plans of attack. I will put the install image on a USB drive and load it off USB onto the unit. IIRC, having done this before long ago, this takes *ages* to do.
If I can get both units happy with the same software versions, VC can reestablish, and things can go along.
I also have two EX4550's not-in-production right now, I'll bring them along just in case. If things are really to the point of no return on the EX4500, I can put in one or two and bring them up, restoring the saved configs onto them. Of course, things will have to shuffle a little then. This will delay me a bit in unmounting them from where they are, but I think it is a safe backup plan.
Another option would be to make the EX4550 standalone, but I have to unhook the VC cables before this can happen, and it only helps along a portion of the members. I don't think this is worthwhile to pursue.
So, it'll be another hour before I can get downtown to my place, get hardware, and get over to 511.
-- Doug McIntyre <merlyn@iphouse.net<mailto:merlyn@iphouse.net>> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
WI:(715) 598-4284<tel:%28715%29%20598-4284> MN:(651) 647-6109<tel:%28651%29%20647-6109> Toll Free:(800) 896-0907<tel:%28800%29%20896-0907> Fax:(866)-280-2356<tel:%28866%29-280-2356> -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815<tel:612-626-0815> Minneapolis, MN 55414-3029 Cell: 612-812-9952<tel:612-812-9952> =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote:
i think we owe you one, Doug
thanks!
No problem. Anthony was there too checking out things and helping, so thanks out to him too. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Kudos to Anthony as well. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson@mankatonetworks.net On Thu, Oct 27, 2016 at 1:29 PM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote:
i think we owe you one, Doug
thanks!
No problem. Anthony was there too checking out things and helping, so thanks out to him too.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Thanks, team! Reid On Oct 27, 2016 2:29 PM, "Doug McIntyre" <merlyn@iphouse.net> wrote:
On Thu, Oct 27, 2016 at 07:03:41AM -0500, Mike O'Connor wrote:
i think we owe you one, Doug
thanks!
No problem. Anthony was there too checking out things and helping, so thanks out to him too.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Thu, Oct 27, 2016 at 01:35:06AM -0500, Doug McIntyre wrote:
If it comes down to it, I should have non-production hardware that could temporarily go in.
Haven't read the rest of the thread yet. Thanks for your work, Doug. -- Mike Horwath, reachable via drechsau@Geeks.ORG
participants (14)
-
Andrew Hoyos -
Ben Wiechman -
Colin Baker -
David Farmer -
Doug McIntyre -
Jason Hanke -
Jeremy Lumby -
Levi Pederson -
Mike Horwath -
Mike O'Connor -
Reid Fishler -
Richard Laager -
Ryan Goldberg -
Shaun Carlson