- mice-discuss - lists.micemn.net

Re: FW: New gen route servers

by Doug McIntyre

On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote: > Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through? Everything has all been RAID1, the old setup had the write-back cache, all the new setups don't have battery backed up cache, so they are all write-through setups. I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Anton Kapela

(Chiming in because storage and OS nerding) Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through? A pair of SLC-type SSD's with ext3/4, or XFS mounted 'sync' atop a write-through raid1 should still be entirely awesome and performant for the job of booting an OS and loading route-server binaries, and perhaps even soaking up a few syslog outputs...while being as 'safe' as the hardware allows. If that sounds tempting, tell me where to fedex the ssd media & caddy adapters :) -Tk > On Mar 27, 2015, at 1:26 AM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote: >> As an alternate option I have several HP dl380 g7's that are maybe a year >> old and still under warranty I could provide. They have licensed ILO on >> them as well and I could provide some hot spare drives for them. > > I think with the current PE R320 and PE R805, that we are probably set, but > thank you for your offer. The R320 is a pretty new box, hmm, pulling the > serial # and going to Dell, it looks like it is only 16 months old. Jeff > mentioned they had some more coming out of cycle, and that people were looking > to get a hot spare at the UG meeting? > > I think the specific problem I was seeing with the old hardware was > due to a very specific configuration. With the PERC6/i battery backed > up RAID controller, and some combination due to the age of the battery, > and that specific design, which periodically (every 2-3 months or so) does > a deep drain and charge cycle on the battery to make sure it can hold its' > intended charge for the writeback log, during this cycle it switches the > writeback log off and then on again, and sometimes the OS glitches on this > sudden change in disk latency. My guess is the batteries were below a certain > threshold point that exacerbated the writelog switchover, since we have been > running on the same hardware for quite some time without issues, but not > below the threshold which would alert the BCM on the RAID battery being bad. > > Both of these new systems do not have a battery backed up RAID system, which > really isn't needed in this use case anyway, and are newer hardware, which I > think would serve us well. > > I have both inhouse, with the OS installed, I just have to copy the configs back over, > which I can get done pretty quickly, as it isn't extensive, and haul them down > to 511. I'll have to schedule up some sort of switchout time, how much lead time > would a graceful changeout of the hardware be required? > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Doug McIntyre

On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote: > As an alternate option I have several HP dl380 g7's that are maybe a year > old and still under warranty I could provide. They have licensed ILO on > them as well and I could provide some hot spare drives for them. I think with the current PE R320 and PE R805, that we are probably set, but thank you for your offer. The R320 is a pretty new box, hmm, pulling the serial # and going to Dell, it looks like it is only 16 months old. Jeff mentioned they had some more coming out of cycle, and that people were looking to get a hot spare at the UG meeting? I think the specific problem I was seeing with the old hardware was due to a very specific configuration. With the PERC6/i battery backed up RAID controller, and some combination due to the age of the battery, and that specific design, which periodically (every 2-3 months or so) does a deep drain and charge cycle on the battery to make sure it can hold its' intended charge for the writeback log, during this cycle it switches the writeback log off and then on again, and sometimes the OS glitches on this sudden change in disk latency. My guess is the batteries were below a certain threshold point that exacerbated the writelog switchover, since we have been running on the same hardware for quite some time without issues, but not below the threshold which would alert the BCM on the RAID battery being bad. Both of these new systems do not have a battery backed up RAID system, which really isn't needed in this use case anyway, and are newer hardware, which I think would serve us well. I have both inhouse, with the OS installed, I just have to copy the configs back over, which I can get done pretty quickly, as it isn't extensive, and haul them down to 511. I'll have to schedule up some sort of switchout time, how much lead time would a graceful changeout of the hardware be required? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Brady Kittel

Correction just checked the tags, they're actually about 2 and a half years old. On Thu, Mar 26, 2015 at 6:24 PM, Brady Kittel <bkittel(a)gmail.com> wrote: > As an alternate option I have several HP dl380 g7's that are maybe a year > old and still under warranty I could provide. They have licensed ILO on > them as well and I could provide some hot spare drives for them. They're > also massively over speced for this use (quad core processors, 8-12GB of > memory, 4 x 15k drives. We've had much better luck with the HP hardware > with regard to firmware stability in that generation range than the dell > hardware we run for what it's worth. Given we only run maybe a thousand > servers but the sample size seems large enough to bare that out. > > On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn(a)iphouse.net> > wrote: > >> On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: >> > This machine is ready to go if MICE would like it. I can drop it by >> this afternoon. >> > Here are the specs: >> > Dell R320 >> > Single socket 4 core 1.8ghz Xeon. >> > PERC H310 Mini with RAID 10 >> > Qty 4 300 gig 10k SAS disks >> > 8Gig RAM >> >> >> Sure! I think that would be an excellent upgrade to newer gear than >> what we have now. Of course, it is total overkill for what it needs, >> but then again just about anything enterprise hardware is. >> >> When the R805 is ready, I could swing out and do up this one at 511, >> probably >> easier than hauling it back and forth, even I'm only 8 blocks away. >> >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> > >

March 26, 2015

Re: FW: New gen route servers

by Brady Kittel

As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them. They're also massively over speced for this use (quad core processors, 8-12GB of memory, 4 x 15k drives. We've had much better luck with the HP hardware with regard to firmware stability in that generation range than the dell hardware we run for what it's worth. Given we only run maybe a thousand servers but the sample size seems large enough to bare that out. On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: > > This machine is ready to go if MICE would like it. I can drop it by this > afternoon. > > Here are the specs: > > Dell R320 > > Single socket 4 core 1.8ghz Xeon. > > PERC H310 Mini with RAID 10 > > Qty 4 300 gig 10k SAS disks > > 8Gig RAM > > > Sure! I think that would be an excellent upgrade to newer gear than > what we have now. Of course, it is total overkill for what it needs, > but then again just about anything enterprise hardware is. > > When the R805 is ready, I could swing out and do up this one at 511, > probably > easier than hauling it back and forth, even I'm only 8 blocks away. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

March 26, 2015

Re: FW: New gen route servers

by Doug McIntyre

On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: > This machine is ready to go if MICE would like it. I can drop it by this afternoon. > Here are the specs: > Dell R320 > Single socket 4 core 1.8ghz Xeon. > PERC H310 Mini with RAID 10 > Qty 4 300 gig 10k SAS disks > 8Gig RAM Sure! I think that would be an excellent upgrade to newer gear than what we have now. Of course, it is total overkill for what it needs, but then again just about anything enterprise hardware is. When the R805 is ready, I could swing out and do up this one at 511, probably easier than hauling it back and forth, even I'm only 8 blocks away. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 26, 2015

Re: FW: New gen route servers

by Mike Horwath

R320 is current. I'd jump on this if I were doing the work. -- Mike, via phone

March 26, 2015

Re: FW: New gen route servers

by Jeff Anderson

This machine is ready to go if MICE would like it. I can drop it by this afternoon. Here are the specs: Dell R320 Single socket 4 core 1.8ghz Xeon. PERC H310 Mini with RAID 10 Qty 4 300 gig 10k SAS disks 8Gig RAM All firmware has been flashed up to appropriate patch levels. Jeff -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Hemphill Sent: Thursday, March 26, 2015 10:30 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: New gen route servers All, Please see Jeff's offer below as well. Mike Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill(a)cologix.com -----Original Message----- From: Jeff Anderson [mailto:jeffa@implex.net] Sent: Wednesday, March 25, 2015 11:08 PM To: mhemphill@mngateway com; Dave Farmer; Jay Hanke Subject: New gen route servers Guys, I have a dell R320 box that we would like to donate as a new route server platform. Its got a single core 1.8ghz. 8gig ram. Raid 10 disks. Ill provide whatever ram or disk you require. I should have a couple more of these available end of month. We're virtualizing a client into our cloud so we should have some extra hardware coming our way shortly. .sparky

March 26, 2015

FW: New gen route servers

by Mike Hemphill

All, Please see Jeff's offer below as well. Mike Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill(a)cologix.com -----Original Message----- From: Jeff Anderson [mailto:jeffa@implex.net] Sent: Wednesday, March 25, 2015 11:08 PM To: mhemphill@mngateway com; Dave Farmer; Jay Hanke Subject: New gen route servers Guys, I have a dell R320 box that we would like to donate as a new route server platform. Its got a single core 1.8ghz. 8gig ram. Raid 10 disks. Ill provide whatever ram or disk you require. I should have a couple more of these available end of month. We're virtualizing a client into our cloud so we should have some extra hardware coming our way shortly. .sparky

March 26, 2015

Re: Mice route server stability

by Doug McIntyre

On Thu, Mar 26, 2015 at 03:16:09PM +0000, Derek Poehls wrote: > In regards to route server hardware. We have 6 older Dell 1950???s we???ve recently pulled out if they could be of use? Nevin@Arcustech has graciously donated a Dell PE R805 to MICE for use as a RR. We specificily choose it with a SAS6/i controller, which is a non battery backed RAID controller, so the deep battery drain/charge cycle that I see glitch out the disks sometimes wouldn't be an issue. After micemn-02 ate it last week, I had actually taken one of ipHouse surplus PE 1950 with a SAS6/i controller down to MICE to replace micemn-02, so it has been up since then. I think that with the refresh of micemn-01 with the newer Dell and with specificly choosing that controller, I think MICE RR's will be in a good spot. I'm currently awaiting a batch of DRAC cards that I bought for it to arrive and I'll take the R805 down to 511 to replace micemn-01. I'm thinking with the different RAID HBAs that the stability will be improved and this won't be an issue. The actual resources required by the RR's are nothing, but it is the redundancy and stability that we need. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 26, 2015