All, Please see Jeff's offer below as well. Mike Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill@cologix.com -----Original Message----- From: Jeff Anderson [mailto:jeffa@implex.net] Sent: Wednesday, March 25, 2015 11:08 PM To: mhemphill@mngateway com; Dave Farmer; Jay Hanke Subject: New gen route servers Guys, I have a dell R320 box that we would like to donate as a new route server platform. Its got a single core 1.8ghz. 8gig ram. Raid 10 disks. Ill provide whatever ram or disk you require. I should have a couple more of these available end of month. We're virtualizing a client into our cloud so we should have some extra hardware coming our way shortly. .sparky
This machine is ready to go if MICE would like it. I can drop it by this afternoon. Here are the specs: Dell R320 Single socket 4 core 1.8ghz Xeon. PERC H310 Mini with RAID 10 Qty 4 300 gig 10k SAS disks 8Gig RAM All firmware has been flashed up to appropriate patch levels. Jeff -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Hemphill Sent: Thursday, March 26, 2015 10:30 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: New gen route servers All, Please see Jeff's offer below as well. Mike Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill@cologix.com -----Original Message----- From: Jeff Anderson [mailto:jeffa@implex.net] Sent: Wednesday, March 25, 2015 11:08 PM To: mhemphill@mngateway com; Dave Farmer; Jay Hanke Subject: New gen route servers Guys, I have a dell R320 box that we would like to donate as a new route server platform. Its got a single core 1.8ghz. 8gig ram. Raid 10 disks. Ill provide whatever ram or disk you require. I should have a couple more of these available end of month. We're virtualizing a client into our cloud so we should have some extra hardware coming our way shortly. .sparky
On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote:
This machine is ready to go if MICE would like it. I can drop it by this afternoon. Here are the specs: Dell R320 Single socket 4 core 1.8ghz Xeon. PERC H310 Mini with RAID 10 Qty 4 300 gig 10k SAS disks 8Gig RAM
Sure! I think that would be an excellent upgrade to newer gear than what we have now. Of course, it is total overkill for what it needs, but then again just about anything enterprise hardware is. When the R805 is ready, I could swing out and do up this one at 511, probably easier than hauling it back and forth, even I'm only 8 blocks away. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them. They're also massively over speced for this use (quad core processors, 8-12GB of memory, 4 x 15k drives. We've had much better luck with the HP hardware with regard to firmware stability in that generation range than the dell hardware we run for what it's worth. Given we only run maybe a thousand servers but the sample size seems large enough to bare that out. On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote:
This machine is ready to go if MICE would like it. I can drop it by this afternoon. Here are the specs: Dell R320 Single socket 4 core 1.8ghz Xeon. PERC H310 Mini with RAID 10 Qty 4 300 gig 10k SAS disks 8Gig RAM
Sure! I think that would be an excellent upgrade to newer gear than what we have now. Of course, it is total overkill for what it needs, but then again just about anything enterprise hardware is.
When the R805 is ready, I could swing out and do up this one at 511, probably easier than hauling it back and forth, even I'm only 8 blocks away.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Correction just checked the tags, they're actually about 2 and a half years old. On Thu, Mar 26, 2015 at 6:24 PM, Brady Kittel <bkittel@gmail.com> wrote:
As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them. They're also massively over speced for this use (quad core processors, 8-12GB of memory, 4 x 15k drives. We've had much better luck with the HP hardware with regard to firmware stability in that generation range than the dell hardware we run for what it's worth. Given we only run maybe a thousand servers but the sample size seems large enough to bare that out.
On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote:
This machine is ready to go if MICE would like it. I can drop it by this afternoon. Here are the specs: Dell R320 Single socket 4 core 1.8ghz Xeon. PERC H310 Mini with RAID 10 Qty 4 300 gig 10k SAS disks 8Gig RAM
Sure! I think that would be an excellent upgrade to newer gear than what we have now. Of course, it is total overkill for what it needs, but then again just about anything enterprise hardware is.
When the R805 is ready, I could swing out and do up this one at 511, probably easier than hauling it back and forth, even I'm only 8 blocks away.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote:
As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them.
I think with the current PE R320 and PE R805, that we are probably set, but thank you for your offer. The R320 is a pretty new box, hmm, pulling the serial # and going to Dell, it looks like it is only 16 months old. Jeff mentioned they had some more coming out of cycle, and that people were looking to get a hot spare at the UG meeting? I think the specific problem I was seeing with the old hardware was due to a very specific configuration. With the PERC6/i battery backed up RAID controller, and some combination due to the age of the battery, and that specific design, which periodically (every 2-3 months or so) does a deep drain and charge cycle on the battery to make sure it can hold its' intended charge for the writeback log, during this cycle it switches the writeback log off and then on again, and sometimes the OS glitches on this sudden change in disk latency. My guess is the batteries were below a certain threshold point that exacerbated the writelog switchover, since we have been running on the same hardware for quite some time without issues, but not below the threshold which would alert the BCM on the RAID battery being bad. Both of these new systems do not have a battery backed up RAID system, which really isn't needed in this use case anyway, and are newer hardware, which I think would serve us well. I have both inhouse, with the OS installed, I just have to copy the configs back over, which I can get done pretty quickly, as it isn't extensive, and haul them down to 511. I'll have to schedule up some sort of switchout time, how much lead time would a graceful changeout of the hardware be required? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
(Chiming in because storage and OS nerding) Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through? A pair of SLC-type SSD's with ext3/4, or XFS mounted 'sync' atop a write-through raid1 should still be entirely awesome and performant for the job of booting an OS and loading route-server binaries, and perhaps even soaking up a few syslog outputs...while being as 'safe' as the hardware allows. If that sounds tempting, tell me where to fedex the ssd media & caddy adapters :) -Tk
On Mar 27, 2015, at 1:26 AM, Doug McIntyre <merlyn@IPHOUSE.NET> wrote:
On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote: As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them.
I think with the current PE R320 and PE R805, that we are probably set, but thank you for your offer. The R320 is a pretty new box, hmm, pulling the serial # and going to Dell, it looks like it is only 16 months old. Jeff mentioned they had some more coming out of cycle, and that people were looking to get a hot spare at the UG meeting?
I think the specific problem I was seeing with the old hardware was due to a very specific configuration. With the PERC6/i battery backed up RAID controller, and some combination due to the age of the battery, and that specific design, which periodically (every 2-3 months or so) does a deep drain and charge cycle on the battery to make sure it can hold its' intended charge for the writeback log, during this cycle it switches the writeback log off and then on again, and sometimes the OS glitches on this sudden change in disk latency. My guess is the batteries were below a certain threshold point that exacerbated the writelog switchover, since we have been running on the same hardware for quite some time without issues, but not below the threshold which would alert the BCM on the RAID battery being bad.
Both of these new systems do not have a battery backed up RAID system, which really isn't needed in this use case anyway, and are newer hardware, which I think would serve us well.
I have both inhouse, with the OS installed, I just have to copy the configs back over, which I can get done pretty quickly, as it isn't extensive, and haul them down to 511. I'll have to schedule up some sort of switchout time, how much lead time would a graceful changeout of the hardware be required?
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote:
Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through?
Everything has all been RAID1, the old setup had the write-back cache, all the new setups don't have battery backed up cache, so they are all write-through setups. I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Sounds good, we could probably pickup the new dell servers under our hardware support for server parts if they don't have coverage currently. We get 24x7x4 parts sourcing from IBM logistics (yes they carry dell parts). I'd just need the service tags and model info off of them. On Mar 27, 2015 10:25 AM, "Doug McIntyre" <merlyn@iphouse.net> wrote:
On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote:
Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through?
Everything has all been RAID1, the old setup had the write-back cache, all the new setups don't have battery backed up cache, so they are all write-through setups.
I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration.
-- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Fri, Mar 27, 2015 at 10:25 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration.
indeed - not much, but quite useful (if one doesn't like things rolling slow) for when one rocks out w/ their / or /usr or whatnot on a sync-mounted fs ;) -Tk
On Fri, Mar 27, 2015 at 11:04:36AM -0500, Anton Kapela wrote:
On Fri, Mar 27, 2015 at 10:25 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration.
indeed - not much, but quite useful (if one doesn't like things rolling slow) for when one rocks out w/ their / or /usr or whatnot on a sync-mounted fs ;)
Still wrong problem. Just sayin'. ZFS and UFS are vastly different with metadata updates anyway so it really isn't part of the problem you are trying to solve. BTW: Still trolling. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote:
(Chiming in because storage and OS nerding)
Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through?
Overkill and wrong solution for the wrong, imaginary, problem.
A pair of SLC-type SSD's with ext3/4, or XFS mounted 'sync' atop a write-through raid1 should still be entirely awesome and performant for the job of booting an OS and loading route-server binaries, and perhaps even soaking up a few syslog outputs...while being as 'safe' as the hardware allows.
This isn't Linux either. Cause. You don't need linux to be your every hammer for every nail. Cause that's my storage and OS nerding coming out. Wanna really go to town? GlusterFS with 3 nodes with replica=3 and one node not on same network. But that won't help the system from crashing because of potential firmware bugs going on as Doug has mentioned. That's the problem, not an issue of corrupted storage. In fact, as far as I know, there has been zero corruption of data during these periods of instability showing that FreeBSD rocks even when using ZFS and I bet would be just as stable with UFS.
If that sounds tempting, tell me where to fedex the ssd media & caddy adapters :)
SSDs always good though. BTW: I'm a troll. -- Mike Horwath, reachable via drechsau@Geeks.ORG
participants (6)
-
Anton Kapela -
Brady Kittel -
Doug McIntyre -
Jeff Anderson -
Mike Hemphill -
Mike Horwath