March 2015 - mice-discuss - lists.micemn.net

Re: MICE Governance Documents

by Richard Laager

I would still like to see operators of remote switches (e.g. Mankato Networks) be allowed to be members. PROPOSAL 1A: In order to qualify for membership, a member shall be: (i) an operator of an Internet Protocol network which has one or more direct, or approved indirect, connections to Midwest Internet Cooperative Exchange LLC’s (the “Company”) switches; or (ii) an operator of equipment providing approved indirect connections. PROPOSAL 1B: [ If we feel it is important to allow for Cologix (and similar companies in the future) to be members, then... ] In order to qualify for membership, a member shall be: (i) an operator of an Internet Protocol network which has one or more direct, or approved indirect, connections to Midwest Internet Cooperative Exchange LLC’s (the “Company”) switches; (ii) an operator of equipment providing approved indirect connections; or (iii) an operator of a colocation data center in which the Company's switches are located. I'd like to see this sentence change: CURRENT: A member may be elected or appointed to membership by the Board. PROPOSAL 2A: A member may be elected or appointed to membership. PROPOSAL 2B: A member may be elected to membership by the members or appointed to membership by the Board. -- Richard

March 27, 2015

MICE Governance Documents

by Shaun Carlson

All, I would invite you to review the attached documents for any concerns or potential changes. Please note one suggested change annotated in red (if you have tracking turned on). If you have any concerns or changes, please send them to the list for comment and review in the near future. Mike, Reid and I will be meeting on the 8th to go over a few things and set the dates for a couple of special meetings to provide an opportunity for comment and review in an immediate forum as well. We intend, at this time, to hold those special meetings via webex and to have the vote following those meetings. Thank you for your time in working through this process! s *Shaun Carlson*Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com

March 27, 2015

Re: FW: New gen route servers

by Anton Kapela

On Fri, Mar 27, 2015 at 10:25 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > I'm not sure SSD would bring much, so far it has been controller failures, > and not disk failures. Everything has all been enterprise grade drives in > every iteration. indeed - not much, but quite useful (if one doesn't like things rolling slow) for when one rocks out w/ their / or /usr or whatnot on a sync-mounted fs ;) -Tk

March 27, 2015

Re: FW: New gen route servers

by Brady Kittel

Sounds good, we could probably pickup the new dell servers under our hardware support for server parts if they don't have coverage currently. We get 24x7x4 parts sourcing from IBM logistics (yes they carry dell parts). I'd just need the service tags and model info off of them. On Mar 27, 2015 10:25 AM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote: > > Given the use case (safety > perf), perhaps the route server storage > would be best matched by something like a sync-mounted file system, raid1 > with no cache enabled, or read cache only with a forced write-through? > > Everything has all been RAID1, the old setup had the write-back cache, > all the new setups don't have battery backed up cache, so they are all > write-through setups. > > I'm not sure SSD would bring much, so far it has been controller failures, > and not disk failures. Everything has all been enterprise grade drives in > every iteration. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

March 27, 2015

Re: FW: New gen route servers

by Doug McIntyre

On Fri, Mar 27, 2015 at 07:13:12AM -0500, Anton Kapela wrote: > Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through? Everything has all been RAID1, the old setup had the write-back cache, all the new setups don't have battery backed up cache, so they are all write-through setups. I'm not sure SSD would bring much, so far it has been controller failures, and not disk failures. Everything has all been enterprise grade drives in every iteration. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Anton Kapela

(Chiming in because storage and OS nerding) Given the use case (safety > perf), perhaps the route server storage would be best matched by something like a sync-mounted file system, raid1 with no cache enabled, or read cache only with a forced write-through? A pair of SLC-type SSD's with ext3/4, or XFS mounted 'sync' atop a write-through raid1 should still be entirely awesome and performant for the job of booting an OS and loading route-server binaries, and perhaps even soaking up a few syslog outputs...while being as 'safe' as the hardware allows. If that sounds tempting, tell me where to fedex the ssd media & caddy adapters :) -Tk > On Mar 27, 2015, at 1:26 AM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote: >> As an alternate option I have several HP dl380 g7's that are maybe a year >> old and still under warranty I could provide. They have licensed ILO on >> them as well and I could provide some hot spare drives for them. > > I think with the current PE R320 and PE R805, that we are probably set, but > thank you for your offer. The R320 is a pretty new box, hmm, pulling the > serial # and going to Dell, it looks like it is only 16 months old. Jeff > mentioned they had some more coming out of cycle, and that people were looking > to get a hot spare at the UG meeting? > > I think the specific problem I was seeing with the old hardware was > due to a very specific configuration. With the PERC6/i battery backed > up RAID controller, and some combination due to the age of the battery, > and that specific design, which periodically (every 2-3 months or so) does > a deep drain and charge cycle on the battery to make sure it can hold its' > intended charge for the writeback log, during this cycle it switches the > writeback log off and then on again, and sometimes the OS glitches on this > sudden change in disk latency. My guess is the batteries were below a certain > threshold point that exacerbated the writelog switchover, since we have been > running on the same hardware for quite some time without issues, but not > below the threshold which would alert the BCM on the RAID battery being bad. > > Both of these new systems do not have a battery backed up RAID system, which > really isn't needed in this use case anyway, and are newer hardware, which I > think would serve us well. > > I have both inhouse, with the OS installed, I just have to copy the configs back over, > which I can get done pretty quickly, as it isn't extensive, and haul them down > to 511. I'll have to schedule up some sort of switchout time, how much lead time > would a graceful changeout of the hardware be required? > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Doug McIntyre

On Thu, Mar 26, 2015 at 06:24:03PM -0500, Brady Kittel wrote: > As an alternate option I have several HP dl380 g7's that are maybe a year > old and still under warranty I could provide. They have licensed ILO on > them as well and I could provide some hot spare drives for them. I think with the current PE R320 and PE R805, that we are probably set, but thank you for your offer. The R320 is a pretty new box, hmm, pulling the serial # and going to Dell, it looks like it is only 16 months old. Jeff mentioned they had some more coming out of cycle, and that people were looking to get a hot spare at the UG meeting? I think the specific problem I was seeing with the old hardware was due to a very specific configuration. With the PERC6/i battery backed up RAID controller, and some combination due to the age of the battery, and that specific design, which periodically (every 2-3 months or so) does a deep drain and charge cycle on the battery to make sure it can hold its' intended charge for the writeback log, during this cycle it switches the writeback log off and then on again, and sometimes the OS glitches on this sudden change in disk latency. My guess is the batteries were below a certain threshold point that exacerbated the writelog switchover, since we have been running on the same hardware for quite some time without issues, but not below the threshold which would alert the BCM on the RAID battery being bad. Both of these new systems do not have a battery backed up RAID system, which really isn't needed in this use case anyway, and are newer hardware, which I think would serve us well. I have both inhouse, with the OS installed, I just have to copy the configs back over, which I can get done pretty quickly, as it isn't extensive, and haul them down to 511. I'll have to schedule up some sort of switchout time, how much lead time would a graceful changeout of the hardware be required? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 27, 2015

Re: FW: New gen route servers

by Brady Kittel

Correction just checked the tags, they're actually about 2 and a half years old. On Thu, Mar 26, 2015 at 6:24 PM, Brady Kittel <bkittel(a)gmail.com> wrote: > As an alternate option I have several HP dl380 g7's that are maybe a year > old and still under warranty I could provide. They have licensed ILO on > them as well and I could provide some hot spare drives for them. They're > also massively over speced for this use (quad core processors, 8-12GB of > memory, 4 x 15k drives. We've had much better luck with the HP hardware > with regard to firmware stability in that generation range than the dell > hardware we run for what it's worth. Given we only run maybe a thousand > servers but the sample size seems large enough to bare that out. > > On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn(a)iphouse.net> > wrote: > >> On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: >> > This machine is ready to go if MICE would like it. I can drop it by >> this afternoon. >> > Here are the specs: >> > Dell R320 >> > Single socket 4 core 1.8ghz Xeon. >> > PERC H310 Mini with RAID 10 >> > Qty 4 300 gig 10k SAS disks >> > 8Gig RAM >> >> >> Sure! I think that would be an excellent upgrade to newer gear than >> what we have now. Of course, it is total overkill for what it needs, >> but then again just about anything enterprise hardware is. >> >> When the R805 is ready, I could swing out and do up this one at 511, >> probably >> easier than hauling it back and forth, even I'm only 8 blocks away. >> >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> > >

March 26, 2015

Re: FW: New gen route servers

by Brady Kittel

As an alternate option I have several HP dl380 g7's that are maybe a year old and still under warranty I could provide. They have licensed ILO on them as well and I could provide some hot spare drives for them. They're also massively over speced for this use (quad core processors, 8-12GB of memory, 4 x 15k drives. We've had much better luck with the HP hardware with regard to firmware stability in that generation range than the dell hardware we run for what it's worth. Given we only run maybe a thousand servers but the sample size seems large enough to bare that out. On Thu, Mar 26, 2015 at 11:50 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: > > This machine is ready to go if MICE would like it. I can drop it by this > afternoon. > > Here are the specs: > > Dell R320 > > Single socket 4 core 1.8ghz Xeon. > > PERC H310 Mini with RAID 10 > > Qty 4 300 gig 10k SAS disks > > 8Gig RAM > > > Sure! I think that would be an excellent upgrade to newer gear than > what we have now. Of course, it is total overkill for what it needs, > but then again just about anything enterprise hardware is. > > When the R805 is ready, I could swing out and do up this one at 511, > probably > easier than hauling it back and forth, even I'm only 8 blocks away. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

March 26, 2015

Re: FW: New gen route servers

by Doug McIntyre

On Thu, Mar 26, 2015 at 10:45:03AM -0500, Jeff Anderson wrote: > This machine is ready to go if MICE would like it. I can drop it by this afternoon. > Here are the specs: > Dell R320 > Single socket 4 core 1.8ghz Xeon. > PERC H310 Mini with RAID 10 > Qty 4 300 gig 10k SAS disks > 8Gig RAM Sure! I think that would be an excellent upgrade to newer gear than what we have now. Of course, it is total overkill for what it needs, but then again just about anything enterprise hardware is. When the R805 is ready, I could swing out and do up this one at 511, probably easier than hauling it back and forth, even I'm only 8 blocks away. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

March 26, 2015