- mice-discuss - lists.micemn.net

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Mike Horwath

On Thu, Dec 01, 2016 at 09:39:43AM -0600, Jason Hanke wrote: > USI has arranged for Arista Networks to donate a 7504 switch with > 32x100G and 96x10G ports with one year of support. In year 2, MICE > would be responsible for the ongoing support costs about 14k per > year. There would be one additional slot for future growth. First - thank you USI for diving into this! https://www.arista.com/en/products/7500-series for those that wanna look into what this chassis is capable of. > Here's what I'd propose: > 1G ports= Free (Only 1 port per member, NO LAG allowed) > Initial 10G port= $250 Annual > Each additional 10G port $1000 Annual per port (up to 8 member LAG > allowed) > 100G port $3000 Annual per port My counter commentary + fee list: 1Gbe ports - $250 annual per port, LAG allowed, max 2 10Gbe ports - $500 annual per port, LAG allowed, max 4 40Gbe ports - $2000 annual per port, cause why not just do 100Gbe!?!, LAG max 2, and above no 40Gbe ports were listed so really, do 100Gbe and let's move on, okay? 100Gbe ports - $3000 annual per port, LAG allowed, max 4 Optics provided by the member and must be on the approved list for the hardware (in this case: Arista 7504) - I know that later on Jay mentions that Arista will remove the lockout for them but I don't think MICE should just allow who-flung-poo optics installed because the member is trying to be cheap at the risk of problems with the fabric. Non-participating member - $100 annual, no voting position, feel good fee Participating member - $250 annual, voting position, no port, extra feel good fee. Users who have ports already assigned into a LAG should be grandfathered in for their configuration regardless of LAG port count limit as proposed. If a member is already at 5 10Gbe ports then let them stay at that level. If member wants to add another port then push them towards the 100Gbe solution but allow for a single port increase in LAG size only. > Remote switch operators pay the fee for their main switch ports. > Remote switch participants do not pay a main switch port fee. Seems fair! But if a remote switch operator is charging at a higher rate than paid into the current fees should a portion of that be sent into the funding as well? > No grandfathering, all types of networks are treated the same. YES! > We will not disconnect for non-payment until port run out and > contention occurs. A policy will need to be developed. I don't agree - non-payment == shutdown IMO. And if continues for 3 months is then removed from the fabric and pulled from site. Suspend and comment out the BGP configuration and IP allocation for up to 6 months before removal, and later reallocation of IP space. > Each carrier provides their own optic. Arista has agreed to remove > the vendor lockout for optics. Will they support non-approved optics then? Jay then brings up revenue projections but I think it should go to everyone as this is a single chassis solution. I'd like to see at least a dual chassis solution and members spread across as needed to keep resiliency high. Yes, I know, I am no longer officially part of MICE and my opinion may not matter to all of you. And that's okay :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Doug McIntyre

On Thu, Dec 01, 2016 at 10:19:05AM -0600, Steve Howard wrote: > How many 1G ports are on the Arista? Any of the 10G ports can be run at 1G. Likewise, any of the 100G ports can be broken out to 4 x 10G, 4 x 25G, 1? x 40G, 2 x 50G or 1 x 100G. with the correct breakout cables. > Would we continue to use the original switch for 1G ports? I think that would be the best course of action. Right now that is dual-10G connected to the main 10G switch stack. > Do the existing switch and Arista support the same form-factor > optics? Yes, SFP+ form factor for 10G. QSFP for higher speeds. > i.e. Is there a requirement to purchase new optics to be > connected to the new switch under your proposal? Arista says they will disable the vendor lockin on the optics for this deployment. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Jason Hanke

> > 1 - How does $1800 / year cover the costs of actually DOING the > accounting, e.g. creating new members & invoices, processing payments, > calling on collections, etc. This number seems low to me. Additional > thought, some entities will want to pay with a credit card, we should > prepare for this and cover the costs of processing payments in the budget. > Neutral Path is currently donating bookkeeping services to the exchange. By Neutral Path I really mean Kayla. This is factored into the current budget. Other than the proposed new invoicing there are less than 10 transactions per year. We can use Paypal to process credit cards the fee is about 3%. > > 2 - Where is hardware refresh in the budget? I see a clipped sentence in > your message ( This would cover hardware support and Arin fees and > accounting expense with the remainder going) - maybe it was in there? > Depending on the response to the request for payment there is between 8k and 24k annually building up for hardware refresh or additions. A large update will have to be dealt with at the time or with a future rate increase. > > We have been lucky to secure donations of equipment, but what about the > future? Your budget should include setting aside dollars for hardware > expansion and refresh, and not just switches, but route servers too. > Minor additions are figured in, Major replacement is not. > > Additional thoughts, if new technologies emerge and a hardware vendor that > has said technology doesn't feel like donating equipment, how would we > proceed without a hardware budget? And... by accepting donations from > hardware vendors, we are setting the precedent of "picking a winner", and > could cause headache/heartache in the future. > We would have to deal with new technologies as they emerge. We've been accepting hardware donations for quite some time without serious issue. The current kit contains a lot of donated parts. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Andrew Hoyos

On Dec 1, 2016, at 10:42 AM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> wrote: > > > If we’re going the annual route - any reason/logic behind why the 20g -> 80g range seems to carry a disproportionate amount of the fee load? > > The networks in this range are moving the bulk of the traffic on the IX Yes, but they are also providing the most value to the IX for all members. I think we need to be agnostic about how many ports a member has or how much traffic they are moving. I still think this fee structure unnecessarily classifies a group of the members into paying more on a $/gbps ratio, which doesn’t make sense. Steve’s logic, in his email, makes way more sense to me, and is the totally agnostic - simplify this into a fixed port level fee, and done. 1Gbps=$35/Year (although, I’d argue that this might as well be free at that point for the 1g members) 10Gbps=$350/Year 100Gbps=$3,500/Year > > Why shouldn’t all 10g ports just be $250/yr? With this layout, I fear that the “Big Guys” will think you’re dumping the fee load (and downstream equipment cost) onto them by forcing 100g ports at that cost for n x 10g. Seems like being agnostic on a port level (no matter how many you have) would be a better move. > > Looking at the overall economics to a member the cross connect cost dominates the spend. Specifically when doing lag. If you take the annual fee and convert it to monthly it works out to be about $83 dollars with a $200 cross connect for a monthly total of $283 on a second port. > > For example 4 member 10G lag (monthly): > > Port 1 $200+$21=$221 > Port 2 $200+$83=$283 > Port 3 $200+$83=$283 > Port 4 $200+$83=$283 > ------------------------------- > Total $1070 per month > > A single 100G > Port 1 $200+$250=$450 per month > > This leaves room to cover the incremental 100G optics cost. If the 10G ports were free the 100G would still be a better deal for the big guys. Right, but you’re leaving out the internal costs for the big guys (or mid tier/small folks even) of router ports. It’s far cheaper for one of the big guys to light another 10g port, I’d suspect, than add 100g interfaces. Maybe that’s changing, but again, I don’t agree with penalizing folks for adding more traffic to the IX on multiple 10g ports vs single 100g ports. MICE isn’t paying cross connects, so seems like that should’t be part of the equation when we look at it from a fee charging perspective.

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Jason Hanke

> If we’re going the annual route - any reason/logic behind why the 20g -> 80g range seems to carry a disproportionate amount of the fee load? The networks in this range are moving the bulk of the traffic on the IX > Why shouldn’t all 10g ports just be $250/yr? With this layout, I fear that the “Big Guys” will think you’re dumping the fee load (and downstream equipment cost) onto them by forcing 100g ports at that cost for n x 10g. Seems like being agnostic on a port level (no matter how many you have) would be a better move. Looking at the overall economics to a member the cross connect cost dominates the spend. Specifically when doing lag. If you take the annual fee and convert it to monthly it works out to be about $83 dollars with a $200 cross connect for a monthly total of $283 on a second port. For example 4 member 10G lag (monthly): Port 1 $200+$21=$221 Port 2 $200+$83=$283 Port 3 $200+$83=$283 Port 4 $200+$83=$283 ------------------------------- Total $1070 per month A single 100G Port 1 $200+$250=$450 per month This leaves room to cover the incremental 100G optics cost. If the 10G ports were free the 100G would still be a better deal for the big guys. > I’d also disagree with not allowing multiple 1g ports for customers into a LAG. For small bootstrapped companies, that 1g -> 10g jump can be expensive, and 2xGigE LAG doesn’t seem like a problem in the case where they are hitting that 60-70% on a GigE. Given the Arista architecture and depth of buffers there, I wouldn’t see any technical issue leaving the EX4550 hanging off the 7504, just for 1g customers, if you’re worried about burning “expensive” ports. > Again the cross connect fees dominate the economics at the 1G level. For the initial hardware costs. 1G lag would be allowed on the remote switches if the remote switch operator allowed it. I think all the remote switches would allow it, Neutral Path certainly would. I don't think we've had a request for a 1G lag since we had 10G available on the main switch. All of the aggregate traffic on the "1G" switch is peaking at less than 1G. [image: Inline image 1] Jay

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Reid Fishler

Bill- The problem is we can't charge more than some members use the exchange, and we can't plan for everything. We may still go to the membership and ask for support if need-be, but we can't have the cost of the exchange be too high...look at the other competitive exchanges of our size...all are doing something similar. Believe me, the board spent ALOT of time working on this...ALOT. :-) Reid On Thu, Dec 1, 2016 at 11:38 AM, Bil MacLeslie <bil(a)iphouse.net> wrote: > On 12/1/16 9:39 AM, Jason Hanke wrote: > > Arin: $300 >> Accounting: $1800 (includes postage for billing) >> Hardware support: $15000 >> > > Two questions. > > 1 - How does $1800 / year cover the costs of actually DOING the > accounting, e.g. creating new members & invoices, processing payments, > calling on collections, etc. This number seems low to me. Additional > thought, some entities will want to pay with a credit card, we should > prepare for this and cover the costs of processing payments in the budget. > > 2 - Where is hardware refresh in the budget? I see a clipped sentence in > your message ( This would cover hardware support and Arin fees and > accounting expense with the remainder going) - maybe it was in there? > > We have been lucky to secure donations of equipment, but what about the > future? Your budget should include setting aside dollars for hardware > expansion and refresh, and not just switches, but route servers too. > > Additional thoughts, if new technologies emerge and a hardware vendor that > has said technology doesn't feel like donating equipment, how would we > proceed without a hardware budget? And... by accepting donations from > hardware vendors, we are setting the precedent of "picking a winner", and > could cause headache/heartache in the future. > > I strongly suggest we use the port fees to pay for hardware refresh as > well as hardware maintenance. > > Bil > > -- > Bil K. MacLeslie | TGWGTD | ipHouse | www.ipHouse.com > Office: 612-337-6337 | Direct: 612-337-6206 | Cell: 612-618-1218 > > -- Reid Fishler Director Hurricane Electric +1-510-580-4178

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Bil MacLeslie

On 12/1/16 9:39 AM, Jason Hanke wrote: > Arin: $300 > Accounting: $1800 (includes postage for billing) > Hardware support: $15000 Two questions. 1 - How does $1800 / year cover the costs of actually DOING the accounting, e.g. creating new members & invoices, processing payments, calling on collections, etc. This number seems low to me. Additional thought, some entities will want to pay with a credit card, we should prepare for this and cover the costs of processing payments in the budget. 2 - Where is hardware refresh in the budget? I see a clipped sentence in your message ( This would cover hardware support and Arin fees and accounting expense with the remainder going) - maybe it was in there? We have been lucky to secure donations of equipment, but what about the future? Your budget should include setting aside dollars for hardware expansion and refresh, and not just switches, but route servers too. Additional thoughts, if new technologies emerge and a hardware vendor that has said technology doesn't feel like donating equipment, how would we proceed without a hardware budget? And... by accepting donations from hardware vendors, we are setting the precedent of "picking a winner", and could cause headache/heartache in the future. I strongly suggest we use the port fees to pay for hardware refresh as well as hardware maintenance. Bil -- Bil K. MacLeslie | TGWGTD | ipHouse | www.ipHouse.com Office: 612-337-6337 | Direct: 612-337-6206 | Cell: 612-618-1218

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Dean Bahls

I really like this proposal! Small users can connect painlessly. (I might suggest that LAG is optional but they have to pay something for the 2nd+ 10G ports. Maybe it's $250 for each additional 1G annually?) >From CNS's perspective, we'd be into it for $2,250 annually for the three 10G ports that we consume on the main switch. This is reasonable and affordable. If we decide to charge our downstream customers to offset the cost, that's on us. Thanks to USI and Arista as well! Dean -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Thursday, December 01, 2016 9:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Fees Proposal/UG Agenda Item/Request for Comment All- Our main switch is nearing the end of its lifespan and the available port count is dwindling. At the current rate of burn we will be out of ports in early-2017. USI has arranged for Arista Networks to donate a 7504 switch with 32x100G and 96x10G ports with one year of support. In year 2, MICE would be responsible for the ongoing support costs about 14k per year. There would be one additional slot for future growth. I would like to make a formal request for comment and add the following item to the UG agenda. Here's what I'd propose: 1G ports= Free (Only 1 port per member, NO LAG allowed) Initial 10G port= $250 Annual Each additional 10G port $1000 Annual per port (up to 8 member LAG allowed) 100G port $3000 Annual per port Remote switch operators pay the fee for their main switch ports. Remote switch participants do not pay a main switch port fee. No grandfathering, all types of networks are treated the same. We will not disconnect for non-payment until port run out and contention occurs. A policy will need to be developed. New ports billed at time of allocation. Each carrier provides their own optic. Arista has agreed to remove the vendor lockout for optics. This will yield about $37k of annual revenue assuming everyone pays. We don't disconnect for non-payment but members in good standing would take priority over non-paying members. This would cover hardware support and Arin fees and accounting expense with the remainder going If we extended the full fees to all 10G ports this would generate $70k annually. Looking out, I don't think we will need to raise that much cash to keep operating as we are now. Ongoing Expense looks something like this: Arin: $300 Accounting: $1800 (includes postage for billing) Hardware support: $15000 This financial projection assumes we add no additional ports than what is currently allocated. I'm seeking input/amendment from the membership prior to the board taking action for the 2017 year. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Steve Howard

Dec. 1, 2016

Re: MICE Fees Proposal/UG Agenda Item/Request for Comment

by Andrew Hoyos

On Dec 1, 2016, at 9:39 AM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> wrote: > > USI has arranged for Arista Networks to donate a 7504 switch with > 32x100G and 96x10G ports with one year of support. In year 2, MICE > would be responsible for the ongoing support costs about 14k per year. > There would be one additional slot for future growth. That’s awesome, thanks USI and Arista! Seems like a no brainer to proceed with that option. > Here's what I'd propose: > 1G ports= Free (Only 1 port per member, NO LAG allowed) > Initial 10G port= $250 Annual > Each additional 10G port $1000 Annual per port (up to 8 member LAG allowed) > 100G port $3000 Annual per port If we’re going the annual route - any reason/logic behind why the 20g -> 80g range seems to carry a disproportionate amount of the fee load? Why shouldn’t all 10g ports just be $250/yr? With this layout, I fear that the “Big Guys” will think you’re dumping the fee load (and downstream equipment cost) onto them by forcing 100g ports at that cost for n x 10g. Seems like being agnostic on a port level (no matter how many you have) would be a better move. For comparison on a IX that charges yearly and is non-profit like MICE, NWAX might be a good example, which is charging: 1gbps Port 1st port FREE - additional, $295/month 10gbps Port $295/month 100gbps Port $1995/month + $6000 NRC I’d also disagree with not allowing multiple 1g ports for customers into a LAG. For small bootstrapped companies, that 1g -> 10g jump can be expensive, and 2xGigE LAG doesn’t seem like a problem in the case where they are hitting that 60-70% on a GigE. Given the Arista architecture and depth of buffers there, I wouldn’t see any technical issue leaving the EX4550 hanging off the 7504, just for 1g customers, if you’re worried about burning “expensive” ports. Food for thought, YMMV, etc. -andrew -- Andrew Hoyos hoyosa(a)gmail.com

Dec. 1, 2016