- mice-discuss - lists.micemn.net

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

So if I understand correctly from your "show virtual-chassis active-topology" below, all traffic from the 4500 to the 4200 is going via the 4550? And if that is the case, is there an easy way in the software that we could tell it to have return traffic from the 4200 to the 4500 also go via the 4550? This would allow us to easily test to see if it is the specific cable that runs from the 4200 to the 4500. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, September 20, 2016 12:24 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. ... Your understanding of the vc port speeds is correct. The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon cards that do those functions. We can devote a port to VCP functionality instead, but it'll be a 10G limit instead of 32G The VCP ports are 32Gbps single direction in each port. So in the dual-ring the cables have are 32Gbps in each direction. You can see which ports are connected to what. (FPC0 is the EX4500, FPC1 is the EX4200, FPC2 is the EX4550). show virtual-chassis active-topology fpc0: -------------------------------------------------------------------------- Destination ID Next-hop 1 2(vcp-1.32768) 1(vcp-0.32768) 2 2(vcp-1.32768) fpc1: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-0.32768) 2(vcp-1.32768) 2 2(vcp-1.32768) fpc2: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-255/2/0.32768) 1 1(vcp-255/2/3.32768) You can monitor each of the VCP ports on each of the members. I don't think we are close to saturation. request session member 1 (ie. rlogin to the EX4200 switch). monitor int vcp-0 Interface: vcp-0, Enabled, Link is Up Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps Traffic statistics: Current delta Input bytes: 25685118521 [12138] Output bytes: 27856843010 [13204] Input packets: 33377693 [16] Output packets: 33371644 [16] Error statistics: Input errors: 0 [0] Input drops: 0 [0] Input framing errors: 0 [0] Carrier transitions: 0 [0] Output errors: 0 [0] Output drops: 0 [0] vcp-1 on FPC1 is much less usage. Still zero error stats. All switches show zero error stats on any of the respective VCP port status. I suppose further troubleshooting could be to replace the VCP cables one by one coming out of the EX4200. Its too bad that we've got the quad fiber expansion in the EX4200, another test could be to put the dual 10G expansion in it, and make those VCP ports and make the VC fabric over those ports instead. But we've got 3 member ports lit on that quad 1G fiber. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. ... Your understanding of the vc port speeds is correct. The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon cards that do those functions. We can devote a port to VCP functionality instead, but it'll be a 10G limit instead of 32G The VCP ports are 32Gbps single direction in each port. So in the dual-ring the cables have are 32Gbps in each direction. You can see which ports are connected to what. (FPC0 is the EX4500, FPC1 is the EX4200, FPC2 is the EX4550). show virtual-chassis active-topology fpc0: -------------------------------------------------------------------------- Destination ID Next-hop 1 2(vcp-1.32768) 1(vcp-0.32768) 2 2(vcp-1.32768) fpc1: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-0.32768) 2(vcp-1.32768) 2 2(vcp-1.32768) fpc2: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-255/2/0.32768) 1 1(vcp-255/2/3.32768) You can monitor each of the VCP ports on each of the members. I don't think we are close to saturation. request session member 1 (ie. rlogin to the EX4200 switch). monitor int vcp-0 Interface: vcp-0, Enabled, Link is Up Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps Traffic statistics: Current delta Input bytes: 25685118521 [12138] Output bytes: 27856843010 [13204] Input packets: 33377693 [16] Output packets: 33371644 [16] Error statistics: Input errors: 0 [0] Input drops: 0 [0] Input framing errors: 0 [0] Carrier transitions: 0 [0] Output errors: 0 [0] Output drops: 0 [0] vcp-1 on FPC1 is much less usage. Still zero error stats. All switches show zero error stats on any of the respective VCP port status. I suppose further troubleshooting could be to replace the VCP cables one by one coming out of the EX4200. Its too bad that we've got the quad fiber expansion in the EX4200, another test could be to put the dual 10G expansion in it, and make those VCP ports and make the VC fabric over those ports instead. But we've got 3 member ports lit on that quad 1G fiber. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

So this brings an issue I was watching for a couple days into a new light. I have a customer that is in Rochester MN on Charter's Fiber. They have 2 locations just a couple miles apart. I was monitoring their connections before we start to do their voice on the 22nd. I was surprised to see I was getting about 0.3% loss to both locations. The loss would happen at the same times, so I was tempted to blame Charter. When Matthew posted this today it reminded me that charter is on the 4500 and I am on the 4200 just like Matthew. So I started two different tests this afternoon, and I now have gathered results for last 6 hours, so I feel confident in the result that there is loss in traffic going from the 4200 to the 4500. Test 1 was to add a static route to one of my customer's Rochester locations, so that traffic from me to them went via HE, and then Charter. The location that I added the static route to has had no loss for the past 6 hours, and the location that I did not change has continued to show 0.3% loss. Obviously the return path still takes MICE, so I do not believe traffic from the 4500 to the 4200 is having a problem. Test 2 was to start a running ping to 9 different interface IP addresses on MICE. I choose 3 from each switch to avoid the possibility that one router was an outlier and overly deprioritizing ICMP. I let that run for almost 6 hours, and all interfaces on the 4200, and the 4550 showed 0 loss, and all three on the 4500 showed 0.2% loss. Looking deeper into the data, the loss that happened on the 4500 would often happen at the same time for all three addresses. This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. Let me preface this by saying I am not a Juniper guy, and I may be wrong, however my understanding is that the stacking modules are 128Gig which would seem like plenty, however it is my understanding that each module has 2 ports, and each one is 64G, and then that 64G is actually in + out, and therefore a single direction of one port could get maxed out at 32G. I remember when we added the 4550 I asked if there was a way to graph the traffic on them, and I believe we were not able to. While I do not think that they are saturated, I wonder if there are small bursts that get them near to capacity that are far too short to show up on the 5 minute averages. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Monday, September 19, 2016 4:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Testing against the UW iperf servebbb, but no loss running the same 10M UDP test). So this (somewhat limited) test seems to show the current path from 4200 -> 4550 is okay. ~Matthew On Mon, Sep 19, 2016 at 4:06 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: UW Madison operates some IPerf servers you could try against too, see: https://kb.wisc.edu/uwsysnet/page.php?id=41947 They are on the 4550, it appears, so more data points to be gleaned there. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: Proposed Membership and Sponsorship Model

by Jason Hanke

I've looked at this and I support the SIX style of user fees for MICE. I also very much like that the remotes can charge their own rates. This should create options for members who prefer a different style of payment (perhaps monthly with little upfront?). The other shared resource is address space. We may want to consider a cap per ASN. Perhaps 2 without board/tech committee approval? In the propose scenario there would be no direct cost to the end user for these. I'd also selfishly support membership voting rights for remote switch operators. Jay On Mon, Sep 19, 2016 at 4:56 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > I fully support this model as well. (see: https://www.seattleix.net/join) > > The large expenditures of new equipment are all member growth or new > member driven. Recouping a 1/nth portion of those switch upgrade costs per > member, plus some small overhead for the rest, seems to be a totally viable > model and has worked well for SIX. (plus donations). The rest of the > expense side of the house is pennies in comparison. > > For those that don’t think an IX can be driven, at least partially by > donations, maybe review this as well: https://www.seattleix.net/ > contributors > > This simplifies things on the backend too: > - one time payment vs monthly deposits via multiple methods that someone > on MICE board would need to manage for 60+ members > - members joining the IX have an easy sell internally of one time fee vs > MRC above and beyond a cross connect (always infinitely harder to justify) > > Solves the root issue of new members and more bandwidth (higher speed > ports) driving growth, and doesn’t penalize the little guys. > > Remote switches can charge their own rates, however, still need to pay the > main core switch port costs. (https://www.seattleix.net/rules) > > -- > Andrew Hoyos > hoyosa(a)gmail.com > > > > > On Sep 19, 2016, at 2:35 PM, Justin Krejci <jkrejci(a)usinternet.com> > wrote: > > > > I am a fan of the SIX model of revenue: you pay to connect to the > exchange up front and bigger ports (ie 10-Gig) cost more than smaller (ie > 1-Gig). This fee applies to MICE core equipment only. Port costs on > extension switches are dictated by the respective extension switch > operators. > > > > As there are already many existing connected ports I would also be in > favor of retroactively charging each connected port based on their current > port type(s) in use at the time of the implementation of said fees. Already > connected organizations that want to take a bit of time to budget can make > arrangements with the board. Also, I would be in favor of implementing this > soon instead of waiting for the desired but elusive non-profit status > unless someone has knowledge of its actual pending completion. > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 19, 2016

Re: Proposed Membership and Sponsorship Model

by Andrew Hoyos

I fully support this model as well. (see: https://www.seattleix.net/join) The large expenditures of new equipment are all member growth or new member driven. Recouping a 1/nth portion of those switch upgrade costs per member, plus some small overhead for the rest, seems to be a totally viable model and has worked well for SIX. (plus donations). The rest of the expense side of the house is pennies in comparison. For those that don’t think an IX can be driven, at least partially by donations, maybe review this as well: https://www.seattleix.net/contributors This simplifies things on the backend too: - one time payment vs monthly deposits via multiple methods that someone on MICE board would need to manage for 60+ members - members joining the IX have an easy sell internally of one time fee vs MRC above and beyond a cross connect (always infinitely harder to justify) Solves the root issue of new members and more bandwidth (higher speed ports) driving growth, and doesn’t penalize the little guys. Remote switches can charge their own rates, however, still need to pay the main core switch port costs. (https://www.seattleix.net/rules) -- Andrew Hoyos hoyosa(a)gmail.com > On Sep 19, 2016, at 2:35 PM, Justin Krejci <jkrejci(a)usinternet.com> wrote: > > I am a fan of the SIX model of revenue: you pay to connect to the exchange up front and bigger ports (ie 10-Gig) cost more than smaller (ie 1-Gig). This fee applies to MICE core equipment only. Port costs on extension switches are dictated by the respective extension switch operators. > > As there are already many existing connected ports I would also be in favor of retroactively charging each connected port based on their current port type(s) in use at the time of the implementation of said fees. Already connected organizations that want to take a bit of time to budget can make arrangements with the board. Also, I would be in favor of implementing this soon instead of waiting for the desired but elusive non-profit status unless someone has knowledge of its actual pending completion.

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Testing against the UW iperf servers seems to be mostly clean. (Occasionally, I get one out-of-order, but no loss running the same 10M UDP test). So this (somewhat limited) test seems to show the current path from 4200 -> 4550 is okay. ~Matthew On Mon, Sep 19, 2016 at 4:06 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > UW Madison operates some IPerf servers you could try against too, see: > https://kb.wisc.edu/uwsysnet/page.php?id=41947 > They are on the 4550, it appears, so more data points to be gleaned there. > >

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

> On Sep 19, 2016, at 2:19 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > On Mon, Sep 19, 2016 at 02:03:41PM -0500, Andrew Hoyos wrote: >> What medium is the VC built on and physical topology? Just VC cables on the rear ringed through all 3? > > Yes, VC cabling in a ring, and using the VCP modules and cabling on the EX4550. What’s the actual topology though? 4500 -> 4200 -> 4550 -> back to 4500? >> Specific to any single 4500 destination or is it doing it to both the 4500 and the 4550? > > He wasn't aware of the EX4550, there aren't many ports lit there yet, > he has just been testing different ports on the the EX4500. UW Madison operates some IPerf servers you could try against too, see: https://kb.wisc.edu/uwsysnet/page.php?id=41947 They are on the 4550, it appears, so more data points to be gleaned there. >> Check ‘show virtual-chassis vc-path source-interface <src int> destination-interface <dest int>’. Common VC port in the mix to any of the reported loss paths? > > I think in general, the VC ports run Active/Passive, so only one is > "active" at a time going out of a box without load balancing, so only > one (ie. vcp-1) typically shows up for all paths. Kinda, there is a SPF calculation that happens based on src/dst interfaces. It matters which PFE of the switch the port is on (1 hop != switch, it’s actually each PFE hop). Depending on the topology here, things could be going a variety of ways. -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

On Sep 19, 2016, at 3:29 PM, Matthew Beckwell <matthewb(a)aitech.net> wrote: > > This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. > Some kind of buffer or queue difference? IIRC on the 4500, there is a PFE mapped to ports 0-19 and another PFE handling 20-39, with 200gbps between. AE5 appears to be built (from the website) out of xe-0/0/29,xe-0/0/38,xe-2/0/15. How that hashing played out for your test and which link it actually took, who knows. Either way, could be a 4500 only thing, or PFE cross traffic related. Seems like it might be a good idea to simplify that topology if possible (given the mixed mode behavior already). -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

One interesting thing that might be playing into it is that ae5 is split across both the 4500, and the 4550. I wonder if the issue is only when traffic goes from the 4200 to the 4500, and not the 4200 to the 4550. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Monday, September 19, 2016 3:30 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants 10M UDP testing to Steve's network at Paul Bunyan (egress ae5), doesn't appear to have any loss. The same 10M UDP test to US Internet, egress xe-0/0/14 shows about 3-5% loss. Again, same 10M UDP test to the same US Internet destination IP forced out another transit provider shows 0 loss. This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. Some kind of buffer or queue difference? ~Matthew On Mon, Sep 19, 2016 at 2:03 PM, Steve Howard <showard(a)paulbunyan.net> wrote: On 09/19/2016 01:49 PM, Matthew Beckwell wrote: As to whether there's loss to one of the "remote" switches, I'm not sure.... If one of the Neutral Path/Mankato Networks/ CNS connected participants would be up for running iperf for a little while, we could find out fairly quickly. We've opened up our iperf3 server on 209.191.196.90 for a few hours. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

10M UDP testing to Steve's network at Paul Bunyan (egress ae5), doesn't appear to have any loss. The same 10M UDP test to US Internet, egress xe-0/0/14 shows about 3-5% loss. Again, same 10M UDP test to the same US Internet destination IP forced out another transit provider shows 0 loss. This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. Some kind of buffer or queue difference? ~Matthew On Mon, Sep 19, 2016 at 2:03 PM, Steve Howard <showard(a)paulbunyan.net> wrote: > > > On 09/19/2016 01:49 PM, Matthew Beckwell wrote: > > > As to whether there's loss to one of the "remote" switches, I'm not > sure.... > If one of the Neutral Path/Mankato Networks/ CNS connected participants > would be up for running iperf for a little while, we could find out fairly > quickly. > > > We've opened up our iperf3 server on 209.191.196.90 for a few hours. >

Sept. 19, 2016