September 2016 - mice-discuss

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

Matthew, The switch throws out an error if I set the destination port as an agg port. However I did all of their member ports individually if that helps you. Personally I think AE bundles on different switches should probably be ignored as data points to keep the troubleshooting straightforward. jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 vc-path from ge-1/0/15 to xe-0/0/35 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/35 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 vc-path from ge-1/0/15 to xe-0/0/3 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/3 {master:1} jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 vc-path from ge-1/0/15 to xe-0/0/13 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/13 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 vc-path from ge-1/0/15 to xe-0/0/29 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/29 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 vc-path from ge-1/0/15 to xe-0/0/38 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/38 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 vc-path from ge-1/0/15 to xe-2/0/15 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/15 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 1:17 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Does the show virtual-chassis vc-path command work for LAG bundles? Specifically-- I know I'm also getting loss to Hurricane (ae6), and not to CNS (ae5). I'd be curious to see these: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae5 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae6 Additionally, I know I'm not getting loss to University of Wisconsin (xe-2/0/11) or Compudyne (xe-0/0/35), so this might be interesting as well. show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/35 The pattern I'm seeing so far is loss on the first half of the 4500 from the 4200. ~Matthew On Tue, Sep 20, 2016 at 11:23 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Here is the switch output: {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 vc-path from ge-1/0/15 to xe-0/0/14 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/14 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 vc-path from ge-1/0/11 to xe-0/0/8 Hop Member PFE-Device Interface 0 1 4 ge-1/0/11 1 2 6 vcp-1 2 0 0 xe-0/0/8 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Does the show virtual-chassis vc-path command work for LAG bundles? Specifically-- I know I'm also getting loss to Hurricane (ae6), and not to CNS (ae5). I'd be curious to see these: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae5 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae6 Additionally, I know I'm not getting loss to University of Wisconsin (xe-2/0/11) or Compudyne (xe-0/0/35), so this might be interesting as well. show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/35 The pattern I'm seeing so far is loss on the first half of the 4500 from the 4200. ~Matthew On Tue, Sep 20, 2016 at 11:23 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Here is the switch output: > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 > > vc-path from ge-1/0/15 to xe-2/0/11 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/11 > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 > > vc-path from ge-1/0/15 to xe-0/0/14 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/14 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 > > vc-path from ge-1/0/11 to xe-0/0/8 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/11 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/8 > > >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Tue, Sep 20, 2016 at 11:27:38AM -0500, Levi Pederson wrote: > I was unaware of the 5200. That sounds like a great solution depending > interconnect solutions etc. No mention of Junipers newish multi-Tbps switches? QFX10002-72Q is even targeted to IXPs. $200k a pop might be a bit much right now though. >> RE: graphing VCP throughput I've found some SLAX scripts, but they mention it puts the load real high on the switch for exporting the data they way they do it. I previously gave some online 'monitor int' stats showing traffic throughput in the mbps on the VCP links on the FPC1. I don't think throughput is an issue on the VCP backplane. Just adding up the traffic on the LG graph in my head quickly for anything on FPC1, I get maybe 600-700Mbps unidirectional traffic. I'd expect more along the lines of a bad cable or bad port. Or it could be something along the lines of going from the 64bps on the 4550 down to the 32gbps port on the 4200, and we could think about ditching the 4200. We could try to transition some members onto the 4550. But that requires new copper SFPs. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Levi Pederson

All, I was unaware of the 5200. That sounds like a great solution depending interconnect solutions etc. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Tue, Sep 20, 2016 at 11:05 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > On Sep 20, 2016, at 10:46 AM, Levi Pederson <levipederson@MANKATONETWORKS. > NET> wrote: > > > Agreed to a point. But the back plane is several Terrabits. Possibly > alleviating the current situation. A pair would be nice, but since the > chassis' can have up to four power supplies and multiple PFEs we'd be > redundant to the port as we are currently. Chaining devices still > continues to provide enigmatic problems and solutions. Where a chassis > would be a better solution all round, including providing the possibility > for 40/100 Gig connections, especially to the Content providers. > > Kinda - but each slot on a 9200 has a 240gbps full duplex connection to > the backplane. That’s 2 100g ports per slot non-oversub’d, leaving you a > handful of 100g ports for alot of space + power. > > For the 32 port 10g cards, it leaves those oversub’d, and that’s where > you’d likely see the traffic (10g ports -> higher bandwidth ports being > used by content providers). > > Not to mention the space/power vs a couple 1RU boxes that have a scalable > fabric by adding more 40/100g connections to the mix for the fabric. (i.e.: > QFX5200 can have 32 100g ports in 1RU) *or* something like the Arista where > you have 3.something tbps for each slot, and the capability for 36 100g > ports in 1 slot (vs 2). > > -- > Andrew Hoyos > hoyosa(a)gmail.com >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

Here is the switch output: {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 vc-path from ge-1/0/15 to xe-0/0/14 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/14 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 vc-path from ge-1/0/11 to xe-0/0/8 Hop Member PFE-Device Interface 0 1 4 ge-1/0/11 1 2 6 vcp-1 2 0 0 xe-0/0/8 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 11:15 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants I think graphing the VC links is a fantastic idea, if we can figure out how to do that. Do we know what JunOS they're running? I think our combined Cacti powers are strong enough to make this happen :) Couple of things I'm thinking of: -Loss is fairly low low and fairly consistent...(e.g. not time of day sensitive...I think Jeremy confirmed this). Also, I don't see a latency spike like you might see with saturation...it feels more like clock slips. (Maybe backplane stuff acts/feels different, I don't know.) -As jay mentioned, a good majority of traffic is CDN -> Eyeball. But I it doesn't seem like there would be a whole lot of CDN traffic leaving the 4200 that would generate enough traffic to cause stack congestion. (We only see loss in one direction). But...that's something we can probably rule out easy enough. For additional data...I'd be curious to see the output of the various "show virtual-chassis" commands Andrew suggested: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/14 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 ~Matthew On Tue, Sep 20, 2016 at 9:30 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: Are we graphing the VC links? SNMP support for that came in 14.something, otherwise, there is a SLAX script that can put values in the util mib: https://kb.juniper.net/InfoCenter/index?page=content&id=KB27711&actp=search https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master/ex-vcp-to-mib.… I know that the interfaces themselves are clean, as Doug showed us, but that doesn’t rule out microbursts that are maxing them out. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

I think graphing the VC links is a fantastic idea, if we can figure out how to do that. Do we know what JunOS they're running? I think our combined Cacti powers are strong enough to make this happen :) Couple of things I'm thinking of: -Loss is fairly low low and fairly consistent...(e.g. not time of day sensitive...I think Jeremy confirmed this). Also, I don't see a latency spike like you might see with saturation...it feels more like clock slips. (Maybe backplane stuff acts/feels different, I don't know.) -As jay mentioned, a good majority of traffic is CDN -> Eyeball. But I it doesn't seem like there would be a whole lot of CDN traffic leaving the 4200 that would generate enough traffic to cause stack congestion. (We only see loss in one direction). But...that's something we can probably rule out easy enough. For additional data...I'd be curious to see the output of the various "show virtual-chassis" commands Andrew suggested: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/14 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 ~Matthew On Tue, Sep 20, 2016 at 9:30 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > Are we graphing the VC links? > > SNMP support for that came in 14.something, otherwise, there is a SLAX > script that can put values in the util mib: > https://kb.juniper.net/InfoCenter/index?page=content&id= > KB27711&actp=search > https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master > /ex-vcp-to-mib.slax > > I know that the interfaces themselves are clean, as Doug showed us, but > that doesn’t rule out microbursts that are maxing them out. >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

On Sep 20, 2016, at 10:46 AM, Levi Pederson <levipederson(a)MANKATONETWORKS.NET> wrote: > Agreed to a point. But the back plane is several Terrabits. Possibly alleviating the current situation. A pair would be nice, but since the chassis' can have up to four power supplies and multiple PFEs we'd be redundant to the port as we are currently. Chaining devices still continues to provide enigmatic problems and solutions. Where a chassis would be a better solution all round, including providing the possibility for 40/100 Gig connections, especially to the Content providers. Kinda - but each slot on a 9200 has a 240gbps full duplex connection to the backplane. That’s 2 100g ports per slot non-oversub’d, leaving you a handful of 100g ports for alot of space + power. For the 32 port 10g cards, it leaves those oversub’d, and that’s where you’d likely see the traffic (10g ports -> higher bandwidth ports being used by content providers). Not to mention the space/power vs a couple 1RU boxes that have a scalable fabric by adding more 40/100g connections to the mix for the fabric. (i.e.: QFX5200 can have 32 100g ports in 1RU) *or* something like the Arista where you have 3.something tbps for each slot, and the capability for 36 100g ports in 1 slot (vs 2). -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Mike Horwath

On Tue, Sep 20, 2016 at 10:29:46AM -0500, Andrew Hoyos wrote: > > On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: > > Cost per port is high compared to running with 4500/45550 setup and > > can stack 10 deep as it is. > > I don???t know if I necessarily agree here either. The VC will > quickly become a bottleneck if not already (32gb unidirectional > traffic). Dump the 4200 and stack via 128Gbe VC cables (2 of them). > Also, no support for 40/100g. Lies. EX4550-EM-2QSFP - expansion with 2 40Gbe QSFP+ No 100Gbe though :( > If money is being spent, something like an Arista 7504 or Juniper > QFX 5100/5200???s would make more sense here and allow for some > growth not entirely based on "more 10g ports". Doug has Arista running already (for a couple years now I think). He hasn't complained about them. I see Arista all over the place as well. Price per port is very reasonable. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Chase Rydberg

I can attest to Edge-Core. We've used them heavily, their support is top notch as I always have someone to reach out to in an instant if there's an issue. They might be a small operation in the US but there's an advantage to that (e.g. no layers to go through for support). Funny, I was just going to recommend them as an option as they have very reasonably priced 10G/40G hardware. If anyone wants a contact (both from Edge-Core and a distributor), let me know. They're worth consideration from my prospective. Chase Rydberg *North Star Fiber* On Tue, Sep 20, 2016 at 10:40 AM, Jason Hanke <jayhanke(a)neutralpath.net> wrote: > I'm starting to think that the new OEM switches are worth a look. > Especially because we don't need a complex feature set. > > Mellanox > EdgeCore > etc. > > On Tue, Sep 20, 2016 at 10:29 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > >> > On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: >> > >> > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: >> > >> >> A full chassis 8200 and 9200 would be a solid upgrade for all >> >> pertinent points. >> > >> > I don't agree. >> >> Also don’t agree with the 8200/9200. Most line cards on the 8200 and 9200 >> are oversub’d anyway, and the space/ports ratio isn’t great. >> >> > Cost per port is high compared to running with 4500/45550 setup and >> > can stack 10 deep as it is. >> >> I don’t know if I necessarily agree here either. The VC will quickly >> become a bottleneck if not already (32gb unidirectional traffic). >> Also, no support for 40/100g. >> >> If money is being spent, something like an Arista 7504 or Juniper QFX >> 5100/5200’s would make more sense here and allow for some growth not >> entirely based on "more 10g ports". >> >> -- >> Andrew Hoyos >> hoyosa(a)gmail.com >> > > > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Levi Pederson

All, Agreed to a point. But the back plane is several Terrabits. Possibly alleviating the current situation. A pair would be nice, but since the chassis' can have up to four power supplies and multiple PFEs we'd be redundant to the port as we are currently. Chaining devices still continues to provide enigmatic problems and solutions. Where a chassis would be a better solution all round, including providing the possibility for 40/100 Gig connections, especially to the Content providers. Chassis architecture can provide some degree of assistance in the current meta and distribution, the main point I would make is the chassis provides a flexible future. I do agree, a full suite of EX4550's with the 128Gig vpc connections would be a solution to our current dilemmas. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Tue, Sep 20, 2016 at 10:18 AM, Mike Horwath <drechsau(a)geeks.org> wrote: > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: > > > A full chassis 8200 and 9200 would be a solid upgrade for all > > pertinent points. > > I don't agree. > > Cost per port is high compared to running with 4500/45550 setup and > can stack 10 deep as it is. > > Get another pair of 4500/4550 (as it is now) and transition the > current people off of the 4200 onto the ports there. This will make it > so that the stack is 100% 10Gbe with VC ports creating the mesh. > > Also, 8200/9200 won't fit in the current space given to the endeavor > today. And you'd want a pair, right? > > 4500 -> 4550 -> 4550 -> 4500 (loop back to 4500) for the rings. > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG >

Sept. 20, 2016