- mice-discuss - lists.micemn.net

Call for Agenda

by Shaun Carlson

All, Please send me any agenda items you'd like added for next week's meeting. I'm working on assembling the agenda. Thanks! s *Shaun Carlson*Director of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

I am in favor, THANKS GUYS!!! From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Tuesday, September 20, 2016 3:41 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Compudyne has offered to donate an EX4200 with a 2x10G uplink card to MICE to help the cause. On Tue, Sep 20, 2016 at 3:31 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: I am in favor of a regular layer 2 uplink. I think the main reason I feel this way is it is industry standard, and then the switch has its own forwarding table. We have seen issues in the past where if the link on a high volume port goes down, the MAC gets removed from the forwarding table, and then all of that traffic gets flooded to all other ports until BGP times out. This seems to be especially problematic when it hits the 1G ports. My vision of a future upgrade would be something along the lines of a 5200, or equiv from another manufacturer that has all of the other switches uplinked to it. Then we would have separate forwarding tables, a place to connect remote switches, as well as something that has some 40G, and 100G capabilities. All of the uplinks would then be easy to monitor/graph without excessive load on the switch. Copper SFP+/QSFP+ cables can be used between switches and they are affordable/easy to come by. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, September 20, 2016 3:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants On Tue, Sep 20, 2016 at 02:23:00PM -0500, Jeremy Lumby wrote: > That would make sense, then we are just down to how traffic can leave the 4200 without going across a vcp Most likely we'd have to kick off (umm. move) the three members that are fiber 1G connections in the expansion slot on the 4200, get a 10G card, optics for both sides, etc. Then decide if we are going to run virtual chassis protocol, or just layer-2 uplink into the two other switches. Or we move the 19 active 1G copper members + 3 1G fiber members off the 4200 altogether and eat up ports on the 4550 with 19 copper SFPs. Then we wouldn't have to do "Mixed-mode Virtual-Chassis" then either. > Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? As I've stated multiple times, I think since VCP is active/passive, there is only one active ring around the the VC through VCP-1 now, and I would expect to see vcp-1 in all paths through the EX4200. We can swap active and passive with the command I posted previously. That way we can troubleshoot if there is a port/cable issue, or if it is an issue with the device itself. Or it could be a defect in this coderev (although I doubt it). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jason Hanke

Compudyne has offered to donate an EX4200 with a 2x10G uplink card to MICE to help the cause. On Tue, Sep 20, 2016 at 3:31 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I am in favor of a regular layer 2 uplink. I think the main reason I feel > this way is it is industry standard, and then the switch has its own > forwarding table. We have seen issues in the past where if the link on a > high volume port goes down, the MAC gets removed from the forwarding table, > and then all of that traffic gets flooded to all other ports until BGP > times out. This seems to be especially problematic when it hits the 1G > ports. > > My vision of a future upgrade would be something along the lines of a > 5200, or equiv from another manufacturer that has all of the other switches > uplinked to it. Then we would have separate forwarding tables, a place to > connect remote switches, as well as something that has some 40G, and 100G > capabilities. All of the uplinks would then be easy to monitor/graph > without excessive load on the switch. Copper SFP+/QSFP+ cables can be used > between switches and they are affordable/easy to come by. > > Jeremy > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Doug McIntyre > Sent: Tuesday, September 20, 2016 3:15 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G > Participants > > On Tue, Sep 20, 2016 at 02:23:00PM -0500, Jeremy Lumby wrote: > > That would make sense, then we are just down to how traffic can leave > the 4200 without going across a vcp > > Most likely we'd have to kick off (umm. move) the three members that > are fiber 1G connections in the expansion slot on the 4200, get a 10G > card, optics for both sides, etc. Then decide if we are going to run > virtual chassis protocol, or just layer-2 uplink into the two other > switches. > > Or we move the 19 active 1G copper members + 3 1G fiber members off the > 4200 > altogether and eat up ports on the 4550 with 19 copper SFPs. Then we > wouldn't > have to do "Mixed-mode Virtual-Chassis" then either. > > > Although- it looks like so far all the paths with loss seem to have > "vcp-1" in them? > > As I've stated multiple times, I think since VCP is active/passive, > there is only one active ring around the the VC through VCP-1 now, > and I would expect to see vcp-1 in all paths through the EX4200. > > We can swap active and passive with the command I posted previously. > That way we can troubleshoot if there is a port/cable issue, or if it is > an issue with the device itself. > > Or it could be a defect in this coderev (although I doubt it). > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

I am in favor of a regular layer 2 uplink. I think the main reason I feel this way is it is industry standard, and then the switch has its own forwarding table. We have seen issues in the past where if the link on a high volume port goes down, the MAC gets removed from the forwarding table, and then all of that traffic gets flooded to all other ports until BGP times out. This seems to be especially problematic when it hits the 1G ports. My vision of a future upgrade would be something along the lines of a 5200, or equiv from another manufacturer that has all of the other switches uplinked to it. Then we would have separate forwarding tables, a place to connect remote switches, as well as something that has some 40G, and 100G capabilities. All of the uplinks would then be easy to monitor/graph without excessive load on the switch. Copper SFP+/QSFP+ cables can be used between switches and they are affordable/easy to come by. Jeremy -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, September 20, 2016 3:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants On Tue, Sep 20, 2016 at 02:23:00PM -0500, Jeremy Lumby wrote: > That would make sense, then we are just down to how traffic can leave the 4200 without going across a vcp Most likely we'd have to kick off (umm. move) the three members that are fiber 1G connections in the expansion slot on the 4200, get a 10G card, optics for both sides, etc. Then decide if we are going to run virtual chassis protocol, or just layer-2 uplink into the two other switches. Or we move the 19 active 1G copper members + 3 1G fiber members off the 4200 altogether and eat up ports on the 4550 with 19 copper SFPs. Then we wouldn't have to do "Mixed-mode Virtual-Chassis" then either. > Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? As I've stated multiple times, I think since VCP is active/passive, there is only one active ring around the the VC through VCP-1 now, and I would expect to see vcp-1 in all paths through the EX4200. We can swap active and passive with the command I posted previously. That way we can troubleshoot if there is a port/cable issue, or if it is an issue with the device itself. Or it could be a defect in this coderev (although I doubt it). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jason Hanke

I'd suggest that we pick up an additional EX4200 and uplink it at layer2 to the main stack at 2x10G. Then move all of the 1G copper links to the new switch. The current switch could then be a spare, VCed into a 4200 only cluster, or uplinked from another suite. If my math is correct, it will free up some VCP bandwidth and also eliminate the mixed mode configuration. On Tue, Sep 20, 2016 at 3:15 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Tue, Sep 20, 2016 at 02:23:00PM -0500, Jeremy Lumby wrote: > > That would make sense, then we are just down to how traffic can leave > the 4200 without going across a vcp > > Most likely we'd have to kick off (umm. move) the three members that > are fiber 1G connections in the expansion slot on the 4200, get a 10G > card, optics for both sides, etc. Then decide if we are going to run > virtual chassis protocol, or just layer-2 uplink into the two other > switches. > > Or we move the 19 active 1G copper members + 3 1G fiber members off the > 4200 > altogether and eat up ports on the 4550 with 19 copper SFPs. Then we > wouldn't > have to do "Mixed-mode Virtual-Chassis" then either. > > > Although- it looks like so far all the paths with loss seem to have > "vcp-1" in them? > > As I've stated multiple times, I think since VCP is active/passive, > there is only one active ring around the the VC through VCP-1 now, > and I would expect to see vcp-1 in all paths through the EX4200. > > We can swap active and passive with the command I posted previously. > That way we can troubleshoot if there is a port/cable issue, or if it is > an issue with the device itself. > > Or it could be a defect in this coderev (although I doubt it). > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Tue, Sep 20, 2016 at 02:23:00PM -0500, Jeremy Lumby wrote: > That would make sense, then we are just down to how traffic can leave the 4200 without going across a vcp Most likely we'd have to kick off (umm. move) the three members that are fiber 1G connections in the expansion slot on the 4200, get a 10G card, optics for both sides, etc. Then decide if we are going to run virtual chassis protocol, or just layer-2 uplink into the two other switches. Or we move the 19 active 1G copper members + 3 1G fiber members off the 4200 altogether and eat up ports on the 4550 with 19 copper SFPs. Then we wouldn't have to do "Mixed-mode Virtual-Chassis" then either. > Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? As I've stated multiple times, I think since VCP is active/passive, there is only one active ring around the the VC through VCP-1 now, and I would expect to see vcp-1 in all paths through the EX4200. We can swap active and passive with the command I posted previously. That way we can troubleshoot if there is a port/cable issue, or if it is an issue with the device itself. Or it could be a defect in this coderev (although I doubt it). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

That would make sense, then we are just down to how traffic can leave the 4200 without going across a vcp From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Tuesday, September 20, 2016 2:11 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants I think the internal port is the opposite pfe in the same chassis. IIRC EX4200 is two pfe in the same box, EX3200 is one pfe On Tue, Sep 20, 2016 at 1:58 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: I agree, I am confused to what the second two mean as well. I was surprised to see internal 1/24, as well as no intermediate interface in the lossless ones. I expected to see all vcp-1 or vcp-0 in the middle From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 1:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Yeah, we should set aside the ae5 test data for the moment...because I agree it's confusing when it's split across 2 switches. I'd have to rely on someone who speaks Juniper as to what the various paths mean.... "vcp-1" vs "internal-1/24" vs nothing?... Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? ~Matthew On Tue, Sep 20, 2016 at 1:28 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Matthew, The switch throws out an error if I set the destination port as an agg port. However I did all of their member ports individually if that helps you. Personally I think AE bundles on different switches should probably be ignored as data points to keep the troubleshooting straightforward. jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 vc-path from ge-1/0/15 to xe-0/0/35 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/35 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 vc-path from ge-1/0/15 to xe-0/0/3 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/3 {master:1} jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 vc-path from ge-1/0/15 to xe-0/0/13 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/13 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 vc-path from ge-1/0/15 to xe-0/0/29 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/29 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 vc-path from ge-1/0/15 to xe-0/0/38 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/38 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 vc-path from ge-1/0/15 to xe-2/0/15 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/15 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jason Hanke

I think the internal port is the opposite pfe in the same chassis. IIRC EX4200 is two pfe in the same box, EX3200 is one pfe On Tue, Sep 20, 2016 at 1:58 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I agree, I am confused to what the second two mean as well. I was > surprised to see internal 1/24, as well as no intermediate interface in the > lossless ones. I expected to see all vcp-1 or vcp-0 in the middle > > > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf > Of *Matthew Beckwell > *Sent:* Tuesday, September 20, 2016 1:51 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G > Participants > > > > Yeah, we should set aside the ae5 test data for the moment...because I > agree it's confusing when it's split across 2 switches. > > > > I'd have to rely on someone who speaks Juniper as to what the various > paths mean.... > > "vcp-1" vs "internal-1/24" vs nothing?... > > > > Although- it looks like so far all the paths with loss seem to have > "vcp-1" in them? > > ~Matthew > > > > > > On Tue, Sep 20, 2016 at 1:28 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > Matthew, > > > > The switch throws out an error if I set the destination > port as an agg port. However I did all of their member ports individually > if that helps you. Personally I think AE bundles on different switches > should probably be ignored as data points to keep the troubleshooting > straightforward. > > > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 > > vc-path from ge-1/0/15 to xe-2/0/11 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/11 > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 > > vc-path from ge-1/0/15 to xe-0/0/35 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/35 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 > > vc-path from ge-1/0/15 to xe-0/0/3 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/3 > > > > {master:1} > > jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 > > vc-path from ge-1/0/15 to xe-0/0/13 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/13 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 > > vc-path from ge-1/0/15 to xe-0/0/29 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/29 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 > > vc-path from ge-1/0/15 to xe-0/0/38 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/38 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 > > vc-path from ge-1/0/15 to xe-2/0/15 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/15 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

I agree, I am confused to what the second two mean as well. I was surprised to see internal 1/24, as well as no intermediate interface in the lossless ones. I expected to see all vcp-1 or vcp-0 in the middle From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 1:51 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Yeah, we should set aside the ae5 test data for the moment...because I agree it's confusing when it's split across 2 switches. I'd have to rely on someone who speaks Juniper as to what the various paths mean.... "vcp-1" vs "internal-1/24" vs nothing?... Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? ~Matthew On Tue, Sep 20, 2016 at 1:28 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Matthew, The switch throws out an error if I set the destination port as an agg port. However I did all of their member ports individually if that helps you. Personally I think AE bundles on different switches should probably be ignored as data points to keep the troubleshooting straightforward. jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 vc-path from ge-1/0/15 to xe-0/0/35 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/35 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 vc-path from ge-1/0/15 to xe-0/0/3 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/3 {master:1} jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 vc-path from ge-1/0/15 to xe-0/0/13 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/13 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 vc-path from ge-1/0/15 to xe-0/0/29 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/29 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 vc-path from ge-1/0/15 to xe-0/0/38 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/38 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 vc-path from ge-1/0/15 to xe-2/0/15 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/15 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Yeah, we should set aside the ae5 test data for the moment...because I agree it's confusing when it's split across 2 switches. I'd have to rely on someone who speaks Juniper as to what the various paths mean.... "vcp-1" vs "internal-1/24" vs nothing?... Although- it looks like so far all the paths with loss seem to have "vcp-1" in them? ~Matthew On Tue, Sep 20, 2016 at 1:28 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Matthew, > > > > The switch throws out an error if I set the destination > port as an agg port. However I did all of their member ports individually > if that helps you. Personally I think AE bundles on different switches > should probably be ignored as data points to keep the troubleshooting > straightforward. > > > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 > > vc-path from ge-1/0/15 to xe-2/0/11 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/11 > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 > > vc-path from ge-1/0/15 to xe-0/0/35 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/35 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 > > vc-path from ge-1/0/15 to xe-0/0/3 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/3 > > > > {master:1} > > jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 > > vc-path from ge-1/0/15 to xe-0/0/13 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/13 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 > > vc-path from ge-1/0/15 to xe-0/0/29 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/29 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 > > vc-path from ge-1/0/15 to xe-0/0/38 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 1 3 internal-1/24 > > 2 0 1 xe-0/0/38 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 > > vc-path from ge-1/0/15 to xe-2/0/15 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/15 >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

Matthew, The switch throws out an error if I set the destination port as an agg port. However I did all of their member ports individually if that helps you. Personally I think AE bundles on different switches should probably be ignored as data points to keep the troubleshooting straightforward. jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/35 vc-path from ge-1/0/15 to xe-0/0/35 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/35 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/3 vc-path from ge-1/0/15 to xe-0/0/3 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/3 {master:1} jlumby@MICE-SW1> ...e-interface ge-1/0/15 destination-interface xe-0/0/13 vc-path from ge-1/0/15 to xe-0/0/13 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/13 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/29 vc-path from ge-1/0/15 to xe-0/0/29 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/29 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-0/0/38 vc-path from ge-1/0/15 to xe-0/0/38 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 1 3 internal-1/24 2 0 1 xe-0/0/38 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/15 destination-interface xe-2/0/15 vc-path from ge-1/0/15 to xe-2/0/15 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/15 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 1:17 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Does the show virtual-chassis vc-path command work for LAG bundles? Specifically-- I know I'm also getting loss to Hurricane (ae6), and not to CNS (ae5). I'd be curious to see these: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae5 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae6 Additionally, I know I'm not getting loss to University of Wisconsin (xe-2/0/11) or Compudyne (xe-0/0/35), so this might be interesting as well. show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/35 The pattern I'm seeing so far is loss on the first half of the 4500 from the 4200. ~Matthew On Tue, Sep 20, 2016 at 11:23 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: Here is the switch output: {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 vc-path from ge-1/0/15 to xe-0/0/14 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/14 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 vc-path from ge-1/0/11 to xe-0/0/8 Hop Member PFE-Device Interface 0 1 4 ge-1/0/11 1 2 6 vcp-1 2 0 0 xe-0/0/8 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Does the show virtual-chassis vc-path command work for LAG bundles? Specifically-- I know I'm also getting loss to Hurricane (ae6), and not to CNS (ae5). I'd be curious to see these: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae5 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface ae6 Additionally, I know I'm not getting loss to University of Wisconsin (xe-2/0/11) or Compudyne (xe-0/0/35), so this might be interesting as well. show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/35 The pattern I'm seeing so far is loss on the first half of the 4500 from the 4200. ~Matthew On Tue, Sep 20, 2016 at 11:23 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Here is the switch output: > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 > > vc-path from ge-1/0/15 to xe-2/0/11 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 xe-2/0/11 > > > > {master:1} > > jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 > > vc-path from ge-1/0/15 to xe-0/0/14 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/15 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/14 > > > > {master:1} > > jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 > > vc-path from ge-1/0/11 to xe-0/0/8 > > Hop Member PFE-Device Interface > > 0 1 4 ge-1/0/11 > > 1 2 6 vcp-1 > > 2 0 0 xe-0/0/8 > > >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Tue, Sep 20, 2016 at 11:27:38AM -0500, Levi Pederson wrote: > I was unaware of the 5200. That sounds like a great solution depending > interconnect solutions etc. No mention of Junipers newish multi-Tbps switches? QFX10002-72Q is even targeted to IXPs. $200k a pop might be a bit much right now though. >> RE: graphing VCP throughput I've found some SLAX scripts, but they mention it puts the load real high on the switch for exporting the data they way they do it. I previously gave some online 'monitor int' stats showing traffic throughput in the mbps on the VCP links on the FPC1. I don't think throughput is an issue on the VCP backplane. Just adding up the traffic on the LG graph in my head quickly for anything on FPC1, I get maybe 600-700Mbps unidirectional traffic. I'd expect more along the lines of a bad cable or bad port. Or it could be something along the lines of going from the 64bps on the 4550 down to the 32gbps port on the 4200, and we could think about ditching the 4200. We could try to transition some members onto the 4550. But that requires new copper SFPs. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Levi Pederson

All, I was unaware of the 5200. That sounds like a great solution depending interconnect solutions etc. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Tue, Sep 20, 2016 at 11:05 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > On Sep 20, 2016, at 10:46 AM, Levi Pederson <levipederson@MANKATONETWORKS. > NET> wrote: > > > Agreed to a point. But the back plane is several Terrabits. Possibly > alleviating the current situation. A pair would be nice, but since the > chassis' can have up to four power supplies and multiple PFEs we'd be > redundant to the port as we are currently. Chaining devices still > continues to provide enigmatic problems and solutions. Where a chassis > would be a better solution all round, including providing the possibility > for 40/100 Gig connections, especially to the Content providers. > > Kinda - but each slot on a 9200 has a 240gbps full duplex connection to > the backplane. That’s 2 100g ports per slot non-oversub’d, leaving you a > handful of 100g ports for alot of space + power. > > For the 32 port 10g cards, it leaves those oversub’d, and that’s where > you’d likely see the traffic (10g ports -> higher bandwidth ports being > used by content providers). > > Not to mention the space/power vs a couple 1RU boxes that have a scalable > fabric by adding more 40/100g connections to the mix for the fabric. (i.e.: > QFX5200 can have 32 100g ports in 1RU) *or* something like the Arista where > you have 3.something tbps for each slot, and the capability for 36 100g > ports in 1 slot (vs 2). > > -- > Andrew Hoyos > hoyosa(a)gmail.com >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

Here is the switch output: {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-2/0/11 vc-path from ge-1/0/15 to xe-2/0/11 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 xe-2/0/11 {master:1} jlumby@MICE-SW1> ...ce ge-1/0/15 destination-interface xe-0/0/14 vc-path from ge-1/0/15 to xe-0/0/14 Hop Member PFE-Device Interface 0 1 4 ge-1/0/15 1 2 6 vcp-1 2 0 0 xe-0/0/14 {master:1} jlumby@MICE-SW1> ...-interface ge-1/0/11 destination-interface xe-0/0/8 vc-path from ge-1/0/11 to xe-0/0/8 Hop Member PFE-Device Interface 0 1 4 ge-1/0/11 1 2 6 vcp-1 2 0 0 xe-0/0/8 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Tuesday, September 20, 2016 11:15 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants I think graphing the VC links is a fantastic idea, if we can figure out how to do that. Do we know what JunOS they're running? I think our combined Cacti powers are strong enough to make this happen :) Couple of things I'm thinking of: -Loss is fairly low low and fairly consistent...(e.g. not time of day sensitive...I think Jeremy confirmed this). Also, I don't see a latency spike like you might see with saturation...it feels more like clock slips. (Maybe backplane stuff acts/feels different, I don't know.) -As jay mentioned, a good majority of traffic is CDN -> Eyeball. But I it doesn't seem like there would be a whole lot of CDN traffic leaving the 4200 that would generate enough traffic to cause stack congestion. (We only see loss in one direction). But...that's something we can probably rule out easy enough. For additional data...I'd be curious to see the output of the various "show virtual-chassis" commands Andrew suggested: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/14 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 ~Matthew On Tue, Sep 20, 2016 at 9:30 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: Are we graphing the VC links? SNMP support for that came in 14.something, otherwise, there is a SLAX script that can put values in the util mib: https://kb.juniper.net/InfoCenter/index?page=content&id=KB27711&actp=search https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master/ex-vcp-to-mib.… I know that the interfaces themselves are clean, as Doug showed us, but that doesn’t rule out microbursts that are maxing them out. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

I think graphing the VC links is a fantastic idea, if we can figure out how to do that. Do we know what JunOS they're running? I think our combined Cacti powers are strong enough to make this happen :) Couple of things I'm thinking of: -Loss is fairly low low and fairly consistent...(e.g. not time of day sensitive...I think Jeremy confirmed this). Also, I don't see a latency spike like you might see with saturation...it feels more like clock slips. (Maybe backplane stuff acts/feels different, I don't know.) -As jay mentioned, a good majority of traffic is CDN -> Eyeball. But I it doesn't seem like there would be a whole lot of CDN traffic leaving the 4200 that would generate enough traffic to cause stack congestion. (We only see loss in one direction). But...that's something we can probably rule out easy enough. For additional data...I'd be curious to see the output of the various "show virtual-chassis" commands Andrew suggested: show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/14 show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11 ~Matthew On Tue, Sep 20, 2016 at 9:30 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > Are we graphing the VC links? > > SNMP support for that came in 14.something, otherwise, there is a SLAX > script that can put values in the util mib: > https://kb.juniper.net/InfoCenter/index?page=content&id= > KB27711&actp=search > https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master > /ex-vcp-to-mib.slax > > I know that the interfaces themselves are clean, as Doug showed us, but > that doesn’t rule out microbursts that are maxing them out. >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

On Sep 20, 2016, at 10:46 AM, Levi Pederson <levipederson(a)MANKATONETWORKS.NET> wrote: > Agreed to a point. But the back plane is several Terrabits. Possibly alleviating the current situation. A pair would be nice, but since the chassis' can have up to four power supplies and multiple PFEs we'd be redundant to the port as we are currently. Chaining devices still continues to provide enigmatic problems and solutions. Where a chassis would be a better solution all round, including providing the possibility for 40/100 Gig connections, especially to the Content providers. Kinda - but each slot on a 9200 has a 240gbps full duplex connection to the backplane. That’s 2 100g ports per slot non-oversub’d, leaving you a handful of 100g ports for alot of space + power. For the 32 port 10g cards, it leaves those oversub’d, and that’s where you’d likely see the traffic (10g ports -> higher bandwidth ports being used by content providers). Not to mention the space/power vs a couple 1RU boxes that have a scalable fabric by adding more 40/100g connections to the mix for the fabric. (i.e.: QFX5200 can have 32 100g ports in 1RU) *or* something like the Arista where you have 3.something tbps for each slot, and the capability for 36 100g ports in 1 slot (vs 2). -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Mike Horwath

On Tue, Sep 20, 2016 at 10:29:46AM -0500, Andrew Hoyos wrote: > > On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: > > Cost per port is high compared to running with 4500/45550 setup and > > can stack 10 deep as it is. > > I don???t know if I necessarily agree here either. The VC will > quickly become a bottleneck if not already (32gb unidirectional > traffic). Dump the 4200 and stack via 128Gbe VC cables (2 of them). > Also, no support for 40/100g. Lies. EX4550-EM-2QSFP - expansion with 2 40Gbe QSFP+ No 100Gbe though :( > If money is being spent, something like an Arista 7504 or Juniper > QFX 5100/5200???s would make more sense here and allow for some > growth not entirely based on "more 10g ports". Doug has Arista running already (for a couple years now I think). He hasn't complained about them. I see Arista all over the place as well. Price per port is very reasonable. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Chase Rydberg

I can attest to Edge-Core. We've used them heavily, their support is top notch as I always have someone to reach out to in an instant if there's an issue. They might be a small operation in the US but there's an advantage to that (e.g. no layers to go through for support). Funny, I was just going to recommend them as an option as they have very reasonably priced 10G/40G hardware. If anyone wants a contact (both from Edge-Core and a distributor), let me know. They're worth consideration from my prospective. Chase Rydberg *North Star Fiber* On Tue, Sep 20, 2016 at 10:40 AM, Jason Hanke <jayhanke(a)neutralpath.net> wrote: > I'm starting to think that the new OEM switches are worth a look. > Especially because we don't need a complex feature set. > > Mellanox > EdgeCore > etc. > > On Tue, Sep 20, 2016 at 10:29 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > >> > On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: >> > >> > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: >> > >> >> A full chassis 8200 and 9200 would be a solid upgrade for all >> >> pertinent points. >> > >> > I don't agree. >> >> Also don’t agree with the 8200/9200. Most line cards on the 8200 and 9200 >> are oversub’d anyway, and the space/ports ratio isn’t great. >> >> > Cost per port is high compared to running with 4500/45550 setup and >> > can stack 10 deep as it is. >> >> I don’t know if I necessarily agree here either. The VC will quickly >> become a bottleneck if not already (32gb unidirectional traffic). >> Also, no support for 40/100g. >> >> If money is being spent, something like an Arista 7504 or Juniper QFX >> 5100/5200’s would make more sense here and allow for some growth not >> entirely based on "more 10g ports". >> >> -- >> Andrew Hoyos >> hoyosa(a)gmail.com >> > > > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Levi Pederson

All, Agreed to a point. But the back plane is several Terrabits. Possibly alleviating the current situation. A pair would be nice, but since the chassis' can have up to four power supplies and multiple PFEs we'd be redundant to the port as we are currently. Chaining devices still continues to provide enigmatic problems and solutions. Where a chassis would be a better solution all round, including providing the possibility for 40/100 Gig connections, especially to the Content providers. Chassis architecture can provide some degree of assistance in the current meta and distribution, the main point I would make is the chassis provides a flexible future. I do agree, a full suite of EX4550's with the 128Gig vpc connections would be a solution to our current dilemmas. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Tue, Sep 20, 2016 at 10:18 AM, Mike Horwath <drechsau(a)geeks.org> wrote: > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: > > > A full chassis 8200 and 9200 would be a solid upgrade for all > > pertinent points. > > I don't agree. > > Cost per port is high compared to running with 4500/45550 setup and > can stack 10 deep as it is. > > Get another pair of 4500/4550 (as it is now) and transition the > current people off of the 4200 onto the ports there. This will make it > so that the stack is 100% 10Gbe with VC ports creating the mesh. > > Also, 8200/9200 won't fit in the current space given to the endeavor > today. And you'd want a pair, right? > > 4500 -> 4550 -> 4550 -> 4500 (loop back to 4500) for the rings. > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jason Hanke

I'm starting to think that the new OEM switches are worth a look. Especially because we don't need a complex feature set. Mellanox EdgeCore etc. On Tue, Sep 20, 2016 at 10:29 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: > > > > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: > > > >> A full chassis 8200 and 9200 would be a solid upgrade for all > >> pertinent points. > > > > I don't agree. > > Also don’t agree with the 8200/9200. Most line cards on the 8200 and 9200 > are oversub’d anyway, and the space/ports ratio isn’t great. > > > Cost per port is high compared to running with 4500/45550 setup and > > can stack 10 deep as it is. > > I don’t know if I necessarily agree here either. The VC will quickly > become a bottleneck if not already (32gb unidirectional traffic). > Also, no support for 40/100g. > > If money is being spent, something like an Arista 7504 or Juniper QFX > 5100/5200’s would make more sense here and allow for some growth not > entirely based on "more 10g ports". > > -- > Andrew Hoyos > hoyosa(a)gmail.com > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

> On Sep 20, 2016, at 10:18 AM, Mike Horwath <drechsau(a)GEEKS.ORG> wrote: > > On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: > >> A full chassis 8200 and 9200 would be a solid upgrade for all >> pertinent points. > > I don't agree. Also don’t agree with the 8200/9200. Most line cards on the 8200 and 9200 are oversub’d anyway, and the space/ports ratio isn’t great. > Cost per port is high compared to running with 4500/45550 setup and > can stack 10 deep as it is. I don’t know if I necessarily agree here either. The VC will quickly become a bottleneck if not already (32gb unidirectional traffic). Also, no support for 40/100g. If money is being spent, something like an Arista 7504 or Juniper QFX 5100/5200’s would make more sense here and allow for some growth not entirely based on "more 10g ports". -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Mike Horwath

On Tue, Sep 20, 2016 at 09:49:48AM -0500, Levi Pederson wrote: > A full chassis 8200 and 9200 would be a solid upgrade for all > pertinent points. I don't agree. Cost per port is high compared to running with 4500/45550 setup and can stack 10 deep as it is. Get another pair of 4500/4550 (as it is now) and transition the current people off of the 4200 onto the ports there. This will make it so that the stack is 100% 10Gbe with VC ports creating the mesh. Also, 8200/9200 won't fit in the current space given to the endeavor today. And you'd want a pair, right? 4500 -> 4550 -> 4550 -> 4500 (loop back to 4500) for the rings. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Sept. 20, 2016

Re: Non-Profit Status

by Richard Laager

On 09/20/2016 10:09 AM, Reid Fishler wrote: > This will be handed off with the new board seat as well. The new board member is going to need the documentation in question. -- Richard

Sept. 20, 2016

Re: Non-Profit Status

by Mike Horwath

On Tue, Sep 20, 2016 at 10:00:46AM -0500, Richard Laager wrote: > I've so far avoided commenting publicly, but enough is enough. We need > to move this along. This is hard and done via volunterring. Public is okay by me! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Sept. 20, 2016

Re: Non-Profit Status

by Reid Fishler

Shaun has been managing this while dealing with his work, and being deployed. This will be handed off with the new board seat as well. Reid On Sep 20, 2016 6:00 PM, "Richard Laager" <rlaager(a)wiktel.com> wrote: > I'd like to request the board provide an actual update on the non-profit > status, preferably here before the meeting. > > I've been mailing off-list regularly since the last UG, and the answer > seems to be that: > > 1) We need to get some form to someone (lawyers or IRS?). > > 2) This form may need some financial details. > > 3) The board member responsible for this is busy and will do this "next > week". > > It's been "next week" every time. > > Another member tells me off-list that they've been doing the same thing, > for the same amount of time, and are getting the same answer. > > I've so far avoided commenting publicly, but enough is enough. We need > to move this along. > > -- > Richard > >

Sept. 20, 2016

Re: Non-Profit Status

by Shaun Carlson

That's fair Richard. It's on me to get the documentation to you and the other member and I will do so in short order. I have been focused on some other things, personally and professionally, and have not made this my priority. I will provide an update NLT Friday of this week. s *Shaun Carlson*Director of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Tue, Sep 20, 2016 at 10:00 AM, Richard Laager <rlaager(a)wiktel.com> wrote: > I'd like to request the board provide an actual update on the non-profit > status, preferably here before the meeting. > > I've been mailing off-list regularly since the last UG, and the answer > seems to be that: > > 1) We need to get some form to someone (lawyers or IRS?). > > 2) This form may need some financial details. > > 3) The board member responsible for this is busy and will do this "next > week". > > It's been "next week" every time. > > Another member tells me off-list that they've been doing the same thing, > for the same amount of time, and are getting the same answer. > > I've so far avoided commenting publicly, but enough is enough. We need > to move this along. > > -- > Richard >

Sept. 20, 2016

Non-Profit Status

by Richard Laager

I'd like to request the board provide an actual update on the non-profit status, preferably here before the meeting. I've been mailing off-list regularly since the last UG, and the answer seems to be that: 1) We need to get some form to someone (lawyers or IRS?). 2) This form may need some financial details. 3) The board member responsible for this is busy and will do this "next week". It's been "next week" every time. Another member tells me off-list that they've been doing the same thing, for the same amount of time, and are getting the same answer. I've so far avoided commenting publicly, but enough is enough. We need to move this along. -- Richard

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Levi Pederson

All, A full chassis 8200 and 9200 would be a solid upgrade for all pertinent points. Thank you, *Levi Pederson* Mankato Networks LLC cell | 612.481.0769 work | 612.787.7392 levipederson(a)mankatonetworks.net On Tue, Sep 20, 2016 at 9:47 AM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > On a partially related note, I think we should keep all of these little > technical details in the back of our mind for when we have nonprofit > status, and can collect the money for a larger switching platform. I think > they can really influence the scalability, redundancy, as well as ease of > monitoring/troubleshooting. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Andrew Hoyos > Sent: Tuesday, September 20, 2016 9:31 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G > Participants > > Option #1 seems to be the most logical (with a 2x10g LAG, split between > the two 4500/4550 switches even MCLAG style). That removes that traffic > from the VC paths and simplifies that side of the house. > > I’d even pony up this EX-UM-2XFP I have sitting here unused. > > Are we graphing the VC links? > > SNMP support for that came in 14.something, otherwise, there is a SLAX > script that can put values in the util mib: > https://kb.juniper.net/InfoCenter/index?page=content& > id=KB27711&actp=search > https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/ > master/ex-vcp-to-mib.slax > > I know that the interfaces themselves are clean, as Doug showed us, but > that doesn’t rule out microbursts that are maxing them out. > > -- > Andrew Hoyos > hoyosa(a)gmail.com > > > > > On Sep 20, 2016, at 9:21 AM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> > wrote: > > > > We've had pretty significant growth (20+ Gig) over the last year or so. > > > > <graph_image.png> > > > > Most of this traffic is CDN->Eyeball. The CDN networks are taking > increasingly larger connections creating a large delta between the largest > and smallest connections on the switch. A 4x10G LAG can put a pretty big > dent in the VC links on its own. > > > > I have two ideas- > > > > 1) Reconfigure the 1G switch to have a LAG group back into the main > switch. > > 2) Offer community strings letting the CDN networks send traffic to the > eyeballs without traversing the VC links. > > > > Jay > > > > > > > > On Tue, Sep 20, 2016 at 8:29 AM, Doug McIntyre <merlyn(a)iphouse.net> > wrote: > > No, that isn't the case, it is a dual ring, so there are paths > > > > FPC0->FPC1->FPC2->FPC0 > > FPC0->FPC2->FPC1->FPC0 > > > > There are ways to tell which is the "active" path through the rings, as I > > stated before, I think one path is mainly "active" and one is mainly > passive. > > I don't know if just disabling one VCP port will work. > > > > But I guess there is a KB on it. > > https://kb.juniper.net/InfoCenter/index?page=content& > id=KB17821&actp=search > > > > So, we could failover the active path to the other ring. > > Or set one of the VCP ports to disable. > > > > Should we do a 'virtual-chassis active path failover' event sometime? > > > > > > On Tue, Sep 20, 2016 at 08:07:00AM -0500, Jeremy Lumby wrote: > > > So if I understand correctly from your "show virtual-chassis > active-topology" below, all traffic from the 4500 to the 4200 is going via > the 4550? And if that is the case, is there an easy way in the software > that we could tell it to have return traffic from the 4200 to the 4500 also > go via the 4550? This would allow us to easily test to see if it is the > specific cable that runs from the 4200 to the 4500. > > > > > > -----Original Message----- > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf > Of Doug McIntyre > > > Sent: Tuesday, September 20, 2016 12:24 AM > > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G > Participants > > > > > > On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > > > > This leads me to wonder if there is possibly an issue with the > stacking modules, or they are being saturated. > > > ... > > > > > > Your understanding of the vc port speeds is correct. > > > > > > The VCP ports are built into the EX4200 and EX4500. The EX4550 has > addon > > > cards that do those functions. We can devote a port to VCP > functionality > > > instead, but it'll be a 10G limit instead of 32G > > > > > > The VCP ports are 32Gbps single direction in each port. So in the > dual-ring > > > the cables have are 32Gbps in each direction. > > > > > > You can see which ports are connected to what. (FPC0 is the EX4500, > > > FPC1 is the EX4200, FPC2 is the EX4550). > > > > > > show virtual-chassis active-topology > > > fpc0: > > > ------------------------------------------------------------ > -------------- > > > Destination ID Next-hop > > > > > > 1 2(vcp-1.32768) 1(vcp-0.32768) > > > > > > 2 2(vcp-1.32768) > > > > > > fpc1: > > > ------------------------------------------------------------ > -------------- > > > Destination ID Next-hop > > > > > > 0 0(vcp-0.32768) 2(vcp-1.32768) > > > > > > 2 2(vcp-1.32768) > > > > > > fpc2: > > > ------------------------------------------------------------ > -------------- > > > Destination ID Next-hop > > > > > > 0 0(vcp-255/2/0.32768) > > > > > > 1 1(vcp-255/2/3.32768) > > > > > > > > > > > > You can monitor each of the VCP ports on each of the members. I don't > think > > > we are close to saturation. > > > > > > request session member 1 > > > (ie. rlogin to the EX4200 switch). > > > > > > monitor int vcp-0 > > > Interface: vcp-0, Enabled, Link is Up > > > Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps > > > Traffic statistics: > Current delta > > > Input bytes: 25685118521 > [12138] > > > Output bytes: 27856843010 > [13204] > > > Input packets: 33377693 > [16] > > > Output packets: 33371644 > [16] > > > Error statistics: > > > Input errors: 0 > [0] > > > Input drops: 0 > [0] > > > Input framing errors: 0 > [0] > > > Carrier transitions: 0 > [0] > > > Output errors: 0 > [0] > > > Output drops: 0 > [0] > > > > > > > > > vcp-1 on FPC1 is much less usage. Still zero error stats. All switches > > > show zero error stats on any of the respective VCP port status. > > > > > > I suppose further troubleshooting could be to replace the VCP cables > > > one by one coming out of the EX4200. > > > > > > Its too bad that we've got the quad fiber expansion in the EX4200, > > > another test could be to put the dual 10G expansion in it, and make > > > those VCP ports and make the VC fabric over those ports instead. But > > > we've got 3 member ports lit on that quad 1G fiber. > > > > > > > > > > > > -- > > > Doug McIntyre <merlyn(a)iphouse.net> > > > ~.~ ipHouse ~.~ > > > Network Engineer/Provisioning/Jack of all Trades > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > > > > > > > -- > > Jay Hanke > > CTO > > Neutral Path Communications > > 3 Civic Center Plaza, Suite 204 > > Mankato, MN 56001 > > (507) 327-2398 mobile > > jayhanke(a)neutralpath.net > > www.neutralpath.net > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

On a partially related note, I think we should keep all of these little technical details in the back of our mind for when we have nonprofit status, and can collect the money for a larger switching platform. I think they can really influence the scalability, redundancy, as well as ease of monitoring/troubleshooting. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 20, 2016 9:31 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Option #1 seems to be the most logical (with a 2x10g LAG, split between the two 4500/4550 switches even MCLAG style). That removes that traffic from the VC paths and simplifies that side of the house. I’d even pony up this EX-UM-2XFP I have sitting here unused. Are we graphing the VC links? SNMP support for that came in 14.something, otherwise, there is a SLAX script that can put values in the util mib: https://kb.juniper.net/InfoCenter/index?page=content&id=KB27711&actp=search https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master/ex-vcp-to-mib.… I know that the interfaces themselves are clean, as Doug showed us, but that doesn’t rule out microbursts that are maxing them out. -- Andrew Hoyos hoyosa(a)gmail.com > On Sep 20, 2016, at 9:21 AM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> wrote: > > We've had pretty significant growth (20+ Gig) over the last year or so. > > <graph_image.png> > > Most of this traffic is CDN->Eyeball. The CDN networks are taking increasingly larger connections creating a large delta between the largest and smallest connections on the switch. A 4x10G LAG can put a pretty big dent in the VC links on its own. > > I have two ideas- > > 1) Reconfigure the 1G switch to have a LAG group back into the main switch. > 2) Offer community strings letting the CDN networks send traffic to the eyeballs without traversing the VC links. > > Jay > > > > On Tue, Sep 20, 2016 at 8:29 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > No, that isn't the case, it is a dual ring, so there are paths > > FPC0->FPC1->FPC2->FPC0 > FPC0->FPC2->FPC1->FPC0 > > There are ways to tell which is the "active" path through the rings, as I > stated before, I think one path is mainly "active" and one is mainly passive. > I don't know if just disabling one VCP port will work. > > But I guess there is a KB on it. > https://kb.juniper.net/InfoCenter/index?page=content&id=KB17821&actp=search > > So, we could failover the active path to the other ring. > Or set one of the VCP ports to disable. > > Should we do a 'virtual-chassis active path failover' event sometime? > > > On Tue, Sep 20, 2016 at 08:07:00AM -0500, Jeremy Lumby wrote: > > So if I understand correctly from your "show virtual-chassis active-topology" below, all traffic from the 4500 to the 4200 is going via the 4550? And if that is the case, is there an easy way in the software that we could tell it to have return traffic from the 4200 to the 4500 also go via the 4550? This would allow us to easily test to see if it is the specific cable that runs from the 4200 to the 4500. > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > > Sent: Tuesday, September 20, 2016 12:24 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants > > > > On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > > > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. > > ... > > > > Your understanding of the vc port speeds is correct. > > > > The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon > > cards that do those functions. We can devote a port to VCP functionality > > instead, but it'll be a 10G limit instead of 32G > > > > The VCP ports are 32Gbps single direction in each port. So in the dual-ring > > the cables have are 32Gbps in each direction. > > > > You can see which ports are connected to what. (FPC0 is the EX4500, > > FPC1 is the EX4200, FPC2 is the EX4550). > > > > show virtual-chassis active-topology > > fpc0: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 1 2(vcp-1.32768) 1(vcp-0.32768) > > > > 2 2(vcp-1.32768) > > > > fpc1: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 0 0(vcp-0.32768) 2(vcp-1.32768) > > > > 2 2(vcp-1.32768) > > > > fpc2: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 0 0(vcp-255/2/0.32768) > > > > 1 1(vcp-255/2/3.32768) > > > > > > > > You can monitor each of the VCP ports on each of the members. I don't think > > we are close to saturation. > > > > request session member 1 > > (ie. rlogin to the EX4200 switch). > > > > monitor int vcp-0 > > Interface: vcp-0, Enabled, Link is Up > > Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps > > Traffic statistics: Current delta > > Input bytes: 25685118521 [12138] > > Output bytes: 27856843010 [13204] > > Input packets: 33377693 [16] > > Output packets: 33371644 [16] > > Error statistics: > > Input errors: 0 [0] > > Input drops: 0 [0] > > Input framing errors: 0 [0] > > Carrier transitions: 0 [0] > > Output errors: 0 [0] > > Output drops: 0 [0] > > > > > > vcp-1 on FPC1 is much less usage. Still zero error stats. All switches > > show zero error stats on any of the respective VCP port status. > > > > I suppose further troubleshooting could be to replace the VCP cables > > one by one coming out of the EX4200. > > > > Its too bad that we've got the quad fiber expansion in the EX4200, > > another test could be to put the dual 10G expansion in it, and make > > those VCP ports and make the VC fabric over those ports instead. But > > we've got 3 member ports lit on that quad 1G fiber. > > > > > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

Option #1 seems to be the most logical (with a 2x10g LAG, split between the two 4500/4550 switches even MCLAG style). That removes that traffic from the VC paths and simplifies that side of the house. I’d even pony up this EX-UM-2XFP I have sitting here unused. Are we graphing the VC links? SNMP support for that came in 14.something, otherwise, there is a SLAX script that can put values in the util mib: https://kb.juniper.net/InfoCenter/index?page=content&id=KB27711&actp=search https://github.com/dgarros/juniper-ex-vcp-to-mib/blob/master/ex-vcp-to-mib.… I know that the interfaces themselves are clean, as Doug showed us, but that doesn’t rule out microbursts that are maxing them out. -- Andrew Hoyos hoyosa(a)gmail.com > On Sep 20, 2016, at 9:21 AM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> wrote: > > We've had pretty significant growth (20+ Gig) over the last year or so. > > <graph_image.png> > > Most of this traffic is CDN->Eyeball. The CDN networks are taking increasingly larger connections creating a large delta between the largest and smallest connections on the switch. A 4x10G LAG can put a pretty big dent in the VC links on its own. > > I have two ideas- > > 1) Reconfigure the 1G switch to have a LAG group back into the main switch. > 2) Offer community strings letting the CDN networks send traffic to the eyeballs without traversing the VC links. > > Jay > > > > On Tue, Sep 20, 2016 at 8:29 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > No, that isn't the case, it is a dual ring, so there are paths > > FPC0->FPC1->FPC2->FPC0 > FPC0->FPC2->FPC1->FPC0 > > There are ways to tell which is the "active" path through the rings, as I > stated before, I think one path is mainly "active" and one is mainly passive. > I don't know if just disabling one VCP port will work. > > But I guess there is a KB on it. > https://kb.juniper.net/InfoCenter/index?page=content&id=KB17821&actp=search > > So, we could failover the active path to the other ring. > Or set one of the VCP ports to disable. > > Should we do a 'virtual-chassis active path failover' event sometime? > > > On Tue, Sep 20, 2016 at 08:07:00AM -0500, Jeremy Lumby wrote: > > So if I understand correctly from your "show virtual-chassis active-topology" below, all traffic from the 4500 to the 4200 is going via the 4550? And if that is the case, is there an easy way in the software that we could tell it to have return traffic from the 4200 to the 4500 also go via the 4550? This would allow us to easily test to see if it is the specific cable that runs from the 4200 to the 4500. > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > > Sent: Tuesday, September 20, 2016 12:24 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants > > > > On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > > > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. > > ... > > > > Your understanding of the vc port speeds is correct. > > > > The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon > > cards that do those functions. We can devote a port to VCP functionality > > instead, but it'll be a 10G limit instead of 32G > > > > The VCP ports are 32Gbps single direction in each port. So in the dual-ring > > the cables have are 32Gbps in each direction. > > > > You can see which ports are connected to what. (FPC0 is the EX4500, > > FPC1 is the EX4200, FPC2 is the EX4550). > > > > show virtual-chassis active-topology > > fpc0: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 1 2(vcp-1.32768) 1(vcp-0.32768) > > > > 2 2(vcp-1.32768) > > > > fpc1: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 0 0(vcp-0.32768) 2(vcp-1.32768) > > > > 2 2(vcp-1.32768) > > > > fpc2: > > -------------------------------------------------------------------------- > > Destination ID Next-hop > > > > 0 0(vcp-255/2/0.32768) > > > > 1 1(vcp-255/2/3.32768) > > > > > > > > You can monitor each of the VCP ports on each of the members. I don't think > > we are close to saturation. > > > > request session member 1 > > (ie. rlogin to the EX4200 switch). > > > > monitor int vcp-0 > > Interface: vcp-0, Enabled, Link is Up > > Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps > > Traffic statistics: Current delta > > Input bytes: 25685118521 [12138] > > Output bytes: 27856843010 [13204] > > Input packets: 33377693 [16] > > Output packets: 33371644 [16] > > Error statistics: > > Input errors: 0 [0] > > Input drops: 0 [0] > > Input framing errors: 0 [0] > > Carrier transitions: 0 [0] > > Output errors: 0 [0] > > Output drops: 0 [0] > > > > > > vcp-1 on FPC1 is much less usage. Still zero error stats. All switches > > show zero error stats on any of the respective VCP port status. > > > > I suppose further troubleshooting could be to replace the VCP cables > > one by one coming out of the EX4200. > > > > Its too bad that we've got the quad fiber expansion in the EX4200, > > another test could be to put the dual 10G expansion in it, and make > > those VCP ports and make the VC fabric over those ports instead. But > > we've got 3 member ports lit on that quad 1G fiber. > > > > > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Richard Laager

On 09/20/2016 09:21 AM, Jason Hanke wrote: > Offer community strings letting the CDN networks send traffic to the > eyeballs without traversing the VC links. I'm not quite following this one. What did you have in mind? -- Richard

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jason Hanke

We've had pretty significant growth (20+ Gig) over the last year or so. [image: Inline image 5] Most of this traffic is CDN->Eyeball. The CDN networks are taking increasingly larger connections creating a large delta between the largest and smallest connections on the switch. A 4x10G LAG can put a pretty big dent in the VC links on its own. I have two ideas- 1) Reconfigure the 1G switch to have a LAG group back into the main switch. 2) Offer community strings letting the CDN networks send traffic to the eyeballs without traversing the VC links. Jay On Tue, Sep 20, 2016 at 8:29 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > No, that isn't the case, it is a dual ring, so there are paths > > FPC0->FPC1->FPC2->FPC0 > FPC0->FPC2->FPC1->FPC0 > > There are ways to tell which is the "active" path through the rings, as I > stated before, I think one path is mainly "active" and one is mainly > passive. > I don't know if just disabling one VCP port will work. > > But I guess there is a KB on it. > https://kb.juniper.net/InfoCenter/index?page=content& > id=KB17821&actp=search > > So, we could failover the active path to the other ring. > Or set one of the VCP ports to disable. > > Should we do a 'virtual-chassis active path failover' event sometime? > > > On Tue, Sep 20, 2016 at 08:07:00AM -0500, Jeremy Lumby wrote: > > So if I understand correctly from your "show virtual-chassis > active-topology" below, all traffic from the 4500 to the 4200 is going via > the 4550? And if that is the case, is there an easy way in the software > that we could tell it to have return traffic from the 4200 to the 4500 also > go via the 4550? This would allow us to easily test to see if it is the > specific cable that runs from the 4200 to the 4500. > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Doug McIntyre > > Sent: Tuesday, September 20, 2016 12:24 AM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G > Participants > > > > On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > > > This leads me to wonder if there is possibly an issue with the > stacking modules, or they are being saturated. > > ... > > > > Your understanding of the vc port speeds is correct. > > > > The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon > > cards that do those functions. We can devote a port to VCP functionality > > instead, but it'll be a 10G limit instead of 32G > > > > The VCP ports are 32Gbps single direction in each port. So in the > dual-ring > > the cables have are 32Gbps in each direction. > > > > You can see which ports are connected to what. (FPC0 is the EX4500, > > FPC1 is the EX4200, FPC2 is the EX4550). > > > > show virtual-chassis active-topology > > fpc0: > > ------------------------------------------------------------ > -------------- > > Destination ID Next-hop > > > > 1 2(vcp-1.32768) 1(vcp-0.32768) > > > > 2 2(vcp-1.32768) > > > > fpc1: > > ------------------------------------------------------------ > -------------- > > Destination ID Next-hop > > > > 0 0(vcp-0.32768) 2(vcp-1.32768) > > > > 2 2(vcp-1.32768) > > > > fpc2: > > ------------------------------------------------------------ > -------------- > > Destination ID Next-hop > > > > 0 0(vcp-255/2/0.32768) > > > > 1 1(vcp-255/2/3.32768) > > > > > > > > You can monitor each of the VCP ports on each of the members. I don't > think > > we are close to saturation. > > > > request session member 1 > > (ie. rlogin to the EX4200 switch). > > > > monitor int vcp-0 > > Interface: vcp-0, Enabled, Link is Up > > Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps > > Traffic statistics: Current > delta > > Input bytes: 25685118521 > [12138] > > Output bytes: 27856843010 > [13204] > > Input packets: 33377693 > [16] > > Output packets: 33371644 > [16] > > Error statistics: > > Input errors: 0 > [0] > > Input drops: 0 > [0] > > Input framing errors: 0 > [0] > > Carrier transitions: 0 > [0] > > Output errors: 0 > [0] > > Output drops: 0 > [0] > > > > > > vcp-1 on FPC1 is much less usage. Still zero error stats. All switches > > show zero error stats on any of the respective VCP port status. > > > > I suppose further troubleshooting could be to replace the VCP cables > > one by one coming out of the EX4200. > > > > Its too bad that we've got the quad fiber expansion in the EX4200, > > another test could be to put the dual 10G expansion in it, and make > > those VCP ports and make the VC fabric over those ports instead. But > > we've got 3 member ports lit on that quad 1G fiber. > > > > > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

On Sep 20, 2016, at 8:29 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > There are ways to tell which is the "active" path through the rings, as I > stated before, I think one path is mainly "active" and one is mainly passive. > I don't know if just disabling one VCP port will work. Let’s figure out of a common VCP port first. Like I said before, this ‘path’ is based on src/dst interfaces in the VC, and the selection algorithm is SPF based (but factors in actual PFE hops on each switch - internal to each switch is a PFE chip controlling a set of ports with a fabric between that is also used for VC - it’s not as simple as FPC0 -> FPC1 -> FPC2) What say: ‘show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-0/0/14’ (original issue of AITech to USInternet - loss evident) ‘show virtual-chassis vc-path source-interface ge-1/0/11 destination-interface xe-0/0/8’ (MNVoip to Charter - loss evident) ‘show virtual-chassis vc-path source-interface ge-1/0/15 destination-interface xe-2/0/11’ (AITech to UW Madison - no loss evident) If there is a common VCP port on the two paths where loss was evident, it’d probably be easier and less intrusive to disable that port during a maintenance window, test, then re-enable. IIRC that active path failover command is for *. 'request virtual-chassis vc-port member 1 set interface vcp-X disable' -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

No, that isn't the case, it is a dual ring, so there are paths FPC0->FPC1->FPC2->FPC0 FPC0->FPC2->FPC1->FPC0 There are ways to tell which is the "active" path through the rings, as I stated before, I think one path is mainly "active" and one is mainly passive. I don't know if just disabling one VCP port will work. But I guess there is a KB on it. https://kb.juniper.net/InfoCenter/index?page=content&id=KB17821&actp=search So, we could failover the active path to the other ring. Or set one of the VCP ports to disable. Should we do a 'virtual-chassis active path failover' event sometime? On Tue, Sep 20, 2016 at 08:07:00AM -0500, Jeremy Lumby wrote: > So if I understand correctly from your "show virtual-chassis active-topology" below, all traffic from the 4500 to the 4200 is going via the 4550? And if that is the case, is there an easy way in the software that we could tell it to have return traffic from the 4200 to the 4500 also go via the 4550? This would allow us to easily test to see if it is the specific cable that runs from the 4200 to the 4500. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Tuesday, September 20, 2016 12:24 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants > > On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. > ... > > Your understanding of the vc port speeds is correct. > > The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon > cards that do those functions. We can devote a port to VCP functionality > instead, but it'll be a 10G limit instead of 32G > > The VCP ports are 32Gbps single direction in each port. So in the dual-ring > the cables have are 32Gbps in each direction. > > You can see which ports are connected to what. (FPC0 is the EX4500, > FPC1 is the EX4200, FPC2 is the EX4550). > > show virtual-chassis active-topology > fpc0: > -------------------------------------------------------------------------- > Destination ID Next-hop > > 1 2(vcp-1.32768) 1(vcp-0.32768) > > 2 2(vcp-1.32768) > > fpc1: > -------------------------------------------------------------------------- > Destination ID Next-hop > > 0 0(vcp-0.32768) 2(vcp-1.32768) > > 2 2(vcp-1.32768) > > fpc2: > -------------------------------------------------------------------------- > Destination ID Next-hop > > 0 0(vcp-255/2/0.32768) > > 1 1(vcp-255/2/3.32768) > > > > You can monitor each of the VCP ports on each of the members. I don't think > we are close to saturation. > > request session member 1 > (ie. rlogin to the EX4200 switch). > > monitor int vcp-0 > Interface: vcp-0, Enabled, Link is Up > Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps > Traffic statistics: Current delta > Input bytes: 25685118521 [12138] > Output bytes: 27856843010 [13204] > Input packets: 33377693 [16] > Output packets: 33371644 [16] > Error statistics: > Input errors: 0 [0] > Input drops: 0 [0] > Input framing errors: 0 [0] > Carrier transitions: 0 [0] > Output errors: 0 [0] > Output drops: 0 [0] > > > vcp-1 on FPC1 is much less usage. Still zero error stats. All switches > show zero error stats on any of the respective VCP port status. > > I suppose further troubleshooting could be to replace the VCP cables > one by one coming out of the EX4200. > > Its too bad that we've got the quad fiber expansion in the EX4200, > another test could be to put the dual 10G expansion in it, and make > those VCP ports and make the VC fabric over those ports instead. But > we've got 3 member ports lit on that quad 1G fiber. > > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

So if I understand correctly from your "show virtual-chassis active-topology" below, all traffic from the 4500 to the 4200 is going via the 4550? And if that is the case, is there an easy way in the software that we could tell it to have return traffic from the 4200 to the 4500 also go via the 4550? This would allow us to easily test to see if it is the specific cable that runs from the 4200 to the 4500. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, September 20, 2016 12:24 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. ... Your understanding of the vc port speeds is correct. The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon cards that do those functions. We can devote a port to VCP functionality instead, but it'll be a 10G limit instead of 32G The VCP ports are 32Gbps single direction in each port. So in the dual-ring the cables have are 32Gbps in each direction. You can see which ports are connected to what. (FPC0 is the EX4500, FPC1 is the EX4200, FPC2 is the EX4550). show virtual-chassis active-topology fpc0: -------------------------------------------------------------------------- Destination ID Next-hop 1 2(vcp-1.32768) 1(vcp-0.32768) 2 2(vcp-1.32768) fpc1: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-0.32768) 2(vcp-1.32768) 2 2(vcp-1.32768) fpc2: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-255/2/0.32768) 1 1(vcp-255/2/3.32768) You can monitor each of the VCP ports on each of the members. I don't think we are close to saturation. request session member 1 (ie. rlogin to the EX4200 switch). monitor int vcp-0 Interface: vcp-0, Enabled, Link is Up Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps Traffic statistics: Current delta Input bytes: 25685118521 [12138] Output bytes: 27856843010 [13204] Input packets: 33377693 [16] Output packets: 33371644 [16] Error statistics: Input errors: 0 [0] Input drops: 0 [0] Input framing errors: 0 [0] Carrier transitions: 0 [0] Output errors: 0 [0] Output drops: 0 [0] vcp-1 on FPC1 is much less usage. Still zero error stats. All switches show zero error stats on any of the respective VCP port status. I suppose further troubleshooting could be to replace the VCP cables one by one coming out of the EX4200. Its too bad that we've got the quad fiber expansion in the EX4200, another test could be to put the dual 10G expansion in it, and make those VCP ports and make the VC fabric over those ports instead. But we've got 3 member ports lit on that quad 1G fiber. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Mon, Sep 19, 2016 at 11:01:00PM -0500, Jeremy Lumby wrote: > This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. ... Your understanding of the vc port speeds is correct. The VCP ports are built into the EX4200 and EX4500. The EX4550 has addon cards that do those functions. We can devote a port to VCP functionality instead, but it'll be a 10G limit instead of 32G The VCP ports are 32Gbps single direction in each port. So in the dual-ring the cables have are 32Gbps in each direction. You can see which ports are connected to what. (FPC0 is the EX4500, FPC1 is the EX4200, FPC2 is the EX4550). show virtual-chassis active-topology fpc0: -------------------------------------------------------------------------- Destination ID Next-hop 1 2(vcp-1.32768) 1(vcp-0.32768) 2 2(vcp-1.32768) fpc1: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-0.32768) 2(vcp-1.32768) 2 2(vcp-1.32768) fpc2: -------------------------------------------------------------------------- Destination ID Next-hop 0 0(vcp-255/2/0.32768) 1 1(vcp-255/2/3.32768) You can monitor each of the VCP ports on each of the members. I don't think we are close to saturation. request session member 1 (ie. rlogin to the EX4200 switch). monitor int vcp-0 Interface: vcp-0, Enabled, Link is Up Encapsulation: Virtual-Chassis-Interface, Speed: 32000mbps Traffic statistics: Current delta Input bytes: 25685118521 [12138] Output bytes: 27856843010 [13204] Input packets: 33377693 [16] Output packets: 33371644 [16] Error statistics: Input errors: 0 [0] Input drops: 0 [0] Input framing errors: 0 [0] Carrier transitions: 0 [0] Output errors: 0 [0] Output drops: 0 [0] vcp-1 on FPC1 is much less usage. Still zero error stats. All switches show zero error stats on any of the respective VCP port status. I suppose further troubleshooting could be to replace the VCP cables one by one coming out of the EX4200. Its too bad that we've got the quad fiber expansion in the EX4200, another test could be to put the dual 10G expansion in it, and make those VCP ports and make the VC fabric over those ports instead. But we've got 3 member ports lit on that quad 1G fiber. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 20, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

So this brings an issue I was watching for a couple days into a new light. I have a customer that is in Rochester MN on Charter's Fiber. They have 2 locations just a couple miles apart. I was monitoring their connections before we start to do their voice on the 22nd. I was surprised to see I was getting about 0.3% loss to both locations. The loss would happen at the same times, so I was tempted to blame Charter. When Matthew posted this today it reminded me that charter is on the 4500 and I am on the 4200 just like Matthew. So I started two different tests this afternoon, and I now have gathered results for last 6 hours, so I feel confident in the result that there is loss in traffic going from the 4200 to the 4500. Test 1 was to add a static route to one of my customer's Rochester locations, so that traffic from me to them went via HE, and then Charter. The location that I added the static route to has had no loss for the past 6 hours, and the location that I did not change has continued to show 0.3% loss. Obviously the return path still takes MICE, so I do not believe traffic from the 4500 to the 4200 is having a problem. Test 2 was to start a running ping to 9 different interface IP addresses on MICE. I choose 3 from each switch to avoid the possibility that one router was an outlier and overly deprioritizing ICMP. I let that run for almost 6 hours, and all interfaces on the 4200, and the 4550 showed 0 loss, and all three on the 4500 showed 0.2% loss. Looking deeper into the data, the loss that happened on the 4500 would often happen at the same time for all three addresses. This leads me to wonder if there is possibly an issue with the stacking modules, or they are being saturated. Let me preface this by saying I am not a Juniper guy, and I may be wrong, however my understanding is that the stacking modules are 128Gig which would seem like plenty, however it is my understanding that each module has 2 ports, and each one is 64G, and then that 64G is actually in + out, and therefore a single direction of one port could get maxed out at 32G. I remember when we added the 4550 I asked if there was a way to graph the traffic on them, and I believe we were not able to. While I do not think that they are saturated, I wonder if there are small bursts that get them near to capacity that are far too short to show up on the 5 minute averages. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Monday, September 19, 2016 4:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants Testing against the UW iperf servebbb, but no loss running the same 10M UDP test). So this (somewhat limited) test seems to show the current path from 4200 -> 4550 is okay. ~Matthew On Mon, Sep 19, 2016 at 4:06 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: UW Madison operates some IPerf servers you could try against too, see: https://kb.wisc.edu/uwsysnet/page.php?id=41947 They are on the 4550, it appears, so more data points to be gleaned there. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 20, 2016

Re: Proposed Membership and Sponsorship Model

by Jason Hanke

I've looked at this and I support the SIX style of user fees for MICE. I also very much like that the remotes can charge their own rates. This should create options for members who prefer a different style of payment (perhaps monthly with little upfront?). The other shared resource is address space. We may want to consider a cap per ASN. Perhaps 2 without board/tech committee approval? In the propose scenario there would be no direct cost to the end user for these. I'd also selfishly support membership voting rights for remote switch operators. Jay On Mon, Sep 19, 2016 at 4:56 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > I fully support this model as well. (see: https://www.seattleix.net/join) > > The large expenditures of new equipment are all member growth or new > member driven. Recouping a 1/nth portion of those switch upgrade costs per > member, plus some small overhead for the rest, seems to be a totally viable > model and has worked well for SIX. (plus donations). The rest of the > expense side of the house is pennies in comparison. > > For those that don’t think an IX can be driven, at least partially by > donations, maybe review this as well: https://www.seattleix.net/ > contributors > > This simplifies things on the backend too: > - one time payment vs monthly deposits via multiple methods that someone > on MICE board would need to manage for 60+ members > - members joining the IX have an easy sell internally of one time fee vs > MRC above and beyond a cross connect (always infinitely harder to justify) > > Solves the root issue of new members and more bandwidth (higher speed > ports) driving growth, and doesn’t penalize the little guys. > > Remote switches can charge their own rates, however, still need to pay the > main core switch port costs. (https://www.seattleix.net/rules) > > -- > Andrew Hoyos > hoyosa(a)gmail.com > > > > > On Sep 19, 2016, at 2:35 PM, Justin Krejci <jkrejci(a)usinternet.com> > wrote: > > > > I am a fan of the SIX model of revenue: you pay to connect to the > exchange up front and bigger ports (ie 10-Gig) cost more than smaller (ie > 1-Gig). This fee applies to MICE core equipment only. Port costs on > extension switches are dictated by the respective extension switch > operators. > > > > As there are already many existing connected ports I would also be in > favor of retroactively charging each connected port based on their current > port type(s) in use at the time of the implementation of said fees. Already > connected organizations that want to take a bit of time to budget can make > arrangements with the board. Also, I would be in favor of implementing this > soon instead of waiting for the desired but elusive non-profit status > unless someone has knowledge of its actual pending completion. > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Sept. 19, 2016

Re: Proposed Membership and Sponsorship Model

by Andrew Hoyos

I fully support this model as well. (see: https://www.seattleix.net/join) The large expenditures of new equipment are all member growth or new member driven. Recouping a 1/nth portion of those switch upgrade costs per member, plus some small overhead for the rest, seems to be a totally viable model and has worked well for SIX. (plus donations). The rest of the expense side of the house is pennies in comparison. For those that don’t think an IX can be driven, at least partially by donations, maybe review this as well: https://www.seattleix.net/contributors This simplifies things on the backend too: - one time payment vs monthly deposits via multiple methods that someone on MICE board would need to manage for 60+ members - members joining the IX have an easy sell internally of one time fee vs MRC above and beyond a cross connect (always infinitely harder to justify) Solves the root issue of new members and more bandwidth (higher speed ports) driving growth, and doesn’t penalize the little guys. Remote switches can charge their own rates, however, still need to pay the main core switch port costs. (https://www.seattleix.net/rules) -- Andrew Hoyos hoyosa(a)gmail.com > On Sep 19, 2016, at 2:35 PM, Justin Krejci <jkrejci(a)usinternet.com> wrote: > > I am a fan of the SIX model of revenue: you pay to connect to the exchange up front and bigger ports (ie 10-Gig) cost more than smaller (ie 1-Gig). This fee applies to MICE core equipment only. Port costs on extension switches are dictated by the respective extension switch operators. > > As there are already many existing connected ports I would also be in favor of retroactively charging each connected port based on their current port type(s) in use at the time of the implementation of said fees. Already connected organizations that want to take a bit of time to budget can make arrangements with the board. Also, I would be in favor of implementing this soon instead of waiting for the desired but elusive non-profit status unless someone has knowledge of its actual pending completion.

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Testing against the UW iperf servers seems to be mostly clean. (Occasionally, I get one out-of-order, but no loss running the same 10M UDP test). So this (somewhat limited) test seems to show the current path from 4200 -> 4550 is okay. ~Matthew On Mon, Sep 19, 2016 at 4:06 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > UW Madison operates some IPerf servers you could try against too, see: > https://kb.wisc.edu/uwsysnet/page.php?id=41947 > They are on the 4550, it appears, so more data points to be gleaned there. > >

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

> On Sep 19, 2016, at 2:19 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > On Mon, Sep 19, 2016 at 02:03:41PM -0500, Andrew Hoyos wrote: >> What medium is the VC built on and physical topology? Just VC cables on the rear ringed through all 3? > > Yes, VC cabling in a ring, and using the VCP modules and cabling on the EX4550. What’s the actual topology though? 4500 -> 4200 -> 4550 -> back to 4500? >> Specific to any single 4500 destination or is it doing it to both the 4500 and the 4550? > > He wasn't aware of the EX4550, there aren't many ports lit there yet, > he has just been testing different ports on the the EX4500. UW Madison operates some IPerf servers you could try against too, see: https://kb.wisc.edu/uwsysnet/page.php?id=41947 They are on the 4550, it appears, so more data points to be gleaned there. >> Check ‘show virtual-chassis vc-path source-interface <src int> destination-interface <dest int>’. Common VC port in the mix to any of the reported loss paths? > > I think in general, the VC ports run Active/Passive, so only one is > "active" at a time going out of a box without load balancing, so only > one (ie. vcp-1) typically shows up for all paths. Kinda, there is a SPF calculation that happens based on src/dst interfaces. It matters which PFE of the switch the port is on (1 hop != switch, it’s actually each PFE hop). Depending on the topology here, things could be going a variety of ways. -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

On Sep 19, 2016, at 3:29 PM, Matthew Beckwell <matthewb(a)aitech.net> wrote: > > This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. > Some kind of buffer or queue difference? IIRC on the 4500, there is a PFE mapped to ports 0-19 and another PFE handling 20-39, with 200gbps between. AE5 appears to be built (from the website) out of xe-0/0/29,xe-0/0/38,xe-2/0/15. How that hashing played out for your test and which link it actually took, who knows. Either way, could be a 4500 only thing, or PFE cross traffic related. Seems like it might be a good idea to simplify that topology if possible (given the mixed mode behavior already). -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Jeremy Lumby

One interesting thing that might be playing into it is that ae5 is split across both the 4500, and the 4550. I wonder if the issue is only when traffic goes from the 4200 to the 4500, and not the 4200 to the 4550. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Monday, September 19, 2016 3:30 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] EX4200 (1G Switch) Packet Loss -> 10G Participants 10M UDP testing to Steve's network at Paul Bunyan (egress ae5), doesn't appear to have any loss. The same 10M UDP test to US Internet, egress xe-0/0/14 shows about 3-5% loss. Again, same 10M UDP test to the same US Internet destination IP forced out another transit provider shows 0 loss. This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. Some kind of buffer or queue difference? ~Matthew On Mon, Sep 19, 2016 at 2:03 PM, Steve Howard <showard(a)paulbunyan.net> wrote: On 09/19/2016 01:49 PM, Matthew Beckwell wrote: As to whether there's loss to one of the "remote" switches, I'm not sure.... If one of the Neutral Path/Mankato Networks/ CNS connected participants would be up for running iperf for a little while, we could find out fairly quickly. We've opened up our iperf3 server on 209.191.196.90 for a few hours. To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

10M UDP testing to Steve's network at Paul Bunyan (egress ae5), doesn't appear to have any loss. The same 10M UDP test to US Internet, egress xe-0/0/14 shows about 3-5% loss. Again, same 10M UDP test to the same US Internet destination IP forced out another transit provider shows 0 loss. This is certainly interesting....I'm not sure what's special about xe-0/0/14 vs ae5. Some kind of buffer or queue difference? ~Matthew On Mon, Sep 19, 2016 at 2:03 PM, Steve Howard <showard(a)paulbunyan.net> wrote: > > > On 09/19/2016 01:49 PM, Matthew Beckwell wrote: > > > As to whether there's loss to one of the "remote" switches, I'm not > sure.... > If one of the Neutral Path/Mankato Networks/ CNS connected participants > would be up for running iperf for a little while, we could find out fairly > quickly. > > > We've opened up our iperf3 server on 209.191.196.90 for a few hours. >

Sept. 19, 2016

Re: Proposed Membership and Sponsorship Model

by Justin Krejci

I am a fan of the SIX model of revenue: you pay to connect to the exchange up front and bigger ports (ie 10-Gig) cost more than smaller (ie 1-Gig). This fee applies to MICE core equipment only. Port costs on extension switches are dictated by the respective extension switch operators. As there are already many existing connected ports I would also be in favor of retroactively charging each connected port based on their current port type(s) in use at the time of the implementation of said fees. Already connected organizations that want to take a bit of time to budget can make arrangements with the board. Also, I would be in favor of implementing this soon instead of waiting for the desired but elusive non-profit status unless someone has knowledge of its actual pending completion. I feel this method alleviates the problem of non-payment induced disconnects, switch port reservations, tracking payments, etc. I don't know the history of other exchanges with regards to disconnecting organizations, freeing up port and IP number resources, but clearly sometimes companies go under or otherwise have a change in strategy and no longer desire to be connected yet do not formally announce they are disconnecting. So to that end, after some period of time (X months) of being in an unreachable state via their IXP IP addresses, a disconnect is implemented unless said organization otherwise scheduled or coordinated with MICE to be offline for an extended period of time. I am guessing there is some historical data available from other exchanges or other best practices that could be used to guide us on how to do this. With regards to gaining membership/voting rights I like the notion of requiring an active port connection to the exchange (core or extension) with an ASN and reachable exchange assigned IP address(es). An alternate method of gaining membership/voting rights would be from serving on the board of the exchange which would not require one to be an already connected organization of the exchange. Summary of member voting eligibility Provides Eligibility to Vote: * Elected board members. Board members may or may not represent a connected organization, I assume the bylaws cover the 1-vote max per organization but that is implied here * Connected organizations with ASN and at least one reachable exchange assigned IP number Does Not Provide Eligibility to Vote: * Sponsors, 3rd parties, etc (eg Cologix, operators of extension switches) * Organizations exclusively using ancillary exchange services (eg pvlans) ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Steve Howard [showard(a)PAULBUNYAN.NET] Sent: Thursday, September 15, 2016 7:22 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Proposed Membership and Sponsorship Model I'm a little "late to the party" for this discussion, but I'd like to add a few of my thoughts: First off, I'd like to thank Dave for coming up with great ideas and points for discussion. MICE is a cooperative (not a charity). As a cooperative, we need to treat everybody equally. I feel that everybody should have to pay membership dues at some level, perhaps an amount based upon port size. If we fail to require the payment of dues, we will have different classes of users and the potential for discontent as some people don't pay their fair share. Like Martin said, "Classes create inequities". If people don't pay, they need to be disconnected (after a reasonable grace period). The dollar amounts due will likely be very small, the exchange has gotten by without any payments required for several years. I don't think that the fees that would need to be charged would be anything that an organization connecting to the exchange cannot absorb. If they can't handle ~$9/month (using Dave's example numbers), they won't be around long in this industry. I agree with Richard Laager's points about enforcement. We need to enforce whatever policy is decided. I don't like any scenario where the board might have the option to waive fees. That will create the feeling of the "good 'ol boy" system. I also think that all newly connected members should get their initial connect period free, primarily as a sales tool. I'd like to give new members a chance to "try before you buy" to see if it works for them. Perhaps we bill annually and they don't pay until the next billing cycle starts (or some variation thereof)? Until they pay, they should be subject to their port being assigned to somebody else, not be official members, and not able to vote. How do we handle members that are connected to remote switches? Is the switch owner the member? The downstream users? Both? Who pays the port charges? Only the switch owner? Steve On 09/08/2016 01:05 PM, David Farmer wrote: I'm not a candidate and will not be a candidate for the Board. However, I propose the following Membership and Sponsorship Model, including both Membership Dues and Sponsorship levels for MICE Members and Non-Member Sponsors. The dollar values are just my suggestions, I think they should work but they are primarily intended to illustrate the model. With this Model, any network may connect to the Exchange, but all networks participating in governance, known as Members, will have some minimal skin-in-the-game. Further, the Model implies Networks with 10G or larger ports probably should participate at a more significant level, but this is not a requirement for access to 10G or larger ports or even to participate in governance. The Friends of MICE Sponsorship level is intended to be accessible to anyone interested, including individual users of the Internet. My proposal; As an Open Internet Exchange any network with a uniquely registered Autonomous System Number (ASN) may connect to MICE. However, Non-Member networks are provided ports on the Exchange on an as-available basis only and Non-Members do not participate in the governance of the Exchange or the MICE Organization, but may observe and participate in its discussions. Networks connected to the Exchange, or Data Centers hosting an Exchange switch, may become Members and participate in the governance of the Exchange and the MICE Organization by paying Membership Dues at either level listed below. Members are provided prefered access to 1G Exchange ports, and are provided access to 10G or larger Exchange ports on an as-available basis. Sustaining or higher Members are committed to sustaining and growing the Exchange by electing to pay Sustaining Membership Dues and/or paying Basic Membership Dues and providing other Sponsorship to the Exchange of a value totaling $1000 or more. In recognition to this additional commitment, Sustaining or higher Members are provided prefered access to 10G or larger Exchange ports. Basic Membership Dues - $100 Sustaining Membership Dues - $1000 (Dues are paid in cash or cash equivalents only, and may not be provided through in-kind services or equipment) Beyond Dues from Members, MICE accepts Sponsorship both from Members and Non-Members Sponsors as cash, cash equivalents, in-kind services or equipment. Sponsorship is recognized at the following escalating levels based on the total value provided annually. Friends of MICE - Sponsorship of $50 ore more annually Sustaining Members or Sponsors - Sponsorship of $1,000 or more annually Silver Members or Sponsors - Sponsorship of $2,500 or more annually Gold Members or Sponsors - Sponsorship of $5,000 or more annually Platinum Members or Sponsors - Sponsorship of $10,000 or more annually Sponsorship of $20,000 or more are recognized with multiple years at the Platinum level. Friends of MICE are list on a page on the MICE website. Sustaining or higher Members are annotated as such in their listings on the MICE website. All Sustaining or higher Members and Sponsors are listed on the MICE website, with higher levels receiving more prominent placement. Silver or higher Members and Sponsors are recognized in MICE marketing materials, with higher levels receiving more prominent placement. Thanks -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815<tel:612-626-0815> Minneapolis, MN 55414-3029 Cell: 612-812-9952<tel:612-812-9952> =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Matthew Beckwell

Thanks Steve, I haven't done much with iperf3-- it looks like they took away the dual-test option, so I'm a little blind. Are you able to see loss on the iperf3 server side from 199.168.228.123 ? ~Matthew On Mon, Sep 19, 2016 at 2:03 PM, Steve Howard <showard(a)paulbunyan.net> wrote: > > We've opened up our iperf3 server on 209.191.196.90 for a few hours. > > > ------------------------------ > > >

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Doug McIntyre

On Mon, Sep 19, 2016 at 02:03:41PM -0500, Andrew Hoyos wrote: > What medium is the VC built on and physical topology? Just VC cables on the rear ringed through all 3? Yes, VC cabling in a ring, and using the VCP modules and cabling on the EX4550. > Specific to any single 4500 destination or is it doing it to both the 4500 and the 4550? He wasn't aware of the EX4550, there aren't many ports lit there yet, he has just been testing different ports on the the EX4500. > Check ‘show virtual-chassis vc-path source-interface <src int> destination-interface <dest int>’. Common VC port in the mix to any of the reported loss paths? I think in general, the VC ports run Active/Passive, so only one is "active" at a time going out of a box without load balancing, so only one (ie. vcp-1) typically shows up for all paths. But either way, the virtual-chassis stats show zero errors, all the vcp-? ports show zero errors or drops. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Steve Howard

Sept. 19, 2016

Re: EX4200 (1G Switch) Packet Loss -> 10G Participants

by Andrew Hoyos

> On Sep 19, 2016, at 1:56 PM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > > Yes, there is a VC stack of > > EX4200 > EX4500 > EX4550 What medium is the VC built on and physical topology? Just VC cables on the rear ringed through all 3? Specific to any single 4500 destination or is it doing it to both the 4500 and the 4550? Check ‘show virtual-chassis vc-path source-interface <src int> destination-interface <dest int>’. Common VC port in the mix to any of the reported loss paths? -- Andrew Hoyos hoyosa(a)gmail.com

Sept. 19, 2016