- mice-discuss - lists.micemn.net

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 01:55:20PM -0500, Andrew Hoyos wrote: > On May 19, 2011, at 1:43 PM, Doug McIntyre wrote: > > > It looks like the traffic load balancing is a little uneven, but Dave > > and I will keep watch on it. > > I think the Cisco default in a PortChannel is to load balance by src-mac, which might have much help in the current situation, since most of the traffic is coming/going to one mac addr. > > You might want to try changing it to src/dst IP for a better distribution of flows across both links, or even per flow, if the box supports it. The default for the 4500/SupIV is for src-dst-ip, and Dave changed the 3560E to do this as well when we brought up the port-channel. Its converging to something pretty close now Gig2/45 5 minute input rate 325460000 bits/sec, 35790 packets/sec Gig1/2 5 minute input rate 337375000 bits/sec, 36799 packets/sec -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Andrew Hoyos

On May 19, 2011, at 1:43 PM, Doug McIntyre wrote: > It looks like the traffic load balancing is a little uneven, but Dave > and I will keep watch on it. I think the Cisco default in a PortChannel is to load balance by src-mac, which might have much help in the current situation, since most of the traffic is coming/going to one mac addr. You might want to try changing it to src/dst IP for a better distribution of flows across both links, or even per flow, if the box supports it. IIRC, "port-channel load-balance ?" should show you what the capabilities are. Thanks, Andrew ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 10:07:37AM -0500, Mike Hemphill wrote: > It looks like we (Dave Farmer & FWR) will be putting up a second Gig link > over copper between the CNS & core MICE sw later this AM. Dave and I have completed the ether channel to bring it up to a 2Gbps cross link between the two switches.. MICE-gig-sw-1#show etherchannel port-channel Channel-group listing: ---------------------- Group: 1 ---------- Port-channels in the group: --------------------------- Port-channel: Po1 ------------ Age of the Port-channel = 0d:03h:38m:32s Logical slot/port = 11/1 Number of ports = 2 GC = 0x00000000 Port state = Port-channel Ag-Inuse Protocol = - Port security = Disabled Ports in the Port-channel: Index Load Port EC state No of bits ------+------+------+------------------+----------- 1 00 Gi1/2 On 0 0 00 Gi2/45 On 0 Time since last port bundled: 0d:00h:12m:09s Gi1/2 Time since last port Un-bundled: 0d:02h:19m:56s Gi2/45 It looks like the traffic load balancing is a little uneven, but Dave and I will keep watch on it. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Traffic between 10Gig & 1Gig Switches

by James Montz

Switches are now @ 511 in FWR Suite. I did not get a chance to check the SFP. I will be back @ 511 to do install/config around 4pm if noone else gets to it before then. On May 19, 2011 12:20 AM, "Owen DeLong" <owend(a)he.net> wrote: > On May 18, 2011, at 2:52 PM, Steve Howard wrote: > >> The traffic to/from the U shot up today. I'm guessing that is CDN related? >> >> The link between the CNS switch and the MICE switch shot up to ~850Mbps >> today. Too bad that is only a single gig connection! >> >> It is great to have the traffic on the exchange, but we should probably >> deal with this nearly saturated link fast. Here are a few possible >> solutions: >> >> 1) Add another Gig connection. We'd probably need to pay FWR for a >> cross-connect (CNS is paying for the first one) with this option; >> >> 2) Use one of the new Juniper switches with a 10Gig interface and >> connect that to a 10Gig interface on the existing MICE 10Gig switch, >> then move some of the 1Gig members over to those ports. This would >> require purchase of 10Gig optics for the Juniper and the existing 10Gig >> MICE switch; >> >> 3) Move some of the higher bandwidth MICE members to the 1Gig ports on >> the 10Gig switch. Would this increase the FWR cross-connect charges >> for those members if they are going from copper to fiber? >> >> 4) Encourage MICE members on the 1Gig switch to upgrade to a 10Gig >> connection; >> >> 5) Ask the U to not advertise some of the members on the 1Gig switch to >> CDNs (if that is the cause of the traffic load). This is a bummer if >> you are one of the networks that gets left out. I really don't like >> this option -- we are finally getting real traffic on the exchange and >> then we'd be cutting back. >> >> What does everybody think? Any other suggestions? >> > > To the extent that my input is useful, I encourage option 2. > > I believe this is the more scalable longer-term solution and does not represent > a significantly higher cost than any of the others will in relatively short order. > > Owen > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Mike Hemphill

Steve, Happy to accept the shipment & install, if that is the way that it works out. It looks like we (Dave Farmer & FWR) will be putting up a second Gig link over copper between the CNS & core MICE sw later this AM. Mike Mike Hemphill FWR Communication Networks VP Sales 612-333-1922 voice 612-812-5242 cell 612-333-1945 fax mhemphill(a)mngateway.com Visit us at: www.mngateway.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Thursday, May 19, 2011 9:42 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On 05/19/2011 07:58 AM, Mitch Elfstrand wrote: > SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. > That sounds like this might be an ideal short-term fix. Feel free to plug into Gig 0/2 on the CNS switch. We didn't leave any optics down there, you will need to provide a Cisco (or compatible) SFP. Alternatively, you could consider doing a 10Gig connection. Your peak was almost 600Mbps last night! If you want to do 10Gig, you'll need X2 optics for the CNS switch. Let me know and I'll assign a 10Gig port. CNS will donate use of 10Gig LR optics to MICE for use in the CNS switch to connect to the Juniper switch. Unfortunately, those optics are in Bemidji. I will check to see if anybody here is going to the Cities today, if so I will send the optics with them, if not, I will overnight them to Mike Hemphill at FWR. (Mike, I hope that is OK with you). Thanks, Steve > Regards, > > Mitch Elfstrand | Director of Network Engineering > mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA > Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 > > > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Thursday, May 19, 2011 7:54 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE > > On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: >> Some of us are organizing to rectify the situation ASAP. There are >> some workarounds that could help in the very short-term... > Specificly one thing that could be done is if SDN could be moved from > the 4500 onto the 3560, and FWR could string another cross-connect > temporarily between the 3560 and the other fiber "uplink" port on the > 4500, we could change those two gig ports into an Etherchannel. > > This would move the biggest destination of the Akamai traffic closer > to the source, as well as preventing one endpoint from swamping the > interconnect.. > > Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by David Farmer

It looks like Akamai has found a sweet spot they tend to do that, the link between the switches peaked up to about 850 and backed off and is hovering abut 650Mbps for the last hour or so. From what I've seen in the past for Amakai it will probably do that the rest of the day. On 5/19/11 07:23 CDT, David Farmer wrote: > The 10G Akamai stack at the University is having an affect. > > The University turned up routing from the GigaPOP to MICE on its 10G > link Tuesday around 8pm. The new paths were also announced to the Akamai > route collector around that time too. Akamai started using these new > paths around 1:30 in the afternoon on Wednesday. The University's 10G > link peaked a little over 1G around 9:30pm last night. The bottleneck is > the 1G link between the CNS 10G switch and the 4500. It was saturated > from about 6:30pm to 9:30pm at about 950Mbps. > > Some of us are organizing to rectify the situation ASAP. There are some > workarounds that could help in the very short-term, yet this morning, > and we are looking to get the new Juniper switches with 10G up-links > installed ASAP However, the earliest that can happen it this afternoon, > as the switches are down in Mankato with Jay. There is a plan to get > them up to 511 by mid-afternoon. > > Also I'll be watching the links and can modify our announcement to the > Akamai route collector if necessary. Things are going to be fluid today. > > I'll just say that this is the kind of problem I like to see, if you are > going to have to deal with any problem. That is the problem of over > succeeding. > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Steve Howard

On 05/19/2011 07:58 AM, Mitch Elfstrand wrote: > SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. > That sounds like this might be an ideal short-term fix. Feel free to plug into Gig 0/2 on the CNS switch. We didn't leave any optics down there, you will need to provide a Cisco (or compatible) SFP. Alternatively, you could consider doing a 10Gig connection. Your peak was almost 600Mbps last night! If you want to do 10Gig, you'll need X2 optics for the CNS switch. Let me know and I'll assign a 10Gig port. CNS will donate use of 10Gig LR optics to MICE for use in the CNS switch to connect to the Juniper switch. Unfortunately, those optics are in Bemidji. I will check to see if anybody here is going to the Cities today, if so I will send the optics with them, if not, I will overnight them to Mike Hemphill at FWR. (Mike, I hope that is OK with you). Thanks, Steve > Regards, > > Mitch Elfstrand | Director of Network Engineering > mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA > Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 > > > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Thursday, May 19, 2011 7:54 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE > > On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: >> Some of us are organizing to rectify the situation ASAP. There are >> some workarounds that could help in the very short-term... > Specificly one thing that could be done is if SDN could be moved from > the 4500 onto the 3560, and FWR could string another cross-connect > temporarily between the 3560 and the other fiber "uplink" port on the > 4500, we could change those two gig ports into an Etherchannel. > > This would move the biggest destination of the Akamai traffic closer > to the source, as well as preventing one endpoint from swamping the > interconnect.. > > Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Mitch Elfstrand

SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. Regards, Mitch Elfstrand | Director of Network Engineering mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Thursday, May 19, 2011 7:54 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: > Some of us are organizing to rectify the situation ASAP. There are > some workarounds that could help in the very short-term... Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel. This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect.. Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: > Some of us are organizing to rectify the situation ASAP. There are > some workarounds that could help in the very short-term... Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel. This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect.. Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Akamai Traffic on MICE

by David Farmer

The 10G Akamai stack at the University is having an affect. The University turned up routing from the GigaPOP to MICE on its 10G link Tuesday around 8pm. The new paths were also announced to the Akamai route collector around that time too. Akamai started using these new paths around 1:30 in the afternoon on Wednesday. The University's 10G link peaked a little over 1G around 9:30pm last night. The bottleneck is the 1G link between the CNS 10G switch and the 4500. It was saturated from about 6:30pm to 9:30pm at about 950Mbps. Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term, yet this morning, and we are looking to get the new Juniper switches with 10G up-links installed ASAP However, the earliest that can happen it this afternoon, as the switches are down in Mankato with Jay. There is a plan to get them up to 511 by mid-afternoon. Also I'll be watching the links and can modify our announcement to the Akamai route collector if necessary. Things are going to be fluid today. I'll just say that this is the kind of problem I like to see, if you are going to have to deal with any problem. That is the problem of over succeeding. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011