May 2011 - mice-discuss - lists.micemn.net

Re: Akamai Traffic on MICE

by David Farmer

It looks like Akamai has found a sweet spot they tend to do that, the link between the switches peaked up to about 850 and backed off and is hovering abut 650Mbps for the last hour or so. From what I've seen in the past for Amakai it will probably do that the rest of the day. On 5/19/11 07:23 CDT, David Farmer wrote: > The 10G Akamai stack at the University is having an affect. > > The University turned up routing from the GigaPOP to MICE on its 10G > link Tuesday around 8pm. The new paths were also announced to the Akamai > route collector around that time too. Akamai started using these new > paths around 1:30 in the afternoon on Wednesday. The University's 10G > link peaked a little over 1G around 9:30pm last night. The bottleneck is > the 1G link between the CNS 10G switch and the 4500. It was saturated > from about 6:30pm to 9:30pm at about 950Mbps. > > Some of us are organizing to rectify the situation ASAP. There are some > workarounds that could help in the very short-term, yet this morning, > and we are looking to get the new Juniper switches with 10G up-links > installed ASAP However, the earliest that can happen it this afternoon, > as the switches are down in Mankato with Jay. There is a plan to get > them up to 511 by mid-afternoon. > > Also I'll be watching the links and can modify our announcement to the > Akamai route collector if necessary. Things are going to be fluid today. > > I'll just say that this is the kind of problem I like to see, if you are > going to have to deal with any problem. That is the problem of over > succeeding. > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Steve Howard

On 05/19/2011 07:58 AM, Mitch Elfstrand wrote: > SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. > That sounds like this might be an ideal short-term fix. Feel free to plug into Gig 0/2 on the CNS switch. We didn't leave any optics down there, you will need to provide a Cisco (or compatible) SFP. Alternatively, you could consider doing a 10Gig connection. Your peak was almost 600Mbps last night! If you want to do 10Gig, you'll need X2 optics for the CNS switch. Let me know and I'll assign a 10Gig port. CNS will donate use of 10Gig LR optics to MICE for use in the CNS switch to connect to the Juniper switch. Unfortunately, those optics are in Bemidji. I will check to see if anybody here is going to the Cities today, if so I will send the optics with them, if not, I will overnight them to Mike Hemphill at FWR. (Mike, I hope that is OK with you). Thanks, Steve > Regards, > > Mitch Elfstrand | Director of Network Engineering > mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA > Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 > > > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Thursday, May 19, 2011 7:54 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE > > On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: >> Some of us are organizing to rectify the situation ASAP. There are >> some workarounds that could help in the very short-term... > Specificly one thing that could be done is if SDN could be moved from > the 4500 onto the 3560, and FWR could string another cross-connect > temporarily between the 3560 and the other fiber "uplink" port on the > 4500, we could change those two gig ports into an Etherchannel. > > This would move the biggest destination of the Akamai traffic closer > to the source, as well as preventing one endpoint from swamping the > interconnect.. > > Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Mitch Elfstrand

SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. Regards, Mitch Elfstrand | Director of Network Engineering mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Thursday, May 19, 2011 7:54 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: > Some of us are organizing to rectify the situation ASAP. There are > some workarounds that could help in the very short-term... Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel. This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect.. Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: > Some of us are organizing to rectify the situation ASAP. There are > some workarounds that could help in the very short-term... Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel. This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect.. Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Akamai Traffic on MICE

by David Farmer

The 10G Akamai stack at the University is having an affect. The University turned up routing from the GigaPOP to MICE on its 10G link Tuesday around 8pm. The new paths were also announced to the Akamai route collector around that time too. Akamai started using these new paths around 1:30 in the afternoon on Wednesday. The University's 10G link peaked a little over 1G around 9:30pm last night. The bottleneck is the 1G link between the CNS 10G switch and the 4500. It was saturated from about 6:30pm to 9:30pm at about 950Mbps. Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term, yet this morning, and we are looking to get the new Juniper switches with 10G up-links installed ASAP However, the earliest that can happen it this afternoon, as the switches are down in Mankato with Jay. There is a plan to get them up to 511 by mid-afternoon. Also I'll be watching the links and can modify our announcement to the Akamai route collector if necessary. Things are going to be fluid today. I'll just say that this is the kind of problem I like to see, if you are going to have to deal with any problem. That is the problem of over succeeding. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: MICE Traffic

by Tom Drill

We can help with lead time. On Wed, May 18, 2011 at 10:24 PM, Jay Hanke <jayhanke(a)mankatonetworks.net>wrote: > We have seen a dramatic growth in traffic on the exchange. The GE link > between the CNS remote switch and the Main switch is running about > ~950 Mb/s. We're going to need to make some upgrades shortly to avoid > degrading services. Perhaps more quickly than originally planned. > > We may need some volunteers and stuff depending on the course of action. > > Jay > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- MyTELEPATH | Tom Drill tdrill(a)mytelepath.com 11970 Portland Ave. Burnsville, MN 55337 952-400-6004 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: MICE Traffic

by Jeremy Lumby

I would be happy to help out at 511 today if needed. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Wednesday, May 18, 2011 10:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Traffic We have seen a dramatic growth in traffic on the exchange. The GE link between the CNS remote switch and the Main switch is running about ~950 Mb/s. We're going to need to make some upgrades shortly to avoid degrading services. Perhaps more quickly than originally planned. We may need some volunteers and stuff depending on the course of action. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Traffic between 10Gig & 1Gig Switches

by Owen DeLong

On May 18, 2011, at 2:52 PM, Steve Howard wrote: > The traffic to/from the U shot up today. I'm guessing that is CDN related? > > The link between the CNS switch and the MICE switch shot up to ~850Mbps > today. Too bad that is only a single gig connection! > > It is great to have the traffic on the exchange, but we should probably > deal with this nearly saturated link fast. Here are a few possible > solutions: > > 1) Add another Gig connection. We'd probably need to pay FWR for a > cross-connect (CNS is paying for the first one) with this option; > > 2) Use one of the new Juniper switches with a 10Gig interface and > connect that to a 10Gig interface on the existing MICE 10Gig switch, > then move some of the 1Gig members over to those ports. This would > require purchase of 10Gig optics for the Juniper and the existing 10Gig > MICE switch; > > 3) Move some of the higher bandwidth MICE members to the 1Gig ports on > the 10Gig switch. Would this increase the FWR cross-connect charges > for those members if they are going from copper to fiber? > > 4) Encourage MICE members on the 1Gig switch to upgrade to a 10Gig > connection; > > 5) Ask the U to not advertise some of the members on the 1Gig switch to > CDNs (if that is the cause of the traffic load). This is a bummer if > you are one of the networks that gets left out. I really don't like > this option -- we are finally getting real traffic on the exchange and > then we'd be cutting back. > > What does everybody think? Any other suggestions? > To the extent that my input is useful, I encourage option 2. I believe this is the more scalable longer-term solution and does not represent a significantly higher cost than any of the others will in relatively short order. Owen ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

MICE Traffic

by Jay Hanke

We have seen a dramatic growth in traffic on the exchange. The GE link between the CNS remote switch and the Main switch is running about ~950 Mb/s. We're going to need to make some upgrades shortly to avoid degrading services. Perhaps more quickly than originally planned. We may need some volunteers and stuff depending on the course of action. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Traffic between 10Gig & 1Gig Switches

by Jay Hanke

On Wed, May 18, 2011 at 5:43 PM, Steve Howard <showard(a)paulbunyan.net> wrote: > What type? LX? Should be, I didn't pull them out to verify. James can you take a quick look at those? ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 18, 2011