SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. Regards, Mitch Elfstrand | Director of Network Engineering mitch.elfstrand@sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Thursday, May 19, 2011 7:54 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote:

Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term...

Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel. This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect.. Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. -- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Steve, Happy to accept the shipment & install, if that is the way that it works out. It looks like we (Dave Farmer & FWR) will be putting up a second Gig link over copper between the CNS & core MICE sw later this AM. Mike Mike Hemphill FWR Communication Networks VP Sales 612-333-1922 voice 612-812-5242 cell 612-333-1945 fax mhemphill@mngateway.com Visit us at: www.mngateway.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Thursday, May 19, 2011 9:42 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On 05/19/2011 07:58 AM, Mitch Elfstrand wrote:

SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything.

That sounds like this might be an ideal short-term fix. Feel free to plug into Gig 0/2 on the CNS switch. We didn't leave any optics down there, you will need to provide a Cisco (or compatible) SFP. Alternatively, you could consider doing a 10Gig connection. Your peak was almost 600Mbps last night! If you want to do 10Gig, you'll need X2 optics for the CNS switch. Let me know and I'll assign a 10Gig port. CNS will donate use of 10Gig LR optics to MICE for use in the CNS switch to connect to the Juniper switch. Unfortunately, those optics are in Bemidji. I will check to see if anybody here is going to the Cities today, if so I will send the optics with them, if not, I will overnight them to Mike Hemphill at FWR. (Mike, I hope that is OK with you). Thanks, Steve

Regards,

Mitch Elfstrand | Director of Network Engineering mitch.elfstrand@sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782

-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Thursday, May 19, 2011 7:54 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE

On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote:

...

Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term... Specificly one thing that could be done is if SDN could be moved from the 4500 onto the 3560, and FWR could string another cross-connect temporarily between the 3560 and the other fiber "uplink" port on the 4500, we could change those two gig ports into an Etherchannel.

This would move the biggest destination of the Akamai traffic closer to the source, as well as preventing one endpoint from swamping the interconnect..

Even if SDN could be moved to the 3560 would do a lot to alleviate the issue.

-- Doug McIntyre <merlyn@iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades

########################################################################

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

########################################################################

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

On May 19, 2011, at 1:43 PM, Doug McIntyre wrote:

It looks like the traffic load balancing is a little uneven, but Dave and I will keep watch on it.

I think the Cisco default in a PortChannel is to load balance by src-mac, which might have much help in the current situation, since most of the traffic is coming/going to one mac addr. You might want to try changing it to src/dst IP for a better distribution of flows across both links, or even per flow, if the box supports it. IIRC, "port-channel load-balance ?" should show you what the capabilities are. Thanks, Andrew ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

On Thu, May 19, 2011 at 7:58 AM, Mitch Elfstrand <Mitch.Elfstrand@sdncommunications.com> wrote:

SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything.

Has SDN considered moving to a 10 G connection? You're running 800 Mb/s of traffic consistently. When the Akamai cluster gets upgraded at the U you'll likely be blocking pretty bad. We've got the new Junipers about ready to go so you would have your choice of connection points. CNS, Are you planning on moving to 10G for your switch as well? Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Yes, the optics are ready to go in port Ten 0/10.  (Note that this port was previously as Hurricane Electric).

Mike at FWR is running an interconnect to the old switch to the new Junipers. I would like to at least move SDN if possible at the same time. Maybe it would be best to move everything at once. We don't have any 1G SFP modules. The only two links on fiber are SDN and CNS. Are we making progress on graphing? Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

As was noted we installed a 2nd GigE between the two switches yesterday. Things are looked good last night we peaked at 1.7Gbps most of that coming from the U's Akamai Stack. Akamai tells me we are maxing out our servers again, it looks like we were maxing them out most of the spring anyway. The Akamai servers are putting out a little less than 2G from about 7am to 11pm. I believe we are seeing more traffic in the evening going to MICE because most of the other Gigapop networks have more of a business day demand curve, so Akamai serves more of the demand out of MICE in the evening as demand from the Gigapop networks slack off. So I think things are much better now, we should get the new switches installed and people moved over but we shouldn't have to rush it. The University will work with Akamai to get more servers over the summer, but I would hope we would have everyone moved over to the juniper switches long before that happens. On 5/19/11 09:56 CDT, David Farmer wrote:

It looks like Akamai has found a sweet spot they tend to do that, the link between the switches peaked up to about 850 and backed off and is hovering abut 650Mbps for the last hour or so. From what I've seen in the past for Amakai it will probably do that the rest of the day.

On 5/19/11 07:23 CDT, David Farmer wrote:

...

The 10G Akamai stack at the University is having an affect.

The University turned up routing from the GigaPOP to MICE on its 10G link Tuesday around 8pm. The new paths were also announced to the Akamai route collector around that time too. Akamai started using these new paths around 1:30 in the afternoon on Wednesday. The University's 10G link peaked a little over 1G around 9:30pm last night. The bottleneck is the 1G link between the CNS 10G switch and the 4500. It was saturated from about 6:30pm to 9:30pm at about 950Mbps.

Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term, yet this morning, and we are looking to get the new Juniper switches with 10G up-links installed ASAP However, the earliest that can happen it this afternoon, as the switches are down in Mankato with Jay. There is a plan to get them up to 511 by mid-afternoon.

Also I'll be watching the links and can modify our announcement to the Akamai route collector if necessary. Things are going to be fluid today.

I'll just say that this is the kind of problem I like to see, if you are going to have to deal with any problem. That is the problem of over succeeding.

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

I was planing to make that part of the discussion with Akamai. However, even without MICE we were going to need more servers for next fall. It seems our servers were shedding demand much of the spring. Since school is out, our demand has dropped by about 50% over the last 2 weeks, and is down about 65% from our Spring peak about 6 weeks ago. On 5/20/11 10:23 CDT, Owen DeLong wrote:

Is there any way we can use the amount of U/Akamai traffic being pointed at MICE as an incentive for Akamai to come to MICE themselves?

Owen

On May 20, 2011, at 8:12 AM, David Farmer wrote:

...

As was noted we installed a 2nd GigE between the two switches yesterday. Things are looked good last night we peaked at 1.7Gbps most of that coming from the U's Akamai Stack. Akamai tells me we are maxing out our servers again, it looks like we were maxing them out most of the spring anyway.

The Akamai servers are putting out a little less than 2G from about 7am to 11pm. I believe we are seeing more traffic in the evening going to MICE because most of the other Gigapop networks have more of a business day demand curve, so Akamai serves more of the demand out of MICE in the evening as demand from the Gigapop networks slack off.

So I think things are much better now, we should get the new switches installed and people moved over but we shouldn't have to rush it. The University will work with Akamai to get more servers over the summer, but I would hope we would have everyone moved over to the juniper switches long before that happens.

On 5/19/11 09:56 CDT, David Farmer wrote:

...

It looks like Akamai has found a sweet spot they tend to do that, the link between the switches peaked up to about 850 and backed off and is hovering abut 650Mbps for the last hour or so. From what I've seen in the past for Amakai it will probably do that the rest of the day.

On 5/19/11 07:23 CDT, David Farmer wrote:

...

The 10G Akamai stack at the University is having an affect.

The University turned up routing from the GigaPOP to MICE on its 10G link Tuesday around 8pm. The new paths were also announced to the Akamai route collector around that time too. Akamai started using these new paths around 1:30 in the afternoon on Wednesday. The University's 10G link peaked a little over 1G around 9:30pm last night. The bottleneck is the 1G link between the CNS 10G switch and the 4500. It was saturated from about 6:30pm to 9:30pm at about 950Mbps.

Some of us are organizing to rectify the situation ASAP. There are some workarounds that could help in the very short-term, yet this morning, and we are looking to get the new Juniper switches with 10G up-links installed ASAP However, the earliest that can happen it this afternoon, as the switches are down in Mankato with Jay. There is a plan to get them up to 511 by mid-afternoon.

Also I'll be watching the links and can modify our announcement to the Akamai route collector if necessary. Things are going to be fluid today.

I'll just say that this is the kind of problem I like to see, if you are going to have to deal with any problem. That is the problem of over succeeding.

-- =============================================== David Farmer Email:farmer@umn.edu Networking& Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

########################################################################

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

########################################################################

To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1