The MICE Akamai cluster is currently running at reduced capacity. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Do you need me to go to 511 today to assist? From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Tuesday, October 25, 2011 8:32 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Akamai Issues The MICE Akamai cluster is currently running at reduced capacity. Jay To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Tue, Oct 25, 2011 at 8:42 AM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
** Do you need me to go to 511 today to assist?
Thanks for the offer. We've exceeded the allotment of cache flow bandwidth. We're working with Akamai on solutions. jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Did this get resolved? Frank From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Tuesday, October 25, 2011 9:18 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Akamai Issues On Tue, Oct 25, 2011 at 8:42 AM, Jeremy Lumby <jlumby@mnvoip.com<mailto:jlumby@mnvoip.com>> wrote: Do you need me to go to 511 today to assist? Thanks for the offer. We've exceeded the allotment of cache flow bandwidth. We're working with Akamai on solutions. jay ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Wed, Oct 26, 2011 at 10:37 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Did this get resolved?
From the graphs it would appear not as there is basically no traffic on the links to Akamai.
Justin/Dave/Marty, do you have an update? Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, 2011-10-27 at 02:01 -0500, Jay Hanke wrote:
On Wed, Oct 26, 2011 at 10:37 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Did this get resolved?
From the graphs it would appear not as there is basically no traffic on the links to Akamai.
Justin/Dave/Marty, do you have an update?
Jay
I sent over our proposal to Marty for Akamai's approval this morning. If he says its OK then we should be able to get implement without much delay, depending on what processes and procedures are needed on the UMN side. Will keep this list updated. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Oct 27, 2011 at 11:30:22AM -0500, Justin Krejci wrote:
I sent over our proposal to Marty for Akamai's approval this morning. If he says its OK then we should be able to get implement without much delay, depending on what processes and procedures are needed on the UMN side.
Will keep this list updated.
I must not understand the problem. What does USI have to do with the Akamai stuff? Is there a problem between the two? Help me to understand, please. -- Mike Horwath ipHouse - Welcome home! drechsau@iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, 2011-10-27 at 13:10 -0500, Mike Horwath wrote:
What does USI have to do with the Akamai stuff? Is there a problem between the two?
I believe they are donating the bandwidth to feed the Akamai cluster. Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Oct 27, 2011 at 01:42:43PM -0500, Richard Laager wrote:
On Thu, 2011-10-27 at 13:10 -0500, Mike Horwath wrote:
What does USI have to do with the Akamai stuff? Is there a problem between the two?
I believe they are donating the bandwidth to feed the Akamai cluster.
Ok. But what's the problem that occured to take the Akamai cluster offline? -- Mike Horwath ipHouse - Welcome home! drechsau@iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
It's not offline. Our NOC inadvertently shut it off after I had asked them to simply place a cap on the amount of capacity it would push. That was fixed this morning. I would've seen it sooner myself, but I'm in Senegal at the moment. Best, -M< On 10/27/11 6:46 PM, "Mike Horwath" <drechsau@IPHOUSE.NET> wrote:
On Thu, Oct 27, 2011 at 01:42:43PM -0500, Richard Laager wrote:
On Thu, 2011-10-27 at 13:10 -0500, Mike Horwath wrote:
What does USI have to do with the Akamai stuff? Is there a problem between the two?
I believe they are donating the bandwidth to feed the Akamai cluster.
Ok.
But what's the problem that occured to take the Akamai cluster offline?
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Oct 27, 2011 at 01:49:36PM -0500, Hannigan, Martin wrote:
It's not offline. Our NOC inadvertently shut it off after I had asked them to simply place a cap on the amount of capacity it would push. That was fixed this morning. I would've seen it sooner myself, but I'm in Senegal at the moment.
So this had nothing to do with USI then? I am still pretty confused. I have 2 reasons right now for outage: USI has a proposal into Akamai regarding <something> Akamai inadvertently shut it off after a cap was put into place but neither is really giving me an answer. Is the problem that USI donated bandwdith and some cap was exceeded for feeding the Akamai cluster? If that's the case, what is the cap? Can the load be shared amongst some/all providers for the benefit of the exchange point? Is the problem that Akamai exceeded an internal cap for feeding into MICE's network and it needed to be adjusted? If that's the case, what was the cap? What is it now? Is there anything others can do to help alleviate this in the future? -- Mike Horwath ipHouse - Welcome home! drechsau@iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 10/27/11 13:55 CDT, Mike Horwath wrote:
On Thu, Oct 27, 2011 at 01:49:36PM -0500, Hannigan, Martin wrote:
It's not offline. Our NOC inadvertently shut it off after I had asked them to simply place a cap on the amount of capacity it would push. That was fixed this morning. I would've seen it sooner myself, but I'm in Senegal at the moment.
So this had nothing to do with USI then?
I am still pretty confused.
I have 2 reasons right now for outage:
USI has a proposal into Akamai regarding<something>
The amount of transit necessary for cache fill is more the USI was told and is causing some congestion for them. We are proposing to Akamai and UMN supplements the inbound transit for cache fill of the node.
Akamai inadvertently shut it off after a cap was put into place
This is the cause of the outage, not the cause of the initial problem that most people weren't aware of.
but neither is really giving me an answer.
Is the problem that USI donated bandwdith and some cap was exceeded for feeding the Akamai cluster?
If that's the case, what is the cap?
There isn't a cap but there was congestion.
Can the load be shared amongst some/all providers for the benefit of the exchange point?
We are talking to Akamai about the issue.
Is the problem that Akamai exceeded an internal cap for feeding into MICE's network and it needed to be adjusted?
If that's the case, what was the cap? What is it now?
Is there anything others can do to help alleviate this in the future?
Over the summer we were averaging 800Mb to 1Gb of combine cache fill and TCP ACK traffic, and serving a total of 2 to 2.5G of content out of our node to both MICE and GigaPOP participants. At peak the two nodes seem to be serving 2.5 to 3G, but I'm not really confident of those number because we really haven't got everything settled in yet, I'd like to see a week or two of sustained numbers before I believe we've settled in. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On 10/27/11 7:07 PM, "David Farmer" <farmer@UMN.EDU> wrote:
On 10/27/11 13:55 CDT, Mike Horwath wrote:
On Thu, Oct 27, 2011 at 01:49:36PM -0500, Hannigan, Martin wrote:
It's not offline. Our NOC inadvertently shut it off after I had asked them to simply place a cap on the amount of capacity it would push. That was fixed this morning. I would've seen it sooner myself, but I'm in Senegal at the moment.
So this had nothing to do with USI then?
I am still pretty confused.
I have 2 reasons right now for outage:
USI has a proposal into Akamai regarding<something>
The amount of transit necessary for cache fill is more the USI was told and is causing some congestion for them.
It shouldn't be anymore as I had the cap "fixed" and adjusted to the expected amount. If it's still an issue, I haven't heard it so let me know. [ clip ]
If that's the case, what is the cap?
There isn't a cap but there was congestion.
There is a cap on my side at the moment and it's 1.25 Gb/s. With respect to settled numbers, we're settled. Our system reacts to changes typically within 48 hours. We usually see more use when things get more local for a variety of reasons. Best, -M< ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Thanks, Dave! -- Mike Horwath ipHouse - Welcome home! drechsau@iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Apologies for the confusion. Been quite busy the last few days and it was not immediately clear to my subconscious that the list was not already familiar with the issue. As already explained well by Dave, nobody probably would have even noticed it if the capacity was just reduced as opposed to completely disabled. On Thu, 2011-10-27 at 15:28 -0500, Mike Horwath wrote:
Thanks, Dave!
######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (8)
-
David Farmer -
Frank Bulk -
Hannigan, Martin -
Jay Hanke -
Jeremy Lumby -
Justin Krejci -
Mike Horwath -
Richard Laager