- mice-discuss - lists.micemn.net

Re: Cacti Graph using SNMPv1?

by Steve Howard

On 05/25/2011 02:15 PM, Mike Horwath wrote: > On 5/25/11 11:39 AM, "Andrew Hoyos" <hoyosa(a)GMAIL.COM> wrote: > >> On May 25, 2011, at 11:33 AM, David Farmer wrote: >> >>> On 5/25/11 11:07 CDT, Jay Hanke wrote: >>>>> Did you check that the Data Source for the graph is also using the >>>>> 64-bit counters? >>>>> >>>>>> Nothing special in logs either showing a problem. >>>>>> >>>>>> Steve - your own monitoring showing problems at all with data >>>> Are you looking at Gi-0/2? >>> it is possible this is a problem on the Cisco side too, that port >>> Gi-0/2 on the 3560E got changed to being in a port channel sometime this >>> creates weird issues. They usually go away after a reboot. >> Yeah, I've seen this same thing on a few occasions when adding/removing >> interfaces from port channels. >> >> Also, double check you've got "snmp-server ifindex persist" set in the >> config, as you might be polling something else too. > Shit, that's a great point! > > We aways have that set. > > Steve, can you check that? > > Could very well be a different port now :( > > But once you set it to persist (if it wasn't set), I can look into things > further. > It was not there. It is now. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Mike Horwath

On 5/25/11 11:39 AM, "Andrew Hoyos" <hoyosa(a)GMAIL.COM> wrote: >On May 25, 2011, at 11:33 AM, David Farmer wrote: > >> On 5/25/11 11:07 CDT, Jay Hanke wrote: >>>> Did you check that the Data Source for the graph is also using the >>>> 64-bit counters? >>>> >>>>> >>>>> Nothing special in logs either showing a problem. >>>>> >>>>> Steve - your own monitoring showing problems at all with data >>> >>> Are you looking at Gi-0/2? >> >> it is possible this is a problem on the Cisco side too, that port >>Gi-0/2 on the 3560E got changed to being in a port channel sometime this >>creates weird issues. They usually go away after a reboot. > >Yeah, I've seen this same thing on a few occasions when adding/removing >interfaces from port channels. > >Also, double check you've got "snmp-server ifindex persist" set in the >config, as you might be polling something else too. Shit, that's a great point! We aways have that set. Steve, can you check that? Could very well be a different port now :( But once you set it to persist (if it wasn't set), I can look into things further. -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Mike Horwath

On 5/25/11 11:01 AM, "Justin Krejci" <jkrejci(a)USINTERNET.COM> wrote: >On Wed, 2011-05-25 at 15:53 +0000, Mike Horwath wrote: >> On 5/24/11 6:28 PM, "Justin Krejci" <jkrejci(a)USINTERNET.COM> wrote: >> >> > >> >'micemn-gateway-2.micemn.net - Traffic - Gi0/2' in Cacti is exhibiting >> >signs of using SNMPv1 counters (frequent gaps at higher bandwidth >>usage) >> > >> >Is this a known issue? If not I presume someone at iphouse can fix this >> >easily. >> > >> >The "Fix64bit" plugin for Cacti is great for correcting this condition >>on >> >the fly if my assumption on the problem is correct. >> >> Your assumption is wrong unfortunately. >> >> I wish I had made a mistake, but stats.micemn.net's Cacti config is >>64bit >> for gathering of stuff. >> >> And using SNMPv2 according to the configuration. > >Did you check that the Data Source for the graph is also using the >64-bit counters? Verified! -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: New Member's Configured on BIRD

by Mike Horwath

On 5/25/11 11:46 AM, "James Montz" <jamesmontz(a)MANKATONETWORKS.NET> wrote: >Added IPv4 and IPv6 Configuration for City of Windomw (ASN: 46692). >Welcome! >Also noticed that IPv6 config for UMN (ASN: 57) was not in, so added that >as well. > >Maybe its a matter of semantics, but the website lists Northern Lights >GigaPoP as ASN: 57, NOT UMN. Then fix0r! You has the power! -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

New Member's Configured on BIRD

by James Montz

Added IPv4 and IPv6 Configuration for City of Windomw (ASN: 46692). Welcome! Also noticed that IPv6 config for UMN (ASN: 57) was not in, so added that as well. Maybe its a matter of semantics, but the website lists Northern Lights GigaPoP as ASN: 57, NOT UMN. --------------------------------------------------- *James C. Montz* Senior Consultant *Mankato Networks LLC* <http://www.mankatonetworks.net/>320 East Liberty Street <http://maps.google.com/maps?q=320+East+Liberty+Street%2CMankato%2CMN+56001&…>Mankato, MN 56001 *Phone:* *(320)-496-6689* *Email: jamesmontz(a)mankatonetworks.net* *Web: http://www.mankatonetworks.net Skype: james.montz* * http://www.linkedin.com/in/jmontz* ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Andrew Hoyos

On May 25, 2011, at 11:33 AM, David Farmer wrote: > On 5/25/11 11:07 CDT, Jay Hanke wrote: >>> Did you check that the Data Source for the graph is also using the >>> 64-bit counters? >>> >>>> >>>> Nothing special in logs either showing a problem. >>>> >>>> Steve - your own monitoring showing problems at all with data >> >> Are you looking at Gi-0/2? > > it is possible this is a problem on the Cisco side too, that port Gi-0/2 on the 3560E got changed to being in a port channel sometime this creates weird issues. They usually go away after a reboot. Yeah, I've seen this same thing on a few occasions when adding/removing interfaces from port channels. Also, double check you've got "snmp-server ifindex persist" set in the config, as you might be polling something else too. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by David Farmer

On 5/25/11 11:07 CDT, Jay Hanke wrote: >> Did you check that the Data Source for the graph is also using the >> 64-bit counters? >> >>> >>> Nothing special in logs either showing a problem. >>> >>> Steve - your own monitoring showing problems at all with data > > Are you looking at Gi-0/2? it is possible this is a problem on the Cisco side too, that port Gi-0/2 on the 3560E got changed to being in a port channel sometime this creates weird issues. They usually go away after a reboot. > Usually SNMPv1 issues will appear at right about 100 Mb/s on a five > minute average. We're pulling the po1 interface well over a Gig > without an issue. Also, the traffic will show up with spike and then > nothing (when the counter rolls over). > > Jay > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Steve Howard

On 05/25/2011 10:53 AM, Mike Horwath wrote: > Steve - your own monitoring showing problems at all with data I haven't got around to setting up the monitoring. Yours has worked great so I didn't bother (yet). Steve ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Owen DeLong

On May 25, 2011, at 9:01 AM, Justin Krejci wrote: > On Wed, 2011-05-25 at 15:53 +0000, Mike Horwath wrote: >> On 5/24/11 6:28 PM, "Justin Krejci" <jkrejci(a)USINTERNET.COM> wrote: >> >>> >>> 'micemn-gateway-2.micemn.net - Traffic - Gi0/2' in Cacti is exhibiting >>> signs of using SNMPv1 counters (frequent gaps at higher bandwidth usage) >>> >>> Is this a known issue? If not I presume someone at iphouse can fix this >>> easily. >>> >>> The "Fix64bit" plugin for Cacti is great for correcting this condition on >>> the fly if my assumption on the problem is correct. >> >> Your assumption is wrong unfortunately. >> >> I wish I had made a mistake, but stats.micemn.net's Cacti config is 64bit >> for gathering of stuff. >> >> And using SNMPv2 according to the configuration. > > Did you check that the Data Source for the graph is also using the > 64-bit counters? > This would be the question of ifInOctets (32 bit) vs. ifInHCOctets (64 bit) IIRC. In case the reference is helpful. Owen >> >> Nothing special in logs either showing a problem. >> >> Steve - your own monitoring showing problems at all with data? >> > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Jay Hanke

> Did you check that the Data Source for the graph is also using the > 64-bit counters? > >> >> Nothing special in logs either showing a problem. >> >> Steve - your own monitoring showing problems at all with data Are you looking at Gi-0/2? Usually SNMPv1 issues will appear at right about 100 Mb/s on a five minute average. We're pulling the po1 interface well over a Gig without an issue. Also, the traffic will show up with spike and then nothing (when the counter rolls over). Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Justin Krejci

On Wed, 2011-05-25 at 15:53 +0000, Mike Horwath wrote: > On 5/24/11 6:28 PM, "Justin Krejci" <jkrejci(a)USINTERNET.COM> wrote: > > > > >'micemn-gateway-2.micemn.net - Traffic - Gi0/2' in Cacti is exhibiting > >signs of using SNMPv1 counters (frequent gaps at higher bandwidth usage) > > > >Is this a known issue? If not I presume someone at iphouse can fix this > >easily. > > > >The "Fix64bit" plugin for Cacti is great for correcting this condition on > >the fly if my assumption on the problem is correct. > > Your assumption is wrong unfortunately. > > I wish I had made a mistake, but stats.micemn.net's Cacti config is 64bit > for gathering of stuff. > > And using SNMPv2 according to the configuration. Did you check that the Data Source for the graph is also using the 64-bit counters? > > Nothing special in logs either showing a problem. > > Steve - your own monitoring showing problems at all with data? > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: Cacti Graph using SNMPv1?

by Mike Horwath

On 5/24/11 6:28 PM, "Justin Krejci" <jkrejci(a)USINTERNET.COM> wrote: > >'micemn-gateway-2.micemn.net - Traffic - Gi0/2' in Cacti is exhibiting >signs of using SNMPv1 counters (frequent gaps at higher bandwidth usage) > >Is this a known issue? If not I presume someone at iphouse can fix this >easily. > >The "Fix64bit" plugin for Cacti is great for correcting this condition on >the fly if my assumption on the problem is correct. Your assumption is wrong unfortunately. I wish I had made a mistake, but stats.micemn.net's Cacti config is 64bit for gathering of stuff. And using SNMPv2 according to the configuration. Nothing special in logs either showing a problem. Steve - your own monitoring showing problems at all with data? -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Re: NANOG

by Jay Hanke

>> On Mon, May 23, 2011 at 4:13 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: >>> Is anyone going to NANOG in Denver? >> >> i will be. is there interest in a twin cities networking geeks dinner? > > I'm on the fence. Combo WI/MN networking geeks dinner, that might do the trick? :) I'm in. Let's make it happen! ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 25, 2011

Cacti Graph using SNMPv1?

by Justin Krejci

'micemn-gateway-2.micemn.net - Traffic - Gi0/2' in Cacti is exhibiting signs of using SNMPv1 counters (frequent gaps at higher bandwidth usage) Is this a known issue? If not I presume someone at iphouse can fix this easily. The "Fix64bit" plugin for Cacti is great for correcting this condition on the fly if my assumption on the problem is correct. ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 24, 2011

Re: NANOG

by Andrew Hoyos

On May 23, 2011, at 4:18 PM, steve ulrich wrote: > On Mon, May 23, 2011 at 4:13 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: >> Is anyone going to NANOG in Denver? > > i will be. is there interest in a twin cities networking geeks dinner? I'm on the fence. Combo WI/MN networking geeks dinner, that might do the trick? :) ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 24, 2011

Re: NANOG

by steve ulrich

On Mon, May 23, 2011 at 4:13 PM, Jay Hanke <jayhanke(a)mankatonetworks.net> wrote: > Is anyone going to NANOG in Denver? i will be. is there interest in a twin cities networking geeks dinner? -- steve ulrich (sulrich(a)botwerks.*) ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 23, 2011

NANOG

by Jay Hanke

Is anyone going to NANOG in Denver? Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 23, 2011

Re: Akamai Traffic on MICE

by David Farmer

I was planing to make that part of the discussion with Akamai. However, even without MICE we were going to need more servers for next fall. It seems our servers were shedding demand much of the spring. Since school is out, our demand has dropped by about 50% over the last 2 weeks, and is down about 65% from our Spring peak about 6 weeks ago. On 5/20/11 10:23 CDT, Owen DeLong wrote: > Is there any way we can use the amount of U/Akamai traffic being pointed > at MICE as an incentive for Akamai to come to MICE themselves? > > Owen > > On May 20, 2011, at 8:12 AM, David Farmer wrote: > >> As was noted we installed a 2nd GigE between the two switches yesterday. Things are looked good last night we peaked at 1.7Gbps most of that coming from the U's Akamai Stack. Akamai tells me we are maxing out our servers again, it looks like we were maxing them out most of the spring anyway. >> >> The Akamai servers are putting out a little less than 2G from about 7am to 11pm. >> I believe we are seeing more traffic in the evening going to MICE because most of the other Gigapop networks have more of a business day demand curve, so Akamai serves more of the demand out of MICE in the evening as demand from the Gigapop networks slack off. >> >> So I think things are much better now, we should get the new switches installed and people moved over but we shouldn't have to rush it. The University will work with Akamai to get more servers over the summer, but I would hope we would have everyone moved over to the juniper switches long before that happens. >> >> On 5/19/11 09:56 CDT, David Farmer wrote: >>> It looks like Akamai has found a sweet spot they tend to do that, the >>> link between the switches peaked up to about 850 and backed off and is >>> hovering abut 650Mbps for the last hour or so. From what I've seen in >>> the past for Amakai it will probably do that the rest of the day. >>> >>> On 5/19/11 07:23 CDT, David Farmer wrote: >>>> The 10G Akamai stack at the University is having an affect. >>>> >>>> The University turned up routing from the GigaPOP to MICE on its 10G >>>> link Tuesday around 8pm. The new paths were also announced to the Akamai >>>> route collector around that time too. Akamai started using these new >>>> paths around 1:30 in the afternoon on Wednesday. The University's 10G >>>> link peaked a little over 1G around 9:30pm last night. The bottleneck is >>>> the 1G link between the CNS 10G switch and the 4500. It was saturated >>>> from about 6:30pm to 9:30pm at about 950Mbps. >>>> >>>> Some of us are organizing to rectify the situation ASAP. There are some >>>> workarounds that could help in the very short-term, yet this morning, >>>> and we are looking to get the new Juniper switches with 10G up-links >>>> installed ASAP However, the earliest that can happen it this afternoon, >>>> as the switches are down in Mankato with Jay. There is a plan to get >>>> them up to 511 by mid-afternoon. >>>> >>>> Also I'll be watching the links and can modify our announcement to the >>>> Akamai route collector if necessary. Things are going to be fluid today. >>>> >>>> I'll just say that this is the kind of problem I like to see, if you are >>>> going to have to deal with any problem. That is the problem of over >>>> succeeding. >> >> -- >> =============================================== >> David Farmer Email:farmer@umn.edu >> Networking& Telecommunication Services >> Office of Information Technology >> University of Minnesota >> 2218 University Ave SE Phone: 612-626-0815 >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 >> =============================================== >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 20, 2011

Re: Akamai Traffic on MICE

by Owen DeLong

Is there any way we can use the amount of U/Akamai traffic being pointed at MICE as an incentive for Akamai to come to MICE themselves? Owen On May 20, 2011, at 8:12 AM, David Farmer wrote: > As was noted we installed a 2nd GigE between the two switches yesterday. Things are looked good last night we peaked at 1.7Gbps most of that coming from the U's Akamai Stack. Akamai tells me we are maxing out our servers again, it looks like we were maxing them out most of the spring anyway. > > The Akamai servers are putting out a little less than 2G from about 7am to 11pm. > I believe we are seeing more traffic in the evening going to MICE because most of the other Gigapop networks have more of a business day demand curve, so Akamai serves more of the demand out of MICE in the evening as demand from the Gigapop networks slack off. > > So I think things are much better now, we should get the new switches installed and people moved over but we shouldn't have to rush it. The University will work with Akamai to get more servers over the summer, but I would hope we would have everyone moved over to the juniper switches long before that happens. > > On 5/19/11 09:56 CDT, David Farmer wrote: >> It looks like Akamai has found a sweet spot they tend to do that, the >> link between the switches peaked up to about 850 and backed off and is >> hovering abut 650Mbps for the last hour or so. From what I've seen in >> the past for Amakai it will probably do that the rest of the day. >> >> On 5/19/11 07:23 CDT, David Farmer wrote: >>> The 10G Akamai stack at the University is having an affect. >>> >>> The University turned up routing from the GigaPOP to MICE on its 10G >>> link Tuesday around 8pm. The new paths were also announced to the Akamai >>> route collector around that time too. Akamai started using these new >>> paths around 1:30 in the afternoon on Wednesday. The University's 10G >>> link peaked a little over 1G around 9:30pm last night. The bottleneck is >>> the 1G link between the CNS 10G switch and the 4500. It was saturated >>> from about 6:30pm to 9:30pm at about 950Mbps. >>> >>> Some of us are organizing to rectify the situation ASAP. There are some >>> workarounds that could help in the very short-term, yet this morning, >>> and we are looking to get the new Juniper switches with 10G up-links >>> installed ASAP However, the earliest that can happen it this afternoon, >>> as the switches are down in Mankato with Jay. There is a plan to get >>> them up to 511 by mid-afternoon. >>> >>> Also I'll be watching the links and can modify our announcement to the >>> Akamai route collector if necessary. Things are going to be fluid today. >>> >>> I'll just say that this is the kind of problem I like to see, if you are >>> going to have to deal with any problem. That is the problem of over >>> succeeding. > > -- > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 20, 2011

Re: Akamai Traffic on MICE

by David Farmer

As was noted we installed a 2nd GigE between the two switches yesterday. Things are looked good last night we peaked at 1.7Gbps most of that coming from the U's Akamai Stack. Akamai tells me we are maxing out our servers again, it looks like we were maxing them out most of the spring anyway. The Akamai servers are putting out a little less than 2G from about 7am to 11pm. I believe we are seeing more traffic in the evening going to MICE because most of the other Gigapop networks have more of a business day demand curve, so Akamai serves more of the demand out of MICE in the evening as demand from the Gigapop networks slack off. So I think things are much better now, we should get the new switches installed and people moved over but we shouldn't have to rush it. The University will work with Akamai to get more servers over the summer, but I would hope we would have everyone moved over to the juniper switches long before that happens. On 5/19/11 09:56 CDT, David Farmer wrote: > It looks like Akamai has found a sweet spot they tend to do that, the > link between the switches peaked up to about 850 and backed off and is > hovering abut 650Mbps for the last hour or so. From what I've seen in > the past for Amakai it will probably do that the rest of the day. > > On 5/19/11 07:23 CDT, David Farmer wrote: >> The 10G Akamai stack at the University is having an affect. >> >> The University turned up routing from the GigaPOP to MICE on its 10G >> link Tuesday around 8pm. The new paths were also announced to the Akamai >> route collector around that time too. Akamai started using these new >> paths around 1:30 in the afternoon on Wednesday. The University's 10G >> link peaked a little over 1G around 9:30pm last night. The bottleneck is >> the 1G link between the CNS 10G switch and the 4500. It was saturated >> from about 6:30pm to 9:30pm at about 950Mbps. >> >> Some of us are organizing to rectify the situation ASAP. There are some >> workarounds that could help in the very short-term, yet this morning, >> and we are looking to get the new Juniper switches with 10G up-links >> installed ASAP However, the earliest that can happen it this afternoon, >> as the switches are down in Mankato with Jay. There is a plan to get >> them up to 511 by mid-afternoon. >> >> Also I'll be watching the links and can modify our announcement to the >> Akamai route collector if necessary. Things are going to be fluid today. >> >> I'll just say that this is the kind of problem I like to see, if you are >> going to have to deal with any problem. That is the problem of over >> succeeding. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 20, 2011

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 01:55:20PM -0500, Andrew Hoyos wrote: > On May 19, 2011, at 1:43 PM, Doug McIntyre wrote: > > > It looks like the traffic load balancing is a little uneven, but Dave > > and I will keep watch on it. > > I think the Cisco default in a PortChannel is to load balance by src-mac, which might have much help in the current situation, since most of the traffic is coming/going to one mac addr. > > You might want to try changing it to src/dst IP for a better distribution of flows across both links, or even per flow, if the box supports it. The default for the 4500/SupIV is for src-dst-ip, and Dave changed the 3560E to do this as well when we brought up the port-channel. Its converging to something pretty close now Gig2/45 5 minute input rate 325460000 bits/sec, 35790 packets/sec Gig1/2 5 minute input rate 337375000 bits/sec, 36799 packets/sec -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Andrew Hoyos

On May 19, 2011, at 1:43 PM, Doug McIntyre wrote: > It looks like the traffic load balancing is a little uneven, but Dave > and I will keep watch on it. I think the Cisco default in a PortChannel is to load balance by src-mac, which might have much help in the current situation, since most of the traffic is coming/going to one mac addr. You might want to try changing it to src/dst IP for a better distribution of flows across both links, or even per flow, if the box supports it. IIRC, "port-channel load-balance ?" should show you what the capabilities are. Thanks, Andrew ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Doug McIntyre

On Thu, May 19, 2011 at 10:07:37AM -0500, Mike Hemphill wrote: > It looks like we (Dave Farmer & FWR) will be putting up a second Gig link > over copper between the CNS & core MICE sw later this AM. Dave and I have completed the ether channel to bring it up to a 2Gbps cross link between the two switches.. MICE-gig-sw-1#show etherchannel port-channel Channel-group listing: ---------------------- Group: 1 ---------- Port-channels in the group: --------------------------- Port-channel: Po1 ------------ Age of the Port-channel = 0d:03h:38m:32s Logical slot/port = 11/1 Number of ports = 2 GC = 0x00000000 Port state = Port-channel Ag-Inuse Protocol = - Port security = Disabled Ports in the Port-channel: Index Load Port EC state No of bits ------+------+------+------------------+----------- 1 00 Gi1/2 On 0 0 00 Gi2/45 On 0 Time since last port bundled: 0d:00h:12m:09s Gi1/2 Time since last port Un-bundled: 0d:02h:19m:56s Gi2/45 It looks like the traffic load balancing is a little uneven, but Dave and I will keep watch on it. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Traffic between 10Gig & 1Gig Switches

by James Montz

Switches are now @ 511 in FWR Suite. I did not get a chance to check the SFP. I will be back @ 511 to do install/config around 4pm if noone else gets to it before then. On May 19, 2011 12:20 AM, "Owen DeLong" <owend(a)he.net> wrote: > On May 18, 2011, at 2:52 PM, Steve Howard wrote: > >> The traffic to/from the U shot up today. I'm guessing that is CDN related? >> >> The link between the CNS switch and the MICE switch shot up to ~850Mbps >> today. Too bad that is only a single gig connection! >> >> It is great to have the traffic on the exchange, but we should probably >> deal with this nearly saturated link fast. Here are a few possible >> solutions: >> >> 1) Add another Gig connection. We'd probably need to pay FWR for a >> cross-connect (CNS is paying for the first one) with this option; >> >> 2) Use one of the new Juniper switches with a 10Gig interface and >> connect that to a 10Gig interface on the existing MICE 10Gig switch, >> then move some of the 1Gig members over to those ports. This would >> require purchase of 10Gig optics for the Juniper and the existing 10Gig >> MICE switch; >> >> 3) Move some of the higher bandwidth MICE members to the 1Gig ports on >> the 10Gig switch. Would this increase the FWR cross-connect charges >> for those members if they are going from copper to fiber? >> >> 4) Encourage MICE members on the 1Gig switch to upgrade to a 10Gig >> connection; >> >> 5) Ask the U to not advertise some of the members on the 1Gig switch to >> CDNs (if that is the cause of the traffic load). This is a bummer if >> you are one of the networks that gets left out. I really don't like >> this option -- we are finally getting real traffic on the exchange and >> then we'd be cutting back. >> >> What does everybody think? Any other suggestions? >> > > To the extent that my input is useful, I encourage option 2. > > I believe this is the more scalable longer-term solution and does not represent > a significantly higher cost than any of the others will in relatively short order. > > Owen > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011

Re: Akamai Traffic on MICE

by Mike Hemphill

Steve, Happy to accept the shipment & install, if that is the way that it works out. It looks like we (Dave Farmer & FWR) will be putting up a second Gig link over copper between the CNS & core MICE sw later this AM. Mike Mike Hemphill FWR Communication Networks VP Sales 612-333-1922 voice 612-812-5242 cell 612-333-1945 fax mhemphill(a)mngateway.com Visit us at: www.mngateway.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Steve Howard Sent: Thursday, May 19, 2011 9:42 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE On 05/19/2011 07:58 AM, Mitch Elfstrand wrote: > SDN is willing to move the connection to alleviate the issue. Please let us know if we need to send someone to 511 to help with anything. > That sounds like this might be an ideal short-term fix. Feel free to plug into Gig 0/2 on the CNS switch. We didn't leave any optics down there, you will need to provide a Cisco (or compatible) SFP. Alternatively, you could consider doing a 10Gig connection. Your peak was almost 600Mbps last night! If you want to do 10Gig, you'll need X2 optics for the CNS switch. Let me know and I'll assign a 10Gig port. CNS will donate use of 10Gig LR optics to MICE for use in the CNS switch to connect to the Juniper switch. Unfortunately, those optics are in Bemidji. I will check to see if anybody here is going to the Cities today, if so I will send the optics with them, if not, I will overnight them to Mike Hemphill at FWR. (Mike, I hope that is OK with you). Thanks, Steve > Regards, > > Mitch Elfstrand | Director of Network Engineering > mitch.elfstrand(a)sdncommunications.com | 2900 W 10th St. | Sioux Falls, SD 57104 USA > Direct +1.605.978.3587 | Mobile +1.605.212.3392 | TF +1.800.247.1442 | Fax +1.605.334.4782 > > > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Thursday, May 19, 2011 7:54 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Akamai Traffic on MICE > > On Thu, May 19, 2011 at 07:23:48AM -0500, David Farmer wrote: >> Some of us are organizing to rectify the situation ASAP. There are >> some workarounds that could help in the very short-term... > Specificly one thing that could be done is if SDN could be moved from > the 4500 onto the 3560, and FWR could string another cross-connect > temporarily between the 3560 and the other fiber "uplink" port on the > 4500, we could change those two gig ports into an Etherchannel. > > This would move the biggest destination of the Akamai traffic closer > to the source, as well as preventing one endpoint from swamping the > interconnect.. > > Even if SDN could be moved to the 3560 would do a lot to alleviate the issue. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

May 19, 2011