- mice-discuss - lists.micemn.net

Re: SNMP Measurements for MICE?

by Anthony Anderberg

> > In light of that, is this still a problem? > > > It wasn't really a problem more than a curiosity. Even if it was polling at 1m, > as long as the statistics collection method is noted clearly and publicly it's > shouldn't be a big deal. The hourly graph is just some default view that got turned on and is being exported with everything else, if the group thinks it would somehow be more clear to remove it we could do that - as several have noted its increments are (correctly) 5 minutes. In other graph news, I think we've got the Peers/Prefixs graphs displaying correctly now and I think they look pretty good - thanks for people's help troubleshooting that. They're under the "Route Server" portion of the graph tree. Cheers, anthony

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Hannigan, Martin

> On Aug 12, 2015, at 2:42 PM, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > On 08/12/2015 01:39 PM, Hannigan, Martin wrote: >> You can see here that the reporting claims it’s 1M >> >> http://micelg.usinternet.com/export/graph_385.html > > Even if it was polling at 1 minute (which from other messages in this thread, we seem to think it isn't), that's just for the hourly graph. The daily graph is unambiguously claiming 5 minute averages. > > In light of that, is this still a problem? > It wasn’t really a problem more than a curiosity. Even if it was polling at 1m, as long as the statistics collection method is noted clearly and publicly it’s shouldn’t be a big deal. Best, -M<

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Richard Laager

On 08/12/2015 01:39 PM, Hannigan, Martin wrote: > You can see here that the reporting claims it’s 1M > > http://micelg.usinternet.com/export/graph_385.html Even if it was polling at 1 minute (which from other messages in this thread, we seem to think it isn't), that's just for the hourly graph. The daily graph is unambiguously claiming 5 minute averages. In light of that, is this still a problem? -- Richard

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Hannigan, Martin

You can see here that the reporting claims it’s 1M http://micelg.usinternet.com/export/graph_385.html Best, -M< On Aug 12, 2015, at 2:36 PM, Justin Krejci <JKrejci(a)USINTERNET.COM<mailto:JKrejci@USINTERNET.COM>> wrote: The cacti server is polling on 5 minute interval. ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] on behalf of Hannigan, Martin [marty(a)AKAMAI.COM<mailto:marty@AKAMAI.COM>] Sent: Wednesday, August 12, 2015 1:10 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] SNMP Measurements for MICE? My point was about the polling interval. Thanks! -M< On Aug 12, 2015, at 2:05 PM, Matthew Beckwell <matthewb(a)AITECH.NET<mailto:matthewb@AITECH.NET>> wrote: If we look closely at the 1 minute graph-- the steps are actually in 5 minute chunks. (It's even more apparent if you log into the Cacti site and zoom in). I'm pretty sure it's only polling every 5 minutes, but the 1 minute Cacti template is enabled and being exported. A person could probably pretend the top 1 minute graph doesn't exist, since the 1 minute data appears to be interpolated. I'm not sure I understand the 95th (percentile) reference, though? ~Matthew Matthew Beckwell - V.P. - Network Operations Advanced Integrated Technologies Office: +1.952.829.5511 x204 E-Mail: matthewb(a)aitech.net<mailto:matthewb@aitech.net> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com<mailto:marty@akamai.com>> wrote: All, Isn’t 95th @ 1m unusual and non-standard? http://micelg.usinternet.com/export/graph_385.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__micelg.usinternet.com_e…> Best, -M< ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Justin Krejci

The cacti server is polling on 5 minute interval. ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Hannigan, Martin [marty(a)AKAMAI.COM] Sent: Wednesday, August 12, 2015 1:10 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] SNMP Measurements for MICE? My point was about the polling interval. Thanks! -M< On Aug 12, 2015, at 2:05 PM, Matthew Beckwell <matthewb(a)AITECH.NET<mailto:matthewb@AITECH.NET>> wrote: If we look closely at the 1 minute graph-- the steps are actually in 5 minute chunks. (It's even more apparent if you log into the Cacti site and zoom in). I'm pretty sure it's only polling every 5 minutes, but the 1 minute Cacti template is enabled and being exported. A person could probably pretend the top 1 minute graph doesn't exist, since the 1 minute data appears to be interpolated. I'm not sure I understand the 95th (percentile) reference, though? ~Matthew Matthew Beckwell - V.P. - Network Operations Advanced Integrated Technologies Office: +1.952.829.5511 x204 E-Mail: matthewb(a)aitech.net<mailto:matthewb@aitech.net> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com<mailto:marty@akamai.com>> wrote: All, Isn’t 95th @ 1m unusual and non-standard? http://micelg.usinternet.com/export/graph_385.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__micelg.usinternet.com_e…> Best, -M< ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Hannigan, Martin

My point was about the polling interval. Thanks! -M< On Aug 12, 2015, at 2:05 PM, Matthew Beckwell <matthewb(a)AITECH.NET<mailto:matthewb@AITECH.NET>> wrote: If we look closely at the 1 minute graph-- the steps are actually in 5 minute chunks. (It's even more apparent if you log into the Cacti site and zoom in). I'm pretty sure it's only polling every 5 minutes, but the 1 minute Cacti template is enabled and being exported. A person could probably pretend the top 1 minute graph doesn't exist, since the 1 minute data appears to be interpolated. I'm not sure I understand the 95th (percentile) reference, though? ~Matthew Matthew Beckwell - V.P. - Network Operations Advanced Integrated Technologies Office: +1.952.829.5511 x204 E-Mail: matthewb(a)aitech.net<mailto:matthewb@aitech.net> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com<mailto:marty@akamai.com>> wrote: All, Isn’t 95th @ 1m unusual and non-standard? http://micelg.usinternet.com/export/graph_385.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__micelg.usinternet.com_e…> Best, -M< ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Matthew Beckwell

If we look closely at the 1 minute graph-- the steps are actually in 5 minute chunks. (It's even more apparent if you log into the Cacti site and zoom in). I'm pretty sure it's only polling every 5 minutes, but the 1 minute Cacti template is enabled and being exported. A person could probably pretend the top 1 minute graph doesn't exist, since the 1 minute data appears to be interpolated. I'm not sure I understand the 95th (percentile) reference, though? ~Matthew *Matthew Beckwell* - V.P. - Network Operations Advanced Integrated Technologies Office: +1.952.829.5511 x204 E-Mail: matthewb(a)aitech.net On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com> wrote: > > All, > > Isn’t 95th @ 1m unusual and non-standard? > > http://micelg.usinternet.com/export/graph_385.html > > Best, > > -M< > > > > > >

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Hannigan, Martin

Sure, but why does that matter with respect to the IXP fabric? It’s misleading to some extent, along the lines you’ve explained. The idea behind IXP statistics is about volume. Networks typically use it as a gauge of value and especially when there are two IXPs competing in a market. The 5m sampling benchmark is also not a de-facto standard, it’s a fully fledged standard. It is nice that the graph is clear about the sample rate though so not a complaint, pointing out an oddity as contrasted to 99% of the IXP world. There’s really no need for the MICE to be doing 1m IMHO. Best, -M< > On Aug 12, 2015, at 1:47 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Depends on how you look at it. More sampling shows more extremes. If you sample at 1 minute, and it were to shoot up to 12G for a minute or two, and then drop below 1G for a minute or two, you would see both the 12G high, and the 1G low. If you look at the same traffic on a five min sample rate, you would see it at about 6G, and you would not see the 1G low, or the 12G high. Personally I like higher sample rates since if you are sizing a connection to the exchange, and you do not want to have any loss, in the above example you would have loss if you only purchased a 10G circuit, and if you were looking only at 5 minute sample rates, you would have no clue why you were getting intermittent packet loss. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Hannigan, Martin > Sent: Wednesday, August 12, 2015 12:24 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] SNMP Measurements for MICE? > > Doesn't more sampling make it appear as if there is more traffic? The de facto std for IXPs is 5m polling. > > > >> On Aug 12, 2015, at 12:48, Anton Kapela <tkapela(a)GMAIL.COM> wrote: >> >> a secant line is a secant line, be it a ray/segment/line/etc between >> points spaced 1 second, ten seconds, or ten minutes apart, no? >> >> (sorting line segments by their n-tile magnitude, be it at 95% or some >> other interval, seems decoupled from the measurement/secant interval) >> >> if my graph theory is bad, let me know :) >> >> -Tk >> >>> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com> wrote: >>> >>> All, >>> >>> Isn’t 95th @ 1m unusual and non-standard? >>> >>> http://micelg.usinternet.com/export/graph_385.html >>> >>> Best, >>> >>> -M< >>> >>> >>> >>> >>>

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Jeremy Lumby

Depends on how you look at it. More sampling shows more extremes. If you sample at 1 minute, and it were to shoot up to 12G for a minute or two, and then drop below 1G for a minute or two, you would see both the 12G high, and the 1G low. If you look at the same traffic on a five min sample rate, you would see it at about 6G, and you would not see the 1G low, or the 12G high. Personally I like higher sample rates since if you are sizing a connection to the exchange, and you do not want to have any loss, in the above example you would have loss if you only purchased a 10G circuit, and if you were looking only at 5 minute sample rates, you would have no clue why you were getting intermittent packet loss. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Hannigan, Martin Sent: Wednesday, August 12, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] SNMP Measurements for MICE? Doesn't more sampling make it appear as if there is more traffic? The de facto std for IXPs is 5m polling. > On Aug 12, 2015, at 12:48, Anton Kapela <tkapela(a)GMAIL.COM> wrote: > > a secant line is a secant line, be it a ray/segment/line/etc between > points spaced 1 second, ten seconds, or ten minutes apart, no? > > (sorting line segments by their n-tile magnitude, be it at 95% or some > other interval, seems decoupled from the measurement/secant interval) > > if my graph theory is bad, let me know :) > > -Tk > >> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com> wrote: >> >> All, >> >> Isn’t 95th @ 1m unusual and non-standard? >> >> http://micelg.usinternet.com/export/graph_385.html >> >> Best, >> >> -M< >> >> >> >> >>

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Hannigan, Martin

Doesn't more sampling make it appear as if there is more traffic? The de facto std for IXPs is 5m polling. > On Aug 12, 2015, at 12:48, Anton Kapela <tkapela(a)GMAIL.COM> wrote: > > a secant line is a secant line, be it a ray/segment/line/etc between > points spaced 1 second, ten seconds, or ten minutes apart, no? > > (sorting line segments by their n-tile magnitude, be it at 95% or some > other interval, seems decoupled from the measurement/secant interval) > > if my graph theory is bad, let me know :) > > -Tk > >> On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com> wrote: >> >> All, >> >> Isn’t 95th @ 1m unusual and non-standard? >> >> http://micelg.usinternet.com/export/graph_385.html >> >> Best, >> >> -M< >> >> >> >> >>

Aug. 12, 2015

Re: SNMP Measurements for MICE?

by Anton Kapela

a secant line is a secant line, be it a ray/segment/line/etc between points spaced 1 second, ten seconds, or ten minutes apart, no? (sorting line segments by their n-tile magnitude, be it at 95% or some other interval, seems decoupled from the measurement/secant interval) if my graph theory is bad, let me know :) -Tk On Wed, Aug 12, 2015 at 9:11 AM, Hannigan, Martin <marty(a)akamai.com> wrote: > > All, > > Isn’t 95th @ 1m unusual and non-standard? > > http://micelg.usinternet.com/export/graph_385.html > > Best, > > -M< > > > > >

Aug. 12, 2015

SNMP Measurements for MICE?

by Hannigan, Martin

All, Isn’t 95th @ 1m unusual and non-standard? http://micelg.usinternet.com/export/graph_385.html Best, -M<

Aug. 12, 2015

Re: MICE Switch Upgrade

by Mike Horwath

On Wed, Jul 29, 2015 at 11:46:53AM +0000, Anthony Anderberg wrote: > This upgrade is complete, looks like routes and traffic are > returning to normal levels. Thanks, sir! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 30, 2015

Re: MICE Switch Upgrade

by Dan Olsen

Thank You for the work. Dan Olsen- Operations Manager Windomnet 443 10th Street, PO box 38 Windom MN 56101 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Wednesday, July 29, 2015 6:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Switch Upgrade This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst 320-234-5239 <mailto:anthonyanderberg@nu-telecom.net> anthonyanderberg(a)nu-telecom.net <http://www.nutelecom.net> www.nutelecom.net From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent <https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200> &id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst 320-234-5239 <mailto:anthonyanderberg@nu-telecom.net> anthonyanderberg(a)nu-telecom.net <http://www.nutelecom.net> www.nutelecom.net _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

July 29, 2015

Re: MICE Switch Upgrade

by Hannigan, Martin

Thanks for the work Anthony. Best, -M< On Jul 29, 2015, at 7:46 AM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET<mailto:anthonyanderberg@NU-TELECOM.NET>> wrote: This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder – I’m still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I’ll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I’ll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst <image001.gif> 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nutelecom.net&d=AwM…> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we’re encountering a known bug that has been fixed in later software releases. Here’s the bug description for people with Juniper support ids, as we’d suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200<https://urldefense.proofpoint.com/v2/url?u=https-3A__prsearch.juniper.net_I…> The upgrade will require a short interruption in exchange traffic. We’re currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst <image001.gif> 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nutelecom.net&d=AwM…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

July 29, 2015

Re: MICE Switch Upgrade

by Anthony Anderberg

This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 29, 2015

Re: MICE Switch Upgrade

by Anthony Anderberg

This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 29, 2015

FW: MICE Switch Upgrade

by Anthony Anderberg

As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 28, 2015

Re: Multiple members reporting no light on the main switch

by dangieser

A case is opened with Juniper and files uploaded. Andrew and Anthony are copied on the ticket. Thanks, Dan Gieser Mankato Networks Cell: 507-327-5341 Desk: 507-242-6469 dangieser(a)mankatonetworks.net ------ Original Message ------ From: "Andrew Hoyos" <hoyosa(a)gmail.com> To: MICE-DISCUSS(a)lists.iphouse.net Sent: 7/26/2015 5:29:48 PM Subject: Re: [MICE-DISCUSS] Multiple members reporting no light on the main switch >On Jul 25, 2015, at 11:04 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> >>Sure, I put them up at https://noc.iphouse.com/MICE-072415/ > >Thanks! > >>Anthony & Mankato Networks has been the ones dealing with Juniper >>support, >>I don't have contract info. > >Anthony or Mankato folks - can we get a case opened if not already? Or >I’d be happy to as well, just unicast me the J-Care Contract ID. > >Looking through the logs/RSI info, there were core dumps for the >‘eventd’ process around the same general timeframe as traffic quit >forwarding. > >-rw------- 1 root field 13046 Jul 24 19:31 >/var/tmp/eventd.core-tarball.1.tgz >-rw-rw---- 1 root field 71112 Jul 24 19:27 >/var/tmp/eventd.core.0.gz > >While I suspect the answer from Juniper will be ‘upgrade code please’, >it would be good to confirm and see if this is attributable to a known >PR, that is hopefully fixed in whatever version is planned to be >upgraded to. > >Thanks, >Andrew > > >-------------------------------------------------------------------------------- >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

July 26, 2015

Re: Multiple members reporting no light on the main switch

by Andrew Hoyos

On Jul 25, 2015, at 11:04 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > Sure, I put them up at https://noc.iphouse.com/MICE-072415/ <https://noc.iphouse.com/MICE-072415/> Thanks! > Anthony & Mankato Networks has been the ones dealing with Juniper support, > I don't have contract info. Anthony or Mankato folks - can we get a case opened if not already? Or I’d be happy to as well, just unicast me the J-Care Contract ID. Looking through the logs/RSI info, there were core dumps for the ‘eventd’ process around the same general timeframe as traffic quit forwarding. -rw------- 1 root field 13046 Jul 24 19:31 /var/tmp/eventd.core-tarball.1.tgz -rw-rw---- 1 root field 71112 Jul 24 19:27 /var/tmp/eventd.core.0.gz While I suspect the answer from Juniper will be ‘upgrade code please’, it would be good to confirm and see if this is attributable to a known PR, that is hopefully fixed in whatever version is planned to be upgraded to. Thanks, Andrew

July 26, 2015

Re: Multiple members reporting no light on the main switch

by Doug McIntyre

On Sat, Jul 25, 2015 at 09:13:42AM -0500, Andrew Hoyos wrote: > Can we toss these logs up somewhere semi-public so folks can peek at them, along with all the support info? Sure, I put them up at https://noc.iphouse.com/MICE-072415/ Anthony & Mankato Networks has been the ones dealing with Juniper support, I don't have contract info. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Andrew Hoyos

On Jul 24, 2015, at 10:52 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: >> Can someone log in and pull the alarms? > > The alarm lit right now is just "boot from backup root". > The software upgrade should clear that. or running ‘request system snapshot media internal slice alternate’ will copy the active partition to the primary partition. That might be wise to do now, as it will repair the primary partition immediately. > This is the start of the logs around the time the 4500 rebooted tonight > Lots of info (pages and pages more), probably none too relevent to any issue. > > <snip> Can we toss these logs up somewhere semi-public so folks can peek at them, along with all the support info? ‘request support information | save /var/tmp/MICE-ISSUE-072415.txt’ ‘file archive compress source MICE-ISSUE-072415.txt destination MICE-ISSUE-072415.txt.tgz’ > Probably getting up to the current recommended code base for this > hardware is the best course of action. (JunOS 12.3 has been working > very well for our Juniper EX switch stacks). I know the upgrade was being done for other reasons too, but I’d suggest opening a JTAC case so they can at least look at what happened here, before blindly upgrading without knowing the root cause. I’m pretty sure someone paid for support on these recently, right? I’d happily do so with Juniper and volunteer some time….. -- Andrew Hoyos hoyosa(a)gmail.com

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Frank Bulk

It failed at 6:41 pm -- what brought it back up around ~7:40 pm -- the Cologix tech powering it on? Or cycling power? Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, July 24, 2015 10:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multiple members reporting no light on the main switch On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: > Can someone log in and pull the alarms? The alarm lit right now is just "boot from backup root". The software upgrade should clear that. This is the start of the logs around the time the 4500 rebooted tonight Lots of info (pages and pages more), probably none too relevent to any issue. Jul 24 18:38:15 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:19 MICE-SW1 sfid[1022]: sfi_pfepeer_grace_timer expired Jul 24 18:38:30 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:45 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/26 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/27 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/24 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/25 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c0 on vcp-1.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-1.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-1 went down Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c1 on vcp-0.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-0.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-0 went down .. Probably getting up to the current recommended code base for this hardware is the best course of action. (JunOS 12.3 has been working very well for our Juniper EX switch stacks). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Doug McIntyre

On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: > Can someone log in and pull the alarms? The alarm lit right now is just "boot from backup root". The software upgrade should clear that. This is the start of the logs around the time the 4500 rebooted tonight Lots of info (pages and pages more), probably none too relevent to any issue. Jul 24 18:38:15 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:19 MICE-SW1 sfid[1022]: sfi_pfepeer_grace_timer expired Jul 24 18:38:30 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:45 MICE-SW1 ksyncd[1041]: ksyncd_select_control_plane_proto: rhost_sysctlbyname_get: No such file or directory Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/26 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-1/27 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/24 Jul 24 18:38:57 MICE-SW1 chassism[1021]: cm_ff_ifd_enable: fast failover enabled for internal-0/25 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c0 on vcp-1.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-1.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-1 went down Jul 24 18:38:57 MICE-SW1 vccpd[1023]: VCCPD_PROTOCOL_ADJDOWN: Lost adjacency to a8d0.e5ba.37c1 on vcp-0.32768, Jul 24 18:38:57 MICE-SW1 vccpd[1023]: ifl vcp-0.32768 set down, ifl flags 0, flags 0 Jul 24 18:38:57 MICE-SW1 vccpd[1023]: interface vcp-0 went down .. Probably getting up to the current recommended code base for this hardware is the best course of action. (JunOS 12.3 has been working very well for our Juniper EX switch stacks). -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Jason Hanke

Can someone log in and pull the alarms? On Fri, Jul 24, 2015 at 8:07 PM, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: > Mike called and the Cologix tech sees the following alarms on the switch: > SPD > DPX > ADM > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com > > > > > On Fri, Jul 24, 2015 at 5:44 PM -0700, "Jason Hanke" > <jayhanke(a)neutralpath.net> wrote: > >> Main switch services appear to be coming back online. Do we have a root >> cause? >> >> On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill >> wrote: >> > A Cologix tech is on his way. Our NOC was seeing another issue. He will >> > look >> > at the MICE switch as soon as he can and let me know. >> > Mike >> > >> > On Friday, July 24, 2015, Shaun Carlson wrote: >> >> >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone >> >> from >> >> Cologix can get eyes on the switch. >> >> >> >> In the meantime, is there anyone else who is able to get to the 511 in >> >> a >> >> timely way? >> >> >> >> s >> >> >> >> Shaun Carlson >> >> Senior Manager of Information Technology | Arvig >> >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> >> wrote: >> >> >> >>> There are multiple reports of members not getting light from the main >> >>> switch. >> >>> >> >>> Cacti graphing is down. >> >>> >> >>> -- >> >>> Jay Hanke >> >>> CTO >> >>> Neutral Path Communications >> >>> 3 Civic Center Plaza, Suite 204 >> >>> Mankato, MN 56001 >> >>> (507) 327-2398 mobile >> >>> jayhanke(a)neutralpath.net >> >>> www.neutralpath.net >> >> >> >> >> >> ________________________________ >> >> >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > >> > >> > >> > -- >> > Mike Hemphill >> > General Manager | Cologix, Inc. >> > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 >> > P: 1+612.333.1922 | M: 1+612.812.5242 >> > mike.hemphill(a)cologix.com >> > >> > Sent from my iPhone 6 Plus >> > >> > >> > ________________________________ >> > >> > To unsubscribe from the MICE-DISCUSS list, click the following link: >> > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> >> >> >> -- >> Jay Hanke >> CTO >> Neutral Path Communications >> 3 Civic Center Plaza, Suite 204 >> Mankato, MN 56001 >> (507) 327-2398 mobile >> jayhanke(a)neutralpath.net >> www.neutralpath.net > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Shaun Carlson

Mike called and the Cologix tech sees the following alarms on the switch:SPDDPXADM s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Fri, Jul 24, 2015 at 5:44 PM -0700, "Jason Hanke" <jayhanke(a)neutralpath.net> wrote: Main switch services appear to be coming back online. Do we have a root cause? On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill wrote: > A Cologix tech is on his way. Our NOC was seeing another issue. He will look > at the MICE switch as soon as he can and let me know. > Mike > > On Friday, July 24, 2015, Shaun Carlson wrote: >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone from >> Cologix can get eyes on the switch. >> >> In the meantime, is there anyone else who is able to get to the 511 in a >> timely way? >> >> s >> >> Shaun Carlson >> Senior Manager of Information Technology | Arvig >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> wrote: >> >>> There are multiple reports of members not getting light from the main >>> switch. >>> >>> Cacti graphing is down. >>> >>> -- >>> Jay Hanke >>> CTO >>> Neutral Path Communications >>> 3 Civic Center Plaza, Suite 204 >>> Mankato, MN 56001 >>> (507) 327-2398 mobile >>> jayhanke(a)neutralpath.net >>> www.neutralpath.net >> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Mike Hemphill > General Manager | Cologix, Inc. > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 > P: 1+612.333.1922 | M: 1+612.812.5242 > mike.hemphill(a)cologix.com > > Sent from my iPhone 6 Plus > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Jason Hanke

Main switch services appear to be coming back online. Do we have a root cause? On Fri, Jul 24, 2015 at 7:35 PM, Mike Hemphill <mike.hemphill(a)cologix.com> wrote: > A Cologix tech is on his way. Our NOC was seeing another issue. He will look > at the MICE switch as soon as he can and let me know. > Mike > > On Friday, July 24, 2015, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: >> >> We're seeing the same at Arvig. Mike Hemphill is seeing if someone from >> Cologix can get eyes on the switch. >> >> In the meantime, is there anyone else who is able to get to the 511 in a >> timely way? >> >> s >> >> Shaun Carlson >> Senior Manager of Information Technology | Arvig >> ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com >> >> >> >> >> On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" >> <jayhanke(a)neutralpath.net> wrote: >> >>> There are multiple reports of members not getting light from the main >>> switch. >>> >>> Cacti graphing is down. >>> >>> -- >>> Jay Hanke >>> CTO >>> Neutral Path Communications >>> 3 Civic Center Plaza, Suite 204 >>> Mankato, MN 56001 >>> (507) 327-2398 mobile >>> jayhanke(a)neutralpath.net >>> www.neutralpath.net >> >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Mike Hemphill > General Manager | Cologix, Inc. > 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 > P: 1+612.333.1922 | M: 1+612.812.5242 > mike.hemphill(a)cologix.com > > Sent from my iPhone 6 Plus > > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Mike Hemphill

A Cologix tech is on his way. Our NOC was seeing another issue. He will look at the MICE switch as soon as he can and let me know. Mike On Friday, July 24, 2015, Shaun Carlson <shaun.carlson(a)arvig.com> wrote: > We're seeing the same at Arvig. Mike Hemphill is seeing if someone from > Cologix can get eyes on the switch. > > In the meantime, is there anyone else who is able to get to the 511 in a > timely way? > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com > <javascript:_e(%7B%7D,'cvml','shaun.carlson(a)arvig.com');> > > > > > On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" < > jayhanke(a)neutralpath.net > <javascript:_e(%7B%7D,'cvml','jayhanke(a)neutralpath.net');>> wrote: > > There are multiple reports of members not getting light from the main switch. >> >> Cacti graphing is down. >> >> -- >> Jay Hanke >> CTO >> Neutral Path Communications >> 3 Civic Center Plaza, Suite 204 >> Mankato, MN 56001 >> (507) 327-2398 mobilejayhanke(a)neutralpath.net <javascript:_e(%7B%7D,'cvml','jayhanke(a)neutralpath.net');>www.neutralpath.net >> >> > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 1+612.333.1922 | M: 1+612.812.5242 mike.hemphill(a)cologix.com Sent from my iPhone 6 Plus

July 25, 2015

Re: Multiple members reporting no light on the main switch

by David Farmer

I could get down there quick, still in my office, but don't have access to Cologix. On 7/24/15 19:26 , Shaun Carlson wrote: > We're seeing the same at Arvig. Mike Hemphill is seeing if someone from > Cologix can get eyes on the switch. > > In the meantime, is there anyone else who is able to get to the 511 in a > timely way? > > s > > Shaun Carlson > Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 <tel:(218)%20346-8673> | em: shaun.carlson(a)arvig.com > <mailto:shaun.carlson@arvig.com> > > > > > On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" > <jayhanke(a)neutralpath.net <mailto:jayhanke@neutralpath.net>> wrote: > > There are multiple reports of members not getting light from the main switch. > > Cacti graphing is down. > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Justin Krejci

Our 10gig link to MICE is down therefore the cacti server has no path to the switches/prefix. -----Original Message----- From: Jason Hanke [jayhanke(a)NEUTRALPATH.NET] Received: Friday, 24 Jul 2015, 7:25PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Subject: [MICE-DISCUSS] Multiple members reporting no light on the main switch There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net<http://www.neutralpath.net>

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Shaun Carlson

We're seeing the same at Arvig. Mike Hemphill is seeing if someone from Cologix can get eyes on the switch. In the meantime, is there anyone else who is able to get to the 511 in a timely way? s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com On Fri, Jul 24, 2015 at 5:25 PM -0700, "Jason Hanke" <jayhanke(a)neutralpath.net> wrote: There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Multiple members reporting no light on the main switch

by Jason Hanke

There are multiple reports of members not getting light from the main switch. Cacti graphing is down. -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

Re: Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Jeremy Lumby

>From what I can tell remotely the Juniper 4500 (10 gig switch) is down. I have BGP established to the route servers (they and I are both on the 4200) Is anyone close by that can take a look? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Friday, July 24, 2015 7:12 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN All of my MICE BGP sessions have been down since approximately 18:41. For reference, I'm connected to the Mankato Networks switch. -- Richard

July 25, 2015

Re: Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Richard Laager

All of my MICE BGP sessions have been down since approximately 18:41. For reference, I'm connected to the Mankato Networks switch. -- Richard

July 25, 2015

Fwd: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN

by Jason Hanke

can someone take a peek? ---------- Forwarded message ---------- From: NOC IP <svc_NNOCIP(a)charter.com> Date: Fri, Jul 24, 2015 at 7:04 PM Subject: chn00102685,-,mice,l3ppc,-,tge,MICE_public_switch_fabric is DOWN To: "peering(a)micemn.net" <peering(a)micemn.net> MICE, Charter Communications is not getting any incoming light at the Colo link. Could you please check this link and reply with any issues from your end? Thank you, *[image: CharterCommunicatio#B375E30]* *Katie McCarthy, *CCNA | Network Operations Technician II | 866.248.7662 Opt. 2 9333 Dielman Industrial Drive | St Louis, MO 63132 *Please do your part to reduce unnecessary email volume by directing your request to the appropriate NOC group using the following distribution lists:* · *Critical Facilities*: DLCriticalFacilities(a)Charter.com · *IP, CMTS, Optical*: DLNOCIP(a)Charter.Com · *Video*: DLNOCVideo(a)Charter.com · *Voice*: DLNOCVoice(a)Charter.com -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 25, 2015

CNS Switch - Cacti Graph Anomalies

by Justin Krejci

Looks like there is maybe some kind of trouble with the CNS Switch. There are a lot of cacti graphing gaps displayed but only for the ports being queried on the CNS switch itself. Looks like it started around 2pm yesterday. Looking at the LAG port graph off the main switch, it does not appear to be affecting actual member traffic so it may be a management issue with the CNS switch or some other control plane issue possibly. Could be indicative of a larger, brewing problem.

July 23, 2015

Re: MICE Switch Upgrade

by Jeff Bartig

The Seattle SIX exchange tried out this technique for their last software upgrade. Seemed like an interesting solution. https://ripe67.ripe.net/presentations/374-WH-IXPMaintReduce.pdf On Mon, Jul 20, 2015 at 01:11:34PM -0500, Andrew Hoyos wrote: > Shutting down the route server sessions would at least let BGP hold timers expire gracefully while traffic can still be forwarded for the duration of the hold timer. > > Probably more important for those folks that are layer 2 extended into the IX and don't necessarily have any notion of physical link to the switch. > > -- > Andrew Hoyos > hoyosa(a)gmail.com -- Jeff Bartig University of Wisconsin jeffb(a)doit.wisc.edu (608) 262-8336

July 21, 2015

Re: MICE Switch Upgrade

by Andrew Hoyos

> On Jul 20, 2015, at 12:54 PM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > > Wouldn't that be just as disruptive as just doing the switch reboot after upgrade? Shutting down the route server sessions would at least let BGP hold timers expire gracefully while traffic can still be forwarded for the duration of the hold timer. Probably more important for those folks that are layer 2 extended into the IX and don't necessarily have any notion of physical link to the switch. -- Andrew Hoyos hoyosa(a)gmail.com

July 20, 2015

Re: MICE Switch Upgrade

by Jeremy Lumby

I am in favor of turning down the route servers since it would gracefully stop the traffic flow allowing the members routers to automatically reroute traffic before the switch reboot cuts everyone off. The only issue is it does not address the bi-lateral traffic, and therefore the members that care would be forced to turn down BGP on their own anyway. So for simplicity sake, it would be easiest to leave it up to the individual members. I could see it working either way. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Monday, July 20, 2015 12:55 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Switch Upgrade On Mon, Jul 20, 2015 at 12:48:16PM -0500, Richard Laager wrote: > Will you be turning down the route server BGP sessions in advance of > rebooting the switch? Wouldn't that be just as disruptive as just doing the switch reboot after upgrade? Ie. individual members could do so on the schedule they wanted to control the schedule themselves, but I guess I don't see a benefit for the central route servers. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 20, 2015

Re: MICE Switch Upgrade

by Doug McIntyre

On Mon, Jul 20, 2015 at 12:48:16PM -0500, Richard Laager wrote: > Will you be turning down the route server BGP sessions in advance of > rebooting the switch? Wouldn't that be just as disruptive as just doing the switch reboot after upgrade? Ie. individual members could do so on the schedule they wanted to control the schedule themselves, but I guess I don't see a benefit for the central route servers. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 20, 2015

Re: MICE Switch Upgrade

by Richard Laager

Will you be turning down the route server BGP sessions in advance of rebooting the switch? -- Richard

July 20, 2015

MICE Switch Upgrade

by Anthony Anderberg

As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net>

July 20, 2015

Minutes from UG 17

by Shaun Carlson

Please see attached minutes and attendance record. Please note the attendance record also includes the vote counts for validation purposes. Please let me know if there are any corrections or omissions. Thanks! s *Shaun Carlson*Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com

July 15, 2015

Please log into we webex for meeting

by Mike Horwath

Thank you. -- Mike Horwath via iPad.

July 15, 2015

Re: Agenda and Documents for Tomorrow's Meeting

by Dean Bahls

www.webex.com <http://www.webex.com> and join the meeting with the meeting number listed below. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Shane Carlson Sent: Wednesday, July 15, 2015 3:58 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting What's the link to the Webex? _____ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Colin Baker [colinb(a)SUPRANET.NET] Sent: Wednesday, July 15, 2015 3:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting Join the webex and it should give you a call in number. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net <mailto:colinb@supranet.net> This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality From: Chris Wopat Sent: Wednesday, July 15, 2015 3:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Reply To: MICE Discuss Subject: Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting What's the phone #? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net <mailto:wopat@wiscnet.net> 608-210-3965 From: Shaun Carlson <shaun.carlson(a)ARVIG.COM <mailto:shaun.carlson@ARVIG.COM> > Sent: Jul 14, 2015 4:22 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting All, Please see the attached Agenda and related documents for tomorrow's meeting, let me know if you have any questions. I plan to start the Webex at least 15 minutes prior to the meeting start time, please try to join a few minutes early to test and resolve any audio/video issues. In addition, to facilitate communication, please use a Webcam if at all possible. Here are the Webex meeting details again: Regular, Quarterly Meeting (On-Site and Virtual): Vote on 501.(c)6 - Wednesday, July 15th at 4:00 PM - Location: 511 Building Cafeteria - Webex Meeting Number: 196 840 950 When you sign into the virtual meeting, please log in with your company name and ASN to make tracking attendees easier. As with last time, I'll do a roll call to simplify the process since the virtual element makes things a little tougher. Documents Attached: * MICE UG 17 - Agenda -> Agenda for Tomorrow's Meeting * MICE UG 17 - Proxy Ballot -> Proxy Ballot Document for 501(c)(6) Vote(s) * MICE UG 17 - Attendance -> List of MICE members for attendance tracking * MICE, LLC - Amended and Restated Articles of Organization * MICE, LLC - Bylaws of Midwest Internet Cooperative Exchange LLC * MICE, LLC - Resolution secretary certifying amendment * MICE, LLC - Resolution approving Amended and Restated Articles Please let me know if there are any questions or follow-up correspondence requested. Thank you all and I look forward to seeing you tomorrow! s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: <mailto:shaun.carlson@arvig.com> shaun.carlson(a)arvig.com <https://followup.cc/tracker_events/10023405/68d936eb1400ccde38d2fd86671f745 8/ghost.gif?email=shaun.carlson(a)arvig.com> _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

July 15, 2015

Re: Agenda and Documents for Tomorrow's Meeting

by Chris Wopat

I'm mobile in a vehicle. On Jul 15, 2015 3:56 PM, "Colin Baker" <colinb(a)supranet.net> wrote: > Join the webex and it should give you a call in number. > > -- > Colin Baker > SupraNet Communications, Inc. > (608) 572-7634 > colinb(a)supranet.net > > This message is subject to the SupraNet Email Confidentiality Policy > which is located at http://supranet.net/confidentiality > *From: *Chris Wopat > *Sent: *Wednesday, July 15, 2015 3:53 PM > *To: *MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Reply To: *MICE Discuss > *Subject: *Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting > > What's the phone #? > -- > Chris Wopat > Network Engineer, WiscNet > wopat(a)wiscnet.net 608-210-3965 > > > *From:* Shaun Carlson <shaun.carlson(a)ARVIG.COM> > *Sent:* Jul 14, 2015 4:22 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting > > All, > > Please see the attached Agenda and related documents for tomorrow's > meeting, let me know if you have any questions. > > I plan to start the Webex at least 15 minutes prior to the meeting start > time, please try to join a few minutes early to test and resolve any > audio/video issues. In addition, *to facilitate communication, please > use a Webcam if at all possible*. > > Here are the Webex meeting details again: > > Regular, Quarterly Meeting (On-Site and Virtual): Vote on 501.(c)6 > - Wednesday, July 15th at 4:00 PM > - Location: 511 Building Cafeteria > - Webex Meeting Number: 196 840 950 > > When you sign into the virtual meeting, please log in with your company > name and ASN to make tracking attendees easier. As with last time, I'll do > a roll call to simplify the process since the virtual element makes things > a little tougher. > > Documents Attached: > > - MICE UG 17 - Agenda -> Agenda for Tomorrow's Meeting > - MICE UG 17 - Proxy Ballot -> Proxy Ballot Document for 501(c)(6) > Vote(s) > - MICE UG 17 - Attendance -> List of MICE members for attendance > tracking > - MICE, LLC - Amended and Restated Articles of Organization > - MICE, LLC - Bylaws of Midwest Internet Cooperative Exchange LLC > - MICE, LLC - Resolution secretary certifying amendment > - MICE, LLC - Resolution approving Amended and Restated Articles > > > Please let me know if there are any questions or follow-up > correspondence requested. > > Thank you all and I look forward to seeing you tomorrow! > > s > > > *Shaun Carlson *Senior Manager of Information Technology | Arvig > ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

July 15, 2015

Re: Agenda and Documents for Tomorrow's Meeting

by Shane Carlson

What's the link to the Webex? ________________________________ From: MICE Discuss [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] on behalf of Colin Baker [colinb(a)SUPRANET.NET] Sent: Wednesday, July 15, 2015 3:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting Join the webex and it should give you a call in number. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality From: Chris Wopat Sent: Wednesday, July 15, 2015 3:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Reply To: MICE Discuss Subject: Re: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting What's the phone #? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965 From: Shaun Carlson <shaun.carlson(a)ARVIG.COM> Sent: Jul 14, 2015 4:22 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting All, Please see the attached Agenda and related documents for tomorrow's meeting, let me know if you have any questions. I plan to start the Webex at least 15 minutes prior to the meeting start time, please try to join a few minutes early to test and resolve any audio/video issues. In addition, to facilitate communication, please use a Webcam if at all possible. Here are the Webex meeting details again: Regular, Quarterly Meeting (On-Site and Virtual): Vote on 501.(c)6 - Wednesday, July 15th at 4:00 PM - Location: 511 Building Cafeteria - Webex Meeting Number: 196 840 950 When you sign into the virtual meeting, please log in with your company name and ASN to make tracking attendees easier. As with last time, I'll do a roll call to simplify the process since the virtual element makes things a little tougher. Documents Attached: * MICE UG 17 - Agenda -> Agenda for Tomorrow's Meeting * MICE UG 17 - Proxy Ballot -> Proxy Ballot Document for 501(c)(6) Vote(s) * MICE UG 17 - Attendance -> List of MICE members for attendance tracking * MICE, LLC - Amended and Restated Articles of Organization * MICE, LLC - Bylaws of Midwest Internet Cooperative Exchange LLC * MICE, LLC - Resolution secretary certifying amendment * MICE, LLC - Resolution approving Amended and Restated Articles Please let me know if there are any questions or follow-up correspondence requested. Thank you all and I look forward to seeing you tomorrow! s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com<mailto:shaun.carlson@arvig.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 15, 2015

Re: Agenda and Documents for Tomorrow's Meeting

by Colin Baker

July 15, 2015

Re: Agenda and Documents for Tomorrow's Meeting

by Chris Wopat

What's the phone #? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965 From: Shaun Carlson <shaun.carlson(a)ARVIG.COM> Sent: Jul 14, 2015 4:22 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Agenda and Documents for Tomorrow's Meeting All, Please see the attached Agenda and related documents for tomorrow's meeting, let me know if you have any questions. I plan to start the Webex at least 15 minutes prior to the meeting start time, please try to join a few minutes early to test and resolve any audio/video issues. In addition, to facilitate communication, please use a Webcam if at all possible. Here are the Webex meeting details again: Regular, Quarterly Meeting (On-Site and Virtual): Vote on 501.(c)6 - Wednesday, July 15th at 4:00 PM - Location: 511 Building Cafeteria - Webex Meeting Number: 196 840 950 When you sign into the virtual meeting, please log in with your company name and ASN to make tracking attendees easier. As with last time, I'll do a roll call to simplify the process since the virtual element makes things a little tougher. Documents Attached: * MICE UG 17 - Agenda -> Agenda for Tomorrow's Meeting * MICE UG 17 - Proxy Ballot -> Proxy Ballot Document for 501(c)(6) Vote(s) * MICE UG 17 - Attendance -> List of MICE members for attendance tracking * MICE, LLC - Amended and Restated Articles of Organization * MICE, LLC - Bylaws of Midwest Internet Cooperative Exchange LLC * MICE, LLC - Resolution secretary certifying amendment * MICE, LLC - Resolution approving Amended and Restated Articles Please let me know if there are any questions or follow-up correspondence requested. Thank you all and I look forward to seeing you tomorrow! s Shaun Carlson Senior Manager of Information Technology | Arvig ph: (218) 346-8673 | em: shaun.carlson(a)arvig.com<mailto:shaun.carlson@arvig.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 15, 2015

FYI 11th Ave closed

by David Farmer

11th Ave is closed at the light rail. To get from Washington Ave to 511 use Portland Ave Sent from my iPhone

July 15, 2015