July 2015 - mice-discuss - lists.micemn.net

Re: MICE Switch Upgrade

by Mike Horwath

On Wed, Jul 29, 2015 at 11:46:53AM +0000, Anthony Anderberg wrote: > This upgrade is complete, looks like routes and traffic are > returning to normal levels. Thanks, sir! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 30, 2015

Re: MICE Switch Upgrade

by Dan Olsen

Thank You for the work. Dan Olsen- Operations Manager Windomnet 443 10th Street, PO box 38 Windom MN 56101 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Wednesday, July 29, 2015 6:47 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Switch Upgrade This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst 320-234-5239 <mailto:anthonyanderberg@nu-telecom.net> anthonyanderberg(a)nu-telecom.net <http://www.nutelecom.net> www.nutelecom.net From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent <https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200> &id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst 320-234-5239 <mailto:anthonyanderberg@nu-telecom.net> anthonyanderberg(a)nu-telecom.net <http://www.nutelecom.net> www.nutelecom.net _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

July 29, 2015

Re: MICE Switch Upgrade

by Hannigan, Martin

Thanks for the work Anthony. Best, -M< On Jul 29, 2015, at 7:46 AM, Anthony Anderberg <anthonyanderberg(a)NU-TELECOM.NET<mailto:anthonyanderberg@NU-TELECOM.NET>> wrote: This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder – I’m still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I’ll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I’ll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst <image001.gif> 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nutelecom.net&d=AwM…> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we’re encountering a known bug that has been fixed in later software releases. Here’s the bug description for people with Juniper support ids, as we’d suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200<https://urldefense.proofpoint.com/v2/url?u=https-3A__prsearch.juniper.net_I…> The upgrade will require a short interruption in exchange traffic. We’re currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst <image001.gif> 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nutelecom.net&d=AwM…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

July 29, 2015

Re: MICE Switch Upgrade

by Anthony Anderberg

This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 29, 2015

Re: MICE Switch Upgrade

by Anthony Anderberg

This upgrade is complete, looks like routes and traffic are returning to normal levels. Give me a call if anything seems wrong or silly. 320-583-9604 Cheers, anthony From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Tuesday, July 28, 2015 3:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] FW: MICE Switch Upgrade As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 29, 2015

FW: MICE Switch Upgrade

by Anthony Anderberg

As a reminder - I'm still planning to do the MICE exchange upgrade tomorrow morning between 5am and 7am CDT. I'll be shutting down the route server ahead of time, but as we talked about those of you with bilateral sessions or sensitive SLAs may want to take action ahead of time. I'll send a note out when everything is complete. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, July 20, 2015 12:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Switch Upgrade As everyone will recall the exchange has had problems with the main switch flooding unicast traffic a couple times in the last few years. Juniper believes we're encountering a known bug that has been fixed in later software releases. Here's the bug description for people with Juniper support ids, as we'd suspected it deals with L2 database MAC learning/maintaining: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR695200 The upgrade will require a short interruption in exchange traffic. We're currently planning on performing the upgrade next week on Wednesday morning - July 29th between 5am and 7am, that should be the low point in our daily traffic volume. Can someone forward this to the Announce list? Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 28, 2015

Re: Multiple members reporting no light on the main switch

by dangieser

A case is opened with Juniper and files uploaded. Andrew and Anthony are copied on the ticket. Thanks, Dan Gieser Mankato Networks Cell: 507-327-5341 Desk: 507-242-6469 dangieser(a)mankatonetworks.net ------ Original Message ------ From: "Andrew Hoyos" <hoyosa(a)gmail.com> To: MICE-DISCUSS(a)lists.iphouse.net Sent: 7/26/2015 5:29:48 PM Subject: Re: [MICE-DISCUSS] Multiple members reporting no light on the main switch >On Jul 25, 2015, at 11:04 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: >> >>Sure, I put them up at https://noc.iphouse.com/MICE-072415/ > >Thanks! > >>Anthony & Mankato Networks has been the ones dealing with Juniper >>support, >>I don't have contract info. > >Anthony or Mankato folks - can we get a case opened if not already? Or >I’d be happy to as well, just unicast me the J-Care Contract ID. > >Looking through the logs/RSI info, there were core dumps for the >‘eventd’ process around the same general timeframe as traffic quit >forwarding. > >-rw------- 1 root field 13046 Jul 24 19:31 >/var/tmp/eventd.core-tarball.1.tgz >-rw-rw---- 1 root field 71112 Jul 24 19:27 >/var/tmp/eventd.core.0.gz > >While I suspect the answer from Juniper will be ‘upgrade code please’, >it would be good to confirm and see if this is attributable to a known >PR, that is hopefully fixed in whatever version is planned to be >upgraded to. > >Thanks, >Andrew > > >-------------------------------------------------------------------------------- >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

July 26, 2015

Re: Multiple members reporting no light on the main switch

by Andrew Hoyos

On Jul 25, 2015, at 11:04 AM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > > Sure, I put them up at https://noc.iphouse.com/MICE-072415/ <https://noc.iphouse.com/MICE-072415/> Thanks! > Anthony & Mankato Networks has been the ones dealing with Juniper support, > I don't have contract info. Anthony or Mankato folks - can we get a case opened if not already? Or I’d be happy to as well, just unicast me the J-Care Contract ID. Looking through the logs/RSI info, there were core dumps for the ‘eventd’ process around the same general timeframe as traffic quit forwarding. -rw------- 1 root field 13046 Jul 24 19:31 /var/tmp/eventd.core-tarball.1.tgz -rw-rw---- 1 root field 71112 Jul 24 19:27 /var/tmp/eventd.core.0.gz While I suspect the answer from Juniper will be ‘upgrade code please’, it would be good to confirm and see if this is attributable to a known PR, that is hopefully fixed in whatever version is planned to be upgraded to. Thanks, Andrew

July 26, 2015

Re: Multiple members reporting no light on the main switch

by Doug McIntyre

On Sat, Jul 25, 2015 at 09:13:42AM -0500, Andrew Hoyos wrote: > Can we toss these logs up somewhere semi-public so folks can peek at them, along with all the support info? Sure, I put them up at https://noc.iphouse.com/MICE-072415/ Anthony & Mankato Networks has been the ones dealing with Juniper support, I don't have contract info. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 25, 2015

Re: Multiple members reporting no light on the main switch

by Andrew Hoyos

On Jul 24, 2015, at 10:52 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > On Fri, Jul 24, 2015 at 08:10:24PM -0500, Jason Hanke wrote: >> Can someone log in and pull the alarms? > > The alarm lit right now is just "boot from backup root". > The software upgrade should clear that. or running ‘request system snapshot media internal slice alternate’ will copy the active partition to the primary partition. That might be wise to do now, as it will repair the primary partition immediately. > This is the start of the logs around the time the 4500 rebooted tonight > Lots of info (pages and pages more), probably none too relevent to any issue. > > <snip> Can we toss these logs up somewhere semi-public so folks can peek at them, along with all the support info? ‘request support information | save /var/tmp/MICE-ISSUE-072415.txt’ ‘file archive compress source MICE-ISSUE-072415.txt destination MICE-ISSUE-072415.txt.tgz’ > Probably getting up to the current recommended code base for this > hardware is the best course of action. (JunOS 12.3 has been working > very well for our Juniper EX switch stacks). I know the upgrade was being done for other reasons too, but I’d suggest opening a JTAC case so they can at least look at what happened here, before blindly upgrading without knowing the root cause. I’m pretty sure someone paid for support on these recently, right? I’d happily do so with Juniper and volunteer some time….. -- Andrew Hoyos hoyosa(a)gmail.com

July 25, 2015