- mice-discuss - lists.micemn.net

Re: Juniper Switch upgrade

by Mike Horwath

On Tue, Aug 21, 2012 at 01:37:43PM -0700, Owen DeLong wrote: > As tempting as it is, I would suggest... NO! > > Multiple changes at the same time are rarely a good idea on critical > infrastructure. You end up not knowing which change broke what when > it comes time to troubleshoot. While overlapping problems are > unlikely in this case, I would say it's still best to do the > renumbering as more of an over-time process. There's no reason to > do renumbering as a flag day. Just one switch, is it really 'critical'? :) (I know, I shouldn't poke the sleeping dragon) -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Jay Hanke

> Jay, one problem with a method like that, is that there isn't a > central router. Everybody is talking all in the same flat space. If > somebody changes to talk on their 206.108.255.0/24 IP address, then > those that haven't changed yet won't know how to reach their routes. > Thus breaking the exchange apart into those that have adjusted, and > those that haven't adjusted. What more, the unconfigured people will > see the new routes, but can't reach them. I didn't specifically say it but I meant add an additional IP's to each members interfaces. The next hop should follow the session IP. Then set up sessions to the route-servers using both blocks. On flag day we remove the old IP addresses (stick). New entrants would only be assigned in the new block (carrot). If you're not configured yet, you should see two next hops in BGP. One on the old range and one on the new range with an unreachable next hop which won't install in the FIB. The minor PITA is legacy routers that have definite ideas on what address is secondary. I don't remember off hand if a BGP session will survive a move from secondary to primary on an interface in old school IOS. I think it will drop it and then reestablish causing a little outage. The current primary could be flipped to secondary when the switches are down... <back to fundraising and contracts/> ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Doug McIntyre

On Tue, Aug 21, 2012 at 03:54:19PM -0500, Jay Hanke wrote: > On Tue, Aug 21, 2012 at 3:46 PM, Mike Bushard, Jr > <mike.bushard(a)arvig.com> wrote: > > I typically would agree Owen, but in this case it may make sense. Everyone > > will be down already, and we won't have multiple outages to the entire > > exchange for IP changes. There will be a considerable amount of work if > > everyone changes addressing on different schedules. I think just saying > > that on day x we are on the long term strategy will save time in this > > case. Our topology is pretty basic today so I think the rick would be > > minimal at this time. > > The time frame is pretty tight for everyone to coordinate bilateral > changes. Esp for the bigger folks. > > What about enabling the new IP addresses on the existing topology and > moving one of the route servers to the new block. Then setting a flag > date to remove the existing IP addresses 2 months out. I'm sorry I've been so under the weather, that I've been behind and haven't started coordinating the renumbering with my co-conspirators. Jay, one problem with a method like that, is that there isn't a central router. Everybody is talking all in the same flat space. If somebody changes to talk on their 206.108.255.0/24 IP address, then those that haven't changed yet won't know how to reach their routes. Thus breaking the exchange apart into those that have adjusted, and those that haven't adjusted. What more, the unconfigured people will see the new routes, but can't reach them. What follows is the plan I had in my head (but not communicated yet) until I saw the oppertunity to just do it all at once. We communicate everybody's new 206.108.255.0/24 IP & 2001:504:27::/48 address, which almost certainly will just be reusing the same last octet as the current one. Everybody does a secondary of these IP addresses on their MICE facing interface. This has to be completed by such-and-such date. We'd do spot checks with ping to make sure people have completed this. Meanwhile, we build duplicate BGP sessions in the route reflectors for when people are ready to cut. After the such-and-such date, then people are free to cut over their BGP sessions to originate from their new IP, and connect to the new BGP session on the route reflectors. Finally, there is a drop-dead date when all use of the old IP ranges are done. This way, when somebody cuts onto the new BGP session, the unconfigured old people still have a way to reach the new routes (via. the secondary IP addresses configured on their interface). But their BGP sessions don't change until they want them to. This way is slow, and requires 100% compliance in getting the interface secondary addresses, and then in the final drop-dead date. OOTH, if everybody goes down for the switch cut, then people can bring up their session when they want to after the maintenance window, but that also means everybody has to be available to cut their configs over that night/weekend as well if they want to continue to talk on the exchange. -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Jay Hanke

On Tue, Aug 21, 2012 at 3:46 PM, Mike Bushard, Jr <mike.bushard(a)arvig.com> wrote: > I typically would agree Owen, but in this case it may make sense. Everyone > will be down already, and we won't have multiple outages to the entire > exchange for IP changes. There will be a considerable amount of work if > everyone changes addressing on different schedules. I think just saying > that on day x we are on the long term strategy will save time in this > case. Our topology is pretty basic today so I think the rick would be > minimal at this time. The time frame is pretty tight for everyone to coordinate bilateral changes. Esp for the bigger folks. What about enabling the new IP addresses on the existing topology and moving one of the route servers to the new block. Then setting a flag date to remove the existing IP addresses 2 months out. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Hannigan, Martin

That works for me. If we were talking about hundreds of gigs, I might think otherwise, but I expect that this will go relatively smoothly and less complex is better IMHO. Best, -M< On 8/21/12 4:46 PM, "Mike Bushard, Jr" <mike.bushard(a)ARVIG.COM> wrote: > I typically would agree Owen, but in this case it may make sense. Everyone > will be down already, and we won't have multiple outages to the entire > exchange for IP changes. There will be a considerable amount of work if > everyone changes addressing on different schedules. I think just saying > that on day x we are on the long term strategy will save time in this > case. Our topology is pretty basic today so I think the rick would be > minimal at this time. > > > > Mike Bushard, Jr > > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Owen DeLong > Sent: Tuesday, August 21, 2012 3:38 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Juniper Switch upgrade > > As tempting as it is, I would suggest... NO! > > Multiple changes at the same time are rarely a good idea on critical > infrastructure. > You end up not knowing which change broke what when it comes time to > troubleshoot. While overlapping problems are unlikely in this case, I > would say it's still best to do the renumbering as more of an over-time > process. > > There's no reason to do renumbering as a flag day. > > Owen > > On Aug 21, 2012, at 12:28 , Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > >> On Mon, Aug 20, 2012 at 11:51:10AM -0500, Mike Horwath wrote: >>> On Mon, Aug 20, 2012 at 10:43:43AM -0500, Justin Krejci wrote: >>>> That said any time can be convenient if we have some time to adjust >>>> BGP configs to move traffic away in advance, which during business >>>> hours is typically convenient. >>> >>> A shutdown of the routing daemons on the reflectors should take care >>> of things and no manual BGP configuratin adjustments are needed. >> >> Do we want to maybe do the big renumbering at the same time as this >> switch swapout too? Since things are going to be down, everybody stays >> down until they manually redo their interface and BGP configurations >> after the switch change? >> >> Or do we want to do it more gradually? >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> -- ipHouse/Goldengate/Bitstream/ProNS -- >> Network Engineer/Provisioning/Jack of all Trades >> >> ###################################################################### >> ## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Mike Bushard, Jr

I typically would agree Owen, but in this case it may make sense. Everyone will be down already, and we won't have multiple outages to the entire exchange for IP changes. There will be a considerable amount of work if everyone changes addressing on different schedules. I think just saying that on day x we are on the long term strategy will save time in this case. Our topology is pretty basic today so I think the rick would be minimal at this time. Mike Bushard, Jr -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Owen DeLong Sent: Tuesday, August 21, 2012 3:38 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Juniper Switch upgrade As tempting as it is, I would suggest... NO! Multiple changes at the same time are rarely a good idea on critical infrastructure. You end up not knowing which change broke what when it comes time to troubleshoot. While overlapping problems are unlikely in this case, I would say it's still best to do the renumbering as more of an over-time process. There's no reason to do renumbering as a flag day. Owen On Aug 21, 2012, at 12:28 , Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > On Mon, Aug 20, 2012 at 11:51:10AM -0500, Mike Horwath wrote: >> On Mon, Aug 20, 2012 at 10:43:43AM -0500, Justin Krejci wrote: >>> That said any time can be convenient if we have some time to adjust >>> BGP configs to move traffic away in advance, which during business >>> hours is typically convenient. >> >> A shutdown of the routing daemons on the reflectors should take care >> of things and no manual BGP configuratin adjustments are needed. > > Do we want to maybe do the big renumbering at the same time as this > switch swapout too? Since things are going to be down, everybody stays > down until they manually redo their interface and BGP configurations > after the switch change? > > Or do we want to do it more gradually? > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ###################################################################### > ## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Owen DeLong

As a follow-on, I'll propose a suggested renumbering process... 1. Provide new addresses to all participants. 2. Each participant adds new addresses to their routers in accordance with their internal maintenance procedures and announces their readiness to cut over peering sessions. 3. Each participant reviews all announcements prior to theirs from the list and takes responsibility for contacting those peers that have previously announced and cutting over their peering sessions on a schedule by mutual agreement. 4. Once more than 50% of participants have announced readiness with the new addresses, a flag day for deprecating the old addresses can be projected and determined by the board, announced on this list. The remaining stragglers have until that flag day to cut over their peering sessions. In this way, there's an incentive to do your cutover as early as possible (then you can sit back and cut over peering sessions as your peers become ready, instead of having to reach out to each and every one of them). Additionally, it provides for cutting peering sessions over in a controlled manner without requiring any downtime and still on a fairly aggressive time table. Thoughts? Owen ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Owen DeLong

As tempting as it is, I would suggest... NO! Multiple changes at the same time are rarely a good idea on critical infrastructure. You end up not knowing which change broke what when it comes time to troubleshoot. While overlapping problems are unlikely in this case, I would say it's still best to do the renumbering as more of an over-time process. There's no reason to do renumbering as a flag day. Owen On Aug 21, 2012, at 12:28 , Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > On Mon, Aug 20, 2012 at 11:51:10AM -0500, Mike Horwath wrote: >> On Mon, Aug 20, 2012 at 10:43:43AM -0500, Justin Krejci wrote: >>> That said any time can be convenient if we have some time to adjust BGP >>> configs to move traffic away in advance, which during business hours is >>> typically convenient. >> >> A shutdown of the routing daemons on the reflectors should take care >> of things and no manual BGP configuratin adjustments are needed. > > Do we want to maybe do the big renumbering at the same time as this > switch swapout too? Since things are going to be down, everybody stays > down until they manually redo their interface and BGP configurations > after the switch change? > > Or do we want to do it more gradually? > > -- > Doug McIntyre <merlyn(a)iphouse.net> > -- ipHouse/Goldengate/Bitstream/ProNS -- > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Mike Bushard, Jr

May as well get it all over with if we can. Mike Bushard, Jr | Network Engineer IV | *Arvig* | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile ** NEW NUMBER ** | 320-256-7555 Fax [image: Description: Description: http://ipstats.arvig.net/arviglogo.jpeg] *** Please update your address book with my new work email: mike.bushard(a)arvig.com *** *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] *On Behalf Of *Brian Mort *Sent:* Tuesday, August 21, 2012 2:47 PM *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] Juniper Switch upgrade 2nd On Tue, Aug 21, 2012 at 2:46 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: I am in favor of doing it all at once -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, August 21, 2012 2:29 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Juniper Switch upgrade On Mon, Aug 20, 2012 at 11:51:10AM -0500, Mike Horwath wrote: > On Mon, Aug 20, 2012 at 10:43:43AM -0500, Justin Krejci wrote: > > That said any time can be convenient if we have some time to adjust > > BGP configs to move traffic away in advance, which during business > > hours is typically convenient. > > A shutdown of the routing daemons on the reflectors should take care > of things and no manual BGP configuratin adjustments are needed. Do we want to maybe do the big renumbering at the same time as this switch swapout too? Since things are going to be down, everybody stays down until they manually redo their interface and BGP configurations after the switch change? Or do we want to do it more gradually? -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Brian Mort Network Engineer IV (ENA,ENS,ECDP) visit our website: arvig <http://www.arvig.com> (218) 346-8231 brian.mort(a)arvig.com ------------------------------ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012

Re: Juniper Switch upgrade

by Anthony Anderberg

Doing everything at once certainly does solve a whole bunch of logistical problems, if we all are looking at it at (roughly) the same time we can probably make short work of it. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg(a)nu-telecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Tuesday, August 21, 2012 2:29 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Juniper Switch upgrade On Mon, Aug 20, 2012 at 11:51:10AM -0500, Mike Horwath wrote: > On Mon, Aug 20, 2012 at 10:43:43AM -0500, Justin Krejci wrote: > > That said any time can be convenient if we have some time to adjust > > BGP configs to move traffic away in advance, which during business > > hours is typically convenient. > > A shutdown of the routing daemons on the reflectors should take care > of things and no manual BGP configuratin adjustments are needed. Do we want to maybe do the big renumbering at the same time as this switch swapout too? Since things are going to be down, everybody stays down until they manually redo their interface and BGP configurations after the switch change? Or do we want to do it more gradually? -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 21, 2012