August 2012 - mice-discuss - lists.micemn.net

Re: Juniper Switch upgrade

by Owen DeLong

On Aug 22, 2012, at 14:51 , Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > On Wed, Aug 22, 2012 at 02:47:21PM -0500, Justin Krejci wrote: >> While I agree with the general sentiment that the IP change and switch >> change will likely be smooth, the IP change can be done more >> transparently and has literally nothing to do with the L2 gear. >> To that end, I don't see any reason to not configure/assign the new IP >> block now and let folks start using it right away, even before the >> switch switch. > > But, we don't have a central router? > > If your border to the exchange right now got a prefix with the next > hop of 206.108.255.3, how'd you get there? You'd have no route to host > for 206.108.255.3. > > That is why I brought up the secondary IPs. Once everybody puts in a > secondary IP in 206.108.255.0/24, they'll know how to try to get to > 206.108.255.3 for my prefix announcement. But I can't really do the > prefix announcement with the next-hop in 206.108.255.3 until everybody > knows how to get to 206.108.255.0/24? > Your next hop for peers that are still peering on the old address should be your old address. For peers on the new address, should be your new address. The problem you describe could only apply, therefore, to the routeservers which are providing next hop data other than their own local interface. In that case, there are multiple possibilities: 1. Run two route-server instances on each address set. 2. Run one route-server instance on each address set. The route-server instances on the old address set shouldn't be seeing new address next-hops. The route-server instances on the new address set shouldn't be seeing old-address next-hops. Yes, it means everyone that is using new addresses has to maintain their old sessions with the route-servers until everyone has a new session. As Mr. Tindle said earlier... I've been through a few of these. They're not rocket science, but trying to herd all the cats into doing it as one massive change all at the same time is an almost certain path to large-scale disruption and lots of time debugging stuff that didn't have to be debugged all at once. A little careful planning and adding sessions one at a time and then judiciously removing sessions that are no longer needed along the way can accomplish this in a reasonable timeframe with a minimal amount of fuss and no mass-scale risks. Owen ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Mike Horwath

On Wed, Aug 22, 2012 at 02:40:09PM -0700, Owen DeLong wrote: > Suggestion instead... During switch maintenance... > > conf term > int bajilliongmillibit0/0 > ip address new-ip new-netmask > ip address old-ip old-netmask secondary > ipv6 address new-ip new-netmask > ipv6 address old-ip old-netmask secondary > > In this way, yes, there will be a brief time where the old-ip > disappears from the router and those BGP sessions will shut down > accordingly. However, since they will already be down for the switch > maintenance, no harm. > > Then, when everyone has completed their address migration, > removing the secondary address is easy and doesn't cause any > outage. There we go, smarty pants win. -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Mike Horwath

On Wed, Aug 22, 2012 at 04:33:23PM -0500, Jay Hanke wrote: > > conf term > > int bajlliongmillibit0/0 > > ip address secondary-ip secondary-netmask > > ^z > > write mem > > > > notice I didn't remove the primary address...nor the old secondary. > > > > Should work? > > That will change the primary IP. When it gets removed the neighbor > dies because that is typically the source IP of the TCP session. You > then need to add the "old" ip as a secondary. Causing the previously > mentioned blip. > > There isn't really secondaries in Junos so, not an issue. What is the > behavior on Brocade and Extreme? I'd suggest making sure BGP is already configured in ahead of time cause that seems smart. Not like it hurts to have them both operational concurrently or anything. But then sk00ling happens. Damnit. Doug explained this: neighbor (x.x.x.x|x:x:x:x:x..) update-source INTERFACE So oh well, guess you just need to change it some night. IT IS NOT THE END OF THE WORLD IF YOUR MICE CONNECTION GOES DOWN Says Nevin. -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Doug McIntyre

On Wed, Aug 22, 2012 at 02:47:21PM -0500, Justin Krejci wrote: > While I agree with the general sentiment that the IP change and switch > change will likely be smooth, the IP change can be done more > transparently and has literally nothing to do with the L2 gear. > To that end, I don't see any reason to not configure/assign the new IP > block now and let folks start using it right away, even before the > switch switch. But, we don't have a central router? If your border to the exchange right now got a prefix with the next hop of 206.108.255.3, how'd you get there? You'd have no route to host for 206.108.255.3. That is why I brought up the secondary IPs. Once everybody puts in a secondary IP in 206.108.255.0/24, they'll know how to try to get to 206.108.255.3 for my prefix announcement. But I can't really do the prefix announcement with the next-hop in 206.108.255.3 until everybody knows how to get to 206.108.255.0/24? -- Doug McIntyre <merlyn(a)iphouse.net> -- ipHouse/Goldengate/Bitstream/ProNS -- Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Owen DeLong

On Aug 22, 2012, at 14:26 , Mike Horwath <drechsau(a)iphouse.net> wrote: > On Wed, Aug 22, 2012 at 01:55:53PM -0700, Owen DeLong wrote: >> Agreed. However, I would say that for those with Cisco gear where an >> address swap requires removing the address from the interface (and >> thus downing their BGP sessions), making the new address primary and >> the old address secondary could, ideally, be done during the switch >> upgrade and thus avoid an additional outage. > > conf term > int bajlliongmillibit0/0 > ip address secondary-ip secondary-netmask > ^z > write mem > > notice I didn't remove the primary address...nor the old secondary. > > Should work? > That will replace the old address with the new one. If your peers haven't all done the same thing and you haven't modified all your BGP sessions accordingly, then bad things happen. If you meant: ... ip address secondary-ip secondary-netmask secondary Then that postpones the problem to the day you deprecate the primary address. In either case, you forgot the IPv6 address. ;-) Suggestion instead... During switch maintenance... conf term int bajilliongmillibit0/0 ip address new-ip new-netmask ip address old-ip old-netmask secondary ipv6 address new-ip new-netmask ipv6 address old-ip old-netmask secondary In this way, yes, there will be a brief time where the old-ip disappears from the router and those BGP sessions will shut down accordingly. However, since they will already be down for the switch maintenance, no harm. Then, when everyone has completed their address migration, removing the secondary address is easy and doesn't cause any outage. Owen ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Mike Bushard, Jr

Brocade follows the same form as Juniper, an IP is an IP. Mike Bushard, Jr | Network Engineer IV | Arvig | 320.256.7471 Office | 320.256.0178 Direct | 320.429.0837 Mobile ** NEW NUMBER ** | 320-256-7555 Fax *** Please update your address book with my new work email: mike.bushard(a)arvig.com *** -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Wednesday, August 22, 2012 4:33 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Juniper Switch upgrade > conf term > int bajlliongmillibit0/0 > ip address secondary-ip secondary-netmask ^z write mem > > notice I didn't remove the primary address...nor the old secondary. > > Should work? That will change the primary IP. When it gets removed the neighbor dies because that is typically the source IP of the TCP session. You then need to add the "old" ip as a secondary. Causing the previously mentioned blip. There isn't really secondaries in Junos so, not an issue. What is the behavior on Brocade and Extreme? ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Packet loss between GigaPOP and Onvoy/Zayo

by Mike Hemphill

FYI, no Zayo cross connect ordered as of yet. Mike Mike Hemphill General Manager | Cologix, Inc. 511 11th Ave S, Suite 450 | Minneapolis, MN 55415 P: 612.333.1922 | M: 612.812.5242 mike.hemphill(a)cologix.com cologix.com -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Wednesday, August 22, 2012 3:24 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Packet loss between GigaPOP and Onvoy/Zayo On Wed, Aug 22, 2012 at 3:05 PM, Larry Patterson <larry(a)atomicdata.com> wrote: > I attempted to report this issue between Zayo and UMN with BGP paths > and packet loss examples on August 15 to Zayo. But because I am not a > customer if Zayo, their support couldn't do anything with it. I'm > glad to hear that the issues are resolved. That being said, what is > the method for reporting networking issues amongst peering members? Usually directly to the peering contact or via the MICE-DISCUSS list. Zayo isn't currently directly connected to the MICE fabric they connect via the Gigapop. Zayo is in the process of turning up a direct connection. They have assignments but I'm not sure if the x-conn has been ordered yet. Jay ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Jay Hanke

> conf term > int bajlliongmillibit0/0 > ip address secondary-ip secondary-netmask > ^z > write mem > > notice I didn't remove the primary address...nor the old secondary. > > Should work? That will change the primary IP. When it gets removed the neighbor dies because that is typically the source IP of the TCP session. You then need to add the "old" ip as a secondary. Causing the previously mentioned blip. There isn't really secondaries in Junos so, not an issue. What is the behavior on Brocade and Extreme? ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Mike Horwath

On Wed, Aug 22, 2012 at 01:55:53PM -0700, Owen DeLong wrote: > Agreed. However, I would say that for those with Cisco gear where an > address swap requires removing the address from the interface (and > thus downing their BGP sessions), making the new address primary and > the old address secondary could, ideally, be done during the switch > upgrade and thus avoid an additional outage. conf term int bajlliongmillibit0/0 ip address secondary-ip secondary-netmask ^z write mem notice I didn't remove the primary address...nor the old secondary. Should work? -- Mike Horwath ipHouse - Welcome home! drechsau(a)iphouse.net The universe is an island, surrounded by whatever it is that surrounds universes. - Berkeley Fortune ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012

Re: Juniper Switch upgrade

by Owen DeLong

Agreed. However, I would say that for those with Cisco gear where an address swap requires removing the address from the interface (and thus downing their BGP sessions), making the new address primary and the old address secondary could, ideally, be done during the switch upgrade and thus avoid an additional outage. Owen On Aug 22, 2012, at 12:47 , Justin Krejci <jkrejci(a)USINTERNET.COM> wrote: > While I agree with the general sentiment that the IP change and switch change will likely be smooth, the IP change can be done more transparently and has literally nothing to do with the L2 gear. > To that end, I don't see any reason to not configure/assign the new IP block now and let folks start using it right away, even before the switch switch. > > > > > On Tue, 2012-08-21 at 15:54 -0700, Owen DeLong wrote: >> >> On Aug 21, 2012, at 14:07 , Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: >> >> > On Tue, Aug 21, 2012 at 03:54:19PM -0500, Jay Hanke wrote: >> >> On Tue, Aug 21, 2012 at 3:46 PM, Mike Bushard, Jr >> >> <mike.bushard(a)arvig.com> wrote: >> >>> I typically would agree Owen, but in this case it may make sense. Everyone >> >>> will be down already, and we won't have multiple outages to the entire >> >>> exchange for IP changes. There will be a considerable amount of work if >> >>> everyone changes addressing on different schedules. I think just saying >> >>> that on day x we are on the long term strategy will save time in this >> >>> case. Our topology is pretty basic today so I think the rick would be >> >>> minimal at this time. >> >> >> >> The time frame is pretty tight for everyone to coordinate bilateral >> >> changes. Esp for the bigger folks. >> >> >> >> What about enabling the new IP addresses on the existing topology and >> >> moving one of the route servers to the new block. Then setting a flag >> >> date to remove the existing IP addresses 2 months out. >> > >> > >> > I'm sorry I've been so under the weather, that I've been behind and >> > haven't started coordinating the renumbering with my co-conspirators. >> > >> > Jay, one problem with a method like that, is that there isn't a >> > central router. Everybody is talking all in the same flat space. If >> > somebody changes to talk on their 206.108.255.0/24 IP address, then >> > those that haven't changed yet won't know how to reach their routes. >> > Thus breaking the exchange apart into those that have adjusted, and >> > those that haven't adjusted. What more, the unconfigured people will >> > see the new routes, but can't reach them. >> > >> There's no reason people can't add the new addresses and migrate >> peering sessions individually, one peer at a time, then remove the >> old addresses once that's complete. >> >> I think two months would be a bit narrow on the time frame, however. >> >> > >> > What follows is the plan I had in my head (but not communicated yet) >> > until I saw the oppertunity to just do it all at once. >> > >> > We communicate everybody's new 206.108.255.0/24 IP & 2001:504:27::/48 >> > address, which almost certainly will just be reusing the same last >> > octet as the current one. >> > >> >> Might I suggest on the IPv6 block 2001:504:27:0::/64 be used for the >> exchange (we can use other parts of the /48 later if needed) and that >> we number as follows: >> 2001:504:27:0:AS:NO:XXXX:YYYY >> >> Where AS:NO is the 32 bit as number and XXXX:YYYY is a number chosen >> by the applicable AS to represent the specific router/port combination. >> (Yes, 32 bits for router/port combinations is overkill, but I can't think of >> a better use for the bits). >> >> > Everybody does a secondary of these IP addresses on their MICE facing >> > interface. This has to be completed by such-and-such date. We'd do >> > spot checks with ping to make sure people have completed this. >> > >> >> I would actually suggest making the existing address "secondary" and >> the new addresses "primary" to reduce future downtime. >> >> > Meanwhile, we build duplicate BGP sessions in the route reflectors for >> > when people are ready to cut. >> > >> >> +1 >> >> > After the such-and-such date, then people are free to cut over their >> > BGP sessions to originate from their new IP, and connect to the new >> > BGP session on the route reflectors. >> > >> >> I think the RS peering sessions can be cut this way, but suggest you >> look over my earlier proposal for managing the peer to peer sessions. >> >> > Finally, there is a drop-dead date when all use of the old IP ranges are done. >> > >> > This way, when somebody cuts onto the new BGP session, the >> > unconfigured old people still have a way to reach the new routes >> > (via. the secondary IP addresses configured on their interface). But >> > their BGP sessions don't change until they want them to. >> > >> >> An excellent solution for the RS peering sessions. >> >> > This way is slow, and requires 100% compliance in getting the >> > interface secondary addresses, and then in the final drop-dead date. >> > >> >> Actually, it doesn't require 100% compliance. People can actually add >> the interface secondaries at their convenience and everything continues >> to work with their old address until they choose to cut over. >> >> > OOTH, if everybody goes down for the switch cut, then people can bring >> > up their session when they want to after the maintenance window, but >> > that also means everybody has to be available to cut their configs >> > over that night/weekend as well if they want to continue to talk on >> > the exchange. >> >> If everyone were only peered with the RS, I'd say this might be a feasible >> approach. However, since they aren't, I'd suggest that we point out the >> switch cut as a "recommended time" to apply the new addressing to their >> interfaces and leave it at that. >> >> Owen >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2012