For my own curiosity: I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota). A couple of questions for those of you that are doing this: 1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?). 2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?) I've seen some funny business with these kinds of network arrangements. Google DNS responses coming from SIX......CloudFlare cache response coming from Omaha IX. It seems like it's "the Chicago problem" (that MICE was founded to avoid) all over again-- except this time around we're doing it with private links all over the country instead of public transit? Just curious what people have found to be best practices with this type of arrangement. ~Matthew
On 27.10.2017 15:01, Matthew Beckwell wrote:
For my own curiosity:
I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota).
A couple of questions for those of you that are doing this:
1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?).
For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up. Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us.
2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?)
If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability? -Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Baker Sent: Friday, October 27, 2017 3:29 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management?
On 27.10.2017 15:01, Matthew Beckwell wrote:
For my own curiosity:
I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota).
A couple of questions for those of you that are doing this:
1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?).
For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up.
Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us.
2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?)
If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose.
-- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net
This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Michael Hare Sent: Monday, October 30, 2017 8:27 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability? -Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Baker Sent: Friday, October 27, 2017 3:29 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management?
On 27.10.2017 15:01, Matthew Beckwell wrote:
For my own curiosity:
I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota).
A couple of questions for those of you that are doing this:
1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?).
For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up.
Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us.
2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?)
If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose.
-- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net
This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
As a transport seller, We're seeing most networks most commonly buy linear paths to two or more IXes. Typically going opposite directions from their home base. We're also starting to see smaller operators exclusively doing remote peering. A typical set up is buying identical transport to the 2 adjacent IX buildings. On wavelengths, the link failure can be passed back to from end to end. Granted, a long-haul wavelength isn't as stable as a cross connect. If a network doesn't have any remote gear the convergence is governed by EBGP vs IBGP/IGP if there is an additional router. For remote setups, in particular with Ethernet transport we're seeing more and more BFD on the sessions. Jay On Mon, Oct 30, 2017 at 8:27 AM, Michael Hare <michael.hare@wisc.edu> wrote:
For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability?
-Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Baker Sent: Friday, October 27, 2017 3:29 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management?
On 27.10.2017 15:01, Matthew Beckwell wrote:
For my own curiosity:
I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota).
A couple of questions for those of you that are doing this:
1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?).
For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up.
Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us.
2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?)
If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose.
-- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net
This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
-- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke@neutralpath.net www.neutralpath.net
We (AS 3128) are mostly a Juniper shop. There are situations where our longhaul is not redundant and less diverse than you would hope. It is a problem easily solved with $$. At times this unfortunate situation leaves us with several hundred thousand routes to converge; it takes time. We're looking at JunOS 16 for deployment this winter. In addition to RPD being slightly threaded, JunOS 16.1 has features to optimize the order of processing RIB changes (policy-statement based). Our challenge becomes defining priority, optimizing on bandwidth per AS/subnet doesn't necessarily tell the whole story, some low bandwidth things (DNS) are paramount [and thankfully rather static]. For now we also have our internet tables in global unicast table. JunOS currently has features that allow you to program a secondary forwarding next hop but it is reserved for l3vpns. Doing that for the global unicast table is an uncommitted ER. I haven't yet started to seriously explore putting the internet in a VRF. -Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Monday, October 30, 2017 9:01 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management?
As a transport seller, We're seeing most networks most commonly buy linear paths to two or more IXes. Typically going opposite directions from their home base. We're also starting to see smaller operators exclusively doing remote peering. A typical set up is buying identical transport to the 2 adjacent IX buildings.
On wavelengths, the link failure can be passed back to from end to end. Granted, a long-haul wavelength isn't as stable as a cross connect.
If a network doesn't have any remote gear the convergence is governed by EBGP vs IBGP/IGP if there is an additional router. For remote setups, in particular with Ethernet transport we're seeing more and more BFD on the sessions.
Jay
On Mon, Oct 30, 2017 at 8:27 AM, Michael Hare <michael.hare@wisc.edu> wrote:
For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability?
-Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Colin Baker Sent: Friday, October 27, 2017 3:29 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management?
On 27.10.2017 15:01, Matthew Beckwell wrote:
For my own curiosity:
I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota).
A couple of questions for those of you that are doing this:
1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?).
For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up.
Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us.
2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?)
If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose.
-- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb@supranet.net
This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality
-- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke@neutralpath.net www.neutralpath.net
Matthew, My few cents on your questions: Since there are extension switches for some of these other IXes in Minneapolis, often times the cost is far less than the wavelength, a term contract is not even needed, and by having multiple people sharing the cost of an extension, sometimes it can cover the cost of diverse waves (I have been surprised at how often a wave will go down, when you have thousands of miles of fiber, a lot can go wrong) I personally use prepends, and MEDs to manipulate traffic paths. Many of the CDNs have complicated algorithms that go far beyond BGP, and factor in things like load on the cache, geo-location of the IP, and latency. As for the Chicago problem, I still see my upstreams all trying to take me there. This gives me an option to go the other direction. After the diverse content, the second reason I peer at other exchanges is I have a direct relationship with the peer, so I cutout the delays/red tape of working with an upstream to resolve the issue. Not to mention I can avoid if the upstream gets into a pissing match about who they will/won't peer with, and it causes their links to saturate. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Friday, October 27, 2017 3:02 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Multi-IX Traffic Management? For my own curiosity: I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota). A couple of questions for those of you that are doing this: 1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?). 2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?) I've seen some funny business with these kinds of network arrangements. Google DNS responses coming from SIX......CloudFlare cache response coming from Omaha IX. It seems like it's "the Chicago problem" (that MICE was founded to avoid) all over again-- except this time around we're doing it with private links all over the country instead of public transit? Just curious what people have found to be best practices with this type of arrangement. ~Matthew To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
I peer at SIX only with networks that I can't peer with at MICE. If a network is at both, I filter out their routes from SIX's route server and I ask the SIX route server to not propagate my routes to them (using the relevant communities). -- Richard
participants (6)
-
Colin Baker -
Jason Hanke -
Jeremy Lumby -
Matthew Beckwell -
Michael Hare -
Richard Laager