October 2017 - mice-discuss - lists.micemn.net

Re: Graph Updates

by Richard Laager

On 10/30/2017 12:39 PM, anthonyanderberg(a)nu-telecom.net wrote: > I’ve been watching the Aggregate Ingress/Egress graph values diverge > with increasing discomfort and finally had a couple free hours to look > into it yesterday and found the combination of the last round of CDN LAG > upgrades and cruft left over from migrating from the old Juniper core > was causing the Ingress values to skew high. In the end it was easier > to just create a new Aggregate graph so I’ve done that and will remove > the old graph once the web site is updated: > > http://micelg.usinternet.com/export/graph_1430.html Thanks for looking into this. I have updated the website. -- Richard

Oct. 30, 2017

Graph Updates

by anthonyanderberg＠nu-telecom.net

I've been watching the Aggregate Ingress/Egress graph values diverge with increasing discomfort and finally had a couple free hours to look into it yesterday and found the combination of the last round of CDN LAG upgrades and cruft left over from migrating from the old Juniper core was causing the Ingress values to skew high. In the end it was easier to just create a new Aggregate graph so I've done that and will remove the old graph once the web site is updated: http://micelg.usinternet.com/export/graph_1430.html I also took the time to dig into the IP4/6 Peer/Prefix graphs and think I've got them (mostly) fixed now too, they're under the "Route Server" menu item in Cacti. Thanks for everyone's suggestions on that topic. As always let me know if anything seems wrong or silly. Cheers, anthony Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net>

Oct. 30, 2017

Re: Multi-IX Traffic Management?

by Michael Hare

We (AS 3128) are mostly a Juniper shop. There are situations where our longhaul is not redundant and less diverse than you would hope. It is a problem easily solved with $$. At times this unfortunate situation leaves us with several hundred thousand routes to converge; it takes time. We're looking at JunOS 16 for deployment this winter. In addition to RPD being slightly threaded, JunOS 16.1 has features to optimize the order of processing RIB changes (policy-statement based). Our challenge becomes defining priority, optimizing on bandwidth per AS/subnet doesn't necessarily tell the whole story, some low bandwidth things (DNS) are paramount [and thankfully rather static]. For now we also have our internet tables in global unicast table. JunOS currently has features that allow you to program a secondary forwarding next hop but it is reserved for l3vpns. Doing that for the global unicast table is an uncommitted ER. I haven't yet started to seriously explore putting the internet in a VRF. -Michael >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Jason Hanke >>Sent: Monday, October 30, 2017 9:01 AM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? >> >>As a transport seller, We're seeing most networks most commonly buy >>linear paths to two or more IXes. Typically going opposite directions >>from their home base. We're also starting to see smaller operators >>exclusively doing remote peering. A typical set up is buying identical >>transport to the 2 adjacent IX buildings. >> >>On wavelengths, the link failure can be passed back to from end to >>end. Granted, a long-haul wavelength isn't as stable as a cross >>connect. >> >>If a network doesn't have any remote gear the convergence is governed >>by EBGP vs IBGP/IGP if there is an additional router. For remote >>setups, in particular with Ethernet transport we're seeing more and >>more BFD on the sessions. >> >>Jay >> >>On Mon, Oct 30, 2017 at 8:27 AM, Michael Hare <michael.hare(a)wisc.edu> >>wrote: >>> For those buying wavelengths (compared to an mpls circuit), are you >>buying diverse transport? If not, how are you dealing with convergence >>stability? >>> >>> -Michael >>> >>>>>-----Original Message----- >>>>>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On >>Behalf >>>>>Of Colin Baker >>>>>Sent: Friday, October 27, 2017 3:29 PM >>>>>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>>>>Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? >>>>> >>>>>On 27.10.2017 15:01, Matthew Beckwell wrote: >>>>>> For my own curiosity: >>>>>> >>>>>> I've seen quite a few networks in the last year or so connecting >>>>>> themselves to other out-of-market IX's (I presume getting some cheap >>>>>> wavelengths and bringing it back to Minnesota). >>>>>> >>>>>> A couple of questions for those of you that are doing this: >>>>>> >>>>>> 1. I'm assuming this is enabling you to accommodate requirements >>from >>>>>> other networks that require peering at multiple locations, even though >>>>>> technically your network doesn't really extend that far (or maybe it's >>>>>> something else?). >>>>> >>>>>For us, the goal is to reach networks which are not on exchanges which >>>>>we're already connected. The big ones are at MICE, but there are other >>>>>exchanges we can get to reasonably inexpensively that have such a large >>>>>number of networks connected that it makes sense financially >>(compared >>>>>to buying more transit). Traffic from a huge number of ASNs, >>>>>particularly if most are on a route server, can really add up. >>>>> >>>>>Peering in multiple locations is a nice benefit to this, but usually not >>>>>a primary goal for us. >>>>> >>>>>> 2. How are folks monitoring, managing, or manipulating this >>>>>> out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some >>>>>> other mechanism to "prefer" your traffic enter and exit closer to >>>>>> where you really are?) >>>>> >>>>>If you haven't already, install as-stats >>>>>(https://github.com/manuelkasper/AS-Stats) to monitor things, which >>will >>>>>make it clear if something is out of whack. In most cases, our IGP >>>>>handles outbound traffic in a way the makes sense. Inbound traffic can >>>>>be trickier - a few networks will publish action communities or accept >>>>>MEDs, and we utilize those tools when available. Prepending is >>>>>occasionally useful. Announcing more specifics in certain spots always >>>>>felt like a jerk move so I tend to avoid that, but that's a thing I >>>>>suppose. >>>>> >>>>>-- >>>>>Colin Baker >>>>>SupraNet Communications, Inc. >>>>>(608) 572-7634 >>>>>colinb(a)supranet.net >>>>> >>>>>This message is subject to the SupraNet Email Confidentiality Policy >>>>>which is located at http://supranet.net/confidentiality >> >> >> >>-- >>Jay Hanke >>CTO >>Neutral Path Communications >>3 Civic Center Plaza, Suite 204 >>Mankato, MN 56001 >>(507) 327-2398 mobile >>jayhanke(a)neutralpath.net >>www.neutralpath.net

Oct. 30, 2017

Re: Multi-IX Traffic Management?

by Jason Hanke

As a transport seller, We're seeing most networks most commonly buy linear paths to two or more IXes. Typically going opposite directions from their home base. We're also starting to see smaller operators exclusively doing remote peering. A typical set up is buying identical transport to the 2 adjacent IX buildings. On wavelengths, the link failure can be passed back to from end to end. Granted, a long-haul wavelength isn't as stable as a cross connect. If a network doesn't have any remote gear the convergence is governed by EBGP vs IBGP/IGP if there is an additional router. For remote setups, in particular with Ethernet transport we're seeing more and more BFD on the sessions. Jay On Mon, Oct 30, 2017 at 8:27 AM, Michael Hare <michael.hare(a)wisc.edu> wrote: > For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability? > > -Michael > >>>-----Original Message----- >>>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>>Of Colin Baker >>>Sent: Friday, October 27, 2017 3:29 PM >>>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>>Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? >>> >>>On 27.10.2017 15:01, Matthew Beckwell wrote: >>>> For my own curiosity: >>>> >>>> I've seen quite a few networks in the last year or so connecting >>>> themselves to other out-of-market IX's (I presume getting some cheap >>>> wavelengths and bringing it back to Minnesota). >>>> >>>> A couple of questions for those of you that are doing this: >>>> >>>> 1. I'm assuming this is enabling you to accommodate requirements from >>>> other networks that require peering at multiple locations, even though >>>> technically your network doesn't really extend that far (or maybe it's >>>> something else?). >>> >>>For us, the goal is to reach networks which are not on exchanges which >>>we're already connected. The big ones are at MICE, but there are other >>>exchanges we can get to reasonably inexpensively that have such a large >>>number of networks connected that it makes sense financially (compared >>>to buying more transit). Traffic from a huge number of ASNs, >>>particularly if most are on a route server, can really add up. >>> >>>Peering in multiple locations is a nice benefit to this, but usually not >>>a primary goal for us. >>> >>>> 2. How are folks monitoring, managing, or manipulating this >>>> out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some >>>> other mechanism to "prefer" your traffic enter and exit closer to >>>> where you really are?) >>> >>>If you haven't already, install as-stats >>>(https://github.com/manuelkasper/AS-Stats) to monitor things, which will >>>make it clear if something is out of whack. In most cases, our IGP >>>handles outbound traffic in a way the makes sense. Inbound traffic can >>>be trickier - a few networks will publish action communities or accept >>>MEDs, and we utilize those tools when available. Prepending is >>>occasionally useful. Announcing more specifics in certain spots always >>>felt like a jerk move so I tend to avoid that, but that's a thing I >>>suppose. >>> >>>-- >>>Colin Baker >>>SupraNet Communications, Inc. >>>(608) 572-7634 >>>colinb(a)supranet.net >>> >>>This message is subject to the SupraNet Email Confidentiality Policy >>>which is located at http://supranet.net/confidentiality -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Oct. 30, 2017

Re: Multi-IX Traffic Management?

by Jeremy Lumby

-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Michael Hare Sent: Monday, October 30, 2017 8:27 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability? -Michael >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Colin Baker >>Sent: Friday, October 27, 2017 3:29 PM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? >> >>On 27.10.2017 15:01, Matthew Beckwell wrote: >>> For my own curiosity: >>> >>> I've seen quite a few networks in the last year or so connecting >>> themselves to other out-of-market IX's (I presume getting some cheap >>> wavelengths and bringing it back to Minnesota). >>> >>> A couple of questions for those of you that are doing this: >>> >>> 1. I'm assuming this is enabling you to accommodate requirements from >>> other networks that require peering at multiple locations, even though >>> technically your network doesn't really extend that far (or maybe it's >>> something else?). >> >>For us, the goal is to reach networks which are not on exchanges which >>we're already connected. The big ones are at MICE, but there are other >>exchanges we can get to reasonably inexpensively that have such a large >>number of networks connected that it makes sense financially (compared >>to buying more transit). Traffic from a huge number of ASNs, >>particularly if most are on a route server, can really add up. >> >>Peering in multiple locations is a nice benefit to this, but usually not >>a primary goal for us. >> >>> 2. How are folks monitoring, managing, or manipulating this >>> out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some >>> other mechanism to "prefer" your traffic enter and exit closer to >>> where you really are?) >> >>If you haven't already, install as-stats >>(https://github.com/manuelkasper/AS-Stats) to monitor things, which will >>make it clear if something is out of whack. In most cases, our IGP >>handles outbound traffic in a way the makes sense. Inbound traffic can >>be trickier - a few networks will publish action communities or accept >>MEDs, and we utilize those tools when available. Prepending is >>occasionally useful. Announcing more specifics in certain spots always >>felt like a jerk move so I tend to avoid that, but that's a thing I >>suppose. >> >>-- >>Colin Baker >>SupraNet Communications, Inc. >>(608) 572-7634 >>colinb(a)supranet.net >> >>This message is subject to the SupraNet Email Confidentiality Policy >>which is located at http://supranet.net/confidentiality

Oct. 30, 2017

Re: Multi-IX Traffic Management?

by Michael Hare

For those buying wavelengths (compared to an mpls circuit), are you buying diverse transport? If not, how are you dealing with convergence stability? -Michael >>-----Original Message----- >>From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf >>Of Colin Baker >>Sent: Friday, October 27, 2017 3:29 PM >>To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >>Subject: Re: [MICE-DISCUSS] Multi-IX Traffic Management? >> >>On 27.10.2017 15:01, Matthew Beckwell wrote: >>> For my own curiosity: >>> >>> I've seen quite a few networks in the last year or so connecting >>> themselves to other out-of-market IX's (I presume getting some cheap >>> wavelengths and bringing it back to Minnesota). >>> >>> A couple of questions for those of you that are doing this: >>> >>> 1. I'm assuming this is enabling you to accommodate requirements from >>> other networks that require peering at multiple locations, even though >>> technically your network doesn't really extend that far (or maybe it's >>> something else?). >> >>For us, the goal is to reach networks which are not on exchanges which >>we're already connected. The big ones are at MICE, but there are other >>exchanges we can get to reasonably inexpensively that have such a large >>number of networks connected that it makes sense financially (compared >>to buying more transit). Traffic from a huge number of ASNs, >>particularly if most are on a route server, can really add up. >> >>Peering in multiple locations is a nice benefit to this, but usually not >>a primary goal for us. >> >>> 2. How are folks monitoring, managing, or manipulating this >>> out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some >>> other mechanism to "prefer" your traffic enter and exit closer to >>> where you really are?) >> >>If you haven't already, install as-stats >>(https://github.com/manuelkasper/AS-Stats) to monitor things, which will >>make it clear if something is out of whack. In most cases, our IGP >>handles outbound traffic in a way the makes sense. Inbound traffic can >>be trickier - a few networks will publish action communities or accept >>MEDs, and we utilize those tools when available. Prepending is >>occasionally useful. Announcing more specifics in certain spots always >>felt like a jerk move so I tend to avoid that, but that's a thing I >>suppose. >> >>-- >>Colin Baker >>SupraNet Communications, Inc. >>(608) 572-7634 >>colinb(a)supranet.net >> >>This message is subject to the SupraNet Email Confidentiality Policy >>which is located at http://supranet.net/confidentiality

Oct. 30, 2017

Re: Multi-IX Traffic Management?

by Richard Laager

I peer at SIX only with networks that I can't peer with at MICE. If a network is at both, I filter out their routes from SIX's route server and I ask the SIX route server to not propagate my routes to them (using the relevant communities). -- Richard

Oct. 27, 2017

Re: Multi-IX Traffic Management?

by Jeremy Lumby

Matthew, My few cents on your questions: Since there are extension switches for some of these other IXes in Minneapolis, often times the cost is far less than the wavelength, a term contract is not even needed, and by having multiple people sharing the cost of an extension, sometimes it can cover the cost of diverse waves (I have been surprised at how often a wave will go down, when you have thousands of miles of fiber, a lot can go wrong) I personally use prepends, and MEDs to manipulate traffic paths. Many of the CDNs have complicated algorithms that go far beyond BGP, and factor in things like load on the cache, geo-location of the IP, and latency. As for the Chicago problem, I still see my upstreams all trying to take me there. This gives me an option to go the other direction. After the diverse content, the second reason I peer at other exchanges is I have a direct relationship with the peer, so I cutout the delays/red tape of working with an upstream to resolve the issue. Not to mention I can avoid if the upstream gets into a pissing match about who they will/won't peer with, and it causes their links to saturate. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Friday, October 27, 2017 3:02 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Multi-IX Traffic Management? For my own curiosity: I've seen quite a few networks in the last year or so connecting themselves to other out-of-market IX's (I presume getting some cheap wavelengths and bringing it back to Minnesota). A couple of questions for those of you that are doing this: 1. I'm assuming this is enabling you to accommodate requirements from other networks that require peering at multiple locations, even though technically your network doesn't really extend that far (or maybe it's something else?). 2. How are folks monitoring, managing, or manipulating this out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some other mechanism to "prefer" your traffic enter and exit closer to where you really are?) I've seen some funny business with these kinds of network arrangements. Google DNS responses coming from SIX......CloudFlare cache response coming from Omaha IX. It seems like it's "the Chicago problem" (that MICE was founded to avoid) all over again-- except this time around we're doing it with private links all over the country instead of public transit? Just curious what people have found to be best practices with this type of arrangement. ~Matthew To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Oct. 27, 2017

Re: Multi-IX Traffic Management?

by Colin Baker

On 27.10.2017 15:01, Matthew Beckwell wrote: > For my own curiosity: > > I've seen quite a few networks in the last year or so connecting > themselves to other out-of-market IX's (I presume getting some cheap > wavelengths and bringing it back to Minnesota). > > A couple of questions for those of you that are doing this: > > 1. I'm assuming this is enabling you to accommodate requirements from > other networks that require peering at multiple locations, even though > technically your network doesn't really extend that far (or maybe it's > something else?). For us, the goal is to reach networks which are not on exchanges which we're already connected. The big ones are at MICE, but there are other exchanges we can get to reasonably inexpensively that have such a large number of networks connected that it makes sense financially (compared to buying more transit). Traffic from a huge number of ASNs, particularly if most are on a route server, can really add up. Peering in multiple locations is a nice benefit to this, but usually not a primary goal for us. > 2. How are folks monitoring, managing, or manipulating this > out-of-market traffic? (Using BGP MED, Prepending, Localpref, or some > other mechanism to "prefer" your traffic enter and exit closer to > where you really are?) If you haven't already, install as-stats (https://github.com/manuelkasper/AS-Stats) to monitor things, which will make it clear if something is out of whack. In most cases, our IGP handles outbound traffic in a way the makes sense. Inbound traffic can be trickier - a few networks will publish action communities or accept MEDs, and we utilize those tools when available. Prepending is occasionally useful. Announcing more specifics in certain spots always felt like a jerk move so I tend to avoid that, but that's a thing I suppose. -- Colin Baker SupraNet Communications, Inc. (608) 572-7634 colinb(a)supranet.net This message is subject to the SupraNet Email Confidentiality Policy which is located at http://supranet.net/confidentiality

Oct. 27, 2017

Re: Please Welcome Omni-Tech to MICE!

by Doug McIntyre

On Fri, Oct 27, 2017 at 02:40:19PM -0500, Jason Hanke wrote: >Company Name : Omni-Tech Inc. >ASN: 54821 >IPv4: 206.108.255.90/24 >IPv6: 2001:504:27:0:0:d625::1/64 >Port: Neutral Path Remote port ge-0/0/11 >BFD Support: NO >Route Servers: YES >Peering Contact email: peering(a)omni-tech.net > All set on the route servers for Omni-Tech, Welcome! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Oct. 27, 2017