- mice-discuss - lists.micemn.net

Re: Prefix count

by Doug McIntyre

On Sun, Aug 21, 2016 at 04:50:17PM +0000, Frank Bulk wrote: > The Cacti graph (http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=236…) doesn't show that ... but is there any further confirmation that there was a short burst in routes advertised by the RR's? Hmm, interesting, the BIRD servers are configured for a max prefix count of 1024 on IPv4, and there are a few alarm log entries at 10:21AM & 10:22AMshowing the prefix count going above for a specific session, and a statement that their session was shutdown and restarted. I'd have thought there wouldn't be leakage beyond the config'd limit, but apparently there is? -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 22, 2016

Prefix count

by Frank Bulk

We had a max IPv4 prefix count of 2000 with a warning when it hit 90%. At 10:20 am our BGP session was torn down when it apparently exceeded 2000: Aug 18 14:09:18 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 18 14:09:18 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 18 14:14:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 18 14:14:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 18 14:19:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 18 14:19:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 18 14:24:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 18 14:24:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 18 14:29:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 18 14:29:17 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 21 06:12:20 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 21 06:12:50 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 21 06:17:19 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 21 06:17:50 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 21 06:22:19 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 21 06:27:19 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 21 06:27:50 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 21 06:32:19 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1: exceeds warning limit 1800 Aug 21 06:32:50 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2: exceeds warning limit 1800 Aug 21 10:20:10 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.1 exceeds maximum prefix limit...shutdown Aug 21 10:20:10 192.168.0.85 BGP: Peer 206.108.255.1 DOWN (Peer had exceeded the prefix limit) Aug 21 10:20:11 192.168.0.85 BGP: No. of prefix received from BGP Peer 206.108.255.2 exceeds maximum prefix limit...shutdown Aug 21 10:20:11 192.168.0.85 BGP: Peer 206.108.255.2 DOWN (Peer had exceeded the prefix limit) The Cacti graph (http://micelg.usinternet.com/cacti/graph.php?action=view&local_graph_id=236…) doesn't show that ... but is there any further confirmation that there was a short burst in routes advertised by the RR's? Frank

Aug. 21, 2016

Re: MICE gets a shout-out in CloudFlare blog

by Mike Horwath

On Fri, Aug 19, 2016 at 04:18:35AM +0000, Frank Bulk wrote: > https://blog.cloudflare.com/bandwidth-costs-around-the-world/ Pretty neat overall. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

Aug. 19, 2016

MICE gets a shout-out in CloudFlare blog

by Frank Bulk

https://blog.cloudflare.com/bandwidth-costs-around-the-world/ Frank

Aug. 19, 2016

MICE gets a shoutout in this Cloudflare blog -- as one of the good guys, amongst many bad guys

by Mike O'Connor

we get a nice mention down at the end of the "North America" section of this (pretty grouchy) piece about international bandwidth costs. i hope this doesn't violate list-norms -- apologies in advance if it does... mike CloudFlare protects over 4 million websites using our global network <https://cloudflare.com/network-map> which spans 86 cities across 45 countries. Running this network give us a unique vantage point to track the evolving cost of bandwidth around the world. CC BY-SA 2.0 <https://creativecommons.org/licenses/by-sa/2.0/> image <https://www.flickr.com/photos/53326337@N00/4877664667> by Quinn Dombrowski <https://www.flickr.com/photos/quinnanya/> Recap Two years ago, we previewed the relative cost of bandwidth <https://blog.cloudflare.com/the-relative-cost-of-bandwidth-around-the-world/> that we see in different parts of the world. Bandwidth is the largest recurring cost of providing our service. Compared with Europe and North America, there were considerably higher Internet costs in Australia, Asia and Latin America. Even while bandwidth costs tend to trend down over time <https://www.telegeography.com/press/press-releases/2015/09/09/ip-transit-pr…>, driven by competition and decreases in the costs of underlying hardware, we thought it might be interesting to provide an update. Since August 2014, we have tripled the number of our data centers from 28 to 86, with more to come. CloudFlare hardware is also deployed in new regions such as the Middle East and Africa. Our network spans multiple countries in each continent, and, sometimes, multiple cities in each country. Traffic across 86 data centers in the CloudFlare network There are approximately thirteen networks called “Tier 1 networks” (e.g., Telia, GTT, Tata, Cogent) who sell “transit” to access any of thousands of other networks on the Internet using their global backbones, including networks who are not their customers. We connect to networks by either purchasing transit from a global "Tier 1 network" <http://research.dyn.com/2016/04/a-bakers-dozen-2015-edition/> (or major regional network), or by exchanging traffic directly with a carrier or ISP using “peering”. Typically, peered traffic is exchanged without settlement between the peered parties. We try to make it as easy as possible for networks to interconnect with us. CloudFlare has an “open peering” policy, and participates at nearly 150 internet exchanges <http://bgp.he.net/report/exchanges#_participants>, more than any other company. As a benchmark, let's assume the cost of transit in Europe and North America is 10 units (per Mbps). With that benchmark in place, without disclosing exact pricing, we can compare regions by transit cost, percentage of peering, and their effective blended cost (transit + peering). Europe Europe Transit vs Peering (Last 30 Days) Based on our benchmark, the transit cost is 10 units. The region has a large number of Internet exchanges, typically non-profit, where we peer around 60% of our traffic. This makes for an effective regional cost of 4 units. With perhaps the notable exception of the incumbent in Germany, many networks are supportive of open interconnection. CloudFlare already participates at 40 European internet exchanges <https://www.peeringdb.com/net/4224>, and is in the process of joining at least five more. North America North America Transit vs Peering (Last 30 Days) The cost of transit in North America is equal to the cost in Europe, or 10 units. We peer around 40% of our traffic, resulting in an effective regional cost of 6 units. The level of peering in North America is less than in Europe, but a significant improvement over two years ago. The share of peered traffic is expected to grow. Some material changes have occurred and are occurring in the North American market, such as Frontier acquiring Verizon FiOS customers <http://internet.frontier.com/fios-network-acquisition/> in three U.S. States and Charter preparing to merge with Time Warner Cable <http://ir.charter.com/phoenix.zhtml?c=112298&p=irol-newsArticle&ID=2053012>. We can see these changes making an impact to the regional interconnection landscape <http://arstechnica.com/tech-policy/2016/04/doj-fcc-chairman-ok-chartertime-…>. Notably, our peering has particularly grown in smaller regional locations, closer to the end visitor, leading to an improvement in performance. This could be through private peering, or via an interconnection point such as the Midwest Internet Cooperative Exchange (MICE) <http://www.micemn.net/> in Minneapolis. Africa Africa Transit vs Peering (Last 30 Days) Transit prices in Africa are amongst the highest in the world at 14 times the benchmark or 140 units, with notable variance across the continent, from Cairo <https://blog.cloudflare.com/cairo/> to Mombasa <https://blog.cloudflare.com/mombasa-kenya-cloudflares-43rd-data-center/> to Johannesburg <https://blog.cloudflare.com/johannesburg-cloudflares-30th-data-center/>. Fortunately, of the traffic that we are currently able to serve locally in Africa, we manage to peer about 90% (with a mix of carriers and ISPs), making for an effective cost of 14 units. Our African deployments help us avoid the significant latency of serving websites from London, Paris or Marseille. A particularly promising but challenging region where we hope to deploy a CloudFlare data center is West Africa - specifically Nigeria, which is already at just under 100 million Internet users <http://qz.com/658762/there-arent-as-many-nigerians-on-the-mobile-internet-a…>. Middle East Middle East Transit vs Peering (Last 30 Days) CloudFlare currently has four data centers in the Middle East, each of which are cache deployments with strategic ISP partners <https://blog.cloudflare.com/middle-east-expansion/> to serve their respective customers. We are able to peer all the traffic currently served from these data centers. While these collectively provide significant coverage, there is additional traffic (reaching Europe) that we would like to localize in the region. We hope that the remaining ISPs, such as Saudi Telecom Company, deploy similar caches, and enhance the performance of their customers. Because we can peer 100% of our traffic in the Middle East, our effective pricing for bandwidth in the region is 0 units. There are, of course, other costs to delivering our service beyond bandwidth. However, by driving up peering rates in the Middle East we’ve been able to make our service in the Middle East extremely cost competitive. Asia Asia Transit vs Peering (Last 30 Days) In Asia (excluding the Middle East), transit costs 7 times times the benchmark, or 70 units. However, we peer about 60% of our traffic, resulting in an effective cost of 28 units. Beyond the major meeting points in Hong Kong, Singapore and Tokyo, a significant portion of our interconnection is localized to take place closer to visitors in cities such as Bangkok <https://blog.cloudflare.com/bangkok/>, Chennai <https://blog.cloudflare.com/cloudflare-launches-in-india-with-data-centers-…>, Kuala Lumpur <https://blog.cloudflare.com/kuala-lumpur-malaysia-cloudflares-45th-data-cen…>, Mumbai <https://blog.cloudflare.com/cloudflare-launches-in-india-with-data-centers-…>, Osaka <https://blog.cloudflare.com/osaka-data-center/>, New Delhi <https://blog.cloudflare.com/cloudflare-launches-in-india-with-data-centers-…>, Seoul <https://blog.cloudflare.com/seoul-korea-cloudflares-23rd-data-center/>, and Taipei <https://blog.cloudflare.com/taipei>. These statistics do not include our network of strategically located data centers inside of mainland China <https://www.cloudflare.com/china>, where the dynamics of interconnection are entirely unique. Two Asian locations stand out as being especially expensive: Seoul and Taipei. In these markets, with powerful incumbents (Korea Telecom and HiNet), transit costs 15x as much as in Europe or North America, or 150 units. South Korea is perhaps the only country in the world where bandwidth costs are going up. This may be driven by new regulations from the Ministry of Science, ICT and Future Planning <http://english.msip.go.kr/>, which mandate the commercial terms of domestic interconnection, based on predetermined “Tiers” of participating networks. This is contrary to the model in most parts of the world, where networks self-regulate, and often peer without settlement. The government even prescribes the rate at which prices should decrease per year (-7.5%), which is significantly slower than the annual drop in unit bandwidth costs elsewhere in the world. We are only able to peer 2% of our traffic in South Korea. If you include HiNet and Korea Telecom in our blended bandwidth pricing, and take into account peering, our effective price is 28 units. If you exclude HiNet and Korea Telecom, our effective price is 14 units. South America South America Transit vs Peering (Last 30 Days) Transit prices in South America are very high, costing 17 times the benchmark, or 170 units. We peer about 60% of traffic in South America, making for an effective cost of 68 units. One of the reasons that transit prices are high is that the Tier 1 networks which are newer entrants to this region have yet to pick up significant market share. While markets such as Brazil are less expensive and have greater peering, costs are highest in countries such as Peru and Argentina where, in each, a single incumbent provider, respectively Telefonica and Telecom Argentina, controls access for the last mile delivery of content to the majority of Internet users. As we try to increase our share of peered traffic, one of the challenges we face is that many Internet exchanges (e.g., NAP Colombia) only permit domestically incorporated and licensed networks to publicly peer, or in another case, require a unanimous vote of all members on an IX to permit a new participant, effectively creating a separation between “international content” and “domestic content”. If you include Telecom Argentina and Telefonica, our blended cost of bandwidth in South America is 68 units. If you exclude these two providers then our blended cost is 17 units. Oceania Oceania Transit vs Peering (Last 30 Days) Transit prices in Oceania (Australia and New Zealand) are lower than they used to be, but continue to be extremely high in relative terms, costing 17 times the benchmark from Europe, or 170 units. We peer 50% of our traffic, resulting in an effective cost of 85 units. If you exclude Optus and Telstra, then the price falls to 17 units — because we peer with nearly everyone else. Six Expensive Networks Relative Cost of Bandwidth CloudFlare has always optimized where we serve customers to take into account our effective costs. If you are a free customer using an excessive amount of expensive transit, we would serve you from fewer regions. The good news is that, over the last five years, we’ve been able to negotiate reasonable transit pricing or settlement-free peering with the vast majority of the world’s networks. That allows us to continue to provide the free version of our service as well as to keep prices low for all our paid services. Today, however, there are six expensive networks (HiNet, Korea Telecom, Optus, Telecom Argentina, Telefonica, Telstra) that are more than an order of magnitude more expensive than other bandwidth providers around the globe and refuse to discuss local peering relationships. To give you a sense, these six networks represent less than 6% of the traffic but nearly 50% of our bandwidth costs. While we’ve tried to engage all these providers to reduce their extremely high costs and ensure that even our Free customers can be served across their networks, we’ve hit an impasse. To that end, unfortunately, we’ve made the decision that the only thing that will change these providers’ pricing is to make it clear how out of step they are with the rest of the world. To demonstrate this, we’ve moved our Free customers off these six transit providers. Free customers will still be accessible across our network and served from another regional cache with more reasonable bandwidth pricing. Ironically, this actually increases the cost to several of these providers because they now need to backhaul traffic to another CloudFlare data center and pay more in the process. For instance, if Telstra were to peer with CloudFlare then they would only have to move traffic over about 30 meters of fiber optic cable between our adjoining cages in the same data center. Now Telstra will need to backhaul traffic to Free customers to Los Angeles or Singapore over expensive undersea cables. Their behavior is irrational in any competitive market and so it is not a surprise that each of these providers is a relative monopolist in their home market. If you’re a Free CloudFlare customer who cares about optimizing the best possible performance from one of these six providers then we encourage you to reach out to them and encourage them to follow a core principle of a free and open Internet and not abuse their monopoly position. We are committed to serving all our customers across every network that peers with us. To that end, help us convince these six networks to be on the right side of a free and open Internet by reaching out to your ISP. Ask HiNet to peer with CloudFlare in Taipei <http://service.hinet.net/2004/ncsc/index.htm> Ask Korea Telecom to peer with CloudFlare in Seoul <http://www.kt.com/eng/etc/contact.jsp> Ask Optus to peer with CloudFlare across Australia <http://www.optus.com.au/shop/support/answer/complaints-compliments?requestT…> Ask Telecom Argentina to peer with CloudFlare in Buenos Aires <http://www.telecom.com.ar/hogares/gestion_libro.htm> Ask Telefonica to peer with CloudFlare across South America <https://www.telefonica.com/en/web/press-office/contact-us> Ask Telstra to peer with CloudFlare across Australia <https://say.telstra.com.au/customer/general/forms/Email-Complaint> We’ll post updates as we negotiate with these six networks and are hopeful that we’ll soon be able to serve all our customers across all the networks we interconnect with. WI:(715) 598-4284 MN:(651) 647-6109 Toll Free:(800) 896-0907 Fax:(866)-280-2356

Aug. 18, 2016

Re: Stealth Broadband Updated Information

by Doug McIntyre

On Wed, Aug 17, 2016 at 10:25:54AM -0500, Jason Hanke wrote: > Stealth Broadband has received their ASN from ARIN, > > Please update ASN and IPv6 information in the route-servers and website: > > Name: Stealth Broadband > IPv4: 206.108.255.69/24 > IPv6: 2001:504:27:0:6:07BA::1/64 > ASN: 395194 > Port: NP MSP1 Remote ge-0/0/2 > Contact: james.urwiller(a)stealthbroadband.net Sure, the route servers are all set to go for them. Welcome aboard! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 17, 2016

Stealth Broadband Updated Information

by Jason Hanke

Stealth Broadband has received their ASN from ARIN, Please update ASN and IPv6 information in the route-servers and website: Name: Stealth Broadband IPv4: 206.108.255.69/24 IPv6: 2001:504:27:0:6:07BA::1/64 ASN: 395194 Port: NP MSP1 Remote ge-0/0/2 Contact: james.urwiller(a)stealthbroadband.net Thanks! -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Aug. 17, 2016

Re: ipHouse Remote Switch

by Doug McIntyre

On Thu, Aug 11, 2016 at 03:38:27PM -0500, Jason Hanke wrote: > Is there any update on the ipHouse remote switch? We're still interested, but there has been some snafu with equipment going on that we're trying to sort out first. Thanks. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 11, 2016

ipHouse Remote Switch

by Jason Hanke

Is there any update on the ipHouse remote switch? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

Aug. 11, 2016

Airstream Communications Maintenance 2016/08/11

by Charles Monson

Good Afternoon, Airstream Communications (AS11796) will be performing maintenance tonight 2016/08/11 between the hours of Midnight and 6am. We will be shutting down our BGP sessions for approximately 20 minutes to reroute traffic while work is done and turning them back up when complete. If your session to us has not returned by 6am please send an email to support(a)wins.net<mailto:support@wins.net> with your BGP AS number and we will address it as soon as possible. Charles Monson Associate Network Administrator, Airstream Communications (AS11796)

Aug. 10, 2016