At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis. For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused me some issues. For example, latency-sensitive gaming traffic was tromboning Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather than Wiktel-Chicago-Wiktel. This is the biggest issue I am concerned about. Layer-2 can't tell us how far away the peer is. BGP doesn't have knobs for varying latency across peers on the same Layer-2. The SIX example of really-remote
Jonathan, Very excellent points. Here are some reply thoughts to consider. 1. the member has their own private, long distance, layer 2 link to an IX. The member themself knows it's a long distance but most of the other members of the IX probably will not have any idea, unless they happen to infer it from latency to your router or you otherwise disclose it. 2. if the IX does a long distance layer 2 link to an official/remote switch, it would be a lot more clear to people at a glance that members on such a far away remote switch are indeed far away As suggested in an earlier post of this thread, BGP communities could be added on the route server sessions of members marking the member's routes as originating via switch X. Then you as a fellow member could choose to adjust your BGP knobs based on that data, maybe you want to not take their routes or lower your local pref for them specifically and even choose to AS prepend your routes to them or not advertise to them at all. With members individually connecting a long distance, only the members themselves are respectively aware of the fact that they are far away. For those not using route servers, they can still easily gauge the member distances if they are clearly documented as connected to switches classed as far away switches and therefore can make their own routing decisions based on that knowledge, which may include not peering with distant members at all at said IX. If geographic distance is a concern for connected/connecting members, maybe the conversation should be had on whether remote switches or individual members can be connecting from such a distance greater than XYZ distance or perhaps XYZ avg latency, as you mentioned. But now, what if I setup a router in a nearby geographic location with low latency but only have a single router and a single connection back to my main established network that is far away? I as the member look to be close to the fabric and indeed am close to the fabric, yet most of my eyeballs and/or servers are still far away, in this hypothetical situation. Though I do think, as you imply, there is a point of diminishing returns by adding more and more connectivity at increasing distances, not just in target market / brand but also in the very real operational issues that can arise from two networks taking a highly inefficient path via an IX because it is shorter AS hops than usual transit links but is much longer geographically. I have seen this happen first hand. ________________________________ From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of Jonathan Stewart <jonathan@LES.NET> Sent: Tuesday, April 5, 2022 4:22 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Remote Switch Policy A bit late to the discussion, but here are my thoughts: On Thu., 2022-03-24 3:58 p.m., Richard Laager wrote: peers, with resultant latency higher than Transit, is a performance degradation for SIX and their members, and reduces the quality of SIX, I'd argue. I would suggest that MICE have a limit, say 5 ms RTT, on latency to remote switches. Then we can say that MICE is in a geographic region, not potentially a pan-North-America layer-2. Part of MICE's identity should be its location, and the resultant low-latency across MICE itself. If MICE is in Kansas City, and KCIX is in Minneapolis, what's the difference between them? Why do they both exist if they serve the same datacentres in the same way: a Layer-2 ethernet fabric? I become confused at what problem is being solved when IXes start overlapping and becoming non-geographic, rather than limited to a metro area, or certain radius around a hub. I think there's a difference between an individual member (me) arriving over a long layer-2 link (from Winnipeg) to the IX, and the IX doing long layer-2 links themselves. The difference is that I have knowledge of and control over my long-haul links, but if the IX does it, unexpectedly long-haul paths happen without my decision to use them. Finally, I'm a big fan of what MICE has accomplished so far, I hope this message doesn't read as being overly critical. Cheers, Jonathan AS18451 - LES.NET Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA