- mice-discuss - lists.micemn.net

RPKI deployment survey
by Richard Laager March 17, 2025

March 17, 2025

Two professors, one from Tufts and one from Georgia Tech, are doing a survey of RPKI deployments. They are "working on a research project looking at barriers to RPKI adoption and the experience of different stakeholders involved in RPKI to inform strategies to help promote further adoption." I spoke to them at length about my experience with MICE's IRR/RPKI project, and some with my Wiktel hat too. If any of you are interested in speaking with them, please contact: Josephine Wolff <Josephine.Wolff(a)tufts.edu> Cecilia Testart <ctestart(a)gatech.edu> -- Richard

2 2

Round 5 - MICE Switch-Cabinet Migration
by Jeremy Lumby March 6, 2025

March 6, 2025

On Wednesday March 5th at approximately 1PM CST I will be moving the Arista 7504R (100Gig core switch) into the new cabinet, along with some remaining 10G connections on the switch, and all other equipment in cabinet 937. Prior to the migration BGP will be blocked on the specific participant's port to allow traffic to gracefully reroute. This is the fifth round of weekly moves and should complete migrating everything out of the existing cabinet. During this maintenance software on the 7504R switch, as well as route server 1 will be updated. There will be one final maintenance on the afternoon of Wednesday March 19 to move connections off of temp switches in the new cabinet. If you are interested in upgrading to 400Gig let us know ASAP so we can avoid double moves, and facilitate an easier transition for everyone. The full details of the move are: Virtual Host 1 (includes MICE Webserver, DNS server, IXP Manager, Route Server 1) Management Firewall Arista 7504R: Essentia - 3789 Old Ports: Et4/34 Et4/40 New Ports: E 4/42 E 4/41 30110 - Range Old Port: Et4/42 New Port: E 4/45 Management to Remote Switches Old Port: Et4/43 New Port: E 4/48 Yahoo 10310 Old Ports: Et4/44 Et4/48 New Ports: E 4/43 E 4/44 Virtual Host 1 Old Port: Et4/45 New Port: E 4/47 Xiber LLC 393950 Old Port: Et4/46 New Port: E 4/46 Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com

1 3

MICE Server Updates
by Cameron Skappel Feb. 27, 2025

Feb. 27, 2025

I plan to update the MICE servers and firewalls Wednesday February 26th at 13:00 Central and Thursday February 27th at 13:00 Central. I will do one route server per day. -- Cameron Skappel 218-436-5226 Wikstrom Telephone Company

1 2

UG 46 Meeting & Election
by Richard Laager Feb. 26, 2025

Feb. 26, 2025

The MICE UG 46 meeting will be held Wednesday, October 9, 3:00 pm at UMN and via Zoom. Parking, etc. details same as before: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 * * *Zoom Details* https://umn.zoom.us/j/97601658469?pwd=C6JWkmoYfB9WjZsnRPr9S5u1l4Zkuz.1 ID: 94910223662, passcode: Jp74Et 651-372-8299, meeting ID 97601658469, passcode 864193 * * *Nominations* We will be holding an election for a board seat. The board consists of three seats with staggered terms, such that one seat comes up for re-election each year (at the fall meeting). This year, Reid's seat is up for election. As in the past, we will close nominations in advance of the meeting so people have a chance to instruct a proxy appropriately. *Nominations will be open for two weeks, September 23 - October 6. *I plan to send reminder emails approximately at the start of and just after the nomination period. In the event that the election is uncontested, our practice is to elect by acclimation at the meeting. Proxies are unnecessary in that situation, assuming that the meeting makes quorum. *Proxies * The bylaws state that you send proxies to the Chief Manager (me, Richard Laager), and electronic means are acceptable. So a simple email is sufficient: rlaager(a)wiktel.com or chiefmanager(a)micemn.net. If you send me a /vote/ by email (e.g. "We vote for <X>."), I will treat it as an a proxy appointment, appointing me to vote on your behalf, but restricted to /only/ the election item and to /only/ vote for the candidate you expressed. If you send me a proxy with different instructions (e.g. "We appoint <Y> to vote for us."), obviously that will be honored as written. -- Richard

4 7

Round 2 - MICE Switch-Cabinet Migration
by Jeremy Lumby Feb. 26, 2025

Feb. 26, 2025

On Wednesday February 12th at approximately 1PM CST I will be migrating the following networks off of the Arista 7504R (100Gig core switch) to a temporary switch in the new cabinet/space that the 100G chassis will eventually be moved to at 511. Prior to the migration BGP will be blocked on the specific participant's port to allow traffic to gracefully reroute. This is the second round of weekly moves as everything is migrated out of the existing cabinet. If you are interested in upgrading to 400Gig let us know ASAP so we can avoid double moves, and facilitate an easier transition for everyone. Netflix - 2906: Old Ports: Po3 Et6/20/1 Et5/11/1 Temporary New Ports: Po3 E1/1 E2/1 Meta - 32934: Old Ports: Po27 Et5/12/1 Temporary New Ports: Po27 E3/1 Amazon - 16509: Old Ports: Po24 Et3/21/1 Et5/5/1 Et5/6/1 Et6/1/1 Temporary New Ports: Po24 E4/1 E5/1 E6/1 E7/1 CloudFlare - 13335: Old Ports: Po13 Et6/17/1 Temporary New Ports: Po13 Et8/1 G-Core Labs - 199524: Old Ports: Et3/30/1 Temporary New Ports: E9/1 Apple - 714: Old Ports: Po31 Et4/49/1 Temporary New Ports: E10/1 Inteliquent - 40160: Old Port: Et4/32 Temporary New Port: E33 Inteliquent - 5715: Old Port: Et4/5 Temporary New Port: E34 Hurricane Electric - 6939: Old Ports: Po6 Et3/5/1 Et6/5/1 Et6/18/1 Temporary New Ports: Po6 E11/1 E12/1 E13/1 Metronet - 30600: Old Ports: Po40 Et6/7/1 Temporary New Ports: Po40 E14/1 Cedar Falls Utilities - 13855: Old Port: Et6/16/1 Temporary New Port: E15/1 Savage Communications - 23260: Old Port: Et6/19/1 Temporary New Port: E16/1 Nuvera - 23465: Old Ports: Po29 Et5/8/1 Temporary New Ports: Po29 E17/1 Long Lines - 32867: Old Port: Et6/21/1: Temporary New Port: E18/1 Nsight - 3663: Old Port: Et6/22/1 Temporary New Port: E19/1 Bluepeak - 20412: Old Port: Et6/23/1 Temporary New Port: E20/1 UnitedHealth Group - 10879: Old Port: Et6/24/1 Temporary New Port: E21/1 Hennepin County - 40382: Old Port: Et6/25/1 Temporary New Port: E22/1 TDS - 4181: Old Port: Et6/27/1 Temporary New Port: E23/1 Truestream - 398143: Old Port: Et6/28/1 Temporary New Port: E24/1 HBC - 14828: Old Port: Et6/30/1 Temporary New Port: E25/1 SRT Communications - 14543: Old Port: Et6/33/1 Temporary New Port: E26/1 Let me know if you have any questions Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 eFax 952-873-7425 jlumby(a)mnvoip.com

1 10

Route server feed for bgp.tools
by Jonah Aragon Feb. 19, 2025

Feb. 19, 2025

Hello, I was wondering if MICE has considered delivering a live feed to bgp.tools's route collector. I find this service to be extremely useful, especially when it has access to live data from the networks I'm looking at, so it would be a great benefit to me and probably anyone else looking into MICE with this website. The listing for MICE indicates that the route server operators could contact them to set this up: https://bgp.tools/ixp/MICE Best, Jonah Triplebit – AS 401332

3 2

Acentek MI Disconnect
by Jay Hanke Feb. 17, 2025

Feb. 17, 2025

Acentek MI (AS11904) has disconnected from MICE. Please remove any remaining BGP sessions. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/

1 0

MICE Switch-Cabinet Migration Round 1
by Jeremy Lumby Feb. 5, 2025

Feb. 5, 2025

On Wednesday February 5th at approximately 1PM CST I will be migrating the following networks off of the Arista 7504R (100Gig core switch) to the new 7804R3 (400Gig core switch) at 511. Prior to the migration BGP will be blocked on the specific participant's port to allow traffic to gracefully reroute. This will be the first round of weekly moves as everything is migrated out of the existing cabinet. If you are interested in upgrading to 400Gig let us know ASAP so we can avoid double moves, and facilitate an easier transition for everyone. Ridgeview core switch uplink Old Ports: Port-channel10 Ethernet3/35/1 Ethernet3/36/1 Ethernet5/14/1 Ethernet5/15/1 Ethernet6/35/1 Ethernet6/36/1 New Ports: Port-channel3 Ethernet3/33/1 Ethernet3/34/1 Ethernet4/32/1 Ethernet4/33/1 Ethernet4/34/1 Ethernet4/36/1 MN VoIP extension switch uplink Old Ports: Port-channel15 Ethernet3/6/1 Ethernet6/6/1 Ethernet6/13/1 New Ports: Port-channel1 Ethernet3/1/1 (Native 400G port) Fiber Minnesota extension switch uplink Old Ports: Port-channel22 Ethernet3/7/1 Ethernet5/13/1 Ethernet5/16/1 Ethernet6/3/1 New Ports: Port-channel4 Ethernet3/2/1 (Native 400G port) Ethernet4/2/1 (Native 400G port) Akamai Old Ports: Port-channel19 Ethernet6/8/1 Ethernet6/10/1 Ethernet6/12/1 Ethernet6/31/1 Ethernet5/7/1 New Ports: Port-channel5 Ethernet3/3/1 Ethernet3/4/1 Ethernet3/5/1 Ethernet3/6/1 Ethernet3/7/1 Amazon Old Ports: Port-channel5 Ethernet3/12/1 Ethernet6/4/1 Ethernet5/3/1 Ethernet5/4/1 New Ports: Port-channel6 Ethernet3/8/1 Ethernet3/9/1 Ethernet3/10/1 Ethernet3/11/1 Apple Old Ports: Port-channel30 Ethernet 3/14/1 New Ports: Port-channel7 Ethernet 3/12/1 Arvig Old Ports: Port-channel42 Ethernet3/8/1 Ethernet6/11/1 Ethernet5/1/1 New Ports: Port-channel8 Ethernet3/13/1 Ethernet4/13/1 Ethernet4/14/1 Fastly Old Ports: Port-channel18 Ethernet4/50/1 New Ports: Port-channel9 Ethernet3/14/1 Meta Old Ports: Port-channel26 Ethernet3/10/1 New Ports: Port-channel10 Ethernet3/15/1 Microsoft Old Ports: Port-channel34 Ethernet3/18/1 New Ports: Port-channel11 Ethernet3/16/1 Netflix Old Ports: Port-channel2 Ethernet3/11/1 Ethernet3/27/1 New Ports: Port-channel12 Ethernet3/17/1 Ethernet3/18/1 Northern Lights GigaPOP Old Ports: Ethernet6/14/1 New Ports: Ethernet3/19/1 Qwilt Old Ports: Port-channel12 Ethernet6/29/1 Ethernet5/2/1 New Ports: Port-channel13 Ethernet3/20/1 Ethernet4/20/1 T-Mobile Old Ports: Port-channel41 Ethernet3/2/1 Ethernet6/9/1 Ethernet6/15/1 New Ports: Port-channel14 Ethernet3/21/1 Ethernet4/21/1 Ethernet4/22/1 Let me know if you have any questions Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 eFax 952-873-7425 jlumby(a)mnvoip.com

1 2

1G Disconnections
by Richard Laager Jan. 23, 2025

Jan. 23, 2025

The following participants were disconnected yesterday, as they were still on 1G and the 1G switch is going away as part of the cabinet move and 400G switch upgrade. Coloplast [AS25111] Evolve IP (fka Advanced Integrated Tech / AI Tech) [AS13746] ipHouse [AS7753] ipHouse was going away in any event, due to acquisition activity as previously discussed. The other two did confirm they were aware of the disconnect. They elected not to connect at 10G or move to a remote. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

1 0

2025-01-20 MICE Maintenance - Route Server 2
by Richard Laager Jan. 23, 2025

Jan. 23, 2025

MICE physical server 2 (and thus by extension route server 2) is again having errors on its peering interface. We will be performing maintenance tomorrow to address this. Since this only affects one of the two route servers, participants should not be impacted. Details: Sometime tomorrow (exact time depending on day job schedules), Jeremy and I will: 1. Apply the usual security updates to VMs and the host, since the host is going to be rebooted anyway. 2. Convert server 2's port to a LAG. This will make future changes easier, especially if this error problem continues. This will involve a reboot of the server. The reboot is "necessary" because Linux's netplan does not remove VLAN configuration when changes are made (only adds it). Without removing the old direct-on-the-physical-interface VLAN config, we could (would?) create a loop. You may recall that I created a loop that blew up the exchange in more-or-less that exact way when setting up the servers originally. While I could remove the VLAN configuration manually, in this case, it's easier and safer to simply reboot. That also ensures our running configuration is fully consistent with the configuration files. 3. Replace the optic, also moving to the other port on the NIC. The underlying failure manifests as a decrease in optical transmit power (i.e. power received at the switch) over time. We have replaced the optic previously, which fixes it, but then it fails again after a while. We are open to suggestions about why this might be occurring. One theory is that this is due to heat. We are going to try the second port in case that makes a difference. We are also considering switching to a direct attach cable, as perhaps that would produce less heat. -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

3 2

RE: [MICE-DISCUSS] MICE Server Updates
by nanog＠jima.us Dec. 18, 2024

Dec. 18, 2024

Cameron (or others), A cohort reached out to ask me... Does anyone know when the IPv6 address for micemn.net stopped responding? I foolishly didn't check after the migration; I see the v6 address is announced by the same ASN as the (working) v4 address, but attempts to trace die off after the penultimate hop (versus the v4 path). Happy Eyeballs would generally keep the site loading in browsers, assuming one has native IPv4/464XLAT, but it seems like a non-ideal state of affairs. Thank you! - Jima > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Cameron > Skappel > Sent: Monday, November 25, 2024 13:19 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] MICE Server Updates > > The first round of updates, which included Route Server 1, has been > completed. > > On 11/22/24 11:45, Cameron Skappel wrote: > > I plan to update the MICE servers and firewalls Monday November 25th > > at 13:00 Central and Tuesday November 26th at 14:30 Central. I will do > > one route server per day. > > > -- > Cameron Skappel > 218-436-5226 > Wikstrom Telephone Company

1 0

[MICE-DISCUSS] ipHouse MICE Disconnect
by Jay Hanke Dec. 18, 2024

Dec. 18, 2024

ipHouse is disconnecting from MICE. ipHouse is an OG member of MICE and has been a huge supporter over the years contributing to just about every aspect of the IX. They have contributed everything from donating equipment, transit, mailing list services and TONS of manual Bird config over the years. In particular, Doug McIntyre has contributed a huge amount of effort when MICE was just a pinkie. It would have been much more difficult without his help. Thanks! Jay

4 3

Please welcome Express Data Solutions LLC to MICE!
by Jeremy Lumby Dec. 13, 2024

Dec. 13, 2024

Name: Express Data Solutions LLC Switch: MICE MAIN Port: Ethernet4/2 ASN: 31834 IPv4: 206.108.255.183/24 IPv6: 2001:504:27:0:0:7C5A::1/64 Peering Contact: peering(a)express-ds.com Route Servers- Yes BFD: Yes Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

3 10

[MICE-DISCUSS] MICE Server Updates
by Cameron Skappel Nov. 26, 2024

Nov. 26, 2024

The second round of updates, which included Route Server 2, has been completed. -- Cameron Skappel 218-436-5226 Wikstrom Telephone Company

1 0

[MICE-DISCUSS] Re: MICE Mailing Lists
by Richard Laager Nov. 25, 2024

Nov. 25, 2024

This is a test of the new mice-discuss mailing list. On 2024-11-25 13:56, Richard Laager wrote: > MICE is migrating mailing lists to new infrastructure. > > ipHouse is winding down / spinning off certain operations. I want to > thank ipHouse for their support of MICE over many years. In > particular, I want to thank Doug McIntyre for years of work. > > The web interface will be available at https://lists.micemn.net and > the lists will be the same names but at the new domain, i.e. > mice-discuss(a)lists.micemn.net and mice-announce(a)lists.micemn.net. > > I am transferring all subscriptions. You will shortly receive an > invitation message. In theory, no action is necessary, and hopefully > this all just works. :) > > For those of you with digest subscriptions, I will manually change > your setting to digest after importing your address. So give me a few > minutes. > > The external provider will be importing the archives shortly. Again, > hopefully that works. It will not include this message, since this is > after I exported the archives. > > Also, the MICE board now have personal @micemn.net emails: > > Anthony Anderberg <ant(a)micemn.net> > Reid Fishler <rfishler(a)micemn.net> > Richard Laager <rlaager(a)micemn.net> > -- Richard Laager Chief Manager, Director Midwest Internet Cooperative Exchange LLC

3 3

MICE Server Updates
by Cameron Skappel Nov. 22, 2024

Nov. 22, 2024

I plan to update the MICE servers and firewalls Monday November 25th at 13:00 Central and Tuesday November 26th at 14:30 Central. I will do one route server per day. -- Cameron

1 0

netflix/tyson
by Jeff Wilde Nov. 18, 2024

Nov. 18, 2024

Anyone having issues with it streaming. I think its just the fight not Netflix. I can view other stuff. [A picture containing icon Description automatically generated] Jeff Wilde * Network Engineer * Park Region Office 218.826.6161 * Direct 218.826.8330 * Fax 218.826.6298

8 8

MICE 511 Server Maintenance
by Richard Laager Nov. 12, 2024

Nov. 12, 2024

In some tens of minutes, I will be switching MICE’s management infrastructure, including the website, to a new transit provider. This involves IP renumbering. If you have issues, drop me an email to my day job email here, or call/text me if you have my mobile number. -- Richard

1 1

Game Update?
by Anthony Anderberg Nov. 2, 2024

Nov. 2, 2024

Based on traffic patterns it looks like a big game franchise started an upgrade cycle last night at 3am. Looks like both Call of Duty and Fortnite have been planning upgrades but I can’t find anything definitive. I really only play Kerbal Space Program and am not up to date on FPS games, can anyone confirm?

4 3

Global Net (AS395466) Leaving MICE
by Global Net NOC Oct. 25, 2024

Oct. 25, 2024

Greetings – Unfortunately, due to ongoing issues with our transport into Minneapolis, Global Net will be leaving MICE on Thursday, October 31st, 2024. We plan to begin turning down BGP peering sessions on Monday, October 28th beginning at approximately 0500a CDT. Thank you to all who have established peering sessions with us, and for the MICE team who have provided us a great service since 2018 when we joined. We are: ASN: 395466 Speed: 10G IPv4: 206.53.175.96 IPv6: 2001:504:58::96 Thanks, Scott Reed Global Net NOC noc(a)theglobal.net<mailto:noc@theglobal.net> Global Net Peering peering(a)theglobal.net<mailto:peering@theglobal.net>

1 0

Fwd: [arin-announce] Reminder of Upcoming Deadline for ARIN’s Multifactor Authentication (MFA) Enforcement for ARIN Online
by David Farmer Sept. 9, 2024

Sept. 9, 2024

If you still need to enable MFA on your ARIN Online Account, DO IT NOW! More than 15,000 ARIN Online Accounts have yet to enable MFA, which is nuts! Thanks. ---------- Forwarded message --------- From: ARIN <info(a)arin.net> Date: Mon, Sep 9, 2024 at 11:25 AM Subject: [arin-announce] Reminder of Upcoming Deadline for ARIN’s Multifactor Authentication (MFA) Enforcement for ARIN Online To: <arin-announce(a)arin.net> On 7 October 2024, ARIN will lock all remaining ARIN Online accounts that have not enabled multifactor authentication (MFA). As of 1 February 2023, ARIN requires MFA on all new ARIN Online accounts. Users with accounts created before February 2023 were notified and encouraged to log in at their earliest opportunity, where they would be prompted to set up MFA. This step was taken to provide stronger security for the registry, reduce the risk of fraud, and increase confidence in the integrity of their ARIN resources. On 17 July, we announced that ARIN was continuing to take steps to secure the registry by enforcing MFA. We engaged in direct communications with 19,853 account holders linked to organizations with direct allocations, detailed reassignments, and reallocations who had not yet enabled MFA on their accounts. These users were notified that they needed to set up MFA within 60 days or their accounts would be locked, As of 4 September 2024, 15,593 accounts have not completed MFA setup. We are extending this deadline by two weeks to allow more time for account holders to enable MFA on existing accounts. We strongly encourage ARIN Online account holders who have not set up MFA to do so before the 7 October deadline. Users with locked accounts can contact Registration Services during ARIN operating hours, 7:00 AM to 7:00 PM ET Monday through Friday, to unlock and set up MFA and regain access to ARIN Online. ARIN relies on the support of its customers to ensure the accuracy and security of the registry. We rely on all our customers to follow best practices for account security and to report any suspicious activity to us immediately. Regards, John Curran President and CEO American Registry for Internet Numbers (ARIN) --------- Helpful Resources: Need help setting up your MFA? Visit https://www.arin.net/MFA to get started. You’ll find guides on enabling MFA via an authenticator application, SMS, or security key hardware. We also provide information on how to receive and save your MFA recovery codes, as well as what to do if you’ve lost access to your authenticator app, SMS phone number, or security key. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

1 0

Odd AWS transfer issue
by Danny Meister Sept. 7, 2024

Sept. 7, 2024

8 11

New QCIX Mailing Lists
by Jay Hanke Sept. 6, 2024

Sept. 6, 2024

For those that are interested, QCIX has just added mailing lists. https://qcix.mailman3.com/ -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/ <https://southfront.io/path.html>

1 0

Fwd: [Idr] [job@fastly.com: RFC 9234 route leak prevention in the wild!]
by David Farmer Sept. 4, 2024

Sept. 4, 2024

I think this is something we should add to our list Thanks ---------- Forwarded message --------- From: Job Snijders <job=40fastly.com(a)dmarc.ietf.org> Date: Mon, Sep 2, 2024 at 8:36 AM Subject: [Idr] [job(a)fastly.com: RFC 9234 route leak prevention in the wild!] To: <idr(a)ietf.org> Hello IDR, You might enjoy a report back from the field on route leak blocking based on RFC 9234. Please see below. Kind regards, Job ----- Forwarded message from Job Snijders <job(a)fastly.com> ----- Date: Mon, 2 Sep 2024 13:33:00 +0000 From: Job Snijders <job(a)fastly.com> To: nanog(a)nanog.org Subject: RFC 9234 route leak prevention in the wild! Dear all, I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-) The basic idea of 9234 is that BGP routers (based on their role in the Gao-Rexford inter-domain routing model) attach a special BGP Path attribute, or take action based on the presence and contents of this BGP Path attribute - with the intention to constrain a route's propagation radius to just the downstream customer cone of the neighboring ASN. Most operators will intuitively understand that any route propagating through multiple IX route servers operated by different IXPs is a route leak: ``` IXP_1 IXP_2 / \ / \ / \ / \ ISP_A ISP_B ISP_C (receives) (leaker) (originates) ``` (figure 1. propagation from right to left; leak scenario) In the above example, ISP_A originates a route towards IXP_1's route servers, IXP_1 propagates the route to ISP_B (so far so good); but for one reason or another ISP_B subsequently continues propagation of the route towards IXP_2's route servers, who in turn propagate it to ISP_C. ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue. ISP_A and ISP_C are probably not expecting ISP_B to be in the middle. This situation can happen as a result of a misconfiguration in ISP_B's equipment, even when all participants use IRR & RPKI ROV to attempt to mitigate the worst routing incidents. What does it matter / impact ============================ Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers. As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak is happening involving both YYCIX and FranceIX Route Servers via an ISP connected to both IXP's Route Servers; with the leak being stopped at YYCIX thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes. Let's zoom in on 1 entry: ``` $ bgpctl show rib 157.185.154.0/24 detail BGP routing table entry for 157.185.154.0/24 6939 38040 54994 Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 (216.218.252.194) Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs unknown, external, otc leak Last update: 11:58:08 ago Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029 Ext. Communities: ovs not-found Large Communities: 53339:11:1 53339:11:3 Aggregator: 54994 [163.171.131.254] OTC: 51706 ``` (figure 2. inspecting an leaked route using OpenBGPD's CLI) In figure 2. one can see the route is marked as 'otc leak', this was made possible because FranceIX's route server's attached the OTC attribute with the ASN value set to their Route Server's ASN (51706). ``` YYCIX FranceIX . x <adds OTC> \ . \ / \ ISP_A 6939_38040 54994 ``` (figure 3. right to left: real world example of blocked leak) In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route towards the FranceIX route servers. FranceIX accepts this route (probably because an IRR route object exists) and propagates it onward with the "Only-To-Customer" attribute set to 51706. The route is received by AS 38040, who appear to propagate the route to their upstream 6939. << An AS 38040 router is likely misconfigured! >> Then 6939 sends its customer's routes to the YYCIX route server, but the YYCIX route server recognizes that the route already passed through a 'valley' and thus considers this a leak; and blocks further propagation towards ISP_A. Impact with partial deployment ============================== RFC 9234 is an easy mechanism to configure and debug for both small & large network operators and IXP route server operators. In the above scenario YYCIX and FranceIX are likely the only 2 entities in the entire AS path which support RFC 9234; but we're already seeing leaks being blocked despite partial deployment! It's not hard to imagine that many IX-to-IX route leaks can be blocked with only the IXP operators themselves enabling RFC 9234 support. The world's most populair RS implementations (BIRD and OpenBGPD) already support RFC 9234! Tens of thousands of IX customers would enjoy the benefits of a few hundred IXPs taking action. RFC 9234 has no dependency on the RPKI, this means tha What can you do? ================ Just ask your vendors (your hardware routers and IXPs) to implement and deploy RFC 9234! :-) The more people ask, the more it'll bubble to the top of the priority list. The cost of implementing & deploying RFC 9234 is excellent bang for buck. Closing words ============= Shout out to our friends at FranceIX and MSK-IX for being amongst the first IXPs to deploy RFC 9234! Your effort helped reduce the potential impact of today's route leak! Kind regards, Job (YYCIX volunteer) Appendix A: ----------- Vantage point: YYCIX Route Server 1 (rs1.yycix.ca) Timestamp: Mon Sep 2 12:31:50 UTC 2024 Destination Prefix AS_PATH 102.164.129.0/24 6939 37613 6758 37649 i 102.164.139.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 102.164.140.0/24 6939 37613 6758 37649 i 102.164.141.0/24 6939 37613 6758 37649 i 102.164.182.0/24 6939 37613 6758 37649 i 154.65.33.0/24 6939 37613 6758 37649 i 154.65.34.0/24 6939 37613 6758 37649 i 154.65.35.0/24 6939 37613 6758 37649 i 154.65.36.0/24 6939 37613 6758 37649 i 154.65.37.0/24 6939 37613 6758 37649 i 154.65.38.0/24 6939 37613 6758 37649 i 154.65.39.0/24 6939 37613 6758 37649 i 154.72.35.0/24 6939 37613 37100 37027 37063 327721 i 157.185.151.0/24 6939 38040 54994 i 157.185.154.0/24 6939 38040 54994 i 163.171.164.0/24 6939 38040 54994 i 192.150.250.0/23 6939 4651 4618 4618 63529 4621 3836 i 196.50.8.0/24 6939 37613 6758 37649 i 196.50.9.0/24 6939 37613 6758 37649 i 196.50.11.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.13.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.14.0/24 6939 37613 6758 37649 i 2001:67c:15f4::/48 6939 41103 i 2401:c500:fd08::/48 6939 4637 38040 54994 i 2a01:53c0:ff03::/48 6939 4637 38040 54994 i 2a01:53c0:ff0f::/48 6939 4637 38040 54994 i 2a01:58c0::/32 6939 16347 42487 i 2a03:8920::/32 6939 41103 i 2a03:d602::/31 6939 16347 42487 i 2a03:d606::/31 6939 16347 42487 i 2a0d:ee00::/32 6939 16347 42487 i 2a0e:5b80::/29 6939 16347 42487 i 2a0e:e080::/32 6939 16347 42487 i 2a0f:c540::/29 6939 16347 42487 i ----- End forwarded message ----- _______________________________________________ Idr mailing list -- idr(a)ietf.org To unsubscribe send an email to idr-leave(a)ietf.org -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

2 1

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX
by Jeremy Lumby Aug. 23, 2024

Aug. 23, 2024

I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors.I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade.Jeremy LumbyMinnesota VoIP9217 17th Ave SSuite 216Bloomington MN 55425Main 612-355-7740Direct 612-392-6814EFax 952-873-7425jlumby(a)mnvoip.com -------- Original message --------From: Miles McCredie <Miles.McCredie(a)MIDCO.COM> Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

4 4