- mice-discuss - lists.micemn.net

Re: UG 46 Meeting & Election

by Richard Laager

The MICE UG 46 meeting will be held today, Wednesday, October 9, 3:00 pm at UMN and via Zoom. Parking, etc. details same as before: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 * * *Zoom Details* https://umn.zoom.us/j/97601658469?pwd=C6JWkmoYfB9WjZsnRPr9S5u1l4Zkuz.1 ID: 94910223662, passcode: Jp74Et 651-372-8299, meeting ID 97601658469, passcode 864193 * * *Nominations* We will be holding an election for a board seat. The board consists of three seats with staggered terms, such that one seat comes up for re-election each year (at the fall meeting). This year, Reid's seat is up for election. Reid was the only (self-)nomination, so as the election is uncontested, our practice is to elect by acclimation at the meeting. Proxies are unnecessary in that situation, assuming that the meeting makes quorum. -- Richard

Oct. 9, 2024

Mark Twain Rural Telephone has left MICE

by Jeremy Lumby

Mark Twain Rural Telephone has left MICE. If you have any bilaterals to 206.108.255.225 or 2001:504:27::29fe:0:1 please remove them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com

Oct. 4, 2024

Re: UG 46 Meeting & Election

by Michael Hemphill

Reid, So pleased to hear that you are running. You bring a valuable and unique perspective to the board. MICE is privileged to have your experience and expertise. Mike Hemphill ColoConnect.net Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Reid Fishler <0000000c9d62e487-dmarc-request(a)LISTS.IPHOUSE.NET> Sent: Friday, October 4, 2024 12:20:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] UG 46 Meeting & Election I will run again, and due to the Mets winning (woohoo!) I will be remote for this meeting. Open invitation for people to come join me for the games at Citi Field as well. :-) Reid On Mon, Sep 23, 2024 at 10:12 PM Richard Laager <0000001e0910bbd6-dmarc-request(a)lists.iphouse.net<mailto:0000001e0910bbd6-dmarc-request@lists.iphouse.net>> wrote: I'm a hair late on this, but nominations are now open for the director position. On 2024-06-21 21:23, Richard Laager wrote: The MICE UG 46 meeting will be held Wednesday, October 9, 3:00 pm at UMN and via Zoom. Parking, etc. details same as before: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 Zoom Details https://umn.zoom.us/j/97601658469?pwd=C6JWkmoYfB9WjZsnRPr9S5u1l4Zkuz.1 ID: 94910223662, passcode: Jp74Et 651-372-8299, meeting ID 97601658469, passcode 864193 Nominations We will be holding an election for a board seat. The board consists of three seats with staggered terms, such that one seat comes up for re-election each year (at the fall meeting). This year, Reid's seat is up for election. As in the past, we will close nominations in advance of the meeting so people have a chance to instruct a proxy appropriately. Nominations will be open for two weeks, September 23 - October 6. I plan to send reminder emails approximately at the start of and just after the nomination period. In the event that the election is uncontested, our practice is to elect by acclimation at the meeting. Proxies are unnecessary in that situation, assuming that the meeting makes quorum. Proxies The bylaws state that you send proxies to the Chief Manager (me, Richard Laager), and electronic means are acceptable. So a simple email is sufficient: rlaager(a)wiktel.com<mailto:rlaager@wiktel.com> or chiefmanager(a)micemn.net<mailto:chiefmanager@micemn.net>. If you send me a vote by email (e.g. "We vote for <X>."), I will treat it as an a proxy appointment, appointing me to vote on your behalf, but restricted to only the election item and to only vote for the candidate you expressed. If you send me a proxy with different instructions (e.g. "We appoint <Y> to vote for us."), obviously that will be honored as written. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Oct. 4, 2024

Re: UG 46 Meeting & Election

by Reid Fishler

I will run again, and due to the Mets winning (woohoo!) I will be remote for this meeting. Open invitation for people to come join me for the games at Citi Field as well. :-) Reid On Mon, Sep 23, 2024 at 10:12 PM Richard Laager < 0000001e0910bbd6-dmarc-request(a)lists.iphouse.net> wrote: > I'm a hair late on this, but *nominations are now open for the director > position.* > > On 2024-06-21 21:23, Richard Laager wrote: > > The MICE UG 46 meeting will be held Wednesday, October 9, 3:00 pm at UMN > and via Zoom. > > Parking, etc. details same as before: > https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 > > > *Zoom Details* > > https://umn.zoom.us/j/97601658469?pwd=C6JWkmoYfB9WjZsnRPr9S5u1l4Zkuz.1 > ID: 94910223662, passcode: Jp74Et > 651-372-8299, meeting ID 97601658469, passcode 864193 > > > *Nominations* > > We will be holding an election for a board seat. The board consists of > three seats with staggered terms, such that one seat comes up for > re-election each year (at the fall meeting). This year, Reid's seat is up > for election. > > As in the past, we will close nominations in advance of the meeting so > people have a chance to instruct a proxy appropriately. *Nominations will > be open for two weeks, September 23 - October 6. *I plan to send reminder > emails approximately at the start of and just after the nomination period. > > In the event that the election is uncontested, our practice is to elect by > acclimation at the meeting. Proxies are unnecessary in that situation, > assuming that the meeting makes quorum. > > > > *Proxies * > > The bylaws state that you send proxies to the Chief Manager (me, Richard > Laager), and electronic means are acceptable. So a simple email is > sufficient: rlaager(a)wiktel.com or chiefmanager(a)micemn.net. > > If you send me a *vote* by email (e.g. "We vote for <X>."), I will treat > it as an a proxy appointment, appointing me to vote on your behalf, but > restricted to *only* the election item and to *only* vote for the > candidate you expressed. > > If you send me a proxy with different instructions (e.g. "We appoint <Y> > to vote for us."), obviously that will be honored as written. > > -- > Richard > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178

Oct. 4, 2024

Re: UG 46 Meeting & Election

by Richard Laager

I'm a hair late on this, but *nominations are now open for the director position.* On 2024-06-21 21:23, Richard Laager wrote: > > The MICE UG 46 meeting will be held Wednesday, October 9, 3:00 pm at > UMN and via Zoom. > > Parking, etc. details same as before: > https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 > > * > * > > *Zoom Details* > > https://umn.zoom.us/j/97601658469?pwd=C6JWkmoYfB9WjZsnRPr9S5u1l4Zkuz.1 > ID: 94910223662, passcode: Jp74Et > 651-372-8299, meeting ID 97601658469, passcode 864193 > > * > * > > *Nominations* > > We will be holding an election for a board seat. The board consists of > three seats with staggered terms, such that one seat comes up for > re-election each year (at the fall meeting). This year, Reid's seat is > up for election. > > As in the past, we will close nominations in advance of the meeting so > people have a chance to instruct a proxy appropriately. *Nominations > will be open for two weeks, September 23 - October 6. *I plan to send > reminder emails approximately at the start of and just after the > nomination period. > > In the event that the election is uncontested, our practice is to > elect by acclimation at the meeting. Proxies are unnecessary in that > situation, assuming that the meeting makes quorum. > > > *Proxies > * > > The bylaws state that you send proxies to the Chief Manager (me, > Richard Laager), and electronic means are acceptable. So a simple > email is sufficient: rlaager(a)wiktel.com or chiefmanager(a)micemn.net. > > If you send me a /vote/ by email (e.g. "We vote for <X>."), I will > treat it as an a proxy appointment, appointing me to vote on your > behalf, but restricted to /only/ the election item and to /only/ vote > for the candidate you expressed. > > If you send me a proxy with different instructions (e.g. "We appoint > <Y> to vote for us."), obviously that will be honored as written. > -- Richard

Sept. 24, 2024

Please welcome x6c to MICE!

by Jeremy Lumby

Name: x6c inc Switch: NOCIX Ports: E 5/18 ASN: 46733 IPv4: 206.108.255.235/24 IPv6: 2001:504:27:0:0:B68D::1/64 Peering Contact: support(a)x6c.co Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Sept. 23, 2024

Please welcome Essentia Health to MICE!

by Jeremy Lumby

Name: Essentia Health Switch: MICE Core 511 Ports: E 4/34 & 4/40 have been seaparated out of the portchannel so that they can connect redundant routers ASN: 3789 IPv4: Adding 206.108.255.234 to the existing IPs IPv6: Adding 2001:504:27::ecd:0:2 to the existing IPs Peering Contact: nickolas.stevermer(a)essentiahealth.org Route Servers: Partial Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Sept. 18, 2024

Please welcome Netflix to MICE!

by Jeremy Lumby

Name: Netflix Switch: MICE Core 511 Ports: adding E3/27/1 to portchannel 2 and adding E 6/20/1 to portchannel 3 for a total of 2x2x100G ASN: 2906 IPv4: All IP info remains the same, this is a 2x100G port addition IPv6: Peering Contact: peering(a)netflix.com Route Servers: YES Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Sept. 13, 2024

Please welcome Essentia Health to MICE!

by Jeremy Lumby

Name: Essentia Health Switch: MICE Core 511 Ports: E 4/34 & 4/40 in portchannel 28 for a total of 2x10G ASN: 3789 IPv4: All IP info remains the same, this is a switch move, and port addition. IPv6: Peering Contact: nickolas.stevermer(a)essentiahealth.org Route Servers: Partial Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Sept. 11, 2024

Fwd: [arin-announce] Reminder of Upcoming Deadline for ARIN’s Multifactor Authentication (MFA) Enforcement for ARIN Online

by David Farmer

If you still need to enable MFA on your ARIN Online Account, DO IT NOW! More than 15,000 ARIN Online Accounts have yet to enable MFA, which is nuts! Thanks. ---------- Forwarded message --------- From: ARIN <info(a)arin.net> Date: Mon, Sep 9, 2024 at 11:25 AM Subject: [arin-announce] Reminder of Upcoming Deadline for ARIN’s Multifactor Authentication (MFA) Enforcement for ARIN Online To: <arin-announce(a)arin.net> On 7 October 2024, ARIN will lock all remaining ARIN Online accounts that have not enabled multifactor authentication (MFA). As of 1 February 2023, ARIN requires MFA on all new ARIN Online accounts. Users with accounts created before February 2023 were notified and encouraged to log in at their earliest opportunity, where they would be prompted to set up MFA. This step was taken to provide stronger security for the registry, reduce the risk of fraud, and increase confidence in the integrity of their ARIN resources. On 17 July, we announced that ARIN was continuing to take steps to secure the registry by enforcing MFA. We engaged in direct communications with 19,853 account holders linked to organizations with direct allocations, detailed reassignments, and reallocations who had not yet enabled MFA on their accounts. These users were notified that they needed to set up MFA within 60 days or their accounts would be locked, As of 4 September 2024, 15,593 accounts have not completed MFA setup. We are extending this deadline by two weeks to allow more time for account holders to enable MFA on existing accounts. We strongly encourage ARIN Online account holders who have not set up MFA to do so before the 7 October deadline. Users with locked accounts can contact Registration Services during ARIN operating hours, 7:00 AM to 7:00 PM ET Monday through Friday, to unlock and set up MFA and regain access to ARIN Online. ARIN relies on the support of its customers to ensure the accuracy and security of the registry. We rely on all our customers to follow best practices for account security and to report any suspicious activity to us immediately. Regards, John Curran President and CEO American Registry for Internet Numbers (ARIN) --------- Helpful Resources: Need help setting up your MFA? Visit https://www.arin.net/MFA to get started. You’ll find guides on enabling MFA via an authenticator application, SMS, or security key hardware. We also provide information on how to receive and save your MFA recovery codes, as well as what to do if you’ve lost access to your authenticator app, SMS phone number, or security key. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Sept. 9, 2024

Re: [EXT]: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue

by Danny Meister

Sept. 7, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue

by Miles McCredie

The graph definitely looks better. [cid:image002.png@01DB0083.B32035F0] Danny, Isaiah - Are the downloads back to normal? [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, September 6, 2024 2:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image003.png@01DB0083.B32035F0]<https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Jeremy Lumby

I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Chris Wopat

That is such a high error rate that we should perhaps just shut the port down so the traffic routes elsewhere, properly. Based on the graph it appears to be a failing optic - could perhaps be confirmed if Cacti is trending TX/RX power and TX Bias. --Chris >

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Jay Hanke

Jeremy caught the issue and reached out to Amazon a couple of weeks ago regarding the areas. It might help for those having issues to reach out to Amazon directly to knock it loose on their side. On Fri, Sep 6, 2024, 12:36 PM Isaiah Olson <iolson(a)paulbunyan.net> wrote: > It looks like Amazon’s port Ethernet6/1/1 on the MICE Core switch has some > pretty significant errors. > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Miles > McCredie > *Sent:* Friday, September 06, 2024 12:12 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS > transfer issue > > > > Maybe someone could check for errors on the MICE ports for Atomic Data and > Amazon. > > > > Assuming the MICE ports are clean, it is probably worth following up with > the Amazon NOC, peering-to(a)amazon.com. They did some work on their > Minneapolis peering routers recently though I’m not sure what they were > doing. > > > > > > > > [image: Image removed by sender.] > > *Miles McCredie * > Principal Network Engineer II-Core IP > > Office: 6052755192 > Miles.McCredie(a)midco.com > > *Midco.com* > > Let's go beyond. > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Danny > Meister > *Sent:* Friday, September 6, 2024 11:41 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue > > > > *CAUTION:* This email originated from outside of *MIDCO*. > *Do not click links or open attachments unless you recognize the sender > and know the content is safe.* > > Good Afternoon, > > > > I've been trying to track down a strange packet loss issue when a customer > of mine is downloading files from S3 in the east region. We peer with AWS > via MICE and traffic to east routes via MICE. > > > > Could any of you folks peered with AWS via MICE try downloading this file > a bunch of times and reports the results? > > > > > http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… > > > > > Basically, we see fast transfers maybe 80% of the time and then it will be > slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 > instance also). Captures show lots of retransmits, etc during a slow > transfer. If I shut down my AWS peering the problem goes away. I was able > to test from another client of mine connected to Arvig and I'm pretty sure > I can replicate the issue from there. So, I'm not convinced it's just me. > On the other hand, if there is a problem, I'm surprised no one else has > noticed! > > > > Thanks! > > *Danny Meister * > > Infrastructure Engineer > > 612.466.2071 | DMeister(a)atomicdata.com > > [image: Logo] > <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> > > 250 Marquette Avenue > <https://www.google.com/maps/search/250+Marquette+Avenue?entry=gmail&source=g> > South | Minneapolis, MN | 55401 > > *atomicdata.com* > <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> > | *Services* > <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> > | *Blog* > <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> > *Atomic Data Welcomes Chris Heim as CEO* > <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Isaiah Olson

It looks like Amazon's port Ethernet6/1/1 on the MICE Core switch has some pretty significant errors. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Miles McCredie Sent: Friday, September 06, 2024 12:12 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to(a)amazon.com <mailto:peering-to@amazon.com> . They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign ature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email > Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Miles McCredie

Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to(a)amazon.com<mailto:peering-to@amazon.com>. They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image001.png@01DB0055.EE8B7A60]<https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Jeremy Lumby

Amazon is aware of a condition that is likely causing this intermittent issue. Last I heard it should be repaired early next week. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Isaiah Olson

I was able to replicate this slowness on my first try. It appears that the specific node I was talking to when seeing the slow download was 54.231.233.205 and it took a couple of minutes to complete, averaging 100 KB/s. We only peer with Amazon via MICE. Hope the information is helpful, and please let us know if you find anything. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 06, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? <http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws .com/empty_test_file_50M.txt> http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | <mailto:DMeister@atomicdata.com> DMeister(a)atomicdata.com <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Justin Krejci

My path takes one of our PNI connections to Amazon, not via MICE. The download via PNI in Minneapolis it took 2.3 seconds. -----Original Message----- From: Ryan Malek <rmalek(a)ROUTER12.NET<mailto:Ryan%20Malek%20%3crmalek@ROUTER12.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Odd AWS transfer issue Date: Fri, 06 Sep 2024 11:47:47 -0500 Downloaded nearly instantly, no problem. ________________________________ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 9/6/2024 11:41 AM, Danny Meister wrote: Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws…<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [Logo]<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> | Services<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> | Blog<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> Atomic Data Welcomes Chris Heim as CEO<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…>

Sept. 6, 2024

Re: Odd AWS transfer issue

by Ryan Malek

Downloaded nearly instantly, no problem. ------------------------------------------------------------------------ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 9/6/2024 11:41 AM, Danny Meister wrote: > Good Afternoon, > > I've been trying to track down a strange packet loss issue when a > customer of mine is downloading files from S3 in the east region. We > peer with AWS via MICE and traffic to east routes via MICE. > > Could any of you folks peered with AWS via MICE try downloading this > file a bunch of times and reports the results? > > http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… > > > Basically, we see fast transfers maybe 80% of the time and then it > will be slow, like sub 1Mb slow (we can replicate the issue with iperf > to an EC2 instance also). Captures show lots of retransmits, etc > during a slow transfer. If I shut down my AWS peering the problem goes > away. I was able to test from another client of mine connected to > Arvig and I'm pretty sure I can replicate the issue from there. So, > I'm not convinced it's just me. On the other hand, if there is a > problem, I'm surprised no one else has noticed! > > Thanks! > > *Danny Meister * > > Infrastructure Engineer > > 612.466.2071 | DMeister(a)atomicdata.com > > Logo > <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E…> > > 250 Marquette Avenue South | Minneapolis, MN | 55401 > > *atomicdata.com* > <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…>|*Services* > <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E…>| > *Blog* > <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> > Atomic Data Welcomes Chris Heim as CEO > <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> > > > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> >

Sept. 6, 2024

Odd AWS transfer issue

by Danny Meister

Sept. 6, 2024

New QCIX Mailing Lists

by Jay Hanke

For those that are interested, QCIX has just added mailing lists. https://qcix.mailman3.com/ -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/ <https://southfront.io/path.html>

Sept. 6, 2024

Please welcome Triplebit to MICE!

by Jeremy Lumby

Name: Triplebit Switch: MICE Core - Ridgeview Ports: E 3/31/2 Strands 123-124 for a total of 1x10G ASN: 401332 IPv4: 206.108.255.233/24 IPv6: 2001:504:27:0:6:1FB4::1 Peering Contact: peering(a)triplebit.org Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Sept. 4, 2024

Internet Society IXP grant program

by Mike O'Connor

hi all, just a heads up if we're not already aware of this program. probably too late for this round (deadline: Sept 6th), but might be a good one to track in the future... https://circleid.com/posts/20240903-deadline-of-september-6-for-grant-fundi… the Internet Society might also have a list of other funders we would tap... mike O'C

Sept. 4, 2024

Fwd: [Idr] [job@fastly.com: RFC 9234 route leak prevention in the wild!]

by David Farmer

I think this is something we should add to our list Thanks ---------- Forwarded message --------- From: Job Snijders <job=40fastly.com(a)dmarc.ietf.org> Date: Mon, Sep 2, 2024 at 8:36 AM Subject: [Idr] [job(a)fastly.com: RFC 9234 route leak prevention in the wild!] To: <idr(a)ietf.org> Hello IDR, You might enjoy a report back from the field on route leak blocking based on RFC 9234. Please see below. Kind regards, Job ----- Forwarded message from Job Snijders <job(a)fastly.com> ----- Date: Mon, 2 Sep 2024 13:33:00 +0000 From: Job Snijders <job(a)fastly.com> To: nanog(a)nanog.org Subject: RFC 9234 route leak prevention in the wild! Dear all, I'd like to share an update on RFC 9234 deployment. RFC 9234 titled "BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ... routing security is more than just RPKI! :-) The basic idea of 9234 is that BGP routers (based on their role in the Gao-Rexford inter-domain routing model) attach a special BGP Path attribute, or take action based on the presence and contents of this BGP Path attribute - with the intention to constrain a route's propagation radius to just the downstream customer cone of the neighboring ASN. Most operators will intuitively understand that any route propagating through multiple IX route servers operated by different IXPs is a route leak: ``` IXP_1 IXP_2 / \ / \ / \ / \ ISP_A ISP_B ISP_C (receives) (leaker) (originates) ``` (figure 1. propagation from right to left; leak scenario) In the above example, ISP_A originates a route towards IXP_1's route servers, IXP_1 propagates the route to ISP_B (so far so good); but for one reason or another ISP_B subsequently continues propagation of the route towards IXP_2's route servers, who in turn propagate it to ISP_C. ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue. ISP_A and ISP_C are probably not expecting ISP_B to be in the middle. This situation can happen as a result of a misconfiguration in ISP_B's equipment, even when all participants use IRR & RPKI ROV to attempt to mitigate the worst routing incidents. What does it matter / impact ============================ Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023 using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024. Both IXPs configured their route servers to reject BGP routes that have an OTC attribute attached, and to attach an OTC attribute when propagating routes to the Route Server's peers. As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak is happening involving both YYCIX and FranceIX Route Servers via an ISP connected to both IXP's Route Servers; with the leak being stopped at YYCIX thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes. Let's zoom in on 1 entry: ``` $ bgpctl show rib 157.185.154.0/24 detail BGP routing table entry for 157.185.154.0/24 6939 38040 54994 Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 (216.218.252.194) Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs unknown, external, otc leak Last update: 11:58:08 ago Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029 Ext. Communities: ovs not-found Large Communities: 53339:11:1 53339:11:3 Aggregator: 54994 [163.171.131.254] OTC: 51706 ``` (figure 2. inspecting an leaked route using OpenBGPD's CLI) In figure 2. one can see the route is marked as 'otc leak', this was made possible because FranceIX's route server's attached the OTC attribute with the ASN value set to their Route Server's ASN (51706). ``` YYCIX FranceIX . x <adds OTC> \ . \ / \ ISP_A 6939_38040 54994 ``` (figure 3. right to left: real world example of blocked leak) In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route towards the FranceIX route servers. FranceIX accepts this route (probably because an IRR route object exists) and propagates it onward with the "Only-To-Customer" attribute set to 51706. The route is received by AS 38040, who appear to propagate the route to their upstream 6939. << An AS 38040 router is likely misconfigured! >> Then 6939 sends its customer's routes to the YYCIX route server, but the YYCIX route server recognizes that the route already passed through a 'valley' and thus considers this a leak; and blocks further propagation towards ISP_A. Impact with partial deployment ============================== RFC 9234 is an easy mechanism to configure and debug for both small & large network operators and IXP route server operators. In the above scenario YYCIX and FranceIX are likely the only 2 entities in the entire AS path which support RFC 9234; but we're already seeing leaks being blocked despite partial deployment! It's not hard to imagine that many IX-to-IX route leaks can be blocked with only the IXP operators themselves enabling RFC 9234 support. The world's most populair RS implementations (BIRD and OpenBGPD) already support RFC 9234! Tens of thousands of IX customers would enjoy the benefits of a few hundred IXPs taking action. RFC 9234 has no dependency on the RPKI, this means tha What can you do? ================ Just ask your vendors (your hardware routers and IXPs) to implement and deploy RFC 9234! :-) The more people ask, the more it'll bubble to the top of the priority list. The cost of implementing & deploying RFC 9234 is excellent bang for buck. Closing words ============= Shout out to our friends at FranceIX and MSK-IX for being amongst the first IXPs to deploy RFC 9234! Your effort helped reduce the potential impact of today's route leak! Kind regards, Job (YYCIX volunteer) Appendix A: ----------- Vantage point: YYCIX Route Server 1 (rs1.yycix.ca) Timestamp: Mon Sep 2 12:31:50 UTC 2024 Destination Prefix AS_PATH 102.164.129.0/24 6939 37613 6758 37649 i 102.164.139.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 102.164.140.0/24 6939 37613 6758 37649 i 102.164.141.0/24 6939 37613 6758 37649 i 102.164.182.0/24 6939 37613 6758 37649 i 154.65.33.0/24 6939 37613 6758 37649 i 154.65.34.0/24 6939 37613 6758 37649 i 154.65.35.0/24 6939 37613 6758 37649 i 154.65.36.0/24 6939 37613 6758 37649 i 154.65.37.0/24 6939 37613 6758 37649 i 154.65.38.0/24 6939 37613 6758 37649 i 154.65.39.0/24 6939 37613 6758 37649 i 154.72.35.0/24 6939 37613 37100 37027 37063 327721 i 157.185.151.0/24 6939 38040 54994 i 157.185.154.0/24 6939 38040 54994 i 163.171.164.0/24 6939 38040 54994 i 192.150.250.0/23 6939 4651 4618 4618 63529 4621 3836 i 196.50.8.0/24 6939 37613 6758 37649 i 196.50.9.0/24 6939 37613 6758 37649 i 196.50.11.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.13.0/24 6939 37613 6758 37649 37649 37649 37649 37649 i 196.50.14.0/24 6939 37613 6758 37649 i 2001:67c:15f4::/48 6939 41103 i 2401:c500:fd08::/48 6939 4637 38040 54994 i 2a01:53c0:ff03::/48 6939 4637 38040 54994 i 2a01:53c0:ff0f::/48 6939 4637 38040 54994 i 2a01:58c0::/32 6939 16347 42487 i 2a03:8920::/32 6939 41103 i 2a03:d602::/31 6939 16347 42487 i 2a03:d606::/31 6939 16347 42487 i 2a0d:ee00::/32 6939 16347 42487 i 2a0e:5b80::/29 6939 16347 42487 i 2a0e:e080::/32 6939 16347 42487 i 2a0f:c540::/29 6939 16347 42487 i ----- End forwarded message ----- _______________________________________________ Idr mailing list -- idr(a)ietf.org To unsubscribe send an email to idr-leave(a)ietf.org -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Sept. 2, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by Jeremy Lumby

You can view the ports on the weathermap. The short version is that they have been using BGP to redirect traffic away from MICE for years. Each of the 100G ports ramps up to 90G sits there for about 18 hours, and then drops to reasonable levels for the late night/early morning hours. Repeat every day since shortly after their last upgrade several years ago. By my estimation they need 600G to MICE just to not saturate/not need to redirect traffic away from MICE. While I am not on the board, at those traffic levels Google can justify doing something on their own, and not need to look for handouts to get to MICE. Google has received a lot of free assistance in connecting to MICE in the past. Now they are unwilling to even have an open conversation about anything that would assist keeping them DIRECTLY on MICE as opposed to some sort of workaround where they are not taking on ANY of the recurring costs. Jeremy From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Miles McCredie Sent: Friday, August 23, 2024 9:00 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Interesting. I’d be interested in any/all of average/P95/P99/Peak traffic over the last month on the Google ports but understand it’s not really any of my business. Is there a trend line over the last quarter or two? I have heard second hand that MICE ran into some unpleasantness around cache back in the day but I, for one, think that it would be worthwhile exploring the option of hosting caches again. (Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊) As someone pointed out up thread, there are a handful of issues to sort out in this area. <http://images.midcocomm.com/ES_MidcoLogo.png> Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com <mailto:Miles.McCredie@midco.com> Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Jeremy Lumby Sent: Friday, August 23, 2024 8:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors. I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com <mailto:jlumby@mnvoip.com> -------- Original message -------- From: Miles McCredie <Miles.McCredie(a)MIDCO.COM <mailto:Miles.McCredie@MIDCO.COM> > Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. 1. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket 2. Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… <http://images.midcocomm.com/ES_MidcoLogo.png> Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET <mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e> > Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e> > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ >From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET <mailto:AnthonyAnderberg@NUVERA.NET> > To MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com <mailto:peering-ops-noreply@google.com> >> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by David Farmer

There will be more to come for Google in Minneapolis. They pulled the plug on their Becker DC during COVID; that was a significant setback for us. But it is inconceivable that they won't do something; it just takes time. On Fri, Aug 23, 2024 at 9:29 AM Aaron Wendel <aaron(a)wholesaleinternet.net> wrote: > They were and then they pulled out. > > Google's peering philosophy seems to change about every 3-5 years so it > wouldn't surprise me at all if they build out to the midwest (outside of > Chicago) again. > > As someone who is in the IX, Datacenter and Access businesses I can tell > you that I've seen a major migration of content networks into tier 2/3 > markets, especially in the Midwest/Great Plains areas over the last year or > two. Google will come back eventually. > > Aaron > > > > ------ Original Message ------ > > (Would be even better for Google to build in to Minneapolis so that we > can all turn up more local PNIs but they haven’t been interested in my > views on this. 😊) > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by Aaron Wendel

They were and then they pulled out. Google's peering philosophy seems to change about every 3-5 years so it wouldn't surprise me at all if they build out to the midwest (outside of Chicago) again. As someone who is in the IX, Datacenter and Access businesses I can tell you that I've seen a major migration of content networks into tier 2/3 markets, especially in the Midwest/Great Plains areas over the last year or two. Google will come back eventually. Aaron ------ Original Message ------ > (Would be even better for Google to build in to Minneapolis so that we >can all turn up more local PNIs but they haven’t been interested in my >views on this. 😊) >

Aug. 23, 2024

Re: Peering with Google no longer available at MICE IX

by David Farmer

I have a tentative agreement for the University to host a Google Cache and provide the cache fill for MICE. The plan would be to host it on campus, initially in our Telecom Building, and move it to the new data center being constructed on campus, probably later in 2026. We would provide space, power, cache fill, and fiber back to 511, and we would expect MICE to pay for cross-connects and to provide the needed ports. We might need MICE to sign the contract with Google. The last time we talked with Google, they wanted the NDA to cover the contract itself, but as a government agency, we don't usually do that. I am reaching out to some Google contacts to see if Google has any new flexibility on that issue. Thanks On Thu, Aug 22, 2024 at 4:28 PM Anthony Anderberg < AnthonyAnderberg(a)nuvera.net> wrote: > I've been surprised that folks haven't been talking about this more and am > curious about everyone's thoughts - as MICE user-members but also in terms > of MICE leadership and direction. > > As a user-member I feel somewhat conflicted: on one hand in an effort to > maximize performance I could obtain transport to Chicago and connect to > Google there, but I feel like doing so is making Google's problem into my > problem and that doing so undercuts the goals of MICE. We've seen content > providers move MICE's larger members toward local direct private peering, > and I can understand rational behind that even if it shifts traffic away > from MICE - but Google is proposing remote direct peering at member's cost > which feels different. > > Of course nobody expects Google to subsidize our corner of the world, I am > just curious if there are other aspects we should be thinking about. > > Thanks in advance, > Anthony > > > > On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" < > peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> > wrote: > > Dear Peer, > > This is a reminder that peering with Google will no longer be available at > MICE Internet Exchange from 16th of September. > > As a result of unsustainable overheads associated with connecting to an IX > using remote waves, we have made a decision to stop connecting to IXs that > way. MICE IX falls under the category of IXs we remotely connect to. > > We will withdraw prefixes on 16th of September, and disconnect from the IX > on 24th of September. > > We realize this may cause inconvenience for some peers. If there is > another > mutually present IX that we can peer in, we will be happy to help set up a > bilateral session there - no multilaterals. If there is no mutual IX, > traffic may have to be exchanged via indirect paths (transit links). > > We are not looking to connect to any additional IXs to replace the IX that > we are leaving, but if you are considering connecting to another IX and > not > sure if Google will also be leaving that IX because of this measure, > please > reach out to us and we can share more. > > Thank you for your understanding. > > > Sincerely, > Google Network Operations > > > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by David Farmer

The GigaPoP receives the vast majority of its Google traffic from MICE. Our regular daytime peak is 20 to 25G, occasionally up to 30ish. On Fri, Aug 23, 2024 at 6:34 AM Miles McCredie <Miles.McCredie(a)midco.com> wrote: > Curious how many MICE members are in each of the following buckets. > > > > 1. Receive limited portion of Google traffic via MICE > > Midco and, I assume, US Internet are in > this bucket > > 2. Receive a significant portion of Google traffic via MICE today > > > > I’ve only got a couple years of experience with Google for Midco but my > experience has been that Google will not always provide the cache capacity > that I think is appropriate. The last two augments were triggered 9-12 > months later than I would have liked. Could be my fault for creating new > GNLs but… > > > > > > > > *Miles McCredie * > Principal Network Engineer II-Core IP > > Office: 6052755192 > Miles.McCredie(a)midco.com > > *Midco.com* > > Let's go beyond. > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Justin > Krejci > *Sent:* Thursday, August 22, 2024 5:25 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer > available at MICE IX > > > > *CAUTION:* This email originated from outside of *MIDCO*. > *Do not click links or open attachments unless you recognize the sender > and know the content is safe.* > > I am not sure what Google's stance is on deploying cache servers at an IX > vs remote connection like the ones pending disconnection. > > If they were to deploy caching nodes at an IX, I assume they would still > have a cache-fill requirement in terms of connectivity that has to come > from somewhere... of which they appear to be disinterested in maintaining > long distance connections going forward. So perhaps this is a donation > opportunity for some to entice Google to deploy caching nodes at MICE? > Donate local colo and transit bandwidth along with local MICE connectivity? > > > > We have numerous Google caching nodes in our network as well as multiple > PNIs to Google, so their pending disconnect from MICE will likely have > minimal direct impact on our eyeballs. > > > > > > -----Original Message----- > > *From*: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET > <Aaron%20Wendel%20%3caaron(a)WHOLESALEINTERNET.NET%3e>> > > *Reply-To*: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET > <MICE%20Discuss%20%3cMICE-DISCUSS(a)LISTS.IPHOUSE.NET%3e>> > > *To*: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > *Subject*: Re: [MICE-DISCUSS] Peering with Google no longer available at > MICE IX > > *Date*: Thu, 22 Aug 2024 22:03:01 +0000 > > > > Has MICE requested a cache box from Google? > > > > > > > > ------ Original Message ------ > > From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> > > To MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Date 8/22/2024 4:28:18 PM > > Subject Re: [MICE-DISCUSS] Peering with Google no longer available at > > MICE IX > > > > I've been surprised that folks haven't been talking about this more and am > curious about everyone's thoughts - as MICE user-members but also in terms > of MICE leadership and direction. > > > > As a user-member I feel somewhat conflicted: on one hand in an effort to > maximize performance I could obtain transport to Chicago and connect to > Google there, but I feel like doing so is making Google's problem into my > problem and that doing so undercuts the goals of MICE. We've seen content > providers move MICE's larger members toward local direct private peering, > and I can understand rational behind that even if it shifts traffic away > from MICE - but Google is proposing remote direct peering at member's cost > which feels different. > > > > Of course nobody expects Google to subsidize our corner of the world, I am > just curious if there are other aspects we should be thinking about. > > > > Thanks in advance, > > Anthony > > > > > > > > On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" < > peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> > wrote: > > > > Dear Peer, > > > > This is a reminder that peering with Google will no longer be available at > > MICE Internet Exchange from 16th of September. > > > > As a result of unsustainable overheads associated with connecting to an IX > > using remote waves, we have made a decision to stop connecting to IXs that > > way. MICE IX falls under the category of IXs we remotely connect to. > > > > We will withdraw prefixes on 16th of September, and disconnect from the IX > > on 24th of September. > > > > We realize this may cause inconvenience for some peers. If there is another > > mutually present IX that we can peer in, we will be happy to help set up a > > bilateral session there - no multilaterals. If there is no mutual IX, > > traffic may have to be exchanged via indirect paths (transit links). > > > > We are not looking to connect to any additional IXs to replace the IX that > > we are leaving, but if you are considering connecting to another IX and not > > sure if Google will also be leaving that IX because of this measure, please > > reach out to us and we can share more. > > > > Thank you for your understanding. > > > > > > Sincerely, > > Google Network Operations > > > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 23, 2024

Re: Peering with Google no longer available at MICE IX

by Jeff Wilde

I agree also. [A picture containing icon Description automatically generated] Jeff Wilde • Network Engineer • Park Region Office 218.826.6161 • Direct 218.826.8330 • Fax 218.826.6298 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Ben Wiechman Sent: Friday, August 23, 2024 8:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I concur, as opposed to each member paying for transport back to Chicago or other POP to haul their own traffic. That largely defeats the purpose of the exchange to a large degree. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<http://arvig.com> On Thu, Aug 22, 2024 at 5:39 PM Aaron Wendel <aaron(a)wholesaleinternet.net<mailto:aaron@wholesaleinternet.net>> wrote: For some historical context: A few years ago Google decided to pull their infrastructure out of Minneapolis, Kansas City and Saint Louis. The dominant exchanges in each market were given a choice between back-hauled peering connections or cache boxes. Minneapolis worked a deal for the back-hauled connection to Chicago. Kansas City and Saint Louis opted for cache servers. The main reason for the cache server choice was the concern over the long term sustainability and scalability of a remote peer of Google's size. Over the subsequent years, Google has shipped more and more servers as demand has increased. They have been very accommodating with the equipment. One other benefit to the cache servers is that they were not affected by Google's recent disconnection from ALL route servers. Google bears the cost for hardware upkeep. KCIX and STLIX use donated resources for cache fill, space and power. I understand MICE does not enjoy the resources that KCIX (and to a lesser extent STLIX) has access to but if Google is still shipping new cache servers (they may not be) that seems like a much better choice than remote peering to Chicago. Aaron ------ Original Message ------ From "Justin Krejci" <JKrejci(a)USINTERNET.COM<mailto:JKrejci@USINTERNET.COM>> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 5:25:08 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET<mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET>> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com<mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com>>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by Miles McCredie

Interesting. I’d be interested in any/all of average/P95/P99/Peak traffic over the last month on the Google ports but understand it’s not really any of my business. Is there a trend line over the last quarter or two? I have heard second hand that MICE ran into some unpleasantness around cache back in the day but I, for one, think that it would be worthwhile exploring the option of hosting caches again. (Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊) As someone pointed out up thread, there are a handful of issues to sort out in this area. [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, August 23, 2024 8:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors. I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> -------- Original message -------- From: Miles McCredie <Miles.McCredie(a)MIDCO.COM<mailto:Miles.McCredie@MIDCO.COM>> Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. 1. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket 1. Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com<mailto:Miles.McCredie@midco.com> Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET<mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET>> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com<mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com>>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 23, 2024

Re: Peering with Google no longer available at MICE IX

by Ben Wiechman

I concur, as opposed to each member paying for transport back to Chicago or other POP to haul their own traffic. That largely defeats the purpose of the exchange to a large degree. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Thu, Aug 22, 2024 at 5:39 PM Aaron Wendel <aaron(a)wholesaleinternet.net> wrote: > For some historical context: > > A few years ago Google decided to pull their infrastructure out of > Minneapolis, Kansas City and Saint Louis. The dominant exchanges in each > market were given a choice between back-hauled peering connections or cache > boxes. > > Minneapolis worked a deal for the back-hauled connection to Chicago. > Kansas City and Saint Louis opted for cache servers. > > The main reason for the cache server choice was the concern over the long > term sustainability and scalability of a remote peer of Google's size. > > Over the subsequent years, Google has shipped more and more servers as > demand has increased. They have been very accommodating with the > equipment. > > One other benefit to the cache servers is that they were not affected by > Google's recent disconnection from ALL route servers. > > Google bears the cost for hardware upkeep. KCIX and STLIX use donated > resources for cache fill, space and power. > > I understand MICE does not enjoy the resources that KCIX (and to a lesser > extent STLIX) has access to but if Google is still shipping new cache > servers (they may not be) that seems like a much better choice than remote > peering to Chicago. > > Aaron > > > > > > ------ Original Message ------ > From "Justin Krejci" <JKrejci(a)USINTERNET.COM> > To MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Date 8/22/2024 5:25:08 PM > Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE > IX > > I am not sure what Google's stance is on deploying cache servers at an IX > vs remote connection like the ones pending disconnection. > If they were to deploy caching nodes at an IX, I assume they would still > have a cache-fill requirement in terms of connectivity that has to come > from somewhere... of which they appear to be disinterested in maintaining > long distance connections going forward. So perhaps this is a donation > opportunity for some to entice Google to deploy caching nodes at MICE? > Donate local colo and transit bandwidth along with local MICE connectivity? > > We have numerous Google caching nodes in our network as well as multiple > PNIs to Google, so their pending disconnect from MICE will likely have > minimal direct impact on our eyeballs. > > > -----Original Message----- > *From*: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET > <Aaron%20Wendel%20%3caaron(a)WHOLESALEINTERNET.NET%3e>> > *Reply-To*: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET > <MICE%20Discuss%20%3cMICE-DISCUSS(a)LISTS.IPHOUSE.NET%3e>> > *To*: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject*: Re: [MICE-DISCUSS] Peering with Google no longer available at > MICE IX > *Date*: Thu, 22 Aug 2024 22:03:01 +0000 > > Has MICE requested a cache box from Google? > > > > ------ Original Message ------ > From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> > To MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Date 8/22/2024 4:28:18 PM > Subject Re: [MICE-DISCUSS] Peering with Google no longer available at > MICE IX > > I've been surprised that folks haven't been talking about this more and am > curious about everyone's thoughts - as MICE user-members but also in terms > of MICE leadership and direction. > > As a user-member I feel somewhat conflicted: on one hand in an effort to > maximize performance I could obtain transport to Chicago and connect to > Google there, but I feel like doing so is making Google's problem into my > problem and that doing so undercuts the goals of MICE. We've seen content > providers move MICE's larger members toward local direct private peering, > and I can understand rational behind that even if it shifts traffic away > from MICE - but Google is proposing remote direct peering at member's cost > which feels different. > > Of course nobody expects Google to subsidize our corner of the world, I am > just curious if there are other aspects we should be thinking about. > > Thanks in advance, > Anthony > > > > On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" < > peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> > wrote: > > Dear Peer, > > This is a reminder that peering with Google will no longer be available at > MICE Internet Exchange from 16th of September. > > As a result of unsustainable overheads associated with connecting to an IX > using remote waves, we have made a decision to stop connecting to IXs that > way. MICE IX falls under the category of IXs we remotely connect to. > > We will withdraw prefixes on 16th of September, and disconnect from the IX > on 24th of September. > > We realize this may cause inconvenience for some peers. If there is another > mutually present IX that we can peer in, we will be happy to help set up a > bilateral session there - no multilaterals. If there is no mutual IX, > traffic may have to be exchanged via indirect paths (transit links). > > We are not looking to connect to any additional IXs to replace the IX that > we are leaving, but if you are considering connecting to another IX and not > sure if Google will also be leaving that IX because of this measure, please > reach out to us and we can share more. > > Thank you for your understanding. > > > Sincerely, > Google Network Operations > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by Jeremy Lumby

I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors.I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade.Jeremy LumbyMinnesota VoIP9217 17th Ave SSuite 216Bloomington MN 55425Main 612-355-7740Direct 612-392-6814EFax 952-873-7425jlumby(a)mnvoip.com -------- Original message --------From: Miles McCredie <Miles.McCredie(a)MIDCO.COM> Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 23, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX

by Miles McCredie

Curious how many MICE members are in each of the following buckets. 1. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket 1. Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET<mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET>> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com<mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com>>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 23, 2024

RFP: 511 IP Transit

by Richard Laager

MICE is in need of IP transit at 511. This is for our management connection. We are looking for IPv4 + IPv6 transit delivered over a 1G copper (if in the same suite), 1G optical, or 10G optical. MICE will take care of the cross-connect. No BGP necessary. We need a /29 of IPv4 space and a /60 of IPv6 space. The current IP space is routed to us over a separate point-to-point (e.g. the /29 is over a /31 point-to-point). Our actual bandwidth usage is very minimal; 100 Mbps is more than plenty. Please specify the speed you are offering. I don't know that we have a specific term length in mind. Month-to-month, 1 year, or 3 year is probably fine. Please specify the term length(s) you are offering. You can reply off-list to Anthony Anderberg <AnthonyAnderberg(a)nuvera.net>. He intends NOT to bid on this. Conflict of interest note: My employer will bid on this. I will recuse from the board vote on it. -- Richard

Aug. 23, 2024

Re: Peering with Google no longer available at MICE IX

by Aaron Wendel

For some historical context: A few years ago Google decided to pull their infrastructure out of Minneapolis, Kansas City and Saint Louis. The dominant exchanges in each market were given a choice between back-hauled peering connections or cache boxes. Minneapolis worked a deal for the back-hauled connection to Chicago. Kansas City and Saint Louis opted for cache servers. The main reason for the cache server choice was the concern over the long term sustainability and scalability of a remote peer of Google's size. Over the subsequent years, Google has shipped more and more servers as demand has increased. They have been very accommodating with the equipment. One other benefit to the cache servers is that they were not affected by Google's recent disconnection from ALL route servers. Google bears the cost for hardware upkeep. KCIX and STLIX use donated resources for cache fill, space and power. I understand MICE does not enjoy the resources that KCIX (and to a lesser extent STLIX) has access to but if Google is still shipping new cache servers (they may not be) that seems like a much better choice than remote peering to Chicago. Aaron ------ Original Message ------ >From "Justin Krejci" <JKrejci(a)USINTERNET.COM> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET Date 8/22/2024 5:25:08 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX >I am not sure what Google's stance is on deploying cache servers at an >IX vs remote connection like the ones pending disconnection. >If they were to deploy caching nodes at an IX, I assume they would >still have a cache-fill requirement in terms of connectivity that has >to come from somewhere... of which they appear to be disinterested in >maintaining long distance connections going forward. So perhaps this is >a donation opportunity for some to entice Google to deploy caching >nodes at MICE? Donate local colo and transit bandwidth along with local >MICE connectivity? > >We have numerous Google caching nodes in our network as well as >multiple PNIs to Google, so their pending disconnect from MICE will >likely have minimal direct impact on our eyeballs. > > >-----Original Message----- >From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET ><mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> >Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET ><mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at >MICE IX >Date: Thu, 22 Aug 2024 22:03:01 +0000 > >Has MICE requested a cache box from Google? > > > >------ Original Message ------ >From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> >To MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Date 8/22/2024 4:28:18 PM >Subject Re: [MICE-DISCUSS] Peering with Google no longer available at >MICE IX > >>I've been surprised that folks haven't been talking about this more >>and am curious about everyone's thoughts - as MICE user-members but >>also in terms of MICE leadership and direction. >> >>As a user-member I feel somewhat conflicted: on one hand in an effort >>to maximize performance I could obtain transport to Chicago and >>connect to Google there, but I feel like doing so is making Google's >>problem into my problem and that doing so undercuts the goals of MICE. >> We've seen content providers move MICE's larger members toward local >>direct private peering, and I can understand rational behind that even >>if it shifts traffic away from MICE - but Google is proposing remote >>direct peering at member's cost which feels different. >> >>Of course nobody expects Google to subsidize our corner of the world, >>I am just curious if there are other aspects we should be thinking >>about. >> >>Thanks in advance, >>Anthony >> >> >> >>On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" >><peering-ops-noreply(a)google.com >><mailto:peering-ops-noreply@google.com>> wrote: >> >>Dear Peer, >> >>This is a reminder that peering with Google will no longer be >>available at >>MICE Internet Exchange from 16th of September. >> >>As a result of unsustainable overheads associated with connecting to >>an IX >>using remote waves, we have made a decision to stop connecting to IXs >>that >>way. MICE IX falls under the category of IXs we remotely connect to. >> >>We will withdraw prefixes on 16th of September, and disconnect from >>the IX >>on 24th of September. >> >>We realize this may cause inconvenience for some peers. If there is >>another >>mutually present IX that we can peer in, we will be happy to help set >>up a >>bilateral session there - no multilaterals. If there is no mutual IX, >>traffic may have to be exchanged via indirect paths (transit links). >> >>We are not looking to connect to any additional IXs to replace the IX >>that >>we are leaving, but if you are considering connecting to another IX >>and not >>sure if Google will also be leaving that IX because of this measure, >>please >>reach out to us and we can share more. >> >>Thank you for your understanding. >> >> >>Sincerely, >>Google Network Operations >> >> -------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 22, 2024

Re: Peering with Google no longer available at MICE IX

by Justin Krejci

I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron(a)WHOLESALEINTERNET.NET<mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET>> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com<mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com>>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations

Aug. 22, 2024

Re: Peering with Google no longer available at MICE IX

by Jeremy Lumby

It has been discussed in the past, however Google wanted MICE to cover all of the costs related to it. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Aaron Wendel Sent: Thursday, August 22, 2024 5:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Has MICE requested a cache box from Google? ------ Original Message ------ >From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX >I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. > >As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. > >Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. > >Thanks in advance, >Anthony > > > >On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> wrote: > >Dear Peer, > >This is a reminder that peering with Google will no longer be available at >MICE Internet Exchange from 16th of September. > >As a result of unsustainable overheads associated with connecting to an IX >using remote waves, we have made a decision to stop connecting to IXs that >way. MICE IX falls under the category of IXs we remotely connect to. > >We will withdraw prefixes on 16th of September, and disconnect from the IX >on 24th of September. > >We realize this may cause inconvenience for some peers. If there is another >mutually present IX that we can peer in, we will be happy to help set up a >bilateral session there - no multilaterals. If there is no mutual IX, >traffic may have to be exchanged via indirect paths (transit links). > >We are not looking to connect to any additional IXs to replace the IX that >we are leaving, but if you are considering connecting to another IX and not >sure if Google will also be leaving that IX because of this measure, please >reach out to us and we can share more. > >Thank you for your understanding. > > >Sincerely, >Google Network Operations > >

Aug. 22, 2024

Re: Peering with Google no longer available at MICE IX

by Aaron Wendel

Has MICE requested a cache box from Google? ------ Original Message ------ >From "Anthony Anderberg" <AnthonyAnderberg(a)NUVERA.NET> To MICE-DISCUSS(a)LISTS.IPHOUSE.NET Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX >I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. > >As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. > >Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. > >Thanks in advance, >Anthony > > > >On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> wrote: > >Dear Peer, > >This is a reminder that peering with Google will no longer be available at >MICE Internet Exchange from 16th of September. > >As a result of unsustainable overheads associated with connecting to an IX >using remote waves, we have made a decision to stop connecting to IXs that >way. MICE IX falls under the category of IXs we remotely connect to. > >We will withdraw prefixes on 16th of September, and disconnect from the IX >on 24th of September. > >We realize this may cause inconvenience for some peers. If there is another >mutually present IX that we can peer in, we will be happy to help set up a >bilateral session there - no multilaterals. If there is no mutual IX, >traffic may have to be exchanged via indirect paths (transit links). > >We are not looking to connect to any additional IXs to replace the IX that >we are leaving, but if you are considering connecting to another IX and not >sure if Google will also be leaving that IX because of this measure, please >reach out to us and we can share more. > >Thank you for your understanding. > > >Sincerely, >Google Network Operations > >

Aug. 22, 2024

Re: Peering with Google no longer available at MICE IX

by Anthony Anderberg

I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply(a)google.com <mailto:peering-ops-noreply@google.com>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations

Aug. 22, 2024

Please welcome Warrior Hill to MICE!

by Jeremy Lumby

Name: Warrior Hill, LLC Switch: MICE Core - Ridgeview Ports: E 3/31/3 Strands 125-126 for a total of 1x10G ASN: 26329 IPv4: 206.108.255.232/24 IPv6: 2001:504:27:0:0:66D9::1/64 Peering Contact: hostmaster(a)warriorhill.com Route Servers: YES Welcome Aboard! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

Aug. 9, 2024

Re: StackPath has left MICE

by Chris Wopat

For those that didn't see: * https://www.datacenterdynamics.com/en/news/stackpath-to-close-down-liquidat… * https://43963146.hs-sites.com/important-notice-stackpath-immediate-end-of-l… "After careful consideration, it has been decided to close all StackPath products and liquidate all assets for the benefit of our creditors. Effective immediately, we will begin the decommissioning of all StackPath services..." On Tue, Jul 9, 2024 at 4:06 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > StackPath has left MICE. If you have any bilaterals to > 206.108.255.189 2001:504:27:0:0:829E::1 206.108.255.190 or > 2001:504:27:0:0:829E::2 please remove them. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington MN 55425 > Main 612-355-7740 x211 > Direct 612-392-6814 > Fax 952-873-7425 > jlumby(a)mnvoip.com >

July 9, 2024

StackPath has left MICE

by Jeremy Lumby

StackPath has left MICE. If you have any bilaterals to 206.108.255.189 2001:504:27:0:0:829E::1 206.108.255.190 or 2001:504:27:0:0:829E::2 please remove them. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com

July 9, 2024

Re: bgp down to 206.108.255.117 and 2001:504:27::80a6:0:2 for both as3128 and as2381.. others?

by Anthony Anderberg

It looks like one of Facebook's two links didn't come back up after some kind of overnight maintenance: http://micelg.usinternet.com/cacti/graph.php?rra_id=all&local_graph_id=526 http://micelg.usinternet.com/cacti/graph.php?rra_id=all&local_graph_id=3604 The only other thing I see is that DCN has had one of their links down since yesterday, but I doubt that's related. Cheers, Anthony On 7/9/24, 12:48 PM, "MICE Discuss on behalf of Michael Hare" <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> on behalf of 000000097dab80c5-dmarc-request(a)LISTS.IPHOUSE.NET <mailto:000000097dab80c5-dmarc-request@LISTS.IPHOUSE.NET>> wrote: As the subject says, at least two ASN seem to bgp down v4/v6 as of roughly 03:48:42 local [UTC-5]. Is this a bigger thing? Apologies if I missed something. -Michael

July 9, 2024

Re: bgp down to 206.108.255.117 and 2001:504:27::80a6:0:2 for both as3128 and as2381.. others?

by Jeremy Lumby

I have not heard of any planned maintenance from that peer. Their redundant connection is up and passing traffic. -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Michael Hare Sent: Tuesday, July 9, 2024 12:49 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] bgp down to 206.108.255.117 and 2001:504:27::80a6:0:2 for both as3128 and as2381.. others? As the subject says, at least two ASN seem to bgp down v4/v6 as of roughly 03:48:42 local [UTC-5]. Is this a bigger thing? Apologies if I missed something. -Michael

July 9, 2024

bgp down to 206.108.255.117 and 2001:504:27::80a6:0:2 for both as3128 and as2381.. others?

by Michael Hare

As the subject says, at least two ASN seem to bgp down v4/v6 as of roughly 03:48:42 local [UTC-5]. Is this a bigger thing? Apologies if I missed something. -Michael

July 9, 2024

Please welcome Muscatine Power and Water to MICE!

by Jeremy Lumby

Name: Muscatine Power and Water Switch: SouthFront Networks Ports: E 11/1 for a total of 1x100G ASN: 33734 IPv4: All IP info remains the same, this is a port upgrade to 100G IPv6: Peering Contact: dominickl(a)mpw.org Route Servers: YES Welcome Aboard Again! Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 Main: 612-355-7740 Direct: 612-392-6814 Fax: 952-873-7425 jlumby(a)mnvoip.com

July 4, 2024

Global Net (AS395466) - Transport Outages

by Global Net NOC

MICEMN Members – We have been experiencing ongoing outages with our carrier into Minneapolis (whom I will not name) which directly affects our connectivity with MICE. I want to apologize for the drops in our BGP sessions that seem to be happening more and more frequently as our carrier refuses to do anything to address the problem. We are actively seeking another transport option as a long term solution. Short term, all I can do is apologize for all the flaps and outages. GLOBAL NET AS 395466 IPV4 206.108.255.97 IPV6 2001:504:27:0:6:8ca:0:1 Thanks, Global Net NOC noc(a)theglobal.net<mailto:noc@theglobal.net> Global Net Peering peering(a)theglobal.net<mailto:peering@theglobal.net>

June 24, 2024