Re: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX
I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors.I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade.Jeremy LumbyMinnesota VoIP9217 17th Ave SSuite 216Bloomington MN 55425Main 612-355-7740Direct 612-392-6814EFax 952-873-7425jlumby@mnvoip.com -------- Original message --------From: Miles McCredie <Miles.McCredie@MIDCO.COM> Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron@WHOLESALEINTERNET.NET> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg@NUVERA.NET> To MICE-DISCUSS@LISTS.IPHOUSE.NET Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply@google.com <mailto:peering-ops-noreply@google.com>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Interesting. I’d be interested in any/all of average/P95/P99/Peak traffic over the last month on the Google ports but understand it’s not really any of my business. Is there a trend line over the last quarter or two? I have heard second hand that MICE ran into some unpleasantness around cache back in the day but I, for one, think that it would be worthwhile exploring the option of hosting caches again. (Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊) As someone pointed out up thread, there are a handful of issues to sort out in this area. [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, August 23, 2024 8:40 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors. I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby@mnvoip.com<mailto:jlumby@mnvoip.com> -------- Original message -------- From: Miles McCredie <Miles.McCredie@MIDCO.COM<mailto:Miles.McCredie@MIDCO.COM>> Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. 1. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket 1. Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com<mailto:Miles.McCredie@midco.com> Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron@WHOLESALEINTERNET.NET<mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------ From "Anthony Anderberg" <AnthonyAnderberg@NUVERA.NET<mailto:AnthonyAnderberg@NUVERA.NET>> To MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com<mailto:peering-ops-noreply@google.com>>> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
They were and then they pulled out. Google's peering philosophy seems to change about every 3-5 years so it wouldn't surprise me at all if they build out to the midwest (outside of Chicago) again. As someone who is in the IX, Datacenter and Access businesses I can tell you that I've seen a major migration of content networks into tier 2/3 markets, especially in the Midwest/Great Plains areas over the last year or two. Google will come back eventually. Aaron ------ Original Message ------
(Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊)
There will be more to come for Google in Minneapolis. They pulled the plug on their Becker DC during COVID; that was a significant setback for us. But it is inconceivable that they won't do something; it just takes time. On Fri, Aug 23, 2024 at 9:29 AM Aaron Wendel <aaron@wholesaleinternet.net> wrote:
They were and then they pulled out.
Google's peering philosophy seems to change about every 3-5 years so it wouldn't surprise me at all if they build out to the midwest (outside of Chicago) again.
As someone who is in the IX, Datacenter and Access businesses I can tell you that I've seen a major migration of content networks into tier 2/3 markets, especially in the Midwest/Great Plains areas over the last year or two. Google will come back eventually.
Aaron
------ Original Message ------
(Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊)
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
You can view the ports on the weathermap. The short version is that they have been using BGP to redirect traffic away from MICE for years. Each of the 100G ports ramps up to 90G sits there for about 18 hours, and then drops to reasonable levels for the late night/early morning hours. Repeat every day since shortly after their last upgrade several years ago. By my estimation they need 600G to MICE just to not saturate/not need to redirect traffic away from MICE. While I am not on the board, at those traffic levels Google can justify doing something on their own, and not need to look for handouts to get to MICE. Google has received a lot of free assistance in connecting to MICE in the past. Now they are unwilling to even have an open conversation about anything that would assist keeping them DIRECTLY on MICE as opposed to some sort of workaround where they are not taking on ANY of the recurring costs. Jeremy From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Miles McCredie Sent: Friday, August 23, 2024 9:00 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Interesting. I’d be interested in any/all of average/P95/P99/Peak traffic over the last month on the Google ports but understand it’s not really any of my business. Is there a trend line over the last quarter or two? I have heard second hand that MICE ran into some unpleasantness around cache back in the day but I, for one, think that it would be worthwhile exploring the option of hosting caches again. (Would be even better for Google to build in to Minneapolis so that we can all turn up more local PNIs but they haven’t been interested in my views on this. 😊) As someone pointed out up thread, there are a handful of issues to sort out in this area. <http://images.midcocomm.com/ES_MidcoLogo.png> Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com <mailto:Miles.McCredie@midco.com> Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Jeremy Lumby Sent: Friday, August 23, 2024 8:40 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX I would be willing to bet if you looked just at the eyeball networks on MICE, 90% are getting the majority of their Google traffic across MICE, even if they have caches they are getting the fill across MICE in addition to the non cacheable traffic. Google's caches are not nearly as efficient as many of their competitors. I can confirm with over 10 years of experience dealing with Google that they are almost always very slow to upgrade. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby@mnvoip.com <mailto:jlumby@mnvoip.com> -------- Original message -------- From: Miles McCredie <Miles.McCredie@MIDCO.COM <mailto:Miles.McCredie@MIDCO.COM> > Date: 8/23/24 6:34 AM (GMT-06:00) To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Curious how many MICE members are in each of the following buckets. 1. Receive limited portion of Google traffic via MICE Midco and, I assume, US Internet are in this bucket 2. Receive a significant portion of Google traffic via MICE today I’ve only got a couple years of experience with Google for Midco but my experience has been that Google will not always provide the cache capacity that I think is appropriate. The last two augments were triggered 9-12 months later than I would have liked. Could be my fault for creating new GNLs but… <http://images.midcocomm.com/ES_MidcoLogo.png> Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Justin Krejci Sent: Thursday, August 22, 2024 5:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I am not sure what Google's stance is on deploying cache servers at an IX vs remote connection like the ones pending disconnection. If they were to deploy caching nodes at an IX, I assume they would still have a cache-fill requirement in terms of connectivity that has to come from somewhere... of which they appear to be disinterested in maintaining long distance connections going forward. So perhaps this is a donation opportunity for some to entice Google to deploy caching nodes at MICE? Donate local colo and transit bandwidth along with local MICE connectivity? We have numerous Google caching nodes in our network as well as multiple PNIs to Google, so their pending disconnect from MICE will likely have minimal direct impact on our eyeballs. -----Original Message----- From: Aaron Wendel <aaron@WHOLESALEINTERNET.NET <mailto:Aaron%20Wendel%20%3caaron@WHOLESALEINTERNET.NET%3e> > Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e> > To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX Date: Thu, 22 Aug 2024 22:03:01 +0000 Has MICE requested a cache box from Google? ------ Original Message ------
From "Anthony Anderberg" <AnthonyAnderberg@NUVERA.NET <mailto:AnthonyAnderberg@NUVERA.NET> > To MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Date 8/22/2024 4:28:18 PM Subject Re: [MICE-DISCUSS] Peering with Google no longer available at MICE IX
I've been surprised that folks haven't been talking about this more and am curious about everyone's thoughts - as MICE user-members but also in terms of MICE leadership and direction. As a user-member I feel somewhat conflicted: on one hand in an effort to maximize performance I could obtain transport to Chicago and connect to Google there, but I feel like doing so is making Google's problem into my problem and that doing so undercuts the goals of MICE. We've seen content providers move MICE's larger members toward local direct private peering, and I can understand rational behind that even if it shifts traffic away from MICE - but Google is proposing remote direct peering at member's cost which feels different. Of course nobody expects Google to subsidize our corner of the world, I am just curious if there are other aspects we should be thinking about. Thanks in advance, Anthony On 8/22/24, 7:24 AM, "Google Peering Operations No-reply" <peering-ops-noreply@google.com <mailto:peering-ops-noreply@google.com> <mailto:peering-ops-noreply@google.com <mailto:peering-ops-noreply@google.com> >> wrote: Dear Peer, This is a reminder that peering with Google will no longer be available at MICE Internet Exchange from 16th of September. As a result of unsustainable overheads associated with connecting to an IX using remote waves, we have made a decision to stop connecting to IXs that way. MICE IX falls under the category of IXs we remotely connect to. We will withdraw prefixes on 16th of September, and disconnect from the IX on 24th of September. We realize this may cause inconvenience for some peers. If there is another mutually present IX that we can peer in, we will be happy to help set up a bilateral session there - no multilaterals. If there is no mutual IX, traffic may have to be exchanged via indirect paths (transit links). We are not looking to connect to any additional IXs to replace the IX that we are leaving, but if you are considering connecting to another IX and not sure if Google will also be leaving that IX because of this measure, please reach out to us and we can share more. Thank you for your understanding. Sincerely, Google Network Operations _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
participants (4)
-
Aaron Wendel -
David Farmer -
Jeremy Lumby -
Miles McCredie