Downloaded nearly instantly, no problem. ------------------------------------------------------------------------ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 9/6/2024 11:41 AM, Danny Meister wrote:
Good Afternoon,
I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE.
Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results?
http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws....
Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed!
Thanks!
*Danny Meister *
Infrastructure Engineer
612.466.2071 | DMeister@atomicdata.com
Logo <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=EmailSignature>
250 Marquette Avenue South | Minneapolis, MN | 55401
*atomicdata.com* <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature>|*Services* <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email>| *Blog* <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_source=EmailSignature>
------------------------------------------------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
My path takes one of our PNI connections to Amazon, not via MICE. The download via PNI in Minneapolis it took 2.3 seconds. -----Original Message----- From: Ryan Malek <rmalek@ROUTER12.NET<mailto:Ryan%20Malek%20%3crmalek@ROUTER12.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Odd AWS transfer issue Date: Fri, 06 Sep 2024 11:47:47 -0500 Downloaded nearly instantly, no problem. ________________________________ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 9/6/2024 11:41 AM, Danny Meister wrote: Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws.com/empty_test_file_50M.txt<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=ggvbibif&n=bshedynhz&ms=0.0,0.0,0.0,0.0&u=http%3A%2F%2Fofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws.com%2Fempty_test_file_50M.txt> Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com<mailto:DMeister@atomicdata.com> [Logo]<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=scaxh3gm&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=https%3A%2F%2Fwww.atomicdata.com%2F%3Futm_campaign%3Dsignature%26amp%3Bamp%3Bamp%3Butm_source%3DEmailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=eaijah6f&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=https%3A%2F%2Fwww.atomicdata.com%2F%3Futm_campaign%3Dsignature%26amp%3Bamp%3Butm_source%3DEmailSignature> | Services<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=wjrzwthg&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=https%3A%2F%2Fwww.atomicdata.com%2Fservices%3Futm_campaign%3DSignature%26amp%3Butm_source%3DEmail> | Blog<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=kgptyuzx&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=https%3A%2F%2Fwww.atomicdata.com%2Fblog%3Futm_campaign%3DSignature%26amp%3Butm_source%3DEmail> Atomic Data Welcomes Chris Heim as CEO<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=ujtzlsft&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=https%3A%2F%2Fwww.atomicdata.com%2Fatomic-data-welcomes-chris-heim-as-ceo%2F%3Futm_source%3DEmailSignature> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=pppykv3s&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=1725641274324-084-00147494&s=pppykv3s&n=bshedynh2&ms=0.0,0.0,0.0,0.0&u=http%3A%2F%2Flists.iphouse.net%2Fcgi-bin%2Fwa%3FSUBED1%3DMICE-DISCUSS%26A%3D1>
I was able to replicate this slowness on my first try. It appears that the specific node I was talking to when seeing the slow download was 54.231.233.205 and it took a couple of minutes to complete, averaging 100 KB/s. We only peer with Amazon via MICE. Hope the information is helpful, and please let us know if you find anything. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 06, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? <http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws .com/empty_test_file_50M.txt> http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | <mailto:DMeister@atomicdata.com> DMeister@atomicdata.com <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email
Blog
<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
Amazon is aware of a condition that is likely causing this intermittent issue. Last I heard it should be repaired early next week. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email
Blog
<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email
Blog
<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
The graph definitely looks better. [cid:image002.png@01DB0083.B32035F0] Danny, Isaiah - Are the downloads back to normal? [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, September 6, 2024 2:40 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws.... Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image003.png@01DB0083.B32035F0]<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_source=EmailSignature> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to@amazon.com<mailto:peering-to@amazon.com>. They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws.... Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image001.png@01DB0055.EE8B7A60]<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_source=EmailSignature> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
It looks like Amazon's port Ethernet6/1/1 on the MICE Core switch has some pretty significant errors. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Miles McCredie Sent: Friday, September 06, 2024 12:12 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to@amazon.com <mailto:peering-to@amazon.com> . They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie@midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister@atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&utm_source=Email Signature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign ature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email
Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Blog
<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1
Jeremy caught the issue and reached out to Amazon a couple of weeks ago regarding the areas. It might help for those having issues to reach out to Amazon directly to knock it loose on their side. On Fri, Sep 6, 2024, 12:36 PM Isaiah Olson <iolson@paulbunyan.net> wrote:
It looks like Amazon’s port Ethernet6/1/1 on the MICE Core switch has some pretty significant errors.
*From:* MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *On Behalf Of *Miles McCredie *Sent:* Friday, September 06, 2024 12:12 PM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue
Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon.
Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to@amazon.com. They did some work on their Minneapolis peering routers recently though I’m not sure what they were doing.
[image: Image removed by sender.]
*Miles McCredie * Principal Network Engineer II-Core IP
Office: 6052755192 Miles.McCredie@midco.com
*Midco.com*
Let's go beyond.
*From:* MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> *On Behalf Of *Danny Meister *Sent:* Friday, September 6, 2024 11:41 AM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue
*CAUTION:* This email originated from outside of *MIDCO*. *Do not click links or open attachments unless you recognize the sender and know the content is safe.*
Good Afternoon,
I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE.
Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results?
http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws....
Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed!
Thanks!
*Danny Meister *
Infrastructure Engineer
612.466.2071 | DMeister@atomicdata.com
[image: Logo] <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature>
250 Marquette Avenue <https://www.google.com/maps/search/250+Marquette+Avenue?entry=gmail&source=g> South | Minneapolis, MN | 55401
*atomicdata.com* <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSignature> | *Services* <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | *Blog* <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> *Atomic Data Welcomes Chris Heim as CEO* <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_source=EmailSignature>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
That is such a high error rate that we should perhaps just shut the port down so the traffic routes elsewhere, properly. Based on the graph it appears to be a failing optic - could perhaps be confirmed if Cacti is trending TX/RX power and TX Bias. --Chris
participants (8)
-
Chris Wopat -
Danny Meister -
Isaiah Olson -
Jay Hanke -
Jeremy Lumby -
Justin Krejci -
Miles McCredie -
Ryan Malek