- mice-discuss - lists.micemn.net

Re: [EXT]: Re: [MICE-DISCUSS] [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue

by Danny Meister

Sept. 7, 2024

Re: [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue

by Miles McCredie

The graph definitely looks better. [cid:image002.png@01DB0083.B32035F0] Danny, Isaiah - Are the downloads back to normal? [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Friday, September 6, 2024 2:40 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - Re: [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image003.png@01DB0083.B32035F0]<https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Jeremy Lumby

I have been informed that Amazon has corrected the issue, and things are looking good now. Is everything testing out OK now? From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Chris Wopat

That is such a high error rate that we should perhaps just shut the port down so the traffic routes elsewhere, properly. Based on the graph it appears to be a failing optic - could perhaps be confirmed if Cacti is trending TX/RX power and TX Bias. --Chris >

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Jay Hanke

Jeremy caught the issue and reached out to Amazon a couple of weeks ago regarding the areas. It might help for those having issues to reach out to Amazon directly to knock it loose on their side. On Fri, Sep 6, 2024, 12:36 PM Isaiah Olson <iolson(a)paulbunyan.net> wrote: > It looks like Amazon’s port Ethernet6/1/1 on the MICE Core switch has some > pretty significant errors. > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Miles > McCredie > *Sent:* Friday, September 06, 2024 12:12 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS > transfer issue > > > > Maybe someone could check for errors on the MICE ports for Atomic Data and > Amazon. > > > > Assuming the MICE ports are clean, it is probably worth following up with > the Amazon NOC, peering-to(a)amazon.com. They did some work on their > Minneapolis peering routers recently though I’m not sure what they were > doing. > > > > > > > > [image: Image removed by sender.] > > *Miles McCredie * > Principal Network Engineer II-Core IP > > Office: 6052755192 > Miles.McCredie(a)midco.com > > *Midco.com* > > Let's go beyond. > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Danny > Meister > *Sent:* Friday, September 6, 2024 11:41 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue > > > > *CAUTION:* This email originated from outside of *MIDCO*. > *Do not click links or open attachments unless you recognize the sender > and know the content is safe.* > > Good Afternoon, > > > > I've been trying to track down a strange packet loss issue when a customer > of mine is downloading files from S3 in the east region. We peer with AWS > via MICE and traffic to east routes via MICE. > > > > Could any of you folks peered with AWS via MICE try downloading this file > a bunch of times and reports the results? > > > > > http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… > > > > > Basically, we see fast transfers maybe 80% of the time and then it will be > slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 > instance also). Captures show lots of retransmits, etc during a slow > transfer. If I shut down my AWS peering the problem goes away. I was able > to test from another client of mine connected to Arvig and I'm pretty sure > I can replicate the issue from there. So, I'm not convinced it's just me. > On the other hand, if there is a problem, I'm surprised no one else has > noticed! > > > > Thanks! > > *Danny Meister * > > Infrastructure Engineer > > 612.466.2071 | DMeister(a)atomicdata.com > > [image: Logo] > <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> > > 250 Marquette Avenue > <https://www.google.com/maps/search/250+Marquette+Avenue?entry=gmail&source=g> > South | Minneapolis, MN | 55401 > > *atomicdata.com* > <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> > | *Services* > <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> > | *Blog* > <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> > *Atomic Data Welcomes Chris Heim as CEO* > <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> > > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Isaiah Olson

It looks like Amazon's port Ethernet6/1/1 on the MICE Core switch has some pretty significant errors. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Miles McCredie Sent: Friday, September 06, 2024 12:12 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to(a)amazon.com <mailto:peering-to@amazon.com> . They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign ature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email > Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue

by Miles McCredie

Maybe someone could check for errors on the MICE ports for Atomic Data and Amazon. Assuming the MICE ports are clean, it is probably worth following up with the Amazon NOC, peering-to(a)amazon.com<mailto:peering-to@amazon.com>. They did some work on their Minneapolis peering routers recently though I'm not sure what they were doing. [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com Let's go beyond. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [EXTERNAL] - [MICE-DISCUSS] Odd AWS transfer issue CAUTION: This email originated from outside of MIDCO. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws… Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [cid:image001.png@01DB0055.EE8B7A60]<https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://www.atomicdata.com/?utm_campaign=signature&utm_source=EmailSign…> | Services<https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=Email> | Blog<https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email> Atomic Data Welcomes Chris Heim as CEO<https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Jeremy Lumby

Amazon is aware of a condition that is likely causing this intermittent issue. Last I heard it should be repaired early next week. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 6, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com <mailto:DMeister@atomicdata.com> <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Isaiah Olson

I was able to replicate this slowness on my first try. It appears that the specific node I was talking to when seeing the slow download was 54.231.233.205 and it took a couple of minutes to complete, averaging 100 KB/s. We only peer with Amazon via MICE. Hope the information is helpful, and please let us know if you find anything. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Danny Meister Sent: Friday, September 06, 2024 11:41 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Odd AWS transfer issue Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? <http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws .com/empty_test_file_50M.txt> http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws. com/empty_test_file_50M.txt Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | <mailto:DMeister@atomicdata.com> DMeister(a)atomicdata.com <https://www.atomicdata.com/?utm_campaign=signature&amp;amp;utm_source=E mailSignature> 250 Marquette Avenue South | Minneapolis, MN | 55401 <https://www.atomicdata.com/?utm_campaign=signature&amp;utm_source=Email Signature> atomicdata.com | <https://www.atomicdata.com/services?utm_campaign=Signature&utm_source=E mail> Services | <https://www.atomicdata.com/blog?utm_campaign=Signature&utm_source=Email > Blog <https://www.atomicdata.com/atomic-data-welcomes-chris-heim-as-ceo/?utm_sour ce=EmailSignature> Atomic Data Welcomes Chris Heim as CEO _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

Sept. 6, 2024

Re: Odd AWS transfer issue

by Justin Krejci

My path takes one of our PNI connections to Amazon, not via MICE. The download via PNI in Minneapolis it took 2.3 seconds. -----Original Message----- From: Ryan Malek <rmalek(a)ROUTER12.NET<mailto:Ryan%20Malek%20%3crmalek@ROUTER12.NET%3e>> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE%20Discuss%20%3cMICE-DISCUSS@LISTS.IPHOUSE.NET%3e>> To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Odd AWS transfer issue Date: Fri, 06 Sep 2024 11:47:47 -0500 Downloaded nearly instantly, no problem. ________________________________ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 9/6/2024 11:41 AM, Danny Meister wrote: Good Afternoon, I've been trying to track down a strange packet loss issue when a customer of mine is downloading files from S3 in the east region. We peer with AWS via MICE and traffic to east routes via MICE. Could any of you folks peered with AWS via MICE try downloading this file a bunch of times and reports the results? http://ofw-atomic-data-test-region-us-east-1.s3-website-us-east-1.amazonaws…<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> Basically, we see fast transfers maybe 80% of the time and then it will be slow, like sub 1Mb slow (we can replicate the issue with iperf to an EC2 instance also). Captures show lots of retransmits, etc during a slow transfer. If I shut down my AWS peering the problem goes away. I was able to test from another client of mine connected to Arvig and I'm pretty sure I can replicate the issue from there. So, I'm not convinced it's just me. On the other hand, if there is a problem, I'm surprised no one else has noticed! Thanks! Danny Meister Infrastructure Engineer 612.466.2071 | DMeister(a)atomicdata.com<mailto:DMeister@atomicdata.com> [Logo]<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> 250 Marquette Avenue South | Minneapolis, MN | 55401 atomicdata.com<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> | Services<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> | Blog<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> Atomic Data Welcomes Chris Heim as CEO<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://url-shield.securence.com/?p=1.0&r=jkrejci@usinternet.com&sid=172564…>

Sept. 6, 2024