What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE's own aggregate is the same. [cid:image004.jpg@01D2190B.83B53030] [cid:image003.png@01D2190B.83A5EDF0] Frank
Looks like it's the Google LAG: [cid:image001.png@01D2190B.D3667E90] From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? Saw our own MICE-facing traffic drop down, and I see MICE's own aggregate is the same.
We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too. -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
The tech team has some open tickets at the moment with Google. They said they were escalating them. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too. -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Yesterday I noticed they were running into a limit to the number of IPv6 prefixes they'd accept from MICE (now over 15K) but hadn't seen any issues beyond that. Their GGC team said they were looking into it this morning, I'll ping them and mention these issues as well. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too. -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Thanks for reaching out to your Google contacts. Are you saying that MICE's route reflectors have been seeing their BGP sessions to Google's cache boxes flapping over the last few days? Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too. -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We have bilateral sessions to them, not sure on the route reflectors. Can someone check logs/BGP uptime there? -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:25 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Thanks for reaching out to your Google contacts.
Are you saying that MICE's route reflectors have been seeing their BGP sessions to Google's cache boxes flapping over the last few days?
Frank
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too.
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Looks the same as yesterday on both servers: 206.108.255.37 BGP master up 22:38:09 Established 2001:504:27:0:0:8cc8::1 BGP master start 22:42:34 Idle Received: Maximum number of prefixes reached But I see we've started getting these BIRD log messages: 016-09-26 17:28:45 <RMT> NASN_36040: Received: Other configuration change Additionally my ping generated a "We're actively troubleshooting this issue" reply, so it all seems to fit together. Cheers, anthony Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:40 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? We have bilateral sessions to them, not sure on the route reflectors. Can someone check logs/BGP uptime there? -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:25 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Thanks for reaching out to your Google contacts.
Are you saying that MICE's route reflectors have been seeing their BGP sessions to Google's cache boxes flapping over the last few days?
Frank
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too.
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
That’s what we’re seeing too: er1.511.msp rpd[1461]: bgp_read_v4_message:10672: NOTIFICATION received from 206.108.255.37 (External AS 36040): code 6 (Cease) subcode 6 (Other Configuration Change) (which on JunOS, actually flaps the session) -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:56 PM, anthonyanderberg@nu-telecom.net <anthonyanderberg@NU-TELECOM.NET> wrote:
Looks the same as yesterday on both servers: 206.108.255.37 BGP master up 22:38:09 Established 2001:504:27:0:0:8cc8::1 BGP master start 22:42:34 Idle Received: Maximum number of prefixes reached
But I see we've started getting these BIRD log messages: 016-09-26 17:28:45 <RMT> NASN_36040: Received: Other configuration change
Additionally my ping generated a "We're actively troubleshooting this issue" reply, so it all seems to fit together.
Cheers, anthony
Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:40 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We have bilateral sessions to them, not sure on the route reflectors. Can someone check logs/BGP uptime there?
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:25 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Thanks for reaching out to your Google contacts.
Are you saying that MICE's route reflectors have been seeing their BGP sessions to Google's cache boxes flapping over the last few days?
Frank
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too.
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Traffic started ramping up at 7:45. Thanks to all who worked on this. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 11:00 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening? That’s what we’re seeing too: er1.511.msp rpd[1461]: bgp_read_v4_message:10672: NOTIFICATION received from 206.108.255.37 (External AS 36040): code 6 (Cease) subcode 6 (Other Configuration Change) (which on JunOS, actually flaps the session) -- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:56 PM, anthonyanderberg@nu-telecom.net <anthonyanderberg@NU-TELECOM.NET> wrote:
Looks the same as yesterday on both servers: 206.108.255.37 BGP master up 22:38:09 Established 2001:504:27:0:0:8cc8::1 BGP master start 22:42:34 Idle Received: Maximum number of prefixes reached
But I see we've started getting these BIRD log messages: 016-09-26 17:28:45 <RMT> NASN_36040: Received: Other configuration change
Additionally my ping generated a "We're actively troubleshooting this issue" reply, so it all seems to fit together.
Cheers, anthony
Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:40 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We have bilateral sessions to them, not sure on the route reflectors. Can someone check logs/BGP uptime there?
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:25 PM, Frank Bulk <fbulk@mypremieronline.com> wrote:
Thanks for reaching out to your Google contacts.
Are you saying that MICE's route reflectors have been seeing their BGP sessions to Google's cache boxes flapping over the last few days?
Frank
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Tuesday, September 27, 2016 10:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
We’ve had some issues with our sessions to the Google cache boxes flapping over the last few days. Checking with a few internal folks to Google, seemed like maybe there were a few other open tickets related to these cache boxes too.
-- Andrew Hoyos hoyosa@gmail.com
On Sep 27, 2016, at 10:09 PM, Frank Bulk <fbulk@MYPREMIERONLINE.COM> wrote:
Looks like it’s the Google LAG:
<image001.png>
From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Tuesday, September 27, 2016 10:07 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] What's happening with MICE bandwidth this evening?
Saw our own MICE-facing traffic drop down, and I see MICE’s own aggregate is the same.
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (4)
-
Andrew Hoyos -
anthonyanderberg＠nu-telecom.net -
Frank Bulk -
Jeremy Lumby