- mice-discuss - lists.micemn.net

Re: Route Server 1 - BFD not working

by Richard Laager

On 2023-06-23 10:46, Jonathan Stewart wrote: > Richard's debugging suggested it was BIRD not responding to valid BFD > packets transmitted from our Juniper router. Re: "valid". In looking at the packet capture again, I'm not sure how the discriminator values are supposed to look. So maybe that is related. > The problem for diagnosis is reproduceability. Since a session reset > fixed the problem, we can't reproduce the problem, and that makes > diagnosis impossible without additional data. > > To be clear, I don't expect any further action from MICE, as the > problem is resolved. I sent a write-up to the bird-users mailing list. We'll see if anyone there has any ideas. -- Richard

June 23, 2023

Re: Route Server 1 - BFD not working

by Jonathan Stewart

Richard's debugging suggested it was BIRD not responding to valid BFD packets transmitted from our Juniper router. But almost all BFD sessions are up, so what's wrong with this specific one--or pair in this case? There was a fibre cut which disconnected AS18451 from MICE for 1.5 days--maybe at service restoration there was packet loss for the first few seconds, so BFD handshaking failed? The problem for diagnosis is reproduceability. Since a session reset fixed the problem, we can't reproduce the problem, and that makes diagnosis impossible without additional data. To be clear, I don't expect any further action from MICE, as the problem is resolved. I'm just discussing troubleshooting and the limitations of our knowledge about this problem. Cheers, Jonathan Stewart AS18451 On Thu, 2023-06-22 3:53 p.m., Mike Horwath wrote: > On Thu, Jun 22, 2023 at 02:40:14PM -0500, Jonathan Stewart wrote: >> Disable the BGP (and thus also BFD) session, waited a minute, and >> re-enabled BGP & BFD. >> >> Came back as normal and looks fine now on both IPv4 and IPv6. >> >> Thanks for the assistance Richard. >> >> Though it's still not clear what the real cause of the problem was. > Bugs from the router vs the route-server perhaps? > > I hate magic like this. Magic sucks >

June 23, 2023

Re: [External] [MICE-DISCUSS] Three BGP peers

by Tom Krenn

Not peered with Yahoo but I do see that Google peer down. Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd 206.108.255.167 0 15169 605946 572995 0 0 0 13:06:45 Active Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd 2001:504:27::3b41:0:2 0 15169 598758 572903 0 0 0 13:07:55 Active Tom Krenn Network Architect Enterprise Architecture - Information Technology -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Frank Bulk Sent: Thursday, June 22, 2023 4:42 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [External] [MICE-DISCUSS] Three BGP peers CAUTION: This email was sent from outside of Hennepin County. Unless you recognize the sender and know the content, do not click links or open attachments. Anyone else seeing both Yahoo peers and one Google peer down since early am? I didn't see any log messages about exceeding prefix counts or the like. Frank SiouxCity-Fibernet-Arista(s2)#sh ip bgp sum | ex Estab BGP summary information for VRF default Router identifier 207.32.15.21, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.58 4 10310 2035987 4886675 0 0 12:23:38 Active 206.108.255.59 4 10310 2029469 4886360 0 0 12:23:41 Active 206.108.255.167 4 15169 2050148 4888984 0 0 12:23:40 Active SiouxCity-Fibernet-Arista(s2)#sh ipv6 bgp sum | ex Estab BGP summary information for VRF default Router identifier 207.32.15.21, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 2001:504:27::2846:0:1 4 10310 2209610 1893247 0 0 12:23:40 Active 2001:504:27::2846:0:2 4 10310 2202613 1892818 0 0 12:23:47 Active 2001:504:27::3b41:0:2 4 15169 2043589 1894022 0 0 12:23:48 Active SiouxCity-Fibernet-Arista(s2)# Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

June 22, 2023

Re: Three BGP peers

by Frank Bulk

Thanks for confirming. Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Thursday, June 22, 2023 5:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Three BGP peers Same, suspect fiber cut or maint. Weathermap doesn’t show any traffic on those links. [cid:image001.png@01D9A52B.A1A56C50] [cid:image002.png@01D9A52B.A1A56C50] > On Jun 22, 2023, at 4:42 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM<mailto:fbulk@MYPREMIERONLINE.COM>> wrote: > > Anyone else seeing both Yahoo peers and one Google peer down since early am? I didn't see any log messages about exceeding prefix counts or the like. > > Frank > > SiouxCity-Fibernet-Arista(s2)#sh ip bgp sum | ex Estab > BGP summary information for VRF default > Router identifier 207.32.15.21, local AS number 18883 > Neighbor Status Codes: m - Under maintenance > Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc > 206.108.255.58 4 10310 2035987 4886675 0 0 12:23:38 Active > 206.108.255.59 4 10310 2029469 4886360 0 0 12:23:41 Active > 206.108.255.167 4 15169 2050148 4888984 0 0 12:23:40 Active > SiouxCity-Fibernet-Arista(s2)#sh ipv6 bgp sum | ex Estab > BGP summary information for VRF default > Router identifier 207.32.15.21, local AS number 18883 > Neighbor Status Codes: m - Under maintenance > Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc > 2001:504:27::2846:0:1 4 10310 2209610 1893247 0 0 12:23:40 Active > 2001:504:27::2846:0:2 4 10310 2202613 1892818 0 0 12:23:47 Active > 2001:504:27::3b41:0:2 4 15169 2043589 1894022 0 0 12:23:48 Active > SiouxCity-Fibernet-Arista(s2)#

June 22, 2023

Re: Three BGP peers

by Andrew Hoyos

Same, suspect fiber cut or maint. Weathermap doesn’t show any traffic on those links. > On Jun 22, 2023, at 4:42 PM, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: > > Anyone else seeing both Yahoo peers and one Google peer down since early am? I didn't see any log messages about exceeding prefix counts or the like. > > Frank > > SiouxCity-Fibernet-Arista(s2)#sh ip bgp sum | ex Estab > BGP summary information for VRF default > Router identifier 207.32.15.21, local AS number 18883 > Neighbor Status Codes: m - Under maintenance > Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc > 206.108.255.58 4 10310 2035987 4886675 0 0 12:23:38 Active > 206.108.255.59 4 10310 2029469 4886360 0 0 12:23:41 Active > 206.108.255.167 4 15169 2050148 4888984 0 0 12:23:40 Active > SiouxCity-Fibernet-Arista(s2)#sh ipv6 bgp sum | ex Estab > BGP summary information for VRF default > Router identifier 207.32.15.21, local AS number 18883 > Neighbor Status Codes: m - Under maintenance > Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc > 2001:504:27::2846:0:1 4 10310 2209610 1893247 0 0 12:23:40 Active > 2001:504:27::2846:0:2 4 10310 2202613 1892818 0 0 12:23:47 Active > 2001:504:27::3b41:0:2 4 15169 2043589 1894022 0 0 12:23:48 Active > SiouxCity-Fibernet-Arista(s2)#

June 22, 2023

Three BGP peers

by Frank Bulk

Anyone else seeing both Yahoo peers and one Google peer down since early am? I didn't see any log messages about exceeding prefix counts or the like. Frank SiouxCity-Fibernet-Arista(s2)#sh ip bgp sum | ex Estab BGP summary information for VRF default Router identifier 207.32.15.21, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.58 4 10310 2035987 4886675 0 0 12:23:38 Active 206.108.255.59 4 10310 2029469 4886360 0 0 12:23:41 Active 206.108.255.167 4 15169 2050148 4888984 0 0 12:23:40 Active SiouxCity-Fibernet-Arista(s2)#sh ipv6 bgp sum | ex Estab BGP summary information for VRF default Router identifier 207.32.15.21, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 2001:504:27::2846:0:1 4 10310 2209610 1893247 0 0 12:23:40 Active 2001:504:27::2846:0:2 4 10310 2202613 1892818 0 0 12:23:47 Active 2001:504:27::3b41:0:2 4 15169 2043589 1894022 0 0 12:23:48 Active SiouxCity-Fibernet-Arista(s2)#

June 22, 2023

Re: Route Server 1 - BFD not working

by Mike Horwath

On Thu, Jun 22, 2023 at 02:40:14PM -0500, Jonathan Stewart wrote: > Disable the BGP (and thus also BFD) session, waited a minute, and > re-enabled BGP & BFD. > > Came back as normal and looks fine now on both IPv4 and IPv6. > > Thanks for the assistance Richard. > > Though it's still not clear what the real cause of the problem was. Bugs from the router vs the route-server perhaps? I hate magic like this. Magic sucks -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

June 22, 2023

Re: Route Server 1 - BFD not working

by Jonathan Stewart

On Wed, 2023-06-21 6:54 p.m., Richard Laager wrote: > Before I try that, have you tried flapping your BGP session? I had a chance to try this today. Disable the BGP (and thus also BFD) session, waited a minute, and re-enabled BGP & BFD. Came back as normal and looks fine now on both IPv4 and IPv6. Thanks for the assistance Richard. Though it's still not clear what the real cause of the problem was. Curious. Cheers, Jonathan -- Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA

June 22, 2023

Re: Route Server 1 - BFD not working

by Richard Laager

On 2023-06-22 08:39, Michael Hare wrote: > This March I reported to peering@ a similar problem to RR1 If this was happening in March, the "good" news is that this is not due to the new route servers (different OS, BIRD upgrade, new config, etc.). I looked at the BIRD release notes, and I'm not seeing anything about BFD specifically. There are plenty of mentions of generic bug fixes. I suppose I should take this to the BIRD mailing list. -- Richard

June 22, 2023

Re: Route Server 1 - BFD not working

by Michael Hare

Jonathan, This March I reported to peering@ a similar problem to RR1; I’m not sure when my problem started, I only noticed it while doing CLI jockey work on my router. I have had success removing and adding the BFD session on my Juniper. Over the last three months I saw one recurrence to RR1 but it was over v4 and v6; same solution for me. I didn’t spend time debugging JunOS further [myself or with jtac]. My side is a Juniper MX running 20.4. I have since added BFD session monitoring so I'm not reliant on syslogs. -Michael From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Richard Laager Sent: Wednesday, June 21, 2023 6:54 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Route Server 1 - BFD not working On 2023-06-21 14:06, Jonathan Stewart wrote: On our BGP session to Route Server 1 (206.108.225.1 & 2001:504:27::d1af:0:1) i have lost the BFD session. My Juniper router shows the state as "Down". Is this intentional? An accident of the upgrades? Is my configuration wrong? Anyone else noticing this? My configuration is an interval of 1000 ms, multiplier 3. MICE is configured as 500 * 3 tcpdump shows your BFD control packets coming in, but bird is not responding. I'm seeing the same issue with 206.108.255.137 (Inteliquent AS 40160) on IPv4 and 2001:504:27:0:0:B7F8::1 (Compudyne, AS47096) on IPv6. Everyone else is consistent between the two route servers. This thread from 2020 talks about someone having BFD issues after a reboot, which they fixed by restarting BFD in BIRD. That might be something to try if we run out of other ideas: https://www.mail-archive.com/bird-users@network.cz/msg05382.html Before I try that, have you tried flapping your BGP session? -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

June 22, 2023

Re: Route Server 1 - BFD not working

by Richard Laager

On 2023-06-21 14:06, Jonathan Stewart wrote: > On our BGP session to Route Server 1 (206.108.225.1 & > 2001:504:27::d1af:0:1) i have lost the BFD session. > > My Juniper router shows the state as "Down". > > Is this intentional? An accident of the upgrades? Is my configuration > wrong? Anyone else noticing this? > My configuration is an interval of 1000 ms, multiplier 3. MICE is configured as 500 * 3 tcpdump shows your BFD control packets coming in, but bird is not responding. I'm seeing the same issue with 206.108.255.137 (Inteliquent AS 40160) on IPv4 and 2001:504:27:0:0:B7F8::1 (Compudyne, AS47096) on IPv6. Everyone else is consistent between the two route servers. This thread from 2020 talks about someone having BFD issues after a reboot, which they fixed by restarting BFD in BIRD. That might be something to try if we run out of other ideas: https://www.mail-archive.com/bird-users@network.cz/msg05382.html Before I try that, have you tried flapping your BGP session? -- Richard

June 21, 2023

Route Server 1 - BFD not working

by Jonathan Stewart

Hi MICE, We are AS18451. On our BGP session to Route Server 1 (206.108.225.1 & 2001:504:27::d1af:0:1) i have lost the BFD session. My Juniper router shows the state as "Down". Is this intentional? An accident of the upgrades? Is my configuration wrong? Anyone else noticing this? My configuration is an interval of 1000 ms, multiplier 3. RS 2 is configured the same and BFD is up. Thanks, Jonathan -- Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA

June 21, 2023

Re: FW: Thursday Night Football from Amazon Prime Video - Network Questionnaire

by Yan Berthier

Hi all, Qwilt here. Thanks Frank! Indeed, we are a MICE member. With or without a cache with us, you'll see some traffic from AS20253, so it's good idea to peer with us either way: while we're on the Route Server, we prefer setting up bilats. Hit me up off list with any question you may have, or to set up a BGP session with us. Thanks, - y On Tue, Jun 13, 2023 at 9:27 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > I’m under NDA, but the February webinar was not restricted to their > existing partners. In that webinar they indicated additional content was > coming at the end of Q1. The NCTC sent a letter a few weeks ago that those > who are part of NCTC’s program should have received. > > > > Qwilt is a MICE member and I look forward to receiving more content. =) > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *Ben > Wiechman > *Sent:* Tuesday, June 13, 2023 8:35 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > Amazon was only serving Thursday Night Football out of the caches. Your > traffic is consistent with the football season. Looks like a little bump > there in April, which is curious. > > > > Our traffic is similar. You can see the NFL season up to Jan, then > relatively little. It is possible that one CDN was launched in April. I > should ask. Looks like there has been some amount of activity since around > 4/1. > > > > Qwilt had a lot of promises about new content moving through the cache > beginning in 1H2023 here. NCTC had a good joint webinar on Feb 7th, for > anyone that is a member. I can't find a link back to it. They presented a > summary of the performance from the 2022 NFL season, as well as plans for > 2023. They didn't state it explicitly, but it didn't seem like they were > getting any traction in hosting any content other than TNF from Amazon. > > > > *Ben Wiechman* > > Director of Network Strategy and Engineering > > 320.247.3224 | ben.wiechman(a)arvig.com > > Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com > <https://link.edgepilot.com/s/7f2b178f/1ZuwJz42b0_NlW9cx7EkKA?u=http://arvig…> > > > > > > > > On Tue, Jun 13, 2023 at 7:21 AM Larry Larsen < > 000000186bf54e79-dmarc-request(a)lists.iphouse.net> wrote: > > I have not seen much usage. Here is my peak bandwidth usage on Qwilt > for that month. All amounts are in Gbps. > > > > Oct-22 > > Nov-22 > > Dec-22 > > Jan-23 > > Feb-23 > > Mar-23 > > Apr-23 > > May-23 > > Amazon1 > > 4.45 > > 4.01 > > 4.06 > > 0.101 > > 0.979 > > 0.995 > > 0.07 > > 0.03 > > Amazon2 > > 3.58 > > 1.44 > > 1.93 > > 0.029 > > 0.035 > > 0.267 > > 1.67 > > 0.641 > > > > As you can see, I had some good usage from October – December but after > that not much. April had a little usage. > > > > Thanks! > > > > Larry L. > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Ben > Wiechman > *Sent:* Tuesday, June 13, 2023 7:12 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > Some people who received this message don't often get email from > 0000001a026a3d65-dmarc-request(a)lists.iphouse.net. Learn why this is > important > <https://link.edgepilot.com/s/0f4a9629/Roa4CaCjskyieHEVX7BgDA?u=https://aka.…> > > WARNING!! This message originated from an *External Source*. Please use > proper judgment and caution when opening attachments, clicking links, or > responding to this email. > > Have you seen them serve any other content from your cache yet. They keep > claiming more content is coming. > > > Ben Wiechman > Director of IP Strategy and Engineering > Direct: 320.256.0184 > Cell: 320.247.3224 > ben.wiechman(a)arvig.com > 150 Second Street SW | Perham, MN 56573 | arvig.com > <https://link.edgepilot.com/s/7f2b178f/1ZuwJz42b0_NlW9cx7EkKA?u=http://arvig…> > > > > On Mon, Jun 12, 2023, 16:08 Justin Krejci <JKrejci(a)usinternet.com> wrote: > > I stumbled across this thread again while looking for something else and > thought I'd add a small update that we did get our Qwilt server deployed > on-net for us a while back and it did indeed serve a relatively large burst > of traffic volume during the live streaming episodes of Thursday Night > Football. > > > > > ------------------------------ > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Justin > Krejci <JKrejci(a)USINTERNET.COM> > *Sent:* Wednesday, February 16, 2022 1:08 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > I believe it is just simpler for everyone involved when you are an NCTC > member but I don't think it is a requirement necessarily. As Frank > suggests, if you're interested in their services you should just reach out > to them directly and they can tell you what they can and will do for you > given your network and usage numbers. In my experience so far Qwilt are > decent folks: I am still working through their process to get a deployment > for our network and they have been competent and reasonable to work with so > far. > > > > To contrast the email subject, because they are in the open caching > market, they do not carry only Amazon content. > > > ------------------------------ > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Frank > Bulk <fbulk(a)MYPREMIERONLINE.COM> > *Sent:* Wednesday, February 16, 2022 12:09 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > I don’t know if it’s exclusive to the NCTC or not – you could reach out to > Qwilt and just ask them what options they have for you. > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Ryan > Malek > *Sent:* Wednesday, February 16, 2022 11:28 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > Is this offering going to be available to non NCTC members at a cost? (Is > this maybe part of Qwilts default offering? ) > > TLDR: Is the Qwilt platform worth it outside of being an NCTC Member? :) > > > ------------------------------ > > Ryan Malek - Router12 Networks LLC > Internet, Phone, and Hosted Services > > Ph. 641.420.7180 > > On 2/16/2022 11:08 AM, Frank Bulk wrote: > > If you’re an NCTC member there’s an opportunity to avoid transit costs and > IX bandwidth concerns by serving the traffic from your own network using an > “open” CDN: https://link.edgepilot.com/s/188fed83/ > kSt--kULDUiOMeuW1EQy_Q?u=https://www.qwilt.com/ > nctc-and-qwilt-launch-joint-initiative-to-upgrade-nctc-member-networks-with-next-generation-content-delivery-technology-and-services/ > <https://link.edgepilot.com/s/9d57f0b0/LWiUplIDzEi9j-EIioO0LA?u=https://url-…> > > There’s no cost to NCTC members (just your own rack space, power, cooling, > ports, and time). > > > > Frank > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> *On Behalf Of *Justin Krejci > *Sent:* Tuesday, January 18, 2022 10:56 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > Definitely legit and definitely worth pursuing additional connectivity > with Amazon and CDNs that Amazon will use to distribute their NFL > streaming content. > > > > We have peering active with Amazon at four IX locations (including MICE) > and are trying to get some PNIs with them now too. https://link.edgepilot. > com/s/d1ff7d9b/nHsaPn7VEkOM1Y2TlRuquw?u=https://aws.amazon.com/peering/ > locations/ This page lists Cologix Minneapolis as a site where > interconnection can happen. If you have a lot of eyeballs on your network, > you should definitely pursue adding/augmenting connectivity to Amazon. > > > > > > > ------------------------------ > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Steve > Savoy <stevesavoy(a)WCTATEL.COM> > *Sent:* Tuesday, January 18, 2022 8:55 AM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon > Prime Video - Network Questionnaire > > > > Thanks Jeremy! > > I'll give it another try. > > Steve > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Tuesday, January 18, 2022 8:49 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon Prime > Video - Network Questionnaire > > Steve, > > Amazon can be notoriously difficult to get to respond, and when > they finally do it can often take a couple months. They have recently > moved to regional peering addresses, and it is worth sending the request to > peering-amer(a)amazon.com if you have not already. > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] On Behalf Of Steve Savoy > Sent: Tuesday, January 18, 2022 8:01 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon Prime > Video - Network Questionnaire > > I’m just jealous that you are able to peer with Amazon. > > Been trying for 2 years now and nothing but ‘crickets’………… [image: 😉] > > Steve > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Dean > Bahls > Sent: Tuesday, January 18, 2022 6:28 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] FW: Thursday Night Football from Amazon Prime > Video - Network Questionnaire > > While I haven’t gotten that specific email from them, I’ve gotten several > Telegeography emails since 2012. They are referenced by Light Reading > Daily quite a bit. They *seem* to be a legit research firm contracted by > multiple companies. > > The Amazon / TNF partnership is a real deal….and MICE needs to be prepared > for a potential uptick in traffic on Thursdays. > > Dean > > > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of > AnthonyAnderberg(a)NUVERA.NET > Sent: Tuesday, January 18, 2022 1:27 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] FW: Thursday Night Football from Amazon Prime > Video - Network Questionnaire > > Has anyone else gotten an email like this? > > On the surface it seems legit, and I always like the opportunity to talk > about MICE, but I’m not sure how I feel about sharing such info... > > Thanks in advance, > Anthony > > > On Thu, Dec 16, 2021 at 1:29 PM Nataliya Coll <ncoll(a)telegeography.com> > wrote: > Hi David and Anthony, > > I am a telecoms analyst with TeleGeography in Washington, DC. We have been > contracted by Amazon Prime Video (APV) to survey ISPs across the country in > preparation for Prime Video’s exclusive presentation of NFL Thursday > Night Football (TNF) starting with the 2022 NFL season. > The attached letter from Amazon Prime Video provides more detail about > their network planning initiative. > > To help avoid congestion on your network during upcoming games, Amazon is > asking us to query you so that they may better understand your network > point(s) of contact, service coverage area, CDN provider(s), IP Transit > provider(s), etc. > > Are you the correct contact person within your network operations team to > respond to these questions? If not, can you please direct me to the right > person within your network operations team? > > If you have any questions, feel free to contact me via email or let me > know if you would like to set up a call with me. > > Thank you so much for your cooperation. > > > Nataliya Coll > Research Analyst, Europe & Eurasia > TeleGeography, a Division of PriMetrica, Inc. > One Thomas Circle, NW, Suite 360 > Washington, DC 20005 USA > https://link.edgepilot.com/s/3d0e2e42/J3BFJl3G5EKYo5vqeKddAQ?u=http://www. > telegeography.com/ > <https://link.edgepilot.com/s/968cb011/KiRmgJ-V10m22p-aKPiGlA?u=https://url-…> > +1 (202) 741-0066 > > ________________________________________ > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > ________________________________________ > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > Confidentiality Notice: This correspondence is the property of Winnebago > Cooperative Telecom Association and is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. Any > unauthorized review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by reply email > and destroy all copies of the original message. > > ________________________________________ > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/be9cbd2c/G6PV3oM4uU2Rsw5-lLsmcg?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > <https://link.edgepilot.com/s/b201a908/30MJtHbv-ke8z2L4f2QmMw?u=https://url-…> > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > https://link.edgepilot.com/s/308a7818/glPpQY_-wEmhMvFnVKsSBQ?u=http:// > lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS%26A=1 > > > > Links contained in this email have been replaced. If you click on a link > in the email above, the link will be analyzed for known threats. If a known > threat is found, you will not be able to proceed to the destination. If > suspicious content is detected, you will see a warning. > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

June 16, 2023

Re: RPKI ROAs

by Jay Hanke

>> I think we should hold off publishing one just yet >> This is pretty common among our similar sized IXes in North America. SiX, TorIX, FL-IX, CIX-ATL are all announcing RPKI with ASN 0.

June 14, 2023

Re: RPKI ROAs

by David Farmer

On Wed, Jun 14, 2023 at 15:01 Richard Laager <rlaager(a)wiktel.com> wrote: > On 2023-06-14 02:46, David Farmer wrote: > > I think we should hold off publishing one just yet > > Do you have a particular reason for thinking we should hold off? Is there > something you're worried about breaking? > > While I take your point that not everyone is doing it, that APNIC blog > post definitely shows that some places *are* doing it. > I think I’m just being cautious. I’d feel much better with a published BCP recommending how IXes should publish ROAs or at least a discussion of the pro and cons. I’ll note a conflict between what the APNIC Blog and RFC 6483 say about the proper maxlength for an AS 0 ROA. It’s nagging little details like that, that make me want to be cautious. I think if you publish a ROA with the assigned ASN then you want the maxlength to be the assigned length, but for an AS 0 ROA you want a maxlength of 32 for IPv4 and 128 for IPv6 like RFC 6483 says. However, the idea that some IXes publish both an AS 0 and one for the assigned ASN, makes me wonder, and want to be even more curious. Thanks -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

June 14, 2023

Re: RPKI ROAs

by Richard Laager

I intend to publish the following ROAs on MICE's behalf to indicate that the peering LAN is not to appear in the Internet routing table: On Monday (June 19) around 10:00 AM: 2001:504:27::/48 AS0 maxLength 128 On Wednesday (June 21) around 10:00 AM: 206.108.255.0/24 AS0 maxLength 32 The reason for the maxLength is: "By convention, an AS 0 ROA should have a maxLength value of 32 for IPv4 addresses and a maxlength value of 128 for IPv6 addresses" -- RFC 6483, section 4. This should only be an issue for someone if they are advertising the MICE peering subnet in their iBGP (which we discourage) and are doing RPKI validation of that iBGP (which seems unlikely). If you think this will be a problem for you, let me know. -- Richard

June 14, 2023

Re: RPKI ROAs

by Richard Laager

On 2023-06-14 02:46, David Farmer wrote: > I think we should hold off publishing one just yet Do you have a particular reason for thinking we should hold off? Is there something you're worried about breaking? While I take your point that not everyone is doing it, that APNIC blog post definitely shows that some places /are/ doing it. -- Richard

June 14, 2023

Re: RPKI ROAs

by Richard Laager

On 2023-06-14 07:23, Chris Wopat wrote: > Related to this, is there intent to join the IXP section of MANRS once > we're in good shape with some of this IRR filtering? I submitted the application now. -- Richard

June 14, 2023

Re: IRR Mandatory at MICE / New Route Servers

by Richard Laager

After finishing up with the last minute inquiries, I deployed these changes. As far as I am aware, for everyone that has been in contact with me, things are either: A) resolved, B) been given an exception, or C) are irrelevant (see the "don't count") at the end of this email. The things that are filtered should only be people who never responded. The following routes are being filtered on rs1 (and rs2). That is, this is what is effectively "blocked" (for lack of route/route6 objects, ignoring rejections for other causes) /excluding/ the 4 exceptions: AS5056: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0047_as5056 206.83.43.0/24 AS5056 AS6939: Too many routes to list. They have looser IRR policies <https://routing.he.net/algorithm.html> (specifically, I think the RIR handle matches allow a lot that traditional IRR filtering does not). Bilateral with them if you want those. AS10242: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0028_as10242 23.130.82.0/24 AS398200 AS11796: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0051_as11796 192.236.20.0/24 AS25958 192.236.21.0/24 AS25958 https://ixpmgr.micemn.net/lg/rs1-ipv6/routes/protocol/pb_0051_as11796 2604:f9c0::/32 AS397064 AS16851: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0060_as16851 170.31.248.0/21 AS27534 170.31.160.0/23 AS27534 AS22402: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0088_as22402 206.11.94.0/24 AS22402 AS27204: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0095_as27204 198.98.192.0/21 AS396853 AS40413: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0076_as40413 207.250.207.0/24 AS40413 198.204.66.0/24 AS40413 AS393639: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0112_as393639 192.55.242.0/23 AS393639 These don't really count: AS26794: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0048_as26794 208.85.166.0/24 AS40569 -- IP space and AS recently removed from WHOIS. 208.85.167.0/24 AS40569 -- IP space and AS recently removed from WHOIS. AS55009: https://ixpmgr.micemn.net/lg/rs1-ipv4/routes/protocol/pb_0002_as55009 69.57.200.0/24 AS400080 -- This is going away and they will pull the announcement tomorrow. -- Richard

June 14, 2023

Re: IRR Mandatory at MICE / New Route Servers

by Richard Laager

On 2023-06-14 00:01, Richard Laager wrote: > > I am on track to make this enforcing tomorrow. > I am still answering last minute queries from people. I expect this will be another hour or so. -- Richard

June 14, 2023

Re: RPKI ROAs

by Chris Wopat

On 6/14/23 01:14, Richard Laager wrote: > > In the ARIN web UI, it looks like a ROA can have multiple prefixes. For > an end network such as myself (Wiktel hat on), what is the best > practice... one ROA per prefix, or list multiple prefixes in the same ROA? As August noted it's an exercise of the user. One note on this - you may not MODIFY an ROA (from what I can tell), so if you publish a single ROA with multiple prefixes and need to change something, you have to delete and recreate the entire thing. We've been doing 1:1 ROA:Prefix for this reason (for now, and we're not fully covered) Related to this, is there intent to join the IXP section of MANRS once we're in good shape with some of this IRR filtering? I believe that would be the final 'Action' (action 1). https://www.manrs.org/ixps/actions/ https://www.manrs.org/ixps/participants/ -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

June 14, 2023

Re: RPKI ROAs

by David Farmer

On Wed, Jun 14, 2023 at 01:14 Richard Laager <rlaager(a)wiktel.com> wrote: > As far as I can see, there's no explicit way to say "do not announce" in > a ROA. But listing something with an origin AS of "0" seems to be a > thing that is done for at least one deliberately invalid ROA > (CloudFlare's 103.21.244.0/23). So, if I were to create a ROA for MICE, > would that be an Origin AS of 53679 or 0? An AS 0 ROA is an explicit statement that a prefix is not to be routed, see section 4 of RFC 6483; https://datatracker.ietf.org/doc/html/rfc6483#section-4 To the question, should we issue an AS 0 ROA for the MICE prefix? Well, there isn’t a strong consensus on that within IX Community. See the following APNIC Blog post; https://blog.apnic.net/2023/03/24/rpki-roa-for-ip-resources-in-the-ix-segme… So I think we should hold off publishing one just yet, but I suspect we will eventually want to publish an AS 0 ROA for the MICE prefix. Thanks > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

June 14, 2023

Re: RPKI ROAs

by August Yang

RFC6482 doesn't explicitly specify whether to have one ROA per prefix or vice versa. In a delegated setup, Krill aggregates ROAs whenever possible, so it's left up to your preference. https://console.rpki.co/rpki.co/repo/AS945/1/AS50058.roa.html When it comes to MICE, it's recommended to publish a ROA covering the peering LAN. This not only secures against prefix hijacking but also prevents any accidental leaking, which has happened to NL-IX for example. Per RFC6907, an AS 0 ROA serves as an attestation by a prefix holder that the prefix described in the ROA, along with any more specific prefixes, should not be used in a routing context. AS0 is better in that one can simply prepend AS53679 to deliberately hijack the prefix, which could be prevented by ASPA but that's another story, and A BGP speaker MUST NOT originate or propagate a route with an AS number of zero (RFC7606). > From: nlixannounce(a)simplelists.com <nlixannounce(a)simplelists.com> on behalf of NL-ix Support <support(a)nl-ix.net> > Sent: Tuesday, February 22, 2022, 3:48 AM > To: Support-NLix <support(a)nl-ix.net> > Subject: [NL-ix Announce] Recent incidents in peering VLAN > Dear members, > We have recently experienced some unfortunate events on our peering VLAN due to the announcement of a more-specific prefix that is part of our /22 subnet. One of our members accidentally announced 193.239.117.0/24 to one of their upstreams, propagating this prefix to the rest of the internet. Although one would expect this prefix to be dropped early on due to IRR validation failing, it was instead widely propagated. On 2023-06-14 2:14 a.m., Richard Laager wrote: > In the ARIN web UI, it looks like a ROA can have multiple prefixes. > For an end network such as myself (Wiktel hat on), what is the best > practice... one ROA per prefix, or list multiple prefixes in the same > ROA? > > With my MICE hat on, should MICE publish a ROA for the peering network > space? The theoretical advantage would be some amount of hijacking > protection. Though since it's not supposed to be announced, it > probably doesn't matter much either way? > > As far as I can see, there's no explicit way to say "do not announce" > in a ROA. But listing something with an origin AS of "0" seems to be a > thing that is done for at least one deliberately invalid ROA > (CloudFlare's 103.21.244.0/23). So, if I were to create a ROA for > MICE, would that be an Origin AS of 53679 or 0? > -- Best regards August Yang

June 14, 2023

RPKI ROAs

by Richard Laager

In the ARIN web UI, it looks like a ROA can have multiple prefixes. For an end network such as myself (Wiktel hat on), what is the best practice... one ROA per prefix, or list multiple prefixes in the same ROA? With my MICE hat on, should MICE publish a ROA for the peering network space? The theoretical advantage would be some amount of hijacking protection. Though since it's not supposed to be announced, it probably doesn't matter much either way? As far as I can see, there's no explicit way to say "do not announce" in a ROA. But listing something with an origin AS of "0" seems to be a thing that is done for at least one deliberately invalid ROA (CloudFlare's 103.21.244.0/23). So, if I were to create a ROA for MICE, would that be an Origin AS of 53679 or 0? -- Richard

June 14, 2023

Re: IRR Mandatory at MICE / New Route Servers

by Richard Laager

On 2023-06-07 12:37, Richard Laager wrote: > > It turns out I made an error in the IRR prefix enforcement. So while > we were enforcing the as-set requirement as intended, we were /not/ > enforcing the route object requirement on rs2. Accordingly, I did > /not/ make that enforcing on rs1 today as scheduled. Instead, I > corrected it on rs2 and will wait another week before making it > enforced on rs1. > > I have separately emailed everyone who has prefixes being filtered. > From my perspective, I am on track to make this enforcing tomorrow. I have again emailed everyone who has prefixes being filtered. For the last two emails, I added the ARIN WHOIS contacts for the originating ASes (and for one email, the ARIN WHOIS contacts for the IP space involved). I've had a number of email and phone conversations with people. The list has definitely gotten much better. I have begrudgingly made exceptions for two networks. They have ARIN issues to sort out. (Yes, I am aware they could use some other registry and have mentioned that.) These exceptions will be time limited. These exceptions allow specifically the prefixes they are currently announcing that are being rejected. So these exceptions will not defeat IRR security in general (unlike just turning it off for them entirely). These exceptions will be on rs1 only, which will allow them/me to use the looking glass to rs2 to know if things are fixed. As a consequence, their routes will still be filtered when rs1 is down, e.g. for patching. They have been so informed. -- Richard

June 14, 2023