One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS attack for about 20 minutes and some of the incoming traffic was going over our MICE link. Do the MICE admins have a way to blackhole our IP, if needed? Frank Premier Communications (AS53347) FiberNet Communications (AS18883)
On 4/6/20 4:50 PM, Frank Bulk wrote:
One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS attack for about 20 minutes and some of the incoming traffic was going over our MICE link.
Do the MICE admins have a way to blackhole our IP, if needed?
No. I'd love to see us implement something like this: https://www.seattleix.net/blackholing Let's see if we can get this done. Here's what I see as steps: 1) Pick IPs and MACs. Here's a proposal: IPv4: 206.108.255.0 same idea as SIX IPv6: 2001:504:27:0:0:FFFF::666 from 65535:666 used in RFC 7999 MAC: 66:66:de:ad:be:ef same as SIX 2) Jeremy?: Configure MAC ACLs to drop traffic to that MAC on the core switches. 3) Doug?: Configure the route servers to: accept /32 (IPv4) and /128 (IPv6) set next-hop to the blackhole IP (see above) add no-export community when: next-hop == blackhole IP OR 65535:666 is set See also the BIRD example on the SIX page. 4) Me: Document this on our website & notify the members it's ready. Ask, but not require, that remote switches do the same. -- Richard
This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address. — Andrew Hoyos hoyosa@gmail.com <mailto:hoyosa@gmail.com>
On Apr 6, 2020, at 5:12 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
On 4/6/20 4:50 PM, Frank Bulk wrote:
One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS attack for about 20 minutes and some of the incoming traffic was going over our MICE link.
Do the MICE admins have a way to blackhole our IP, if needed?
No.
I'd love to see us implement something like this: https://www.seattleix.net/blackholing
Let's see if we can get this done. Here's what I see as steps:
1) Pick IPs and MACs. Here's a proposal: IPv4: 206.108.255.0 same idea as SIX IPv6: 2001:504:27:0:0:FFFF::666 from 65535:666 used in RFC 7999 MAC: 66:66:de:ad:be:ef same as SIX
2) Jeremy?: Configure MAC ACLs to drop traffic to that MAC on the core switches.
3) Doug?: Configure the route servers to: accept /32 (IPv4) and /128 (IPv6) set next-hop to the blackhole IP (see above) add no-export community when: next-hop == blackhole IP OR 65535:666 is set See also the BIRD example on the SIX page.
4) Me: Document this on our website & notify the members it's ready. Ask, but not require, that remote switches do the same.
-- Richard
On 4/6/20 5:17 PM, Andrew Hoyos wrote:
This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address.
Nothing is stopping me from announcing 8.8.8.0/24 right now, right? I agree that the IRR filtering is important, but is that absolutely necessary here? -- Richard
On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
On 4/6/20 5:17 PM, Andrew Hoyos wrote:
This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address.
Nothing is stopping me from announcing 8.8.8.0/24 right now, right?
I agree, that’s true in the current scenario. Nothing preventing you from doing that towards the route servers. As it stands right now, though, we wouldn’t necessarily have to accept that route, and we’d have local control to reject/block or filter your ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: operational processes to fix the issue without reliance on MICE intervention, and without completely dropping our route server sessions). Or perhaps folks are already doing self-generated prefix lists and actions towards route server prefixes already (we, for example, passively keep an eye on ASNs/IRR data vs routes learned from MICE route servers). In short, we’ve got tools to deal, if you do.
I agree that the IRR filtering is important, but is that absolutely necessary here?
In the case we enable a RTBH community without strict filtering, we’ve taken the control out of the members hands, and put a powerful tool in any members hands with no regard to security. Say someone does do something nefarious, or inadvertent that breaks something important? How quickly can it be fixed, who can fix it, etc, etc? Operational processes become a lot more important. The fix on a members side is a lot more intrusive (ie: turn down port), and the impact is much greater (blackholed across entire IX fabric - and potentially also affecting bilateral peering traffic). I think it’d be extremely irresponsible of MICE, and I’m vehemently opposed to enabling RTBH on the route servers or fabric, without first having the proper tools in place to ensure it can be used sanely and securely (IRR based filtering + route server looking glass). My $0.02… — Andrew Hoyos hoyosa@gmail.com <mailto:hoyosa@gmail.com>
On 4/6/20 6:38 PM, Andrew Hoyos wrote:
On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager@WIKTEL.COM <mailto:rlaager@WIKTEL.COM>> wrote:
On 4/6/20 5:17 PM, Andrew Hoyos wrote:
This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address.
Nothing is stopping me from announcing 8.8.8.0/24 right now, right?
I agree, that’s true in the current scenario. Nothing preventing you from doing that towards the route servers.
As it stands right now, though, we wouldn’t necessarily have to accept that route, and we’d have local control to reject/block or filter your ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: operational processes to fix the issue without reliance on MICE intervention, and without completely dropping our route server sessions).
Or perhaps folks are already doing self-generated prefix lists and actions towards route server prefixes already (we, for example, passively keep an eye on ASNs/IRR data vs routes learned from MICE route servers).
In short, we’ve got tools to deal, if you do.
...
In the case we enable a RTBH community without strict filtering, we’ve taken the control out of the members hands, and put a powerful tool in any members hands with no regard to security.
I'm still not understanding the difference between someone originating 8.8.8.0/24 and 8.8.8.8/32+65535:666. In either case (with blackholing support added), the route server will accept and propagate that to route server users. Whether they choose to accept that announcement is up to them the same either way. Note that announcements to the route server do /not/ directly result in blackholing traffic across the fabric. It simply sets a special next-hop.* If a participant chooses to accept that BGP route and route traffic to that next-hop*, only then will the traffic will be eaten. To put a finer point on this: if someone does not use the route servers, this change does not affect them in any way no matter what anyone announces as a blackhole route to the route servers. * I forgot to mention that we'll need something on the route servers that will respond to ARP/ND requests for the blackhole IPs and return the blackhole MAC. -- Richard
On Apr 6, 2020, at 6:46 PM, Richard Laager <rlaager@WIKTEL.COM> wrote:
On 4/6/20 6:38 PM, Andrew Hoyos wrote:
On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager@WIKTEL.COM <mailto:rlaager@WIKTEL.COM>> wrote:
On 4/6/20 5:17 PM, Andrew Hoyos wrote:
This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address.
Nothing is stopping me from announcing 8.8.8.0/24 right now, right?
I agree, that’s true in the current scenario. Nothing preventing you from doing that towards the route servers.
As it stands right now, though, we wouldn’t necessarily have to accept that route, and we’d have local control to reject/block or filter your ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: operational processes to fix the issue without reliance on MICE intervention, and without completely dropping our route server sessions).
Or perhaps folks are already doing self-generated prefix lists and actions towards route server prefixes already (we, for example, passively keep an eye on ASNs/IRR data vs routes learned from MICE route servers).
In short, we’ve got tools to deal, if you do.
...
In the case we enable a RTBH community without strict filtering, we’ve taken the control out of the members hands, and put a powerful tool in any members hands with no regard to security. I'm still not understanding the difference between someone originating 8.8.8.0/24 and 8.8.8.8/32+65535:666. In either case (with blackholing support added), the route server will accept and propagate that to route server users. Whether they choose to accept that announcement is up to them the same either way.
Note that announcements to the route server do not directly result in blackholing traffic across the fabric. It simply sets a special next-hop.* If a participant chooses to accept that BGP route and route traffic to that next-hop*, only then will the traffic will be eaten. To put a finer point on this: if someone does not use the route servers, this change does not affect them in any way no matter what anyone announces as a blackhole route to the route servers.
Ack, thanks. Perhaps I misunderstood intent of actually dropping in flight traffic on switch fabric - this makes more sense and lessens my concern. Still think we should get IRR filtering in place first.
participants (3)
-
Andrew Hoyos -
Frank Bulk -
Richard Laager