https://www.seattleix.net/blackholing Does MICE have an blackholng functionality equivalent to SIX? I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. Regards, Frank
On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote:
https://www.seattleix.net/blackholing
Does MICE have an blackholng functionality equivalent to SIX?
I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer.
You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html you could block-hole the AS that is sending you that traffic. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn@IPHOUSE.NET> wrote:
On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote:
https://www.seattleix.net/blackholing
Does MICE have an blackholng functionality equivalent to SIX?
I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer.
You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html
you could block-hole the AS that is sending you that traffic.
unfortunately, that just has the effect of traffic going elsewhere - not a blackhole effect. the communities in place would just cause the route not to be advertised to said peer, and the traffic would just ingress your network via a different path. In the case of a DDOS, it’s likely you have multiple ASN’s targeting you. https://www.seattleix.net/blackholing <https://www.seattleix.net/blackholing> SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE. — Andrew Hoyos hoyosa@gmail.com
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so? Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde@parkregion.com<mailto:jeff.wilde@parkregion.com> From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Andrew Hoyos Sent: Thursday, June 20, 2019 11:48 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn@IPHOUSE.NET<mailto:merlyn@IPHOUSE.NET>> wrote: On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote: https://www.seattleix.net/blackholing Does MICE have an blackholng functionality equivalent to SIX? I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html you could block-hole the AS that is sending you that traffic. unfortunately, that just has the effect of traffic going elsewhere - not a blackhole effect. the communities in place would just cause the route not to be advertised to said peer, and the traffic would just ingress your network via a different path. In the case of a DDOS, it’s likely you have multiple ASN’s targeting you. https://www.seattleix.net/blackholing SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE. — Andrew Hoyos hoyosa@gmail.com<mailto:hoyosa@gmail.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau@Geeks.ORG
No lost icmp to google and others allegedly but I'm not finding anything. Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde@parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Mike Horwath Sent: Thursday, June 20, 2019 12:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau@Geeks.ORG
We also lost google related sites from 6:50 PM until 7:18 PM. I also was unable to explain this. Brian Riggleman Network Administrator Jaguar Communications 507-214-0258 briggleman@jagcom.net On Thu, Jun 20, 2019 at 12:33 PM Jeff Wilde <jeff.wilde@parkregion.com> wrote:
No lost icmp to google and others allegedly but I'm not finding anything.
Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde@parkregion.com
-----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Mike Horwath Sent: Thursday, June 20, 2019 12:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic
On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
You mean for Hulu and Netflix going down? :)
-- Mike Horwath, reachable via drechsau@Geeks.ORG
I was in a hotel in waterloo ia on centruylink and had a similar problem about the same time to fb and google. On Thu, Jun 20, 2019, 1:00 PM Brian Riggleman <briggleman@jagcom.net> wrote:
We also lost google related sites from 6:50 PM until 7:18 PM. I also was unable to explain this.
Brian Riggleman
Network Administrator
Jaguar Communications
507-214-0258 briggleman@jagcom.net
On Thu, Jun 20, 2019 at 12:33 PM Jeff Wilde <jeff.wilde@parkregion.com> wrote:
No lost icmp to google and others allegedly but I'm not finding anything.
Jeff Wilde IT Manager Park Region Telephone Company P: 218-826-8330 E: jeff.wilde@parkregion.com
-----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Mike Horwath Sent: Thursday, June 20, 2019 12:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic
On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
You mean for Hulu and Netflix going down? :)
-- Mike Horwath, reachable via drechsau@Geeks.ORG
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We had issues reported last night... it was 'fixed' by the time I looked at things Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Mike Horwath <drechsau@GEEKS.ORG> Date: 6/20/19 12:32 PM (GMT-06:00) To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic On Thu, Jun 20, 2019 at 05:30:40PM +0000, Jeff Wilde wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
You mean for Hulu and Netflix going down? :) -- Mike Horwath, reachable via drechsau@Geeks.ORG
Yes we saw issues from 6:50 to about 7:20 as well. None of our BGP sessions dropped to cogent, HE, or MICE but our customers noticed it and I did see a drop in traffic during that time.. It's stumped me and I'm glad I'm not the only one since I thought it was internal to our network but it appears not. On Thu, Jun 20, 2019, 12:30 PM Jeff Wilde <jeff.wilde@parkregion.com> wrote:
On a separate note, did anyone experience very intermittent Internet issues last night for about 30 minutes or so?
Jeff Wilde
IT Manager
*Park Region Telephone Company*
P: 218-826-8330
E: jeff.wilde@parkregion.com
*From:* MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> * On Behalf Of *Andrew Hoyos *Sent:* Thursday, June 20, 2019 11:48 AM *To:* MICE-DISCUSS@LISTS.IPHOUSE.NET *Subject:* Re: [MICE-DISCUSS] ADV: [MICE-DISCUSS] Blackholing DoS traffic
On Jun 20, 2019, at 11:39 AM, Doug McIntyre <merlyn@IPHOUSE.NET> wrote:
On Thu, Jun 20, 2019 at 04:33:12PM +0000, Frank Bulk wrote:
https://www.seattleix.net/blackholing
Does MICE have an blackholng functionality equivalent to SIX?
I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer.
You can adjust the routing with communities, ie. in the MICE communities aera of http://micemn.net/technical.html
you could block-hole the AS that is sending you that traffic.
unfortunately, that just has the effect of traffic going elsewhere - not a blackhole effect. the communities in place would just cause the route not to be advertised to said peer, and the traffic would just ingress your network via a different path. In the case of a DDOS, it’s likely you have multiple ASN’s targeting you.
https://www.seattleix.net/blackholing
SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric.
Perhaps something we should look into for MICE.
—
Andrew Hoyos
hoyosa@gmail.com
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
On Thu, Jun 20, 2019 at 11:48:13AM -0500, Andrew Hoyos wrote:
SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE.
A good idea - perahps use a specific community advertisement with correct next-hop into the ether. Then folks can subscribe as needed. You can even automate a few things as well from known blacklists, then another system to accept new networks to be reviewed then blackholed for <period of time>. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Jun 20, 2019, at 12:31 PM, Mike Horwath <drechsau@GEEKS.ORG> wrote:
On Thu, Jun 20, 2019 at 11:48:13AM -0500, Andrew Hoyos wrote:
SIX, as an example, has a blackhole IP address, and the route servers matching a blackhole community to set next hop to this to sink the traffic on the switch fabric. Perhaps something we should look into for MICE.
A good idea - perahps use a specific community advertisement with correct next-hop into the ether.
Yes, exactly. Rough logic = route server accepts /32’s from you tagged with 65535:666 route server sets next hop no matching prefixes to 206.108.255.0 206.108.255.0 is a blackhole/null destination on switching fabric, or at least translates to dst-mac-addr that is layer2 acl’d to drop. I’d think a prerequisite for this, however, is getting to a point on the route-servers where we are validating/strict filtering prefixes that folks are sending (via IRR), otherwise, we run the risk of people essentially being able to blackhole others.
You can even automate a few things as well from known blacklists, then another system to accept new networks to be reviewed then blackholed for <period of time>.
Sure, essentially what many folks do already internally for RTBH, and signaling to upstream. We utilize Kentik to flip back community tagged routes via BGP to for us for dst ip’s under attack matching certain parameters. We blackhole these on our core routers via the same logic above (match community, set next hop to a IP w/ a discard/null route). In turn, you can community match these and send upstream if your upstreams support a RTBH community as well. — Andrew Hoyos hoyosa@gmail.com
Has support for a blackhole community been added? We’d like to start doing that. Frank From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Steve Howard Sent: Friday, July 05, 2019 12:33 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Blackholing DoS traffic Hmmm... That is interesting. It would be nice to see functionality added to MICE someday. I especially like the 65535:666 community support because it would work nicely with our existing DDoS handling configuration. On a somewhat related note, we use the free CYMRU Unwanted Traffic Renewal Service ( http://www.team-cymru.com/utrs.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.team-2Dcymru.com_utrs.html&d=DwMDaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=xqx0yD1kWOETi5_MVFlMPPxb5us_12870MpQFRgCEns&m=Ylyqe-n-_7xOVxx7Qv9TPlttyjMPJKbz4Vu-0DaM6d8&s=cuVc3-u0Gdn3b7B9dICD1Ee2ggDeH-QYe6ioD8_v8J0&e=> ) to reduce our DDoS traffic. I encourage MICE members to check it out -- as more ASes use it, it becomes more effective. On 06/20/2019 11:33 AM, Frank Bulk wrote: https://www.seattleix.net/blackholing<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.seattleix.net_blackholing&d=DwMDaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=xqx0yD1kWOETi5_MVFlMPPxb5us_12870MpQFRgCEns&m=Ylyqe-n-_7xOVxx7Qv9TPlttyjMPJKbz4Vu-0DaM6d8&s=eU2ycSlW26iz0jX0ajtnZme9ZYW7gXtMy_X2UOa7etk&e=> Does MICE have an blackholng functionality equivalent to SIX? I was visiting with a DDoS mitigation vendor this morning and was curious if there was a way we could automatically mitigate DoS attack traffic coming from a MICE peer. Regards, Frank ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2Dbin_wa-3FSUBED1-3DMICE-2DDISCUSS-26A-3D1&d=DwMDaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=xqx0yD1kWOETi5_MVFlMPPxb5us_12870MpQFRgCEns&m=Ylyqe-n-_7xOVxx7Qv9TPlttyjMPJKbz4Vu-0DaM6d8&s=e_J2cRZRD_iFQ1jeM_xoVprZemC8KThGj3PhibKT5-s&e=> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2Dbin_wa-3FSUBED1-3DMICE-2DDISCUSS-26A-3D1&d=DwMDaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=xqx0yD1kWOETi5_MVFlMPPxb5us_12870MpQFRgCEns&m=Ylyqe-n-_7xOVxx7Qv9TPlttyjMPJKbz4Vu-0DaM6d8&s=e_J2cRZRD_iFQ1jeM_xoVprZemC8KThGj3PhibKT5-s&e=>
On 1/23/20 2:45 PM, Frank Bulk wrote:
Has support for a blackhole community been added? We’d like to start doing that.
AFAIK, no. I think we'd need: - A BGP community* - which when set** causes the route servers to set a next hop of a specific IP address (and probably set no-export too) - which the route servers (?) ARP for, returning a specific MAC - which is blocked by a layer 2 ACL on the core switch and any remotes that are able to do so * At least the well-known blackhole community 65535:666 from RFC 7999. ** The route servers would also have to allow smaller prefixes when the blackhole community is set, so that you could blackhole as small as a single address (in IPv4 at least). See also: https://www.seattleix.net/blackholing In practice, this is probably behind IRR filtering in implemetation priority, because we really should be using IRR filtering so that you can only blackhole your own prefixes. -- Richard
This seems reasonable to me -- what is the timeline to IRR filter implementation? Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Richard Laager Sent: Thursday, January 23, 2020 2:54 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Blackholing DoS traffic On 1/23/20 2:45 PM, Frank Bulk wrote:
Has support for a blackhole community been added? We’d like to start doing that.
AFAIK, no. I think we'd need: - A BGP community* - which when set** causes the route servers to set a next hop of a specific IP address (and probably set no-export too) - which the route servers (?) ARP for, returning a specific MAC - which is blocked by a layer 2 ACL on the core switch and any remotes that are able to do so * At least the well-known blackhole community 65535:666 from RFC 7999. ** The route servers would also have to allow smaller prefixes when the blackhole community is set, so that you could blackhole as small as a single address (in IPv4 at least). See also: https://www.seattleix.net/blackholing In practice, this is probably behind IRR filtering in implemetation priority, because we really should be using IRR filtering so that you can only blackhole your own prefixes. -- Richard
On 1/24/20 8:01 AM, Frank Bulk wrote:
This seems reasonable to me -- what is the timeline to IRR filter implementation?
I would suspect this would require reaching out to MICE participants to find their AS-SETs to build IRR off of, which should also help reveal who is actually using IRR, which irrdb their data is at, and so on. I would think the best source of this data would be PeeringDB, instead of MICE generating and maintaining its own list on the participants page. An API call to MICE's internal id shows the needed data: asn, irr_as_set: https://www.peeringdb.com/api/net?ix_id=446 It appears that 50 of the 88 participants have irr_as_set defined in peeringdb, which is a pretty good start ('work' attached as text file) grep name mice.txt | wc -l 88 grep irr_as_set mice.txt | grep \"\" | wc -l 38 Apologies if methods for this have already been discussed, I didn't dig in the archives. Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965
Relevant: http://instituut.net/~job/LINX99_route_servers.pdf
On Jan 24, 2020, at 8:38 AM, Chris Wopat <wopat@wiscnet.net> wrote:
On 1/24/20 8:01 AM, Frank Bulk wrote:
This seems reasonable to me -- what is the timeline to IRR filter implementation?
I would suspect this would require reaching out to MICE participants to find their AS-SETs to build IRR off of, which should also help reveal who is actually using IRR, which irrdb their data is at, and so on.
I would think the best source of this data would be PeeringDB, instead of MICE generating and maintaining its own list on the participants page.
An API call to MICE's internal id shows the needed data: asn, irr_as_set:
https://www.peeringdb.com/api/net?ix_id=446
It appears that 50 of the 88 participants have irr_as_set defined in peeringdb, which is a pretty good start ('work' attached as text file)
grep name mice.txt | wc -l 88
grep irr_as_set mice.txt | grep \"\" | wc -l 38
Apologies if methods for this have already been discussed, I didn't dig in the archives.
Cheers, -- Chris Wopat Network Engineer, WiscNet wopat@wiscnet.net 608-210-3965 <mice-rr-data.txt>
participants (12)
-
Andrew Hoyos -
Brian Riggleman -
Chris Wopat -
Darin Steffl -
Dave Williams -
Doug McIntyre -
Frank Bulk -
Jay Hanke -
Jeff Wilde -
Mike Horwath -
Richard Laager -
Steve Howard