- mice-discuss - lists.micemn.net

Re: DDoS Identification / Mitigation

by Ben Wiechman

Kentik looked rather useful. We've been browsing in this space quite a bit for the last year. We've consistently struggled with the size of the attacks we are seeing, routinely hitting our upstreams at 2-4x our total uplink size. Has anyone set up any automated triggers with tools like Kentik? Does anyone have any experience with FastNetMon? ( https://github.com/pavel-odintsov/fastnetmon) It seemed to potentially be a useful roll your own type of solution. Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman(a)arvig.com On Mon, Aug 1, 2016 at 8:48 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > We’ve had great luck with Kentik (https://www.kentik.com/) as a general > netflow tool to at least identity DDoS sources/targets (not to mention a > very well rounded tool for analyzing flow data coupled with BGP info/sankey > diagrams, as well). > > From a mitigation perspective, hopefully your upstream providers support > D/RTBH at a minimum. If they don’t, vote with your wallet and go somewhere > that does. > > Set up your IBGP mesh with a blackhole community and local null routing, > with respective policies and communities on your transit edges matching > their blackhole communities. In theory, you should be able to add a null > route anywhere in your ibgp mesh, and have network wide black holing that > also triggers upstream blackholing as well. Ideally, you’d have some sort > of standalone trigger router with OOB access that you can use to originate > those routes into BGP. > > Bonus points for automating that process, or giving tech actionable alert > to copy/paste into a router. > > -- > Andrew Hoyos > hoyosa(a)gmail.com > > > > > On Jul 30, 2016, at 4:35 PM, Dave Williams <Dave(a)revnetdata.com> wrote: > > > > Hi all – I know this isn’t a MICE specific question, but I can’t think > of a better group of people to ask! I was wondering if anyone could share > their strategy for DDoS detection and mitigation? We randomly have > troubles with it and as you can imagine it’s quite the pain! > > Thanks in advance! > > d > > > > Dave Williams > > Founder / Visionary > > Revelation Network Management, Inc. > > O: 763.367.6161 > > C: 763.670.5558 > > > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > >

Aug. 1, 2016

Re: DDoS Identification / Mitigation

by Andrew Hoyos

We’ve had great luck with Kentik (https://www.kentik.com/) as a general netflow tool to at least identity DDoS sources/targets (not to mention a very well rounded tool for analyzing flow data coupled with BGP info/sankey diagrams, as well). From a mitigation perspective, hopefully your upstream providers support D/RTBH at a minimum. If they don’t, vote with your wallet and go somewhere that does. Set up your IBGP mesh with a blackhole community and local null routing, with respective policies and communities on your transit edges matching their blackhole communities. In theory, you should be able to add a null route anywhere in your ibgp mesh, and have network wide black holing that also triggers upstream blackholing as well. Ideally, you’d have some sort of standalone trigger router with OOB access that you can use to originate those routes into BGP. Bonus points for automating that process, or giving tech actionable alert to copy/paste into a router. -- Andrew Hoyos hoyosa(a)gmail.com > On Jul 30, 2016, at 4:35 PM, Dave Williams <Dave(a)revnetdata.com> wrote: > > Hi all – I know this isn’t a MICE specific question, but I can’t think of a better group of people to ask! I was wondering if anyone could share their strategy for DDoS detection and mitigation? We randomly have troubles with it and as you can imagine it’s quite the pain! > Thanks in advance! > d > > Dave Williams > Founder / Visionary > Revelation Network Management, Inc. > O: 763.367.6161 > C: 763.670.5558 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 1, 2016

Re: DDoS Identification / Mitigation

by Dave Williams

Steve - that looks like a great thing to start with! Thanks! Sent from my T-Mobile 4G LTE Device -------- Original message -------- From: Steve Howard <showard(a)PAULBUNYAN.NET> Date: 7/30/16 10:18 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] DDoS Identification / Mitigation This isn't exactly a detection/mitigation solution, but we use the CYMRU UTRS service to help us reduce our share of the problem: http://www.team-cymru.org/UTRS/ https://www.cymru.com/jtk/misc/utrs.html I encourage everyone to check it out. It's free, and as more organizations participate it becomes more effective. On Jul 30, 2016, at 15:35, Dave Williams <dave(a)REVNETDATA.COM<mailto:dave@revnetdata.com>> wrote: Hi all – I know this isn’t a MICE specific question, but I can’t think of a better group of people to ask! I was wondering if anyone could share their strategy for DDoS detection and mitigation? We randomly have troubles with it and as you can imagine it’s quite the pain! Thanks in advance! d Dave Williams Founder / Visionary Revelation Network Management, Inc. O: 763.367.6161 C: 763.670.5558 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 1, 2016

Re: DDoS Identification / Mitigation

by Mike Horwath

On Sat, Jul 30, 2016 at 11:05:48PM -0500, Brady Kittel wrote: > I worked at a shop for a while that used Arbor networks solution > with good success. From what I recall it requires the upstream > carrier to support it so you'd have to see if yours do. I replied offlist to the OP. There are so many ways to operate denial of service attacks. Some of them are bandwidth eating (ICMP, SYN, UDP flooding), some of them attack the assets (HTTP slow accept, high fake query/request, etc). Each of these needs a slightly different approach. Arbor Networks works quite well if both ends have it - very true and also very, very expensive. F5 has stuff built in to protect assets but also can be very expensive. Fortigate firewalls have some decent stuff in them for relegating throughput and dropping bad traffic at the edge but will require some tuning. IPS/IDS is quite good as well. But once the traffic is at your edge, and the attack is to burn your bandwidth, you've already lost. But if application attacks are going on then FGT can help. CloudFlare moves the endpoint into their network and have come a long way since I first learned of them. Think of it as a reverse proxy for HTTP-type termination and your global DNS server. There are so many things that can help mitigate depending on the type of attack - that needs to be determined :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 31, 2016

Re: DDoS Identification / Mitigation

by Brady Kittel

I worked at a shop for a while that used Arbor networks solution with good success. From what I recall it requires the upstream carrier to support it so you'd have to see if yours do. On Jul 30, 2016 4:35 PM, "Dave Williams" <dave(a)revnetdata.com> wrote: > Hi all – I know this isn’t a MICE specific question, but I can’t think of > a better group of people to ask! I was wondering if anyone could share > their strategy for DDoS detection and mitigation? We randomly have > troubles with it and as you can imagine it’s quite the pain! > > Thanks in advance! > > d > > > > Dave Williams > > Founder / Visionary > > Revelation Network Management, Inc. > > O: 763.367.6161 > > C: 763.670.5558 > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

July 31, 2016

Re: DDoS Identification / Mitigation

by Steve Howard

This isn't exactly a detection/mitigation solution, but we use the CYMRU UTRS service to help us reduce our share of the problem: http://www.team-cymru.org/UTRS/ https://www.cymru.com/jtk/misc/utrs.html I encourage everyone to check it out. It's free, and as more organizations participate it becomes more effective. > On Jul 30, 2016, at 15:35, Dave Williams <dave(a)REVNETDATA.COM> wrote: > > Hi all – I know this isn’t a MICE specific question, but I can’t think of a better group of people to ask! I was wondering if anyone could share their strategy for DDoS detection and mitigation? We randomly have troubles with it and as you can imagine it’s quite the pain! > Thanks in advance! > d > > Dave Williams > Founder / Visionary > Revelation Network Management, Inc. > O: 763.367.6161 > C: 763.670.5558 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

July 31, 2016

DDoS Identification / Mitigation

by Dave Williams

Hi all - I know this isn't a MICE specific question, but I can't think of a better group of people to ask! I was wondering if anyone could share their strategy for DDoS detection and mitigation? We randomly have troubles with it and as you can imagine it's quite the pain! Thanks in advance! d Dave Williams Founder / Visionary Revelation Network Management, Inc. O: 763.367.6161 C: 763.670.5558

July 30, 2016

LTD Broadband 2nd port

by Jason Hanke

LTD Broadband is adding a second MICE connection on the main switch. Their connection on the NP Remote switch will remain. Port: xe-2/0/4 IPv4: 206.108.255.42/24 IPv6: 2001:504:27:0:6:045A::2/64 ASN: 394330 Contact: corey.hauer(a)ltdbroadband.com They will need route server config for this port as well. These ports will NOT be in a LAG group. Welcome Aboard (again)! -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 30, 2016

Re: State of MN DNS Issues

by Frank Bulk

Doesn't seem happy: http://dnsviz.net/d/state.mn.us/dnssec/ Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Tuesday, July 19, 2016 6:53 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] State of MN DNS Issues On 07/19/2016 03:42 AM, Richard Laager wrote: > FYI: state.mn.us DNS has broken DNSSEC and broken IPv6. I called > yesterday, and they were aware of the DNSSEC issue. The state says it's good, and it looks good from my perspective again. -- Richard

July 20, 2016

Re: State of MN DNS Issues

by Richard Laager

On 07/19/2016 03:42 AM, Richard Laager wrote: > FYI: state.mn.us DNS has broken DNSSEC and broken IPv6. I called > yesterday, and they were aware of the DNSSEC issue. The state says it's good, and it looks good from my perspective again. -- Richard

July 19, 2016

Fwd: [JPTECHS] Fwd: [MICE-DISCUSS] State of MN DNS Issues

by David Farmer

I forwarded this off to the guys at the state, here is a response. Thanks ---------- Forwarded message ---------- From: Reilly, Russ (MNIT) <russ.reilly(a)state.mn.us> Date: Tue, Jul 19, 2016 at 9:19 AM Subject: Re: Fwd: [MICE-DISCUSS] State of MN DNS Issues Thanks, we are aware of some complaints and working the issue. Something possibly related to how IPv6 was enable on the Infoblox and the various addresses. Comcast was broke Saturday and we got that fixed so it seems there are just oddities in various resolvers we are working through this with Infoblox. *Russ Reilly* | Network Support, network Support Manager MN.IT Services, State of Minnesota 651-201-1022 (w) *| * russ.reilly(a)state.mn.us *[image: Title: MN.IT Services logo - Description: MN.IT Services logo]* <http://www.mn.gov/oet> Information Technology for Minnesota Government *|* mn.gov/mnit <http://www.mn.gov/oet> Begin forwarded message: *From:* Doug McIntyre <merlyn(a)IPHOUSE.NET <merlyn(a)iphouse.net>> *Date:* July 19, 2016 at 08:51:27 CDT *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)lists.iphouse.net> *Subject:* *Re: [MICE-DISCUSS] State of MN DNS Issues* *Reply-To:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)lists.iphouse.net>> On Tue, Jul 19, 2016 at 03:42:13AM -0500, Richard Laager wrote: FYI: state.mn.us DNS has broken DNSSEC and broken IPv6. I called yesterday, and they were aware of the DNSSEC issue. I had to push out the early beta BIND 9.11 out to production so I could use the new Negative Trust anchor option for them on my frontline resolvers. rndc nta state.mn.us [view] I think only BIND based resolvers thought it was an error though. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

July 19, 2016

Re: State of MN DNS Issues

by Doug McIntyre

On Tue, Jul 19, 2016 at 03:42:13AM -0500, Richard Laager wrote: > FYI: state.mn.us DNS has broken DNSSEC and broken IPv6. I called > yesterday, and they were aware of the DNSSEC issue. I had to push out the early beta BIND 9.11 out to production so I could use the new Negative Trust anchor option for them on my frontline resolvers. rndc nta state.mn.us [view] I think only BIND based resolvers thought it was an error though. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

July 19, 2016

State of MN DNS Issues

by Richard Laager

FYI: state.mn.us DNS has broken DNSSEC and broken IPv6. I called yesterday, and they were aware of the DNSSEC issue. http://dnsviz.net/d/state.mn.us/dnssec/ -- Richard

July 19, 2016

Re: Good news from Akamai (Finally)

by DeLong, Owen

Quite a bit more likely, too. Owen > On Jul 12, 2016, at 14:32 , Hannigan, Martin <marty(a)AKAMAI.COM> wrote: > > This is better than Boston trouncing the Mets in a World Series! > > Thanks all. > > Best, > > -M< > > > >> On Jul 12, 2016, at 17:22, DeLong, Owen <owen(a)AKAMAI.COM> wrote: >> >> We have resolved the router issue and we are now delivering almost 8 Gbps to MICE. >> >> I expect this will continue to increase as things stabilize over the coming days. >> >> Please let me know if you have any questions. >> >> Also, in case you missed the first few memos… >> >> WE NO LONGER PEER WITH ROUTE SERVERS at MICE. >> >> If you want Akamai Traffic, you must set up bilateral peering with us. >> >> Send email to peering(a)akamai.com >> >> If you send email to peering(a)akamai.com and do not receive a response within 48 hours, please resend your email and CC owen(a)akamai.com. >> >> We will respond to peering requests on a timely basis and we have an open peering policy. >> >> Owen >>

July 13, 2016

Re: Good news from Akamai (Finally)

by Hannigan, Martin

This is better than Boston trouncing the Mets in a World Series! Thanks all. Best, -M< > On Jul 12, 2016, at 17:22, DeLong, Owen <owen(a)AKAMAI.COM> wrote: > > We have resolved the router issue and we are now delivering almost 8 Gbps to MICE. > > I expect this will continue to increase as things stabilize over the coming days. > > Please let me know if you have any questions. > > Also, in case you missed the first few memos… > > WE NO LONGER PEER WITH ROUTE SERVERS at MICE. > > If you want Akamai Traffic, you must set up bilateral peering with us. > > Send email to peering(a)akamai.com > > If you send email to peering(a)akamai.com and do not receive a response within 48 hours, please resend your email and CC owen(a)akamai.com. > > We will respond to peering requests on a timely basis and we have an open peering policy. > > Owen >

July 12, 2016

Good news from Akamai (Finally)

by DeLong, Owen

We have resolved the router issue and we are now delivering almost 8 Gbps to MICE. I expect this will continue to increase as things stabilize over the coming days. Please let me know if you have any questions. Also, in case you missed the first few memos… WE NO LONGER PEER WITH ROUTE SERVERS at MICE. If you want Akamai Traffic, you must set up bilateral peering with us. Send email to peering(a)akamai.com If you send email to peering(a)akamai.com and do not receive a response within 48 hours, please resend your email and CC owen(a)akamai.com. We will respond to peering requests on a timely basis and we have an open peering policy. Owen

July 12, 2016

Akamai @ MICE

by DeLong, Owen

Among the multitude of problems that have plagued this process, we have discovered yet another one. This time, it’s not actually our fault. It seems that the nifty new Juniper we deployed there is suffering from a hardware failure in the form of a memory failure which is causing a number of odd behaviors (such as forwarding packets in ways that don’t agree with the forwarding table). We are working to resolve this latest issue as quickly as possible and will continue to work to bring this site back up to full capacity as soon as possible. We apologize (again) for any inconvenience. Owen

July 12, 2016

Re: Slack

by Mike Horwath

On Fri, Jul 01, 2016 at 11:10:29AM -0500, Chris Wopat wrote: > On 07/01/2016 11:05 AM, Mike Horwath wrote: > > > I dunno - what is there to talk about in realtime? > > It's just basically IRC, which is usually just a competition to see > who can idle the longest :) I'd take that challenge.. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 1, 2016

Re: Slack

by Mike Horwath

On Fri, Jul 01, 2016 at 11:07:40AM -0500, Jason Hanke wrote: > circuit turn up, troubleshooting sounds like something that could be helpful. > We could also argue about port fees. :P Probably the most helpful of all! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 1, 2016

Re: Slack

by Chris Wopat

On 07/01/2016 11:05 AM, Mike Horwath wrote: > I dunno - what is there to talk about in realtime? It's just basically IRC, which is usually just a competition to see who can idle the longest :) -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

July 1, 2016

Re: Slack

by Jason Hanke

circuit turn up, troubleshooting We could also argue about port fees. :P On Fri, Jul 1, 2016 at 11:05 AM, Mike Horwath <drechsau(a)geeks.org> wrote: > On Fri, Jul 01, 2016 at 10:47:46AM -0500, Jason Hanke wrote: > > Does anyone on the list use Slack? > > Yep! > > > Would a MICE Slack channel be useful? > > I dunno - what is there to talk about in realtime? > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG > -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 1, 2016

Re: Slack

by Mike Horwath

On Fri, Jul 01, 2016 at 10:47:46AM -0500, Jason Hanke wrote: > Does anyone on the list use Slack? Yep! > Would a MICE Slack channel be useful? I dunno - what is there to talk about in realtime? -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 1, 2016

Re: Slack

by Ben Wiechman

We do at Arvig Ben Wiechman Network Engineer IV | Arvig Direct: 320.256.0184 Cell: 320.247.3224 Office: 320.256.7471 ben.wiechman(a)arvig.com On Fri, Jul 1, 2016 at 10:47 AM, Jason Hanke <jayhanke(a)neutralpath.net> wrote: > Does anyone on the list use Slack? > > Would a MICE Slack channel be useful? > > -- > Jay Hanke > CTO > Neutral Path Communications > 3 Civic Center Plaza, Suite 204 > Mankato, MN 56001 > (507) 327-2398 mobile > jayhanke(a)neutralpath.net > www.neutralpath.net > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

July 1, 2016

Slack

by Jason Hanke

Does anyone on the list use Slack? Would a MICE Slack channel be useful? -- Jay Hanke CTO Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke(a)neutralpath.net www.neutralpath.net

July 1, 2016

Remote Switch Procedures

by Richard Laager

Does the new "Remote Switch" section on the technical page match everyone's understanding of the remote switch procedure and obligations? If not, please suggest specific corrections and I'll get right on it. http://micemn.net/technical.html I'm not sure if we have a formal agreement document. If I get one, I'll update accordingly, and also upload the document itself. -- Richard

June 28, 2016