March 2014 - mice-discuss - lists.micemn.net

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Michael Horwath

On Mar 13, 2014, at 4:03 PM, Doug McIntyre <merlyn(a)iphouse.net> wrote: > Nevin@Arcus has already volunteered another PE1950 chassis if this was the > case. We just have to coordinate getting it and swapping it in. That Nevin; he rocks! I nominate him as MICEMN mascot for 2014. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Doug McIntyre

On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: > We've noticed this being unresponsive several times this evening. > > I'm into the machine, but it is behaving badly, and from past > experience, (and not direct evidence since it is remote and I can't > get the relavent logs), it feels like a hard drive failing on this > system. This is an actual hardware motherboard failure. Probably only the 2nd I've seen in my life of using Dell hardware. Lots of errors around the PCI bus, even with no cards installed. Not drives, memory, DRAC card. Nothing left to even try. Nevin@Arcus has already volunteered another PE1950 chassis if this was the case. We just have to coordinate getting it and swapping it in. The 2nd route-reflector should just keep up with our needs until this hardware can be fixed up. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

Re: DDoS Attacks

by Mike O'Connor

hi all, there’s a recent report from ICANN’s SSAC (Security and Stability Advisory Committee) on DDoS attacks using DNS infrastructure that may be useful https://www.icann.org/en/groups/ssac/documents/sac-065-en.pdf here’s the Executive Summary to give you a sense of what’s in it. Executive Summary This document is intended for the Internet technical community, particularly authoritative and recursive Domain Name System (DNS) operators, network operators, the Internet Corporation for Assigned Names and Numbers (ICANN), and policy makers. It explores several unresolved critical design and deployment issues that have enabled increasingly large and severe Distributed Denial of Service (DDoS) attacks using the DNS. While DDoS attacks can exploit multiple characteristics of network infrastructure and operations, the prevalence and criticality of the DNS means that securing it is both challenging and urgent. These unresolved DNS issues and related DDoS attacks pose a real and present danger to the security and stability of the Internet. The first recommendation below is made to ICANN, while others are made to operators of Internet infrastructure and manufacturers. While in many instances they reflect actions not under ICANN’s control and actors not necessarily within ICANN’s usual community, they are meant to address the overall responsibilities of the multi-stakeholder community and encourage ICANN to take action where it is relevant to do so. In particular, this means ICANN should be looking for ways to increase the effectiveness of steps already being taken against DNS abuse and promoting the participation of others as well as pursuing the measures suggested here. The Security and Stability Advisory Committee (SSAC) strongly recommends that all types of DNS operators and network operators take immediate steps to mitigate the design and deployment issues that make large scale DDoS attacks possible. Specifically, the SSAC strongly recommends that: • ICANN should help facilitate an Internet-wide community effort to reduce the number of open resolvers and networks that allow network spoofing. This effort should involve measurement efforts and outreach. • All network operators should take immediate steps to prevent network address spoofing. • Recursive DNS server operators should take immediate steps to secure open recursive DNS servers. • Authoritative DNS server operators should support efforts to investigate authoritative response rate limiting. • DNS server operators should put in place operational processes to ensure that their DNS software is regularly updated and communicate with their software vendors to keep abreast of the latest developments. • Manufacturers and/or configurators of customer premise networking equipment, including home networking equipment, should take immediate steps to secure these devices and ensure that they are field upgradable when new software is available to fix security vulnerabilities, and aggressively replace the installed base of non-upgradeable devices with upgradeable devices. On Mar 13, 2014, at 2:04 PM, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > Last night, we got hit by a ~3 Gbps DDoS attack. It's been a while since > this has happened to us, so I'd like to make sure I'm still up on the > state of the art. > > Is there anything more to be done than the following? > 1. Identify the victim. > 2. Null route the victim. > 3. Propagate the null route to your upstreams (via BGP, if > supported, otherwise a phone call to their NOC). > 4. Move the victim to a new IP. > > > To avoid participating in at least some classes of DDoS attacks, we: > * long ago implemented uRPF (and/or similar ACLs) to block spoofed > outbound packets, as recommended by BCP 38 (RFC 2827). > * ensured our NTP servers (and any NTP servers of our customers) > are not responding to monlist queries. The openntpproject.org > website is useful here. They list vulnerable NTP servers by IP > range, or you can get all NTP servers by AS (replace YOUR_AS > with your AS, and optionally, add &csv=1) and then query with > ntpdc -n -c monlist IP: > http://openntpproject.org/searchby-asn.cgi?search_asn=YOUR_AS > * just this week started addressing customers with open DNS > resolvers, which can also be used in amplification attacks: > http://openresolverproject.org/searchby-asn.cgi?search_asn=YOUR_AS > > Is there anything else we should be doing? > > -- > Richard > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 PHONE: 651-647-6109, FAX: 866-280-2356, WEB: www.haven2.com, HANDLE: OConnorStP (ID for Twitter, Facebook, LinkedIn, etc.) ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

DDoS Attacks

by Richard Laager

Last night, we got hit by a ~3 Gbps DDoS attack. It's been a while since this has happened to us, so I'd like to make sure I'm still up on the state of the art. Is there anything more to be done than the following? 1. Identify the victim. 2. Null route the victim. 3. Propagate the null route to your upstreams (via BGP, if supported, otherwise a phone call to their NOC). 4. Move the victim to a new IP. To avoid participating in at least some classes of DDoS attacks, we: * long ago implemented uRPF (and/or similar ACLs) to block spoofed outbound packets, as recommended by BCP 38 (RFC 2827). * ensured our NTP servers (and any NTP servers of our customers) are not responding to monlist queries. The openntpproject.org website is useful here. They list vulnerable NTP servers by IP range, or you can get all NTP servers by AS (replace YOUR_AS with your AS, and optionally, add &csv=1) and then query with ntpdc -n -c monlist IP: http://openntpproject.org/searchby-asn.cgi?search_asn=YOUR_AS * just this week started addressing customers with open DNS resolvers, which can also be used in amplification attacks: http://openresolverproject.org/searchby-asn.cgi?search_asn=YOUR_AS Is there anything else we should be doing? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

Re: bgp flapping to 206.108.255.1

by Frank DiGravina

Hello, I am shutting our peering to 206.108.255.1 until I see notice that that your route-reflector issue has been addressed. Thanks, Frank -- Frank DiGravina University of Minnesota Networking & Telecommunications Network Operations Phone: 612-626-9074 Cell: 612-386-0449 E-Mail: digravin(a)umn.edu Helpline: 612-301-HELP ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Brady Kittel

I've also seen issues with failing drives not flagging because of a buggy firmware version on the raid controller or drive at least a few times on Dell servers around that vintage. On Mar 13, 2014 12:19 AM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > We've noticed this being unresponsive several times this evening. > > I'm into the machine, but it is behaving badly, and from past > experience, (and not direct evidence since it is remote and I can't > get the relavent logs), it feels like a hard drive failing on this > system. > > Ie. it may run some of its process out of memory for some functions, > but other functions are probably locking up and not behaving correctly. > Even RAIDed as it is. > > Probably the best course of action is to bring out a fresh drive to it > and swap whatever drive is failing and let the mirror rebuild. I think > we've got some of the 80GB WD800AAJS that is part of the mirror on the > other route reflector around somewhere... > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

micemn-01.micemn.net route reflector at 206.108.255.1

by Doug McIntyre

We've noticed this being unresponsive several times this evening. I'm into the machine, but it is behaving badly, and from past experience, (and not direct evidence since it is remote and I can't get the relavent logs), it feels like a hard drive failing on this system. Ie. it may run some of its process out of memory for some functions, but other functions are probably locking up and not behaving correctly. Even RAIDed as it is. Probably the best course of action is to bring out a fresh drive to it and swap whatever drive is failing and let the mirror rebuild. I think we've got some of the 80GB WD800AAJS that is part of the mirror on the other route reflector around somewhere... -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

XO issues Sunday morning?

by Frank Bulk

This isn't a MICE-related question, but since so many MICE members are in the Minnesota area I thought I would ask. Sunday morning from 1:11 am to about 4:20 am we saw an unusually high number of websites time out over IPv6. We monitor 50 IPv6-enabled web sites over IPv6, performing both an ICMP check (if possible) and then HTTP over IPv6. There were at least 17 sites during that time frame that timed out on our monitoring systems' HTTP GET request. One pattern emerged -- 14 of them traceroute through an XO link that goes through Minneapolis. Here's a sample: traceroute to ipv6.test.Level3.com (2001:1900:2018:3000::105) from 2607:fe28:0:1000::5, port 80, from port 36274, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.395 ms 0.325 ms 0.319 ms 2 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 0.188 ms 0.135 ms 0.185 ms 3 v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1) 1.612 ms 1.545 ms 1.613 ms 4 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 6.667 ms 6.566 ms 8.002 ms 5 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 6.624 ms 7.995 ms 6.602 ms 6 2610:18:18b:c000::11 (2610:18:18b:c000::11) 12.108 ms 12.075 ms 12.137 ms 7 mcr1.minneapolis-mn.us.xo.net (2610:18::30b8) 21.431 ms 21.452 ms 21.421 ms 8 vb1710.rar3.chicago-il.us.xo.net (::ffff:216.156.0.169) 30.701 ms 23.639 ms 23.771 ms .... I then looked at the daily traceroutes we have on file for all 50 sites we monitor and it looks like 15 of them go through XO. So over 93% (14 out of 15) of the sites that go through XO timed out during the 1:11 am to 4:20 am time period. Did anyone with XO connectivity see issues Sunday morning? Frank ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 10, 2014

Re: 2 x 10GE Contact?

by Steve Howard

Hi Martin, I just wanted to make sure that somebody followed up with you. Did you get a response to your question? Thanks, Steve On 03/05/2014 02:55 PM, Hannigan, Martin wrote: > Who can I talk to about adding a second 10G? > > Best, > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 7, 2014

2 x 10GE Contact?

by Hannigan, Martin

Who can I talk to about adding a second 10G? Best, , -M< ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 5, 2014