March 2014 - mice-discuss - lists.micemn.net

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Nevin Lyne

Doug, Thanks for the update, and the work on this. Glad I have a lot of spare 1950 era gear on the stack for re-use. -Nevin On Monday, March 17, 2014 3:10pm, "Doug McIntyre" <merlyn(a)IPHOUSE.NET> said: > On Fri, Mar 14, 2014 at 06:29:44PM -0500, Doug McIntyre wrote: >> On Thu, Mar 13, 2014 at 04:03:03PM -0500, Doug McIntyre wrote: >> > On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: >> > > We've noticed this being unresponsive several times this evening. >> > >> > This is an actual hardware motherboard failure. >> >> I brought out the new machine this early evening, and it came up >> without a hitch. > > > Just an update on this, as I'm sure people have noticed that this > machine died horribly over the weekend, without any remedy by remote > reboot. > > Same sort of issues, with PCI bus errors, and potential disk failure > log entries. > > I went and visited the machine before lunch, brought another good disk > controller, and now neither controller card will work in the slot, and > PCI errors on either controller. > > I can only assume that the original disk controller (which was moved over > due to the existing setup requiring that type of controller) is poison > and destroying PCIe slots and thus systems that it plugs into. > > So, I think we'll try yet again, and burn in a system for a few days > first before hauling it back out to MICE again. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 17, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Doug McIntyre

On Fri, Mar 14, 2014 at 06:29:44PM -0500, Doug McIntyre wrote: > On Thu, Mar 13, 2014 at 04:03:03PM -0500, Doug McIntyre wrote: > > On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: > > > We've noticed this being unresponsive several times this evening. > > > > This is an actual hardware motherboard failure. > > I brought out the new machine this early evening, and it came up > without a hitch. Just an update on this, as I'm sure people have noticed that this machine died horribly over the weekend, without any remedy by remote reboot. Same sort of issues, with PCI bus errors, and potential disk failure log entries. I went and visited the machine before lunch, brought another good disk controller, and now neither controller card will work in the slot, and PCI errors on either controller. I can only assume that the original disk controller (which was moved over due to the existing setup requiring that type of controller) is poison and destroying PCIe slots and thus systems that it plugs into. So, I think we'll try yet again, and burn in a system for a few days first before hauling it back out to MICE again. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 17, 2014

NTIA announcement about ICANN transition to independent entity

by Mike O'Connor

hi all, i’m presuming most of you have seen the NTIA announcement by now. since i hang “MICE" on my nametag when i participate in ICANN meetings, i thought i’d push along some observations, and some links to detailed info that may not find its way into the press coverage. the announcement: http://www.ntia.doc.gov/press-release/2014/ntia-announces-intent-transition… WASHINGTON – To support and enhance the multistakeholder model of Internet policymaking and governance, the U.S. Commerce Department’s National Telecommunications and Information Administration (NTIA) today announces its intent to transition key Internet domain name functions to the global multistakeholder community. As the first step, NTIA is asking the Internet Corporation for Assigned Names and Numbers (ICANN) to convene global stakeholders to develop a proposal to transition the current role played by NTIA in the coordination of the Internet’s domain name system (DNS). the FAQ on ICANN’s site; http://www.icann.org/en/about/agreements/iana/functions-transfer-faqs-14mar… want to know a LOT about these issues (like PhD level)? Here’s Samantha Dickinson’s summary of a boatload of submissions from all over the place that’s going into the upcoming Net Mundial meeting in Brazil. read all these (like i am right now) and you’ll know where the proposals lie. http://linguasynaptica.com/netmundial-submissions-on-iana/ my view is mostly “keep calm and peer on.” i’m hoping the transition could be the same level of non-event that taking down the NSF backbone network was. on the other hand, we don’t know how this is actually going to turn out until it’s done. the fearmongers in Washington aren’t being helpful. my view “from the bottom of the bottom-up process” is that i’ll be an interested observer and i’ll probably continue to put most of my energy into the day-to-day policy stuff. that’s the stuff that makes ICANN legitimate. i was on a conference call this morning where the CEO (Fadi Chehadi) and the Chair of the Board (Steve Crocker - author of RFC 0001) talked to community members about what’s going on. the points i raised were mostly about bandwidth (the community is already really overloaded) and process (a good project plan would be nice). happy to try to answer questions, although i’m still coming up to speed myself — this came as a surprise to almost everybody. i’m headed to the ICANN meeting in Singapore (leaving this Wednesday, back the following Friday). the outlines will be clearer after that. mikey PHONE: 651-647-6109, FAX: 866-280-2356, WEB: www.haven2.com, HANDLE: OConnorStP (ID for Twitter, Facebook, LinkedIn, etc.) ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 15, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Mike Hemphill

Doug, Thank you! You are a great asset & supporter of the exchange. You are appreciated! Mike On Mar 14, 2014 6:29 PM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > On Thu, Mar 13, 2014 at 04:03:03PM -0500, Doug McIntyre wrote: > > On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: > > > We've noticed this being unresponsive several times this evening. > > > > This is an actual hardware motherboard failure. > > I brought out the new machine this early evening, and it came up > without a hitch. > > I see most people's BGP sessions up, we have 923 prefixes announced into > -01 > now, and things are looking good. > > This should complete out this repair now. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 15, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Nevin Lyne

Thanks, Doug! On Friday, March 14, 2014 6:29pm, "Doug McIntyre" <merlyn(a)IPHOUSE.NET> said: > On Thu, Mar 13, 2014 at 04:03:03PM -0500, Doug McIntyre wrote: >> On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: >> > We've noticed this being unresponsive several times this evening. >> >> This is an actual hardware motherboard failure. > > I brought out the new machine this early evening, and it came up > without a hitch. > > I see most people's BGP sessions up, we have 923 prefixes announced into -01 > now, and things are looking good. > > This should complete out this repair now. > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 14, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Doug McIntyre

On Thu, Mar 13, 2014 at 04:03:03PM -0500, Doug McIntyre wrote: > On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: > > We've noticed this being unresponsive several times this evening. > > This is an actual hardware motherboard failure. I brought out the new machine this early evening, and it came up without a hitch. I see most people's BGP sessions up, we have 923 prefixes announced into -01 now, and things are looking good. This should complete out this repair now. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 14, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Nevin Lyne

Have about 2 dozen of them that have been retired, and in IPHouse's data center so easy to get to Doug. -Nevin On Thursday, March 13, 2014 7:44pm, "Brady Kittel" <bkittel(a)GMAIL.COM> said: > We probably have a couple 1950s lying around too if needed. Drop line if > you need me to take a look. > On Mar 13, 2014 4:03 PM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > >> On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: >> > We've noticed this being unresponsive several times this evening. >> > >> > I'm into the machine, but it is behaving badly, and from past >> > experience, (and not direct evidence since it is remote and I can't >> > get the relavent logs), it feels like a hard drive failing on this >> > system. >> >> >> This is an actual hardware motherboard failure. >> Probably only the 2nd I've seen in my life of using Dell hardware. >> >> Lots of errors around the PCI bus, even with no cards installed. >> Not drives, memory, DRAC card. Nothing left to even try. >> >> Nevin@Arcus has already volunteered another PE1950 chassis if this was the >> case. We just have to coordinate getting it and swapping it in. >> >> The 2nd route-reflector should just keep up with our needs until this >> hardware can be fixed up. >> >> >> -- >> Doug McIntyre <merlyn(a)iphouse.net> >> ~.~ ipHouse ~.~ >> Network Engineer/Provisioning/Jack of all Trades >> >> ######################################################################## >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >> > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 14, 2014

Re: micemn-01.micemn.net route reflector at 206.108.255.1

by Brady Kittel

We probably have a couple 1950s lying around too if needed. Drop line if you need me to take a look. On Mar 13, 2014 4:03 PM, "Doug McIntyre" <merlyn(a)iphouse.net> wrote: > On Thu, Mar 13, 2014 at 12:19:08AM -0500, Doug McIntyre wrote: > > We've noticed this being unresponsive several times this evening. > > > > I'm into the machine, but it is behaving badly, and from past > > experience, (and not direct evidence since it is remote and I can't > > get the relavent logs), it feels like a hard drive failing on this > > system. > > > This is an actual hardware motherboard failure. > Probably only the 2nd I've seen in my life of using Dell hardware. > > Lots of errors around the PCI bus, even with no cards installed. > Not drives, memory, DRAC card. Nothing left to even try. > > Nevin@Arcus has already volunteered another PE1950 chassis if this was the > case. We just have to coordinate getting it and swapping it in. > > The 2nd route-reflector should just keep up with our needs until this > hardware can be fixed up. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > > ######################################################################## > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 14, 2014

Re: DDoS Attacks

by David Farmer

On 3/13/14, 14:04 , Richard Laager wrote: > Last night, we got hit by a ~3 Gbps DDoS attack. It's been a while since > this has happened to us, so I'd like to make sure I'm still up on the > state of the art. > > Is there anything more to be done than the following? > 1. Identify the victim. > 2. Null route the victim. > 3. Propagate the null route to your upstreams (via BGP, if > supported, otherwise a phone call to their NOC). > 4. Move the victim to a new IP. 3a. Notify your (MICE) peer if they are participating in the attack. We for one want to respond and deal with issues to the extent possible. FYI, specifically for us abuse(a)umn.edu, sometimes we are not the quickest, but we really don't just ignore issues reported to us. > To avoid participating in at least some classes of DDoS attacks, we: > * long ago implemented uRPF (and/or similar ACLs) to block spoofed > outbound packets, as recommended by BCP 38 (RFC 2827). Yes, please implement BGP38 to the extent possible, we do. Yes, not everyone realistically can, but if everyone that can did, the world would have a better Internet. > * ensured our NTP servers (and any NTP servers of our customers) > are not responding to monlist queries. The openntpproject.org > website is useful here. They list vulnerable NTP servers by IP > range, or you can get all NTP servers by AS (replace YOUR_AS > with your AS, and optionally, add &csv=1) and then query with > ntpdc -n -c monlist IP: We addressed this in January for our authoritative public NTP servers. But, there were a lot of other NTP servers on campus, this took us a while to mitigate. We think we have this squelched now, but let us know if you see moles that need whacking, especially if the mole is bothering you. > http://openntpproject.org/searchby-asn.cgi?search_asn=YOUR_AS > * just this week started addressing customers with open DNS > resolvers, which can also be used in amplification attacks: > http://openresolverproject.org/searchby-asn.cgi?search_asn=YOUR_AS A continual battle for years, if not a decade. Again, let us know if you see moles that need whacking, especially if the mole is bothering you. :( > Is there anything else we should be doing? If you don't already, block other UDP ports that probably shouldn't talk to the Internet because they are also targets of UDP amplification attacks too, SNMP, Chargen, SSDP, QOTD, etc... https://www.us-cert.gov/ncas/alerts/TA14-017A -- ================================================ David Farmer Email: farmer(a)umn.edu Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 1-612-626-0815 Minneapolis, MN 55414-3029 Cell: 1-612-812-9952 ================================================ ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014

Re: DDoS Attacks

by Frank Bulk

I've documented over 60 attacks to customers in our network in the last 22 months. Some are small, just 200 Mbps, the largest has been 2.6 Gbps. In 98% of all cases the attack is just six to eight minutes long, so by the time I respond to any alarms it's already over. So there's one more thing you can do. 5. Increase your uplink so that the small attacks end up affecting only the subset of access technology that's part of the targeted customer. Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Thursday, March 13, 2014 2:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] DDoS Attacks Last night, we got hit by a ~3 Gbps DDoS attack. It's been a while since this has happened to us, so I'd like to make sure I'm still up on the state of the art. Is there anything more to be done than the following? 1. Identify the victim. 2. Null route the victim. 3. Propagate the null route to your upstreams (via BGP, if supported, otherwise a phone call to their NOC). 4. Move the victim to a new IP. To avoid participating in at least some classes of DDoS attacks, we: * long ago implemented uRPF (and/or similar ACLs) to block spoofed outbound packets, as recommended by BCP 38 (RFC 2827). * ensured our NTP servers (and any NTP servers of our customers) are not responding to monlist queries. The openntpproject.org website is useful here. They list vulnerable NTP servers by IP range, or you can get all NTP servers by AS (replace YOUR_AS with your AS, and optionally, add &csv=1) and then query with ntpdc -n -c monlist IP: http://openntpproject.org/searchby-asn.cgi?search_asn=YOUR_AS * just this week started addressing customers with open DNS resolvers, which can also be used in amplification attacks: http://openresolverproject.org/searchby-asn.cgi?search_asn=YOUR_AS Is there anything else we should be doing? -- Richard ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ######################################################################## To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 13, 2014