March 2022 - mice-discuss - lists.micemn.net

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by David Farmer

Michael, Actually, the credit for looking at Netflow goes to Colin. Richard, Thanks for getting that project moving forward, and as Michael said, it happens Everyone, Attached are the detailed reports for the 3 alerts we got in that timeframe last night if you want more details. On Fri, Mar 18, 2022 at 9:24 AM Michael Hare < 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > Dave, thanks for the cluebat about looking at netflow. I didn’t think > that one through myself. I included the top talker we saw below. > > > > Richard, thanks for responding and letting us know what happened. Having > been in similar situations myself, “it happens”. > > > > -Michael > > > > ===/========== > > > > ** nfdump -M /var/local/flows/live/core -T -r nfcapd.202203172000 -n 10 > -s record/packets -A srcip,dstip -6 > > nfdump filter: > > router ip 143.235.32.110 and proto icmp6 > > Aggregated flows 116 > > Top 10 flows ordered by packets: > > Date first seen Duration Src IP > Addr Dst IP Addr Packets Bytes bps > Bpp Flows > > 2022-03-17 19:59:59.104 19.456 fe80::1a2a:d300:64dd:ed24 > <https://flows-1.uwsys.net/nfsen/nfsen.php#null> > ff02::1:ff00:254 <https://flows-1.uwsys.net/nfsen/nfsen.php#null> 9.4 > M 714.9 M 294.0 M 76 2 > > > > > > *From:* MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> * On Behalf Of *David > Farmer > *Sent:* Thursday, March 17, 2022 8:34 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > > > Yes we say it and it reset a bunch of our BGP session on MICE. > > > > Our Arbor Sightling Netflow say the sources were > > > > 2001:504:27::d1af:0:241/128 > > fe80::8618:88ff:fea4:d301/128 > > e80::a66c:2aff:fe76:b400/128 > > > > Destin to; > > All routers ff02::1 > > All MLD Routers ff02::16 > > > > And then a solicited-node address of > > ff02::1:ff00:254 > > Don't know the source of that > > > > > > On Thu, Mar 17, 2022 at 8:22 PM Michael Hare < > 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > > I presume I wasn't the only one that felt the arp/nd storm that began ~ > 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have > info about mac addrs but our peering device reported 20kpps of icmp > neighbor discovery. > > -Michael > [AS3128] > > > > > -- > > =============================================== > David Farmer Email:farmer@umn.edu > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Michael Hare

Dave, thanks for the cluebat about looking at netflow. I didn’t think that one through myself. I included the top talker we saw below. Richard, thanks for responding and letting us know what happened. Having been in similar situations myself, “it happens”. -Michael ===/========== ** nfdump -M /var/local/flows/live/core -T -r nfcapd.202203172000 -n 10 -s record/packets -A srcip,dstip -6 nfdump filter: router ip 143.235.32.110 and proto icmp6 Aggregated flows 116 Top 10 flows ordered by packets: Date first seen Duration Src IP Addr Dst IP Addr Packets Bytes bps Bpp Flows 2022-03-17 19:59:59.104 19.456 fe80::1a2a:d300:64dd:ed24<https://flows-1.uwsys.net/nfsen/nfsen.php#null> ff02::1:ff00:254<https://flows-1.uwsys.net/nfsen/nfsen.php#null> 9.4 M 714.9 M 294.0 M 76 2 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Thursday, March 17, 2022 8:34 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? Yes we say it and it reset a bunch of our BGP session on MICE. Our Arbor Sightling Netflow say the sources were 2001:504:27::d1af:0:241/128 fe80::8618:88ff:fea4:d301/128 e80::a66c:2aff:fe76:b400/128 Destin to; All routers ff02::1 All MLD Routers ff02::16 And then a solicited-node address of ff02::1:ff00:254 Don't know the source of that On Thu, Mar 17, 2022 at 8:22 PM Michael Hare <000000097dab80c5-dmarc-request(a)lists.iphouse.net<mailto:000000097dab80c5-dmarc-request@lists.iphouse.net>> wrote: I presume I wasn't the only one that felt the arp/nd storm that began ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have info about mac addrs but our peering device reported 20kpps of icmp neighbor discovery. -Michael [AS3128] -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Chris Wopat

Seems to jive with some traffic shift on the Compudyne 511 remote switch: http://micelg.usinternet.com/cacti/graph_view.php?action=tree&tree_id=1&lea… -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

FWIW, traffic numbers overall look good. I called Jeremy to get his thoughts. He hasn’t seen anything concerning outside of the initial issue. Out of an abundance of caution, he is shutting down the ports facing the new servers, cutting them (the thing we changed today) off completely. They are not in production use, so there is no reason not to take this step. You might want to do some ping testing to various peers interface IPs and/or check any Juniper-specific storm control stuff. -- Richard > On Mar 17, 2022, at 21:37, Ryan Nelson <ryan.nelson(a)integrisit.com> wrote: > > > Something still doesn't seem right. Really slow traffic via MICE. I just shut down our MICE sessions and the "weirdness" cleared up. > > Ryan > Compudyne/Integris/AS47096 > Ryan Nelson > Network Services Senior Architect > Office: 218-336-2220 > ryan.nelson(a)integrisit.com > > Empowering people through technology. > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Richard Laager <rlaager(a)WIKTEL.COM> > Sent: Thursday, March 17, 2022 9:09 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > EXTERNAL MESSAGE > > We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. > > -- > Richard > > > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > > > That was me. I believe I created a loop with the new servers. > > > > As soon as I realized, I yanked the cables. > > > > Sorry everyone! > > > > -- > > Richard > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Ryan Nelson

Something still doesn't seem right. Really slow traffic via MICE. I just shut down our MICE sessions and the "weirdness" cleared up. Ryan Compudyne/Integris/AS47096 ________________________________ Ryan Nelson Network Services Senior Architect Office: 218-336-2220 ryan.nelson(a)integrisit.com Empowering people through technology. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Richard Laager <rlaager(a)WIKTEL.COM> Sent: Thursday, March 17, 2022 9:09 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? EXTERNAL MESSAGE We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. -- Richard > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > That was me. I believe I created a loop with the new servers. > > As soon as I realized, I yanked the cables. > > Sorry everyone! > > -- > Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

We are hands off as of a few minutes ago. I will provide a more detailed explanation of my mistakes later. -- Richard > On Mar 17, 2022, at 20:34, Richard Laager <rlaager(a)wiktel.com> wrote: > > That was me. I believe I created a loop with the new servers. > > As soon as I realized, I yanked the cables. > > Sorry everyone! > > -- > Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Ryan Malek

We also saw it. We flapped with 2001:504:27::51cc:0:1 (Akamai) at the same time. Not sure they're related. Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 3/17/2022 8:29 PM, Jeff Wilde wrote: > Your correct we did > > > > Sent from my Verizon, Samsung Galaxy smartphone > > > > -------- Original message -------- > From: Michael Hare <000000097dab80c5-dmarc-request(a)LISTS.IPHOUSE.NET> > Date: 3/17/22 8:22 PM (GMT-06:00) > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? > > I presume I wasn't the only one that felt the arp/nd storm that began > ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't > have info about mac addrs but our peering device reported 20kpps of > icmp neighbor discovery. > > -Michael > [AS3128] > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> >

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Richard Laager

That was me. I believe I created a loop with the new servers. As soon as I realized, I yanked the cables. Sorry everyone! -- Richard

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by David Farmer

Yes we say it and it reset a bunch of our BGP session on MICE. Our Arbor Sightling Netflow say the sources were 2001:504:27::d1af:0:241/128 fe80::8618:88ff:fea4:d301/128 e80::a66c:2aff:fe76:b400/128 Destin to; All routers ff02::1 All MLD Routers ff02::16 And then a solicited-node address of ff02::1:ff00:254 Don't know the source of that On Thu, Mar 17, 2022 at 8:22 PM Michael Hare < 000000097dab80c5-dmarc-request(a)lists.iphouse.net> wrote: > I presume I wasn't the only one that felt the arp/nd storm that began ~ > 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have > info about mac addrs but our peering device reported 20kpps of icmp > neighbor discovery. > > -Michael > [AS3128] > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

March 18, 2022

Re: icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT?

by Jeff Wilde

Your correct we did Sent from my Verizon, Samsung Galaxy smartphone -------- Original message -------- From: Michael Hare <000000097dab80c5-dmarc-request(a)LISTS.IPHOUSE.NET> Date: 3/17/22 8:22 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] icmp v6 nd storm ~ 00:58:01 2022/03/18 GMT? I presume I wasn't the only one that felt the arp/nd storm that began ~ 00:58:01 2022/03/18 GMT? Event stopped for us by 01:03:02. I don't have info about mac addrs but our peering device reported 20kpps of icmp neighbor discovery. -Michael [AS3128]

March 18, 2022