Summary: Yesterday afternoon we had a ~20M spike in egress traffic on MICE that seems to have been caused by the main switch not populating a MAC address or two in its L2 table. Since it appears we've seen this before and none of the recent storm control features got triggered it seems like a bug that will need a little research. I don't believe traffic exchange across the network was affected. Gory Details: In late afternoon I noticed that normally quiet ports all had identical graphs showing ~20M of egress traffic. Seeing that it started at about the same time as several changes on the CNS remote switch I called Steve Howard and together we worked through the issue. Counters indicated that it was unicast traffic and Steve wasn't seeing the same patterns on his switch so we zero'ed in on the idea that it was unknown unicast packets (unicasts who's destination MAC wasn't in the L2 forwarding database.) Since that doesn't make any sense in our environment and logs didn't show continuous storm control activity we cleared the L2 forwarding table on the main switch at about 18:41 and traffic immediately returned to normal and has remained normal since. Interestingly there was one burst of logs for a couple seconds like this at the beginning: Nov 5 13:41:04 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:5c:5e:ab:d5:9e:f0/48:IFL=137 I did a search on the error and one of the few hits was some yahoos in Minnesota trying to setup an Internet exchange... but that'll never work. :-) https://www.google.com/search?q=juniper+%22mrvl_fdb_mac_entry_uc_set%28%29%2C966%3AFDb+SP+HW-overwrite+failed%283%29+for+VLAN%22&gws_rd=ssl Other links suggest there is a Juniper bug ID and associated software upgrade so we'll need to do a little research to see what makes sense on that front. Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg@nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net>
Does our support contract include software tech support where it could be emailed off to Juniper? From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Thursday, November 06, 2014 8:53 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] MICE Traffic Summary: Yesterday afternoon we had a ~20M spike in egress traffic on MICE that seems to have been caused by the main switch not populating a MAC address or two in its L2 table. Since it appears we’ve seen this before and none of the recent storm control features got triggered it seems like a bug that will need a little research. I don’t believe traffic exchange across the network was affected. Gory Details: In late afternoon I noticed that normally quiet ports all had identical graphs showing ~20M of egress traffic. Seeing that it started at about the same time as several changes on the CNS remote switch I called Steve Howard and together we worked through the issue. Counters indicated that it was unicast traffic and Steve wasn’t seeing the same patterns on his switch so we zero’ed in on the idea that it was unknown unicast packets (unicasts who’s destination MAC wasn’t in the L2 forwarding database.) Since that doesn’t make any sense in our environment and logs didn’t show continuous storm control activity we cleared the L2 forwarding table on the main switch at about 18:41 and traffic immediately returned to normal and has remained normal since. Interestingly there was one burst of logs for a couple seconds like this at the beginning: Nov 5 13:41:04 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:5c:5e:ab:d5:9e:f0/48:IFL=137 I did a search on the error and one of the few hits was some yahoos in Minnesota trying to setup an Internet exchange… but that’ll never work. :-) https://www.google.com/search?q=juniper+%22mrvl_fdb_mac_entry_uc_set%28%29%2C966%3AFDb+SP+HW-overwrite+failed%283%29+for+VLAN%22&gws_rd=ssl Other links suggest there is a Juniper bug ID and associated software upgrade so we’ll need to do a little research to see what makes sense on that front. Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
The Mankato Network guys were checking our support contract status, once we get that in order bouncing an email off to Juniper is certainly the way to go. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Thursday, November 06, 2014 9:10 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Traffic Does our support contract include software tech support where it could be emailed off to Juniper? From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Thursday, November 06, 2014 8:53 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Traffic Summary: Yesterday afternoon we had a ~20M spike in egress traffic on MICE that seems to have been caused by the main switch not populating a MAC address or two in its L2 table. Since it appears we've seen this before and none of the recent storm control features got triggered it seems like a bug that will need a little research. I don't believe traffic exchange across the network was affected. Gory Details: In late afternoon I noticed that normally quiet ports all had identical graphs showing ~20M of egress traffic. Seeing that it started at about the same time as several changes on the CNS remote switch I called Steve Howard and together we worked through the issue. Counters indicated that it was unicast traffic and Steve wasn't seeing the same patterns on his switch so we zero'ed in on the idea that it was unknown unicast packets (unicasts who's destination MAC wasn't in the L2 forwarding database.) Since that doesn't make any sense in our environment and logs didn't show continuous storm control activity we cleared the L2 forwarding table on the main switch at about 18:41 and traffic immediately returned to normal and has remained normal since. Interestingly there was one burst of logs for a couple seconds like this at the beginning: Nov 5 13:41:04 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:5c:5e:ab:d5:9e:f0/48:IFL=137 I did a search on the error and one of the few hits was some yahoos in Minnesota trying to setup an Internet exchange... but that'll never work. :-) https://www.google.com/search?q=juniper+%22mrvl_fdb_mac_entry_uc_set%28%29%2C966%3AFDb+SP+HW-overwrite+failed%283%29+for+VLAN%22&gws_rd=ssl Other links suggest there is a Juniper bug ID and associated software upgrade so we'll need to do a little research to see what makes sense on that front. Anthony Anderberg Sr. Systems Analyst [NUtel_email_logo_1] 320-234-5239 anthonyanderberg@nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net> www.nutelecom.net<http://www.nutelecom.net> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (2)
-
Anthony Anderberg -
Jeremy Lumby