Summary:
Yesterday afternoon we had a ~20M spike in egress traffic on MICE that seems to have been caused by the main switch not populating a MAC address or two in its L2 table. Since it appears we've seen this before and none of the recent storm control features got triggered it seems like a bug that will need a little research. I don't believe traffic exchange across the network was affected.
Gory Details:
In late afternoon I noticed that normally quiet ports all had identical graphs showing ~20M of egress traffic. Seeing that it started at about the same time as several changes on the CNS remote switch I called Steve Howard and together we worked through the issue.
Counters indicated that it was unicast traffic and Steve wasn't seeing the same patterns on his switch so we zero'ed in on the idea that it was unknown unicast packets (unicasts who's destination MAC wasn't in the L2 forwarding database.) Since that doesn't make any sense in our environment and logs didn't show continuous storm control activity we cleared the L2 forwarding table on the main switch at about 18:41 and traffic immediately returned to normal and has remained normal since.
Interestingly there was one burst of logs for a couple seconds like this at the beginning:
Nov 5 13:41:04 MICE-SW1 backup MRVL-L2:mrvl_fdb_mac_entry_uc_set(),966:FDb SP HW-overwrite failed(3) for VLANIdx=3:5c:5e:ab:d5:9e:f0/48:IFL=137
I did a search on the error and one of the few hits was some yahoos in Minnesota trying to setup an Internet exchange... but that'll never work. :-)
https://www.google.com/search?q=juniper+%22mrvl_fdb_mac_entry_uc_set%28%29%…
Other links suggest there is a Juniper bug ID and associated software upgrade so we'll need to do a little research to see what makes sense on that front.
Anthony Anderberg
Sr. Systems Analyst
[NUtel_email_logo_1]
320-234-5239
anthonyanderberg(a)nu-telecom.net<mailto:anthonyanderberg@nu-telecom.net>
www.nutelecom.net<http://www.nutelecom.net>
New Core Wireless
ASN: 27204
IPv4: 206.108.255.128/24
IPv6: 2001:504:27:0:0:6a44::1/64
Contact: network(a)7sigma.com
They will be connecting via the CNS switch. Steve, do you have a port
assignment?
Route server config please. :)
Welcome Aboard!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
We topped out over 111G last night on MICE!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
Steve Howard posted this handy URL:
http://redirector.c.youtube.com/report_mapping
When I visit that URL with IPv4, I'm now seeing "mice-msp1". Yay! But
with IPv6, I get lvlt-bna1 (2607:f850::/32) [u].
Are others seeing Google at MICE via IPv6?
--
Richard
I'm looking for some stats on the percentage of traffic that MICE is
handling vs transit links. I will use the average of the numbers to assist
in bringing in additional members.
You can unicast me if you don't want the info public.
Thanks!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
Please allocate xe-0/0/27 and xe-0/0/28 to the Google AE group on the main
switch.
Thanks!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
Well it looks like yesterday they started an introductory level of traffic on MICE then today ramped up a couple of times and are now sitting in the 9+ gbps range.
It does not look like any of my eyeballs have even begun to use it much at all based on my MICE traffic and searching my DNS server query logs for 208.65.155.0/24 queries so all this traffic appears to be going to Gigapop and maybe a few others.
Hopefully this means Google will add one or more 10-gig links soon.
Please allocate xe-0/0/35 on the main switch to Compudyne.
Thanks!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
Savage Communications has been allocated xe-0/0/33 on the main switch
to upgrade their existing service to 10G. All IP address information
will remain the same.
Thanks!
--
Jay Hanke
CTO
Neutral Path Communications
3 Civic Center Plaza, Suite 204
Mankato, MN 56001
(507) 327-2398 mobile
jayhanke(a)neutralpath.net
www.neutralpath.net
