I have been watching the broadcast traffic on MICE steadily increase. We are now at the point where our broadcast traffic is 10X that of many exchanges that are larger than us. I would encourage the membership to look over the configuration of their peering routers to make sure that they are configured in a way that they are not generating excessive broadcast traffic. In the past on the list we have referenced the Amsterdam Internet Exchange configuration guidelines. I feel that the router specific configuration hints are a great starting point to be sure that your router is configured properly. https://ams-ix.net/technical/specifications-descriptions/config-guide Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
Has the Tech Committee looked at ARP Sponge at all? https://github.com/AMS-IX/arpsponge On Mon, Apr 2, 2018 at 10:30 AM, Jeremy Lumby <jlumby@mnvoip.com> wrote:
I have been watching the broadcast traffic on MICE steadily increase. We are now at the point where our broadcast traffic is 10X that of many exchanges that are larger than us. I would encourage the membership to look over the configuration of their peering routers to make sure that they are configured in a way that they are not generating excessive broadcast traffic. In the past on the list we have referenced the Amsterdam Internet Exchange configuration guidelines. I feel that the router specific configuration hints are a great starting point to be sure that your router is configured properly. https://ams-ix.net/technical/ specifications-descriptions/config-guide
Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
-- Jay Hanke Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke@neutralpath.net www.neutralpath.net
I looked at ARP sponge briefly, and while I think it can help (when a network drops offline), I was a bit disappointed that there was no IPv6 support. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jason Hanke Sent: Monday, April 02, 2018 10:35 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Broadcast Traffic Has the Tech Committee looked at ARP Sponge at all? https://github.com/AMS-IX/arpsponge On Mon, Apr 2, 2018 at 10:30 AM, Jeremy Lumby <jlumby@mnvoip.com> wrote: I have been watching the broadcast traffic on MICE steadily increase. We are now at the point where our broadcast traffic is 10X that of many exchanges that are larger than us. I would encourage the membership to look over the configuration of their peering routers to make sure that they are configured in a way that they are not generating excessive broadcast traffic. In the past on the list we have referenced the Amsterdam Internet Exchange configuration guidelines. I feel that the router specific configuration hints are a great starting point to be sure that your router is configured properly. https://ams-ix.net/technical/specifications-descriptions/config-guide Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com -- Jay Hanke Neutral Path Communications 3 Civic Center Plaza, Suite 204 Mankato, MN 56001 (507) 327-2398 mobile jayhanke@neutralpath.net www.neutralpath.net To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
+1 to AMSIX guidelines. For context, on our port, I’m seeing about 80pps of broadcast/multicast traffic, with a fairly recent jump. FWIW, for comparison purposes: SIX - ~10pps NWAX - ~30pps Equinix ORD - ~200pps (this is likely lots of ARP fail due to current IP Migration, previous levels were ~60pps) MadIX - 3pps MKEIX - 1pps I just checked what some of the spew was, and it seems like a large majority of it was the pair of RR’s sending out neighbor solicitation messages. Is there some timers we can adjust on those? — Andrew Hoyos hoyosa@gmail.com
On Apr 2, 2018, at 10:30 AM, Jeremy Lumby <jlumby@MNVOIP.COM> wrote:
I have been watching the broadcast traffic on MICE steadily increase. We are now at the point where our broadcast traffic is 10X that of many exchanges that are larger than us. I would encourage the membership to look over the configuration of their peering routers to make sure that they are configured in a way that they are not generating excessive broadcast traffic. In the past on the list we have referenced the Amsterdam Internet Exchange configuration guidelines. I feel that the router specific configuration hints are a great starting point to be sure that your router is configured properly. https://ams-ix.net/technical/specifications-descriptions/config-guide
Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington, MN 55425 Main: 612-355-7740 x211 Direct: 612-392-6814 EFax: 952-873-7425 jlumby@mnvoip.com
On Mon, Apr 02, 2018 at 10:41:37AM -0500, Andrew Hoyos wrote:
I just checked what some of the spew was, and it seems like a large majority of it was the pair of RR’s sending out neighbor solicitation messages. Is there some timers we can adjust on those?
I think a lot of it is from BIRD6 looking for route peers that aren't there. Traditionally, we've configured up IPv4 & IPv6 dual stack route peers in the route servers. Doing a tcpdump on the ICMP6 protocol traffic on them, it is all ICMPv6 ND looking for route peers that aren't configured on the far end. Should we disable all the IPv6 route peers that people aren't using? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Can you post a list and/or communicate with the SP that are not set up for peering and let them either fix it or indicate whether they have any immediate plans to implement IPv6 peering. Maybe make it an onboarding question for new peers - do you intend to peer with the route servers using IPv6 and configure BIRD appropriate for new members? If off document the process to request IPv6 be turned up. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Mon, Apr 2, 2018 at 10:52 AM, Doug McIntyre <merlyn@iphouse.net> wrote:
On Mon, Apr 02, 2018 at 10:41:37AM -0500, Andrew Hoyos wrote:
I just checked what some of the spew was, and it seems like a large majority of it was the pair of RR’s sending out neighbor solicitation messages. Is there some timers we can adjust on those?
I think a lot of it is from BIRD6 looking for route peers that aren't there.
Traditionally, we've configured up IPv4 & IPv6 dual stack route peers in the route servers.
Doing a tcpdump on the ICMP6 protocol traffic on them, it is all ICMPv6 ND looking for route peers that aren't configured on the far end. Should we disable all the IPv6 route peers that people aren't using? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
Can BIRD be configured to be passive with BGP? That is, it would respond to the participant’s router starting the session, but never try to initiate a session. -- Richard
On Mon, Apr 02, 2018 at 10:57:08AM -0500, Richard Laager wrote:
Can BIRD be configured to be passive with BGP? That is, it would respond to the participant’s router starting the session, but never try to initiate a session.
Yes, that is an option. So, you're suggesting that we kick over the non-configured IPv6 peers over to passive BGP on the BIRD6 side? I don't think that should be a default option, although I don't think it matters too much if it is BIRD trying to initiate the BGP or the client. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
On Apr 2, 2018, at 11:03 PM, Doug McIntyre <merlyn@IPHOUSE.NET> wrote:
Yes, that is an option.
So, you're suggesting that we kick over the non-configured IPv6 peers over to passive BGP on the BIRD6 side? I don't think that should be a default option, although I don't think it matters too much if it is BIRD trying to initiate the BGP or the client.
I’d vote for passive BGP sessions (v4/v6) on the route servers across the board, already established or not. Easy solution to the issue at hand and sane default. “Quiet unless spoken to”. Many other IX’s do this by default (AMSIX, SIX, Denver-IX, etc). I’m not sure I see any downsides to this behavior. -- Andrew Hoyos hoyosa@gmail.com
My 2c is proceed but consider the following 1) space out changes the route servers by a day or so. 2) force reset bgp on the changed route server. As someone who has previously screwed up control plane BGP filters (such that sessions only came up if remove side was active), I believe step #2 will shake out anyone who has their control plane filters wrong. Not really MICE's fault per se, but a nice gesture. -Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Tuesday, April 03, 2018 12:56 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Broadcast Traffic
On 04/02/2018 11:23 PM, Andrew Hoyos wrote:
I’d vote for passive BGP sessions (v4/v6) on the route servers across the board, already established or not. Easy solution to the issue at hand and sane default. “Quiet unless spoken to”.
This is what I was thinking too.
-- Richard
Agreed on both the passive configuration for the route servers and implementation suggestions. Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Tue, Apr 3, 2018 at 9:23 AM, Michael Hare <michael.hare@wisc.edu> wrote:
My 2c is proceed but consider the following
1) space out changes the route servers by a day or so. 2) force reset bgp on the changed route server.
As someone who has previously screwed up control plane BGP filters (such that sessions only came up if remove side was active), I believe step #2 will shake out anyone who has their control plane filters wrong. Not really MICE's fault per se, but a nice gesture.
-Michael
-----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Richard Laager Sent: Tuesday, April 03, 2018 12:56 AM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Broadcast Traffic
I’d vote for passive BGP sessions (v4/v6) on the route servers across
On 04/02/2018 11:23 PM, Andrew Hoyos wrote: the board, already established or not. Easy solution to the issue at hand and sane default. “Quiet unless spoken to”.
This is what I was thinking too.
-- Richard
While I support looking into the source of broadcast and multicast traffic and eliminating any that is unnecessary. I'd also like to add a little perspective; we (AS57) currently on our MICE interface are receiving about 1.5Mpps and sending about 1Mpps, for a total of about 2.5Mpps on our interface. So even at 100pps that is (100/2,500,000) or 0.004% broadcast traffic. So let's not panic either. Thanks. On Mon, Apr 2, 2018 at 10:41 AM, Andrew Hoyos <hoyosa@gmail.com> wrote:
+1 to AMSIX guidelines.
For context, on our port, I’m seeing about 80pps of broadcast/multicast traffic, with a fairly recent jump.
FWIW, for comparison purposes: SIX - ~10pps NWAX - ~30pps Equinix ORD - ~200pps (this is likely lots of ARP fail due to current IP Migration, previous levels were ~60pps) MadIX - 3pps MKEIX - 1pps
I just checked what some of the spew was, and it seems like a large majority of it was the pair of RR’s sending out neighbor solicitation messages. Is there some timers we can adjust on those?
— Andrew Hoyos hoyosa@gmail.com
On Apr 2, 2018, at 10:30 AM, Jeremy Lumby <jlumby@MNVOIP.COM> wrote:
I have been watching the broadcast traffic on MICE steadily increase. We are now at the point where our broadcast traffic is 10X that of many exchanges that are larger than us. I would encourage the membership to look over the configuration of their peering routers to make sure that they are configured in a way that they are not generating excessive broadcast traffic. In the past on the list we have referenced the Amsterdam Internet Exchange configuration guidelines. I feel that the router specific configuration hints are a great starting point to be sure that your router is configured properly. https://ams-ix.net/technical/ specifications-descriptions/config-guide
Jeremy Lumby Minnesota VoIP 9217 17th Ave S <https://maps.google.com/?q=9217+17th+Ave+S+Suite+216+Bloomington,+MN+55425&entry=gmail&source=g> Suite 216 <https://maps.google.com/?q=9217+17th+Ave+S+Suite+216+Bloomington,+MN+55425&entry=gmail&source=g> Bloomington, MN 55425 <https://maps.google.com/?q=9217+17th+Ave+S+Suite+216+Bloomington,+MN+55425&entry=gmail&source=g> Main: 612-355-7740 x211 <(612)%20355-7740> Direct: 612-392-6814 <(612)%20392-6814> EFax: 952-873-7425 <(952)%20873-7425> jlumby@mnvoip.com
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================
participants (8)
-
Andrew Hoyos -
Ben Wiechman -
David Farmer -
Doug McIntyre -
Jason Hanke -
Jeremy Lumby -
Michael Hare -
Richard Laager