Hi all, We are migrating our connections to 10G, and I'm wondering if I should request jumbo frames. I'm trying for due diligence, and apologize if this is a rudimentary question. Nanog searches do not have clear recommendations, or maybe I did not search enough. My experience with jumbo frames indicates to me that I should be safe with 9000 byte frames, but there could be PMTU discovery and fragmentation issues during transit or on the receiving side. Would you or would you not enable 9K frames with IXP and ISP connections? I just do not have enough experience in this regard, so I am thankful for any thoughts here. My inclination is to go with 9K frames, yet I do not want to deal with ancillary issues that I cannot anticipate, discern, or troubleshoot - especially during a cut. Any experience from the group is appreciated. Thank you, Jamison Masters Verus Corporation jamison@veruscorp.com<mailto:jamison@veruscorp.com>
The biggest thing I have run across is on an IX where everyone may be running a different MTU if you just set yours to 9K then your router will utilize all of it in it's BGP messaging. If the peer is not also set to 9K then things will get unstable once your router generates a BGP message that is larger than your peer's MTU. You would want to think about using something like "ip tcp adjust-mss 1460" to keep the BGP messaging smaller. Than 1500 while still supporting packets up to 9000 From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jamison Masters Sent: Friday, June 18, 2021 1:09 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Jumbo Frames Hi all, We are migrating our connections to 10G, and I'm wondering if I should request jumbo frames. I'm trying for due diligence, and apologize if this is a rudimentary question. Nanog searches do not have clear recommendations, or maybe I did not search enough. My experience with jumbo frames indicates to me that I should be safe with 9000 byte frames, but there could be PMTU discovery and fragmentation issues during transit or on the receiving side. Would you or would you not enable 9K frames with IXP and ISP connections? I just do not have enough experience in this regard, so I am thankful for any thoughts here. My inclination is to go with 9K frames, yet I do not want to deal with ancillary issues that I cannot anticipate, discern, or troubleshoot - especially during a cut. Any experience from the group is appreciated. Thank you, Jamison Masters Verus Corporation jamison@veruscorp.com ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Hey Jamison, The consensus view is this: 1500 Bytes is the MTU of the internet, and MTU1500 has so much inertia that you or me or all this mailing list together could not change it. The sages also tell us that PMTUd--path MTU discovery--does not work with any reliability and thus does not work in reality. Things like NAT, firewalls and deep packet inspectors routinely break ICMP of all kinds. Most crushingly for an IX: there's no such thing as layer-2 MTU discovery. In the same L2 domain, all routers need to use the same MTU, or stuff will break (Jeremy pointed to one example of failing BGP message exchange). If I set my router to MTU1500, and you set yours to MTU9000, how would we ever know? This is a misconfiguration, so what could coordinate those values on an IX? Thus, we should all use MTU 1500. We could have a separate VLAN that's MTU 9000 where all the routers connected would set their interface to MTU 9000. The routers wouldn't have any MTU breakage over that L2 domain. SIX does this, and it seems the best and most rational way to allow MTU 9000 for those who want it. Finally, views do differ, i'm sure others have opinions that do not match mine. Cheers, Jonathan Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA On Fri., 2021-06-18 1:09 p.m., Jamison Masters wrote:
Hi all,
We are migrating our connections to 10G, and I’m wondering if I should request jumbo frames.
I’m trying for due diligence, and apologize if this is a rudimentary question. Nanog searches do not have clear recommendations, or maybe I did not search enough.
My experience with jumbo frames indicates to me that I should be safe with 9000 byte frames, but there could be PMTU discovery and fragmentation issues during transit or on the receiving side.
Would you or would you not enable 9K frames with IXP and ISP connections? I just do not have enough experience in this regard, so I am thankful for any thoughts here.
My inclination is to go with 9K frames, yet I do not want to deal with ancillary issues that I cannot anticipate, discern, or troubleshoot – especially during a cut.
Any experience from the group is appreciated.
Thank you,
Jamison Masters
Verus Corporation
jamison@veruscorp.com <mailto:jamison@veruscorp.com>
------------------------------------------------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>
Thanks for that input Jonathan! Steve From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jonathan Stewart Sent: Friday, June 18, 2021 2:16 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Jumbo Frames Hey Jamison, The consensus view is this: 1500 Bytes is the MTU of the internet, and MTU1500 has so much inertia that you or me or all this mailing list together could not change it. The sages also tell us that PMTUd--path MTU discovery--does not work with any reliability and thus does not work in reality. Things like NAT, firewalls and deep packet inspectors routinely break ICMP of all kinds. Most crushingly for an IX: there's no such thing as layer-2 MTU discovery. In the same L2 domain, all routers need to use the same MTU, or stuff will break (Jeremy pointed to one example of failing BGP message exchange). If I set my router to MTU1500, and you set yours to MTU9000, how would we ever know? This is a misconfiguration, so what could coordinate those values on an IX? Thus, we should all use MTU 1500. We could have a separate VLAN that's MTU 9000 where all the routers connected would set their interface to MTU 9000. The routers wouldn't have any MTU breakage over that L2 domain. SIX does this, and it seems the best and most rational way to allow MTU 9000 for those who want it. Finally, views do differ, i'm sure others have opinions that do not match mine. Cheers, Jonathan Jonathan Stewart Network Engineer LES.NET - AS18451 Desk: 1-204-666-6191 Mobile: 1-204-990-2120 130 Portage Avenue E Winnipeg, MB R3C 0A1 CANADA On Fri., 2021-06-18 1:09 p.m., Jamison Masters wrote: Hi all, We are migrating our connections to 10G, and I'm wondering if I should request jumbo frames. I'm trying for due diligence, and apologize if this is a rudimentary question. Nanog searches do not have clear recommendations, or maybe I did not search enough. My experience with jumbo frames indicates to me that I should be safe with 9000 byte frames, but there could be PMTU discovery and fragmentation issues during transit or on the receiving side. Would you or would you not enable 9K frames with IXP and ISP connections? I just do not have enough experience in this regard, so I am thankful for any thoughts here. My inclination is to go with 9K frames, yet I do not want to deal with ancillary issues that I cannot anticipate, discern, or troubleshoot - especially during a cut. Any experience from the group is appreciated. Thank you, Jamison Masters Verus Corporation jamison@veruscorp.com<mailto:jamison@veruscorp.com> ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
On 6/18/21 2:16 PM, Jonathan Stewart wrote:
Finally, views do differ, i'm sure others have opinions that do not match mine.
I don't see anything wrong in what you said. I'll add some detail on what we're doing in case it's helpful (or if someone can educate me if I'm doing something bad): I use MTU 9000 on my transit connections, /coordinated with the transit provider/, because there seems to be no harm in doing that. I use MTU 9000* on all my router-to-router links because I need to be able to carry larger-than-1500-byte circuit traffic. If nothing else, circuit VLANs all need to be a bit over 1500 bytes on the outside (i.e. with my tags and possibly tags of other carriers involved) so that they are 1500 bytes on the inside. * Maybe I should/will-need-to change this to 9000 + a bit to allow for 9000 inside. I'm 1500 bytes at the customer edge for Internet. I'm also 1500 bytes at IXes. -- Richard
I use MTU 9000* on all my router-to-router links because I need to be able to carry larger-than-1500-byte circuit traffic. If nothing else, circuit VLANs all need to be a bit over 1500 bytes on the outside (i.e. with my tags and possibly tags of other carriers involved) so that they are 1500 bytes on the inside. * Maybe I should/will-need-to change this to 9000 + a bit to allow for 9000 inside.
We don't typically see carriers looking for anything greater than about 2000 bytes in practice, though we also follow a similar 9000 byte configuration on our internal transport links. 1500 to the internet edge. You could fiddle with the MTU to eek out a few extra bytes for circuits, but what you're doing today already ~4x the real world use cases that we've seen. Maybe experiences differ for providers that carry a lot of traffic between data centers. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman@arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Fri, Jun 18, 2021 at 2:53 PM Richard Laager <rlaager@wiktel.com> wrote:
On 6/18/21 2:16 PM, Jonathan Stewart wrote:
Finally, views do differ, i'm sure others have opinions that do not match mine.
I don't see anything wrong in what you said. I'll add some detail on what we're doing in case it's helpful (or if someone can educate me if I'm doing something bad):
I use MTU 9000 on my transit connections, *coordinated with the transit provider*, because there seems to be no harm in doing that.
I use MTU 9000* on all my router-to-router links because I need to be able to carry larger-than-1500-byte circuit traffic. If nothing else, circuit VLANs all need to be a bit over 1500 bytes on the outside (i.e. with my tags and possibly tags of other carriers involved) so that they are 1500 bytes on the inside. * Maybe I should/will-need-to change this to 9000 + a bit to allow for 9000 inside.
I'm 1500 bytes at the customer edge for Internet. I'm also 1500 bytes at IXes.
-- Richard
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
Thanks everyone. Safe word: “1500 bytes.“ Gratefully, Jamison From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Ben Wiechman Sent: Friday, June 18, 2021 2:57 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Jumbo Frames I use MTU 9000* on all my router-to-router links because I need to be able to carry larger-than-1500-byte circuit traffic. If nothing else, circuit VLANs all need to be a bit over 1500 bytes on the outside (i.e. with my tags and possibly tags of other carriers involved) so that they are 1500 bytes on the inside. * Maybe I should/will-need-to change this to 9000 + a bit to allow for 9000 inside. We don't typically see carriers looking for anything greater than about 2000 bytes in practice, though we also follow a similar 9000 byte configuration on our internal transport links. 1500 to the internet edge. You could fiddle with the MTU to eek out a few extra bytes for circuits, but what you're doing today already ~4x the real world use cases that we've seen. Maybe experiences differ for providers that carry a lot of traffic between data centers. Ben Wiechman Director of IP Strategy and Engineering 320.247.3224 | ben.wiechman@arvig.com<mailto:ben.wiechman@arvig.com> Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com<http://arvig.com> On Fri, Jun 18, 2021 at 2:53 PM Richard Laager <rlaager@wiktel.com<mailto:rlaager@wiktel.com>> wrote: On 6/18/21 2:16 PM, Jonathan Stewart wrote: Finally, views do differ, i'm sure others have opinions that do not match mine. I don't see anything wrong in what you said. I'll add some detail on what we're doing in case it's helpful (or if someone can educate me if I'm doing something bad): I use MTU 9000 on my transit connections, coordinated with the transit provider, because there seems to be no harm in doing that. I use MTU 9000* on all my router-to-router links because I need to be able to carry larger-than-1500-byte circuit traffic. If nothing else, circuit VLANs all need to be a bit over 1500 bytes on the outside (i.e. with my tags and possibly tags of other carriers involved) so that they are 1500 bytes on the inside. * Maybe I should/will-need-to change this to 9000 + a bit to allow for 9000 inside. I'm 1500 bytes at the customer edge for Internet. I'm also 1500 bytes at IXes. -- Richard ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (6)
-
Ben Wiechman -
Jamison Masters -
Jeremy Lumby -
Jonathan Stewart -
Richard Laager -
Steve Savoy