Not learning all routes from RR#2
We (AS53347) moved to a new border router overnight and had some issues with getting our MICE route reflector peerings up -- we'd learn a route or two per second. We had to tweak our Arista's BGP configuration to not "enforce-first-as". Most other direct peers were fine. Eventually RR #1 came up just fine, but RR #2 continues to languish, learning just learning about 10,900 routes. SiouxCenter-Arista-North(s1)#show ip bgp sum BGP summary information for VRF default Router identifier 207.32.15.22, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.1 4 53679 44493 4138 0 0 10:56:46 Estab 56188 56188 206.108.255.2 4 53679 39124 7640 0 0 11:38:40 Estab 10929 10929 I'm not sure if we have remote access to the RR's, but whoever does, any hints as to why we're experiencing this behavior? Frank
On Wed, Jun 27, 2018 at 06:25:02PM +0000, Frank Bulk wrote:
Eventually RR #1 came up just fine, but RR #2 continues to languish, learning just learning about 10,900 routes.
206.108.255.1 4 53679 44493 4138 0 0 10:56:46 Estab 56188 56188 206.108.255.2 4 53679 39124 7640 0 0 11:38:40 Estab 10929 10929
Hmm, odd. The configs are kept in sync with each other. The machines are virtually identical, plus or minus some minor hardware difference. The # of prefixes each has isn't signifigantly different. micemn-01 58611 micemn-02 58581 I show no errors logged by BIRD for AS 18883 The # of routes I see from RR#2 matches. 206.108.255.1 53679 196371 19684 0 98 6d 11:17:58 Establ inet.0: 79/56094/55912/182 206.108.255.2 53679 196440 19686 0 72 6d 11:18:03 Establ inet.0: 1562/56092/55910/182 Also, I see I had more active routes get established via RR#2, but that is just whatever my router choose. So, not seeing that at the RR's, or here? -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
The two MICE-attached routers I'm responsible for look OK, and about the same as they have looked recently: 206.108.255.1 4 53679 10130010 700710 305989456 0 0 1y11w 56003 206.108.255.2 4 53679 9531122 656345 305989456 0 0 1y7w 56001 -- 206.108.255.1 0 53679 14571189 909087 587718403 0 0 46w5d 55942 206.108.255.2 0 53679 14593796 908792 587718403 0 0 46w5d 55940 The MICE Looking Glass seems about the same: telnet://rviews:@route-server.micemn.net:23 206.108.255.1 4 53679 5001870 255450 11721640 0 0 23w0d 56004 206.108.255.2 4 53679 4998551 255464 11721640 0 0 23w0d 56002 The "no bgp enforce-first-as" config is expected on some platforms: http://www.micemn.net/technical.html Is traffic flowing normally otherwise? Perhaps you're running into an L1/L2 or MTU issue such that this is a symptom of another problem? Cheers, anthony -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, June 27, 2018 1:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Not learning all routes from RR#2 We (AS53347) moved to a new border router overnight and had some issues with getting our MICE route reflector peerings up -- we'd learn a route or two per second. We had to tweak our Arista's BGP configuration to not "enforce-first-as". Most other direct peers were fine. Eventually RR #1 came up just fine, but RR #2 continues to languish, learning just learning about 10,900 routes. SiouxCenter-Arista-North(s1)#show ip bgp sum BGP summary information for VRF default Router identifier 207.32.15.22, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.1 4 53679 44493 4138 0 0 10:56:46 Estab 56188 56188 206.108.255.2 4 53679 39124 7640 0 0 11:38:40 Estab 10929 10929 I'm not sure if we have remote access to the RR's, but whoever does, any hints as to why we're experiencing this behavior? Frank
Anthony mentioned MTU. I have had issues in the past with BGP, and larger MTUs. Especially when my router had a larger MTU than the path/destination router. BGP would establish, get a few prefixes, and then time out. I just took a look, and the route servers have a 1500 MTU. Are you set larger? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of AnthonyAnderberg@nuvera.net Sent: Wednesday, June 27, 2018 5:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Not learning all routes from RR#2 The two MICE-attached routers I'm responsible for look OK, and about the same as they have looked recently: 206.108.255.1 4 53679 10130010 700710 305989456 0 0 1y11w 56003 206.108.255.2 4 53679 9531122 656345 305989456 0 0 1y7w 56001 -- 206.108.255.1 0 53679 14571189 909087 587718403 0 0 46w5d 55942 206.108.255.2 0 53679 14593796 908792 587718403 0 0 46w5d 55940 The MICE Looking Glass seems about the same: telnet://rviews:@route-server.micemn.net:23 206.108.255.1 4 53679 5001870 255450 11721640 0 0 23w0d 56004 206.108.255.2 4 53679 4998551 255464 11721640 0 0 23w0d 56002 The "no bgp enforce-first-as" config is expected on some platforms: http://www.micemn.net/technical.html Is traffic flowing normally otherwise? Perhaps you're running into an L1/L2 or MTU issue such that this is a symptom of another problem? Cheers, anthony -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, June 27, 2018 1:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Not learning all routes from RR#2 We (AS53347) moved to a new border router overnight and had some issues with getting our MICE route reflector peerings up -- we'd learn a route or two per second. We had to tweak our Arista's BGP configuration to not "enforce-first-as". Most other direct peers were fine. Eventually RR #1 came up just fine, but RR #2 continues to languish, learning just learning about 10,900 routes. SiouxCenter-Arista-North(s1)#show ip bgp sum BGP summary information for VRF default Router identifier 207.32.15.22, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.1 4 53679 44493 4138 0 0 10:56:46 Estab 56188 56188 206.108.255.2 4 53679 39124 7640 0 0 11:38:40 Estab 10929 10929 I'm not sure if we have remote access to the RR's, but whoever does, any hints as to why we're experiencing this behavior? Frank
Thanks for the feedback. Interface MTU is showing 1500 bytes. Traffic is running -- peering with others without any known issues. This "received unexpected EOF" in the "show ip bgp neighbors 206.108.255.2" jumps out at me, but my google-fu doesn't reveal anything obvious other than a hint at a TCP checksum issue. Resetting the session again (not the first time) has appeared to resolve it. Interesting that IPv6 is showing a count discrepancy, too: 2001:504:27::d1af:0:1 4 53679 49967 7442 0 0 20:39:29 Estab 2199 2199 2001:504:27::d1af:0:2 4 53679 49962 7442 0 0 20:39:21 Estab 8993 8993 SiouxCenter-Arista-North(s1)#show int e 3/24 Ethernet3/24 is up, line protocol is up (connected) Hardware is Ethernet, address is 001c.73d7.6391 Description: to MICE cia CYAN 7/4 Internet address is 206.108.255.67/24 Broadcast address is 255.255.255.255 IPv6 link-local address is fe80::21c:73ff:fed7:6391/64 IPv6 global unicast address(es): 2001:504:27::d063:0:1, subnet is 2001:504:27::/64 IP MTU 1500 bytes , BW 10000000 kbit Full-duplex, 10Gb/s, auto negotiation: off, uni-link: disabled Up 21 hours, 29 minutes, 16 seconds Loopback Mode : None 5 link status changes since last clear Last clearing of "show interface" counters never 30 seconds input rate 1.85 Gbps (18.8% with framing overhead), 168059 packets/sec 30 seconds output rate 285 Mbps (3.0% with framing overhead), 118650 packets/sec 9241433949 packets input, 12515818295584 bytes Received 1960477 broadcasts, 5282001 multicast 0 runts, 0 giants 0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards 0 PAUSE input 7518589614 packets output, 2302004527878 bytes Sent 373 broadcasts, 17753 multicast 0 output errors, 0 collisions 0 late collision, 0 deferred, 0 output discards 0 PAUSE output SiouxCenter-Arista-North(s1)# SiouxCenter-Arista-North(s1)#show ip bgp neighbors 206.108.255.2 BGP neighbor is 206.108.255.2, remote AS 53679, external link Description: MICE_RR_#2 BGP version 4, remote router ID 206.108.255.2, VRF default Negotiated BGP version 4 Last read 00:00:05, last write 00:00:01 Hold time is 30, keepalive interval is 10 seconds Configured hold time is 30, keepalive interval is 10 seconds Connect timer is inactive Idle-restart timer is inactive BGP state is Established, up for 21:10:51 Number of transitions to established: 3 Last state was OpenConfirm Last event was RecvKeepAlive Last rcvd notification:Cease/maximum number of prefixes reached, Last time 21:28:28 Last rcvd socket-error:received unexpected EOF, Last time 21:23:42, First time 21:26:16, Repeats 7 Neighbor Capabilities: Multiprotocol IPv4 Unicast: advertised and received and negotiated Four Octet ASN: advertised and received Route Refresh: advertised and received and negotiated Send End-of-RIB messages: advertised and received and negotiated Additional-paths recv capability: IPv4 Unicast: advertised Graceful Restart advertised: Restart-time is 300 Restarting: no IPv4 Unicast is enabled, Forwarding State is not preserved Restart timer is inactive End of rib timer is inactive Message Statistics: InQ depth is 0 OutQ depth is 0 Sent Rcvd Opens: 18 3 Notifications: 0 1 Updates: 3426 44814 Keepalives: 7629 8472 Route-Refresh: 0 0 Total messages: 11073 53290 Prefix Statistics: Sent Rcvd IPv4 Unicast: 50 11660 IPv6 Unicast: 0 0 IPv4 SR-TE: 0 0 IPv6 SR-TE: 0 0 Inbound updates dropped by reason: AS path loop detection: 0 Enforced First AS: 10746 Originator ID matches local router ID: 0 Nexthop matches local IP address: 0 Unexpected IPv6 nexthop for IPv4 routes: 0 Nexthop invalid for single hop eBGP: 0 Inbound updates with attribute errors: Resulting in removal of all paths in update (treat-as-withdraw): 0 Resulting in AFI/SAFI disable: 0 Resulting in attribute ignore: 0 Inbound paths dropped by reason: IPv4 labeled-unicast NLRIs dropped due to excessive labels: 0 IPv6 labeled-unicast NLRIs dropped due to excessive labels: 0 Outbound paths dropped by reason: IPv4 local address not available: 0 IPv6 local address not available: 0 Inbound route map is MICE_INBOUND Inbound prefix list for IPv4 unicast is PREMIER_IN_FROM_MICEv4 Outbound prefix list for IPv4 unicast is PREMIER_OUT_TO_MICEv4 Local AS is 53347, local router ID 207.32.15.22 TTL is 1 Local TCP address is 206.108.255.67, local port is 179 Remote TCP address is 206.108.255.2, remote port is 23143 Auto-Local-Addr is disabled TCP Socket Information: TCP state is ESTABLISHED Recv-Q: 0/32768 Send-Q: 0/32768 Outgoing Maximum Segment Size (MSS): 1448 Total Number of TCP retransmissions: 0 Options: Timestamps enabled: yes Selective Acknowledgments enabled: yes Window Scale enabled: yes Explicit Congestion Notification (ECN) enabled: no Socket Statistics: Window Scale (wscale): 6,7 Retransmission Timeout (rto): 312.0ms Round-trip Time (rtt/rtvar): 108.6ms/2.2ms Delayed Ack Timeout (ato): 40.0ms Congestion Window (cwnd): 10 TCP Throughput: 1.07 Mbps Recv Round-trip Time (rcv_rtt): 64.0ms Advertised Recv Window (rcv_space): 28960 SiouxCenter-Arista-North(s1)# Frank -----Original Message----- From: MICE Discuss <MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, June 27, 2018 7:34 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Not learning all routes from RR#2 Anthony mentioned MTU. I have had issues in the past with BGP, and larger MTUs. Especially when my router had a larger MTU than the path/destination router. BGP would establish, get a few prefixes, and then time out. I just took a look, and the route servers have a 1500 MTU. Are you set larger? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of AnthonyAnderberg@nuvera.net Sent: Wednesday, June 27, 2018 5:32 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Not learning all routes from RR#2 The two MICE-attached routers I'm responsible for look OK, and about the same as they have looked recently: 206.108.255.1 4 53679 10130010 700710 305989456 0 0 1y11w 56003 206.108.255.2 4 53679 9531122 656345 305989456 0 0 1y7w 56001 -- 206.108.255.1 0 53679 14571189 909087 587718403 0 0 46w5d 55942 206.108.255.2 0 53679 14593796 908792 587718403 0 0 46w5d 55940 The MICE Looking Glass seems about the same: telnet://rviews:@route-server.micemn.net:23 206.108.255.1 4 53679 5001870 255450 11721640 0 0 23w0d 56004 206.108.255.2 4 53679 4998551 255464 11721640 0 0 23w0d 56002 The "no bgp enforce-first-as" config is expected on some platforms: http://www.micemn.net/technical.html Is traffic flowing normally otherwise? Perhaps you're running into an L1/L2 or MTU issue such that this is a symptom of another problem? Cheers, anthony -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Wednesday, June 27, 2018 1:25 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Not learning all routes from RR#2 We (AS53347) moved to a new border router overnight and had some issues with getting our MICE route reflector peerings up -- we'd learn a route or two per second. We had to tweak our Arista's BGP configuration to not "enforce-first-as". Most other direct peers were fine. Eventually RR #1 came up just fine, but RR #2 continues to languish, learning just learning about 10,900 routes. SiouxCenter-Arista-North(s1)#show ip bgp sum BGP summary information for VRF default Router identifier 207.32.15.22, local AS number 18883 Neighbor Status Codes: m - Under maintenance Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc 206.108.255.1 4 53679 44493 4138 0 0 10:56:46 Estab 56188 56188 206.108.255.2 4 53679 39124 7640 0 0 11:38:40 Estab 10929 10929 I'm not sure if we have remote access to the RR's, but whoever does, any hints as to why we're experiencing this behavior? Frank
participants (4)
-
AnthonyAnderberg＠nuvera.net -
Doug McIntyre -
Frank Bulk -
Jeremy Lumby