The exchange *seems* fine, but I can't reliably ping the route servers right now. My BGP sessions to them are up and stable, but ICMP isn't operating "normally" - has anything changed? This started at 15:24 CDT today: [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.1 PING 206.108.255.1 (206.108.255.1) 56(84) bytes of data. 64 bytes from 206.108.255.1: icmp_seq=7 ttl=59 time=7.06 ms --- 206.108.255.1 ping statistics --- 10 packets transmitted, 1 received, 90% packet loss, time 9000ms rtt min/avg/max/mdev = 7.062/7.062/7.062/0.000 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.2 PING 206.108.255.2 (206.108.255.2) 56(84) bytes of data. 64 bytes from 206.108.255.2: icmp_seq=2 ttl=59 time=7.13 ms 64 bytes from 206.108.255.2: icmp_seq=6 ttl=59 time=6.98 ms 64 bytes from 206.108.255.2: icmp_seq=7 ttl=59 time=6.94 ms 64 bytes from 206.108.255.2: icmp_seq=10 ttl=59 time=7.05 ms --- 206.108.255.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9001ms rtt min/avg/max/mdev = 6.942/7.028/7.135/0.093 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.52 PING 206.108.255.52 (206.108.255.52) 56(84) bytes of data. 64 bytes from 206.108.255.52: icmp_seq=1 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=2 ttl=61 time=14.5 ms 64 bytes from 206.108.255.52: icmp_seq=3 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=4 ttl=61 time=13.6 ms 64 bytes from 206.108.255.52: icmp_seq=5 ttl=61 time=13.0 ms 64 bytes from 206.108.255.52: icmp_seq=6 ttl=61 time=12.7 ms 64 bytes from 206.108.255.52: icmp_seq=7 ttl=61 time=12.2 ms 64 bytes from 206.108.255.52: icmp_seq=8 ttl=61 time=11.7 ms 64 bytes from 206.108.255.52: icmp_seq=9 ttl=61 time=11.2 ms 64 bytes from 206.108.255.52: icmp_seq=10 ttl=61 time=11.9 ms --- 206.108.255.52 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9002ms rtt min/avg/max/mdev = 11.289/12.913/14.502/1.052 ms [stahr@ctg-bastion ~]$ Any new policing policy applied today? I seem to be able to ping normally if I use my 206.108.255.x address, but not reliably from any other address: r-pop-min-1#ping 206.108.255.1 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.1 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/2 ms r-pop-min-1# -James
On Mon, Oct 05, 2015 at 04:10:54PM -0500, James Stahr wrote:
The exchange *seems* fine, but I can't reliably ping the route servers right now.
Is this /24 announced via BGP into the exchange or just over some of the connected members? -- Mike Horwath, reachable via drechsau@Geeks.ORG
Observing same issues here pinging same route severs from any source netblock in AS16842. -Tk
On Oct 5, 2015, at 4:10 PM, James Stahr <stahr@MAILBAG.COM> wrote:
The exchange *seems* fine, but I can't reliably ping the route servers right now. My BGP sessions to them are up and stable, but ICMP isn't operating "normally" - has anything changed? This started at 15:24 CDT today:
[stahr@ctg-bastion ~]$ ping -c 10 206.108.255.1 PING 206.108.255.1 (206.108.255.1) 56(84) bytes of data. 64 bytes from 206.108.255.1: icmp_seq=7 ttl=59 time=7.06 ms
--- 206.108.255.1 ping statistics --- 10 packets transmitted, 1 received, 90% packet loss, time 9000ms rtt min/avg/max/mdev = 7.062/7.062/7.062/0.000 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.2 PING 206.108.255.2 (206.108.255.2) 56(84) bytes of data. 64 bytes from 206.108.255.2: icmp_seq=2 ttl=59 time=7.13 ms 64 bytes from 206.108.255.2: icmp_seq=6 ttl=59 time=6.98 ms 64 bytes from 206.108.255.2: icmp_seq=7 ttl=59 time=6.94 ms 64 bytes from 206.108.255.2: icmp_seq=10 ttl=59 time=7.05 ms
--- 206.108.255.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9001ms rtt min/avg/max/mdev = 6.942/7.028/7.135/0.093 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.52 PING 206.108.255.52 (206.108.255.52) 56(84) bytes of data. 64 bytes from 206.108.255.52: icmp_seq=1 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=2 ttl=61 time=14.5 ms 64 bytes from 206.108.255.52: icmp_seq=3 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=4 ttl=61 time=13.6 ms 64 bytes from 206.108.255.52: icmp_seq=5 ttl=61 time=13.0 ms 64 bytes from 206.108.255.52: icmp_seq=6 ttl=61 time=12.7 ms 64 bytes from 206.108.255.52: icmp_seq=7 ttl=61 time=12.2 ms 64 bytes from 206.108.255.52: icmp_seq=8 ttl=61 time=11.7 ms 64 bytes from 206.108.255.52: icmp_seq=9 ttl=61 time=11.2 ms 64 bytes from 206.108.255.52: icmp_seq=10 ttl=61 time=11.9 ms
--- 206.108.255.52 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9002ms rtt min/avg/max/mdev = 11.289/12.913/14.502/1.052 ms [stahr@ctg-bastion ~]$
Any new policing policy applied today? I seem to be able to ping normally if I use my 206.108.255.x address, but not reliably from any other address:
r-pop-min-1#ping 206.108.255.1 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.1 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/2 ms r-pop-min-1#
-James
Just noticed that the utilization graphs were missing data from this afternoon, however not the graphs from the CNS switch, which I assume is on another netblock. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anton Kapela Sent: Monday, October 05, 2015 4:33 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? Observing same issues here pinging same route severs from any source netblock in AS16842. -Tk
On Oct 5, 2015, at 4:10 PM, James Stahr <stahr@MAILBAG.COM> wrote:
The exchange *seems* fine, but I can't reliably ping the route servers right now. My BGP sessions to them are up and stable, but ICMP isn't operating "normally" - has anything changed? This started at 15:24 CDT today:
[stahr@ctg-bastion ~]$ ping -c 10 206.108.255.1 PING 206.108.255.1 (206.108.255.1) 56(84) bytes of data. 64 bytes from 206.108.255.1: icmp_seq=7 ttl=59 time=7.06 ms
--- 206.108.255.1 ping statistics --- 10 packets transmitted, 1 received, 90% packet loss, time 9000ms rtt min/avg/max/mdev = 7.062/7.062/7.062/0.000 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.2 PING 206.108.255.2 (206.108.255.2) 56(84) bytes of data. 64 bytes from 206.108.255.2: icmp_seq=2 ttl=59 time=7.13 ms 64 bytes from 206.108.255.2: icmp_seq=6 ttl=59 time=6.98 ms 64 bytes from 206.108.255.2: icmp_seq=7 ttl=59 time=6.94 ms 64 bytes from 206.108.255.2: icmp_seq=10 ttl=59 time=7.05 ms
--- 206.108.255.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9001ms rtt min/avg/max/mdev = 6.942/7.028/7.135/0.093 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.52 PING 206.108.255.52 (206.108.255.52) 56(84) bytes of data. 64 bytes from 206.108.255.52: icmp_seq=1 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=2 ttl=61 time=14.5 ms 64 bytes from 206.108.255.52: icmp_seq=3 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=4 ttl=61 time=13.6 ms 64 bytes from 206.108.255.52: icmp_seq=5 ttl=61 time=13.0 ms 64 bytes from 206.108.255.52: icmp_seq=6 ttl=61 time=12.7 ms 64 bytes from 206.108.255.52: icmp_seq=7 ttl=61 time=12.2 ms 64 bytes from 206.108.255.52: icmp_seq=8 ttl=61 time=11.7 ms 64 bytes from 206.108.255.52: icmp_seq=9 ttl=61 time=11.2 ms 64 bytes from 206.108.255.52: icmp_seq=10 ttl=61 time=11.9 ms
--- 206.108.255.52 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9002ms rtt min/avg/max/mdev = 11.289/12.913/14.502/1.052 ms [stahr@ctg-bastion ~]$
Any new policing policy applied today? I seem to be able to ping normally if I use my 206.108.255.x address, but not reliably from any other address:
r-pop-min-1#ping 206.108.255.1 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.1 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/2 ms r-pop-min-1#
-James
My internal monitoring (via SmokePing) shows ping packet loss between 15:20 and 16:10 to the MICE route servers. As others have mentioned I don't show any BGP or traffic volume issues, and everything looks OK now. I believe the only change today was adding ports to the Netflix LAG this morning at ~11:15, the last time the MICE route server configuration changed was August 13th. Will continue to look for relevant data points... Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Monday, October 05, 2015 4:56 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? Just noticed that the utilization graphs were missing data from this afternoon, however not the graphs from the CNS switch, which I assume is on another netblock. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anton Kapela Sent: Monday, October 05, 2015 4:33 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? Observing same issues here pinging same route severs from any source netblock in AS16842. -Tk
On Oct 5, 2015, at 4:10 PM, James Stahr <stahr@MAILBAG.COM> wrote:
The exchange *seems* fine, but I can't reliably ping the route servers right now. My BGP sessions to them are up and stable, but ICMP isn't operating "normally" - has anything changed? This started at 15:24 CDT today:
[stahr@ctg-bastion ~]$ ping -c 10 206.108.255.1 PING 206.108.255.1 (206.108.255.1) 56(84) bytes of data. 64 bytes from 206.108.255.1: icmp_seq=7 ttl=59 time=7.06 ms
--- 206.108.255.1 ping statistics --- 10 packets transmitted, 1 received, 90% packet loss, time 9000ms rtt min/avg/max/mdev = 7.062/7.062/7.062/0.000 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.2 PING 206.108.255.2 (206.108.255.2) 56(84) bytes of data. 64 bytes from 206.108.255.2: icmp_seq=2 ttl=59 time=7.13 ms 64 bytes from 206.108.255.2: icmp_seq=6 ttl=59 time=6.98 ms 64 bytes from 206.108.255.2: icmp_seq=7 ttl=59 time=6.94 ms 64 bytes from 206.108.255.2: icmp_seq=10 ttl=59 time=7.05 ms
--- 206.108.255.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9001ms rtt min/avg/max/mdev = 6.942/7.028/7.135/0.093 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.52 PING 206.108.255.52 (206.108.255.52) 56(84) bytes of data. 64 bytes from 206.108.255.52: icmp_seq=1 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=2 ttl=61 time=14.5 ms 64 bytes from 206.108.255.52: icmp_seq=3 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=4 ttl=61 time=13.6 ms 64 bytes from 206.108.255.52: icmp_seq=5 ttl=61 time=13.0 ms 64 bytes from 206.108.255.52: icmp_seq=6 ttl=61 time=12.7 ms 64 bytes from 206.108.255.52: icmp_seq=7 ttl=61 time=12.2 ms 64 bytes from 206.108.255.52: icmp_seq=8 ttl=61 time=11.7 ms 64 bytes from 206.108.255.52: icmp_seq=9 ttl=61 time=11.2 ms 64 bytes from 206.108.255.52: icmp_seq=10 ttl=61 time=11.9 ms
--- 206.108.255.52 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9002ms rtt min/avg/max/mdev = 11.289/12.913/14.502/1.052 ms [stahr@ctg-bastion ~]$
Any new policing policy applied today? I seem to be able to ping normally if I use my 206.108.255.x address, but not reliably from any other address:
r-pop-min-1#ping 206.108.255.1 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.1 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/2 ms r-pop-min-1#
-James
The route servers operating system, and switches do not use the BGP routes from BIRD, therefore the return traffic goes via static route through IPHouse since they are donating bandwidth to manage the servers/switches externally since the MICE /24 does not exist on the internet. If they were making changes this afternoon, that could explain it. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anthony Anderberg Sent: Monday, October 05, 2015 5:38 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? My internal monitoring (via SmokePing) shows ping packet loss between 15:20 and 16:10 to the MICE route servers. As others have mentioned I don't show any BGP or traffic volume issues, and everything looks OK now. I believe the only change today was adding ports to the Netflix LAG this morning at ~11:15, the last time the MICE route server configuration changed was August 13th. Will continue to look for relevant data points... Anthony Anderberg Sr. Systems Analyst 320-234-5239 anthonyanderberg@nu-telecom.net www.nutelecom.net -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeremy Lumby Sent: Monday, October 05, 2015 4:56 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? Just noticed that the utilization graphs were missing data from this afternoon, however not the graphs from the CNS switch, which I assume is on another netblock. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Anton Kapela Sent: Monday, October 05, 2015 4:33 PM To: MICE-DISCUSS@LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] MICE Route Server issues? Observing same issues here pinging same route severs from any source netblock in AS16842. -Tk
On Oct 5, 2015, at 4:10 PM, James Stahr <stahr@MAILBAG.COM> wrote:
The exchange *seems* fine, but I can't reliably ping the route servers right now. My BGP sessions to them are up and stable, but ICMP isn't operating "normally" - has anything changed? This started at 15:24 CDT today:
[stahr@ctg-bastion ~]$ ping -c 10 206.108.255.1 PING 206.108.255.1 (206.108.255.1) 56(84) bytes of data. 64 bytes from 206.108.255.1: icmp_seq=7 ttl=59 time=7.06 ms
--- 206.108.255.1 ping statistics --- 10 packets transmitted, 1 received, 90% packet loss, time 9000ms rtt min/avg/max/mdev = 7.062/7.062/7.062/0.000 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.2 PING 206.108.255.2 (206.108.255.2) 56(84) bytes of data. 64 bytes from 206.108.255.2: icmp_seq=2 ttl=59 time=7.13 ms 64 bytes from 206.108.255.2: icmp_seq=6 ttl=59 time=6.98 ms 64 bytes from 206.108.255.2: icmp_seq=7 ttl=59 time=6.94 ms 64 bytes from 206.108.255.2: icmp_seq=10 ttl=59 time=7.05 ms
--- 206.108.255.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9001ms rtt min/avg/max/mdev = 6.942/7.028/7.135/0.093 ms [stahr@ctg-bastion ~]$ ping -c 10 206.108.255.52 PING 206.108.255.52 (206.108.255.52) 56(84) bytes of data. 64 bytes from 206.108.255.52: icmp_seq=1 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=2 ttl=61 time=14.5 ms 64 bytes from 206.108.255.52: icmp_seq=3 ttl=61 time=14.0 ms 64 bytes from 206.108.255.52: icmp_seq=4 ttl=61 time=13.6 ms 64 bytes from 206.108.255.52: icmp_seq=5 ttl=61 time=13.0 ms 64 bytes from 206.108.255.52: icmp_seq=6 ttl=61 time=12.7 ms 64 bytes from 206.108.255.52: icmp_seq=7 ttl=61 time=12.2 ms 64 bytes from 206.108.255.52: icmp_seq=8 ttl=61 time=11.7 ms 64 bytes from 206.108.255.52: icmp_seq=9 ttl=61 time=11.2 ms 64 bytes from 206.108.255.52: icmp_seq=10 ttl=61 time=11.9 ms
--- 206.108.255.52 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9002ms rtt min/avg/max/mdev = 11.289/12.913/14.502/1.052 ms [stahr@ctg-bastion ~]$
Any new policing policy applied today? I seem to be able to ping normally if I use my 206.108.255.x address, but not reliably from any other address:
r-pop-min-1#ping 206.108.255.1 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source gig0/0/5 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 206.108.255.26 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.1 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.1, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/1 ms r-pop-min-1#ping 206.108.255.2 repeat 100 source loop1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 206.108.255.2, timeout is 2 seconds: Packet sent with a source address of 205.173.182.62 ...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 97 percent (97/100), round-trip min/avg/max = 1/1/2 ms r-pop-min-1#
-James
On Mon, Oct 05, 2015 at 06:51:00PM -0500, Jeremy Lumby wrote:
The route servers operating system, and switches do not use the BGP routes from BIRD, therefore the return traffic goes via static route through IPHouse since they are donating bandwidth to manage the servers/switches externally since the MICE /24 does not exist on the internet. If they were making changes this afternoon, that could explain it.
No, that's not the case. I'll let Doug explain. -- Mike Horwath, reachable via drechsau@Geeks.ORG
On Mon, Oct 05, 2015 at 06:51:00PM -0500, Jeremy Lumby wrote:
The route servers operating system, and switches do not use the BGP routes from BIRD, therefore the return traffic goes via static route through IPHouse since they are donating bandwidth to manage the servers/switches externally since the MICE /24 does not exist on the internet. If they were making changes this afternoon, that could explain it.
ipHouse did have some routing issues this afternoon during that time frame. We found a lurking issue that had been sitting dormant for 2.5 years that something flapped and it reared its ugly head. As Jeremy said, the route reflectors run BIRD, and BIRD by design doesn't install the routes into the routetable of the OS, although I believe that is an option now-a-days. The RR's are dual interfaced, one interface lives in ipHouse IP address space. The other interface lives on the IXC netblock. All the BGP talkers live on the IXC netblock too, so anything talking BGP to them are just fine. But, the RRs only can talk directly to 206.108.255.0/24, if you are pinging it from beyond your interface to the IXC, it will default route over ipHouse IP space out again, as it hasn't learned the routes it talks, it has to route external back out. Overall, I think the BIRD model works well, but depends on us not having any issues.. -- Doug McIntyre <merlyn@iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades
participants (6)
-
Anthony Anderberg -
Anton Kapela -
Doug McIntyre -
James Stahr -
Jeremy Lumby -
Mike Horwath