December 2017 - mice-discuss - lists.micemn.net

Super Bowl Change Freeze

by Richard Laager

As you have probably heard, the Super Bowl is being held in Minneapolis on February 4. We discussed at UG 26 that it would be a good idea to institute a change freeze for a few days to a week before. If you need to upgrade your port capacity or make other changes at MICE, please plan to have those *complete* by January 26th. If it were me, I wouldn't even cut it that close. -- Richard

Dec. 12, 2017

Re: Passive BFD support on Route-Server #1

by Doug McIntyre

On Mon, Dec 04, 2017 at 03:12:28PM -0600, Doug McIntyre wrote: >Just to close this project out, I'll set a schedule to get this done. > >On Monday, 12/11/2017 at 12:30AM, I'll turn on passive BFD on the MICE route >server #1 (206.108.255.1/ 2001:504:27::d1af:0:1) for all connections. This was completed. No complications as far as I can see. I'll let this config run for a week and schedule up the other route server. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 11, 2017

Re: Attribute Length Error today

by Andrew Hoyos

On Dec 9, 2017, at 7:00 PM, Jason Hanke <jayhanke(a)NEUTRALPATH.NET> wrote: > > it's time for secure route servers. Job sent out a list of ixp and quite a few of them are already doing rpki or irr filtering. I second this. IXPmanager makes this easy. > >> On Dec 8, 2017 11:06 PM, "Richard Laager" <rlaager(a)wiktel.com> wrote: >> Zero length is a subset of “doesn’t start with their AS”, so if we filter on that condition, hopefully we will catch that too. >> >> -- >> Richard >> >> > On Dec 8, 2017, at 23:04, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: >> > >> > If the issue was that the AS PATH was zero length, can BIRD filter those out? >> > >> > Frank >> > >> > -----Original Message----- >> > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre >> > Sent: Friday, December 8, 2017 6:15 PM >> > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >> > Subject: Re: [MICE-DISCUSS] Attribute Length Error today >> > >> >> On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: >> >> The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. >> > >> > They do have filters for bogons + default route. >> > >> > I suspect bad AS attribute processing is part of what made it get leaked onwards. >> > The BIRD servers were logging that as well during this period. >> > >> >> On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. >> > >> > Possibly. >> > >> > >> > -- >> > Doug McIntyre <merlyn(a)iphouse.net> >> > ~.~ ipHouse ~.~ >> > Network Engineer/Provisioning/Jack of all Trades >> > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Dec. 10, 2017

Re: Attribute Length Error today

by Jason Hanke

it's time for secure route servers. Job sent out a list of ixp and quite a few of them are already doing rpki or irr filtering. On Dec 8, 2017 11:06 PM, "Richard Laager" <rlaager(a)wiktel.com> wrote: > Zero length is a subset of “doesn’t start with their AS”, so if we filter > on that condition, hopefully we will catch that too. > > -- > Richard > > > On Dec 8, 2017, at 23:04, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: > > > > If the issue was that the AS PATH was zero length, can BIRD filter those > out? > > > > Frank > > > > -----Original Message----- > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Doug McIntyre > > Sent: Friday, December 8, 2017 6:15 PM > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > > > >> On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: > >> The more important question - why didn’t the route servers drop that? > I’d assume there should be inbound filters to drop > bogons+default+$otherbadstuff. > > > > They do have filters for bogons + default route. > > > > I suspect bad AS attribute processing is part of what made it get leaked > onwards. > > The BIRD servers were logging that as well during this period. > > > >> On a larger scale, this sort of thing begs the question - do we need to > have folks in some sort of isolated VLAN with test sessions to the route > servers upon turnup? SIX does this, as well as others, I suspect to prevent > these exact issues from happening. > > > > Possibly. > > > > > > -- > > Doug McIntyre <merlyn(a)iphouse.net> > > ~.~ ipHouse ~.~ > > Network Engineer/Provisioning/Jack of all Trades > > >

Dec. 10, 2017

Re: Attribute Length Error today

by Richard Laager

Zero length is a subset of “doesn’t start with their AS”, so if we filter on that condition, hopefully we will catch that too. -- Richard > On Dec 8, 2017, at 23:04, Frank Bulk <fbulk(a)MYPREMIERONLINE.COM> wrote: > > If the issue was that the AS PATH was zero length, can BIRD filter those out? > > Frank > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre > Sent: Friday, December 8, 2017 6:15 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > >> On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: >> The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. > > They do have filters for bogons + default route. > > I suspect bad AS attribute processing is part of what made it get leaked onwards. > The BIRD servers were logging that as well during this period. > >> On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. > > Possibly. > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades >

Dec. 9, 2017

Re: Attribute Length Error today

by Frank Bulk

If the issue was that the AS PATH was zero length, can BIRD filter those out? Frank -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Doug McIntyre Sent: Friday, December 8, 2017 6:15 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Attribute Length Error today On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: >The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. They do have filters for bogons + default route. I suspect bad AS attribute processing is part of what made it get leaked onwards. The BIRD servers were logging that as well during this period. >On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. Possibly. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 9, 2017

Re: Attribute Length Error today

by Andrew Hoyos

> On Dec 8, 2017, at 6:14 PM, Doug McIntyre <merlyn(a)IPHOUSE.NET> wrote: > > On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: >> The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. > > They do have filters for bogons + default route. Cool. > I suspect bad AS attribute processing is part of what made it get leaked onwards. > The BIRD servers were logging that as well during this period. Gotcha, while extra work, would it be easy enough to add first hop AS path filters on sessions on the route servers? Or do these already exist as well? Without testing, I’m not sure if that even would have prevented this specific scenario, but it seems like one we should prevent in the future, somehow. -- Andrew Hoyos hoyosa(a)gmail.com

Dec. 9, 2017

Re: Attribute Length Error today

by Doug McIntyre

On Fri, Dec 08, 2017 at 06:03:21PM -0600, Andrew Hoyos wrote: >The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. They do have filters for bogons + default route. I suspect bad AS attribute processing is part of what made it get leaked onwards. The BIRD servers were logging that as well during this period. >On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. Possibly. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Dec. 9, 2017

Re: Attribute Length Error today

by Jeremy Lumby

The route servers are dropping it, however I wonder if the error is being generated in the process of dropping it. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Andrew Hoyos Sent: Friday, December 08, 2017 6:03 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Attribute Length Error today The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. And yes, "set-bgp-prepend=0” on the inbound filter on the session is the correct way to handle a similar behavior to ‘no bgp enforce first-as’ in Tik world. This causes the MICE ASN to be removed from the AS Path. -- Andrew Hoyos hoyosa(a)gmail.com > On Dec 8, 2017, at 5:55 PM, Darin Steffl <darin.steffl(a)MNWIFI.COM> wrote: > > It's possible it was sending default earlier and our consultant fixed it when he logged in between 1:27pm and 1:45pm. If someone wants to verify, that'd be great. > > On Fri, Dec 8, 2017 at 5:48 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I do not know much about how Mikrotik works. I agree that the filters do not look like you are sending default, I was just seeing it from Andy's post. > > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Darin Steffl > Sent: Friday, December 08, 2017 5:39 PM > > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > > > > Looking at route filters right now and that I attached, I don't see that I'm advertising a default route unless I'm looking in the wrong spot. I did not setup any portion of the BGP as I know I'm not an expert or comfortable setting it up, thus why I hired a consultant to do it for me. They're a big Mikrotik consultant from Missouri. > > > > I only see us advertising 204.73.77.0/24, 162.255.252.0/22 in our route filters right now that I sent. > > > > > > > > On Fri, Dec 8, 2017 at 5:29 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > In looking at it, do you happen to know why you were sending a default route? Has that been corrected? > > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Darin Steffl > Sent: Friday, December 08, 2017 5:16 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > > > > Hey all, > > > > I sent the following to my Mikrotik consultant who set this up for me and his response is below. Anyone familiar with Mikrotik that knows the easy way to do this? We have MICE filters setup but not sure the correct way to do this without messing anything up further. He had the route filters out of order in our Mikrotik which is why I wasn't seeing MICE traffic. Once he fixed that, we're now receiving 500+ mbps from MICE and its stable on our end. But if we're still sending incorrect route announcements, I want to fix that so I don't cause issues. I found two related forum posts on how to change this in my router but I have a couple MICE filters as shown in the attachment. Thanks > > > > My cell is 507-261-7690 > > > > > > My email to consultant: > > Under the technical page of their website http://micemn.net/technical.html I see the following note "Note that your BGP sessions to the route servers likely need no bgp enforce-first-as or similar. The route servers do not add the MICE AS to the AS path." A quick search of Mikrotik forums shows that the equivalent to this setting is to "set-bgp-prepend=0" for the MICE peer. I'm not sure of this so want clarification to see if we need to make this change in order for traffic to start going through MICE. > > https://forum.mikrotik.com/viewtopic.php?t=55798 > > https://forum.mikrotik.com/viewtopic.php?t=43498 > > > > > > Consultant response: > > Your MICE connection should have had traffic assuming that you were advertising your prefixes to them. Their BGP session should not add their AS, so that should not matter. You should not need to change that, I would have to look to see why you are not seeing traffic. > > > > > > On Fri, Dec 8, 2017 at 4:55 PM, Andy Koch <akoch(a)wiscnet.net> wrote: > > On 08 December 2017 at 16:15:59, Justin Krejci wrote: > > Yup > > > > > > Dec 8 13:41:40.262 CST: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.1 3/11 (invalid or corrupt AS path) 3 bytes 400200 > > Dec 8 13:41:40.262 CST: BGP: 206.108.255.1 Bad attributes FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0200 0000 0E40 0101 0040 0200 4003 04CE 6CFF 5918 CC49 4D16 A2FF E8 > > > >From this UPDATE dump, this parser helps quite a bit to decode what is > going on: http://bgpaste.convergence.cx/ > > Looks like the AS_PATH was 0 bytes long, which is invalid and Frank's > router did the only thing it knows to do on BGP errors - drop the session. > > Further from the decode: > The NEXT_HOP was 206.108.255.89 > And the Routes were 204.73.77.0/24, 162.255.252.0/22 and 0.0.0.0/0 > > Looks like Minnesota WiFi should check on their route announcements. > > HTH, > Andy > > > > > > > -- > > Darin Steffl > > Minnesota WiFi > > www.mnwifi.com > > 507-634-WiFi > > Like us on Facebook > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > > > -- > > Darin Steffl > > Minnesota WiFi > > www.mnwifi.com > > 507-634-WiFi > > Like us on Facebook > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > Like us on Facebook > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Dec. 9, 2017

Re: Attribute Length Error today

by Andrew Hoyos

The more important question - why didn’t the route servers drop that? I’d assume there should be inbound filters to drop bogons+default+$otherbadstuff. On a larger scale, this sort of thing begs the question - do we need to have folks in some sort of isolated VLAN with test sessions to the route servers upon turnup? SIX does this, as well as others, I suspect to prevent these exact issues from happening. And yes, "set-bgp-prepend=0” on the inbound filter on the session is the correct way to handle a similar behavior to ‘no bgp enforce first-as’ in Tik world. This causes the MICE ASN to be removed from the AS Path. -- Andrew Hoyos hoyosa(a)gmail.com > On Dec 8, 2017, at 5:55 PM, Darin Steffl <darin.steffl(a)MNWIFI.COM> wrote: > > It's possible it was sending default earlier and our consultant fixed it when he logged in between 1:27pm and 1:45pm. If someone wants to verify, that'd be great. > > On Fri, Dec 8, 2017 at 5:48 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > I do not know much about how Mikrotik works. I agree that the filters do not look like you are sending default, I was just seeing it from Andy's post. > > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Darin Steffl > Sent: Friday, December 08, 2017 5:39 PM > > > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > > > > Looking at route filters right now and that I attached, I don't see that I'm advertising a default route unless I'm looking in the wrong spot. I did not setup any portion of the BGP as I know I'm not an expert or comfortable setting it up, thus why I hired a consultant to do it for me. They're a big Mikrotik consultant from Missouri. > > > > I only see us advertising 204.73.77.0/24, 162.255.252.0/22 in our route filters right now that I sent. > > > > > > > > On Fri, Dec 8, 2017 at 5:29 PM, Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > > In looking at it, do you happen to know why you were sending a default route? Has that been corrected? > > > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Darin Steffl > Sent: Friday, December 08, 2017 5:16 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] Attribute Length Error today > > > > Hey all, > > > > I sent the following to my Mikrotik consultant who set this up for me and his response is below. Anyone familiar with Mikrotik that knows the easy way to do this? We have MICE filters setup but not sure the correct way to do this without messing anything up further. He had the route filters out of order in our Mikrotik which is why I wasn't seeing MICE traffic. Once he fixed that, we're now receiving 500+ mbps from MICE and its stable on our end. But if we're still sending incorrect route announcements, I want to fix that so I don't cause issues. I found two related forum posts on how to change this in my router but I have a couple MICE filters as shown in the attachment. Thanks > > > > My cell is 507-261-7690 > > > > > > My email to consultant: > > Under the technical page of their website http://micemn.net/technical.html I see the following note "Note that your BGP sessions to the route servers likely need no bgp enforce-first-as or similar. The route servers do not add the MICE AS to the AS path." A quick search of Mikrotik forums shows that the equivalent to this setting is to "set-bgp-prepend=0" for the MICE peer. I'm not sure of this so want clarification to see if we need to make this change in order for traffic to start going through MICE. > > https://forum.mikrotik.com/viewtopic.php?t=55798 > > https://forum.mikrotik.com/viewtopic.php?t=43498 > > > > > > Consultant response: > > Your MICE connection should have had traffic assuming that you were advertising your prefixes to them. Their BGP session should not add their AS, so that should not matter. You should not need to change that, I would have to look to see why you are not seeing traffic. > > > > > > On Fri, Dec 8, 2017 at 4:55 PM, Andy Koch <akoch(a)wiscnet.net> wrote: > > On 08 December 2017 at 16:15:59, Justin Krejci wrote: > > Yup > > > > > > Dec 8 13:41:40.262 CST: %BGP-3-NOTIFICATION: sent to neighbor 206.108.255.1 3/11 (invalid or corrupt AS path) 3 bytes 400200 > > Dec 8 13:41:40.262 CST: BGP: 206.108.255.1 Bad attributes FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0200 0000 0E40 0101 0040 0200 4003 04CE 6CFF 5918 CC49 4D16 A2FF E8 > > > >From this UPDATE dump, this parser helps quite a bit to decode what is > going on: http://bgpaste.convergence.cx/ > > Looks like the AS_PATH was 0 bytes long, which is invalid and Frank's > router did the only thing it knows to do on BGP errors - drop the session. > > Further from the decode: > The NEXT_HOP was 206.108.255.89 > And the Routes were 204.73.77.0/24, 162.255.252.0/22 and 0.0.0.0/0 > > Looks like Minnesota WiFi should check on their route announcements. > > HTH, > Andy > > > > > > > -- > > Darin Steffl > > Minnesota WiFi > > www.mnwifi.com > > 507-634-WiFi > > Like us on Facebook > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > > > -- > > Darin Steffl > > Minnesota WiFi > > www.mnwifi.com > > 507-634-WiFi > > Like us on Facebook > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > Like us on Facebook > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Dec. 9, 2017