May 2020 - mice-discuss - lists.micemn.net

Re: Record Traffic

by Dean Bahls

Curious if the new 100G link coming from Po5 to Amazon is backwards. If not, why is so much traffic flow TO Amazon instead of the other way around? Thanks, Dean -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, May 11, 2020 8:27 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Record Traffic Looks like MICE has hit a new high traffic level on MICE 920Gig (in+out) and climbing. Amazon moved one of their links over to 100G, and this is contributing. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

May 12, 2020

Re: Record Traffic

by Jay Hanke

Flirting with 1/2 Tb of "real" traffic! On Mon, May 11, 2020 at 8:28 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > Looks like MICE has hit a new high traffic level on MICE 920Gig (in+out) > and climbing. Amazon moved one of their links over to 100G, and this is > contributing. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 12, 2020

Record Traffic

by Jeremy Lumby

Looks like MICE has hit a new high traffic level on MICE 920Gig (in+out) and climbing. Amazon moved one of their links over to 100G, and this is contributing. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

May 12, 2020

Global Net (AS395466) - Upcoming CARRIER Maintenance Windows 5/10 + 5/19

by Global Net NOC

MICEMN Members - Our carrier, Zayo Networks, has two upcoming maintenance windows which will affect our service to the MICE: Maintenance 5/10/20 Start: 5/10/20 0500 GMT End: 5/10/20 1000 GMT Down for up to 2 hours. Maintenance 5/19/20 Start: 5/19/20 0500 GMT End: 5/19/20 1000 GMT Down for the duration of the window. I apologize for the short notice on the first window. The carrier continues to schedule, cancel then re-schedule maintenance windows quicker than I can keep up. Thanks, Global Net NOC noc(a)theglobal.net Global Net Peering peering(a)theglobal.net

May 8, 2020

Re: Please Welcome Genesis Wireless to MICE!

by Richard Laager

On 5/7/20 1:38 PM, Doug McIntyre wrote: > On Wed, May 06, 2020 at 09:16:49AM -0500, Jay Hanke wrote: >> Update, Genesis requested to use a previously allocated address to >> them which had not been used. >> >> Name: Genesis Technology Communication, LLC >> Port: MN VoIP Remote Switch (Port Assignment Pending) >> ASN: 30032 >> IPv4: 206.108.255.103/24 >> IPv6: 2001:504:27:0:0:7550::1/64 >> Peering Contact: admin(a)genesiswireless.com > > This was already setup as 'RevNet'. I guess we are renaming RevNet to > Genesis for this one. Yes, that sounds correct. -- Richard

May 7, 2020

Re: Please Welcome Genesis Wireless to MICE!

by Doug McIntyre

On Wed, May 06, 2020 at 09:16:49AM -0500, Jay Hanke wrote: >Update, Genesis requested to use a previously allocated address to >them which had not been used. > >Name: Genesis Technology Communication, LLC >Port: MN VoIP Remote Switch (Port Assignment Pending) >ASN: 30032 >IPv4: 206.108.255.103/24 >IPv6: 2001:504:27:0:0:7550::1/64 >Peering Contact: admin(a)genesiswireless.com This was already setup as 'RevNet'. I guess we are renaming RevNet to Genesis for this one. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 7, 2020

Re: route server down?

by Richard Laager

On 5/6/20 12:09 PM, David Farmer wrote: > BIRD put a malformed BGP message on the wire >From what I'm seeing here, I would agree. Ideally, someone should report this to the BIRD developers. That said, for such a bug report to be useful, we may need more information. If it was just the RFC 1918 next-hop that triggered this, then this should be trivially reproducible. But if it was something else, we may not have enough information to get to the bottom of this. -- Richard

May 7, 2020

Re: route server down?

by Jay Hanke

> IOS XR said "hey look that's bad, and I'm going to ignore it", and Brocade said, "hey that's bad, and you're bad so I'm not going to talk to you", I think the Junipers reacted much as IOS XR did, but that doesn't mean there wasn't a bad message. I can confirm the Juniper reaction, it got marked as "invalid" with unreachable next hop. I don't think it threw an actual error, just put it in the naughty pile and ignored it.

May 6, 2020

Re: route server down?

by David Farmer

My IOS XR router detected a malformed BGP message, logs are in the thread below, but it treated it as a withdrawal per RFC7606. The Brocades saw the error and closed the BGP session, which is what many older BGP implementations do. I'm not sure the reason for the malformed BGP message, it could have something to do the next-hop, or it could be for some completely unrelated reason, but it as far as I can tell, BIRD put a malformed BGP message on the wire. IOS XR said "hey look that's bad, and I'm going to ignore it", and Brocade said, "hey that's bad, and you're bad so I'm not going to talk to you", I think the Junipers reacted much as IOS XR did, but that doesn't mean there wasn't a bad message. Thanks On Wed, May 6, 2020 at 11:49 AM Doug McIntyre <merlyn(a)iphouse.net> wrote: > I think we've come up with some methods for potentially creating > filters for this situation to roll out in the future. > > I don't think its' BIRD place to impose restrictions on what a BGP > peer is advertising, even if it is bogus. > > Clearly its only a bug with Brocade, because other vendors (especially > Juniper) didn't > care that it was announced this way, I just had a bogus route inserted > from them, but > it didn't affect any of my BGP sessions with MICE. > > > > > > On Wed, May 06, 2020 at 04:45:33PM +0000, Frank Bulk wrote: > >Do we need to submit a bug to the developers of BIRD? > > > >Frank > > > >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David > Farmer > >Sent: Thursday, April 30, 2020 8:37 AM > >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > >Subject: Re: [MICE-DISCUSS] route server down? > > > >I think it was more than just an invalid next-hop. If it was simply an > invalid next-hop that shouldn't have created a malformed BGP update. Unless > the invalid next-hop caused BIRD to send out a malformed BGP update. > > > >On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io<mailto: > jayhanke(a)southfront.io>> wrote: > >I emailed xcel about the invalid next-hop address. > > > >We should filter invalid next hops on the route servers. > > > >There also appears to be an issue with how some routers handle the > >invalid next hop. > > > >Are all the peers with the issue of losing the session to RS2 running > Brocade? > > > >On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu<mailto: > farmer(a)umn.edu>> wrote: > >> > >> someone with Access should see what route server 2 sees for that > prefix, and maybe kick it over after look at it. > >> > >> On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io > <mailto:jayhanke@southfront.io>> wrote: > >>> > >>> We're seeing the same with a good next-hop from RS1. > >>> > >>> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net<mailto: > wopat(a)wiscnet.net>> wrote: > >>> > > >>> > On 4/30/20 7:49 AM, David Farmer wrote: > >>> > > >>> > > We're running IOS XR, I found these droppings in our logs; > >>> > > > >>> > > >>> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: > >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received > from > >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, > error > >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF… > > > >>> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: > >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received > from > >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, > error > >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF… > > > >>> > > > >>> > > Maybe try resting you BGP sessions. > >>> > > > >>> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its > hidden > >>> > on our net. > >>> > > >>> > Is 10.223.129.2 something internal to route server #2? > >>> > > >>> > > show route 198.179.154.0 hidden detail > >>> > > >>> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 > holddown, > >>> > 1604 hidden) > >>> > 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> > (3 entries, 1 announced) > >>> > BGP > >>> > Next hop type: Router, Next hop index: 0 > >>> > Address: 0x113614cc > >>> > Next-hop reference count: 1 > >>> > Source: 206.108.255.2 > >>> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected > >>> > Session Id: 0x0 > >>> > State: <Hidden Ext> > >>> > Inactive reason: Unusable path > >>> > Local AS: 65400 Peer AS: 53679 > >>> > Age: 10:02:05 > >>> > Validation State: unverified > >>> > Task: BGP_53679.206.108.255.2 > >>> > AS path: I > >>> > Communities: target:21693:1000 > >>> > Router ID: 206.108.255.2 > >>> > Hidden reason: protocol nexthop is not on the > interface > >>> > > >>> > > >>> > -- > >>> > Chris Wopat > >>> > Network Engineer, WiscNet > >>> > wopat(a)wiscnet.net<mailto:wopat@wiscnet.net> 608-210-3965 > >>> > >>> > >>> > >>> -- > >>> Jay Hanke, President > >>> South Front Networks > >>> jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> > >>> Phone 612-204-0000 > >> > >> > >> > >> -- > >> =============================================== > >> David Farmer Email:farmer@umn.edu<mailto: > Email%3Afarmer(a)umn.edu> > >> Networking & Telecommunication Services > >> Office of Information Technology > >> University of Minnesota > >> 2218 University Ave SE Phone: 612-626-0815 > >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 > >> =============================================== > >> > >> ________________________________ > >> > >> To unsubscribe from the MICE-DISCUSS list, click the following link: > >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1< > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2… > > > > > > > > > >-- > >Jay Hanke, President > >South Front Networks > >jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> > >Phone 612-204-0000 > > > > > >-- > >=============================================== > >David Farmer Email:farmer@umn.edu<mailto: > Email%3Afarmer(a)umn.edu> > >Networking & Telecommunication Services > >Office of Information Technology > >University of Minnesota > >2218 University Ave SE Phone: 612-626-0815 > >Minneapolis, MN 55414-3029 Cell: 612-812-9952 > >=============================================== > > > >________________________________ > > > >To unsubscribe from the MICE-DISCUSS list, click the following link: > >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1< > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2… > > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

May 6, 2020

Re: route server down?

by Doug McIntyre

I think we've come up with some methods for potentially creating filters for this situation to roll out in the future. I don't think its' BIRD place to impose restrictions on what a BGP peer is advertising, even if it is bogus. Clearly its only a bug with Brocade, because other vendors (especially Juniper) didn't care that it was announced this way, I just had a bogus route inserted from them, but it didn't affect any of my BGP sessions with MICE. On Wed, May 06, 2020 at 04:45:33PM +0000, Frank Bulk wrote: >Do we need to submit a bug to the developers of BIRD? > >Frank > >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer >Sent: Thursday, April 30, 2020 8:37 AM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] route server down? > >I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. > >On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: >I emailed xcel about the invalid next-hop address. > >We should filter invalid next hops on the route servers. > >There also appears to be an issue with how some routers handle the >invalid next hop. > >Are all the peers with the issue of losing the session to RS2 running Brocade? > >On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu<mailto:farmer@umn.edu>> wrote: >> >> someone with Access should see what route server 2 sees for that prefix, and maybe kick it over after look at it. >> >> On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: >>> >>> We're seeing the same with a good next-hop from RS1. >>> >>> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net<mailto:wopat@wiscnet.net>> wrote: >>> > >>> > On 4/30/20 7:49 AM, David Farmer wrote: >>> > >>> > > We're running IOS XR, I found these droppings in our logs; >>> > > >>> > >>> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >>> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >>> > > >>> > > Maybe try resting you BGP sessions. >>> > > >>> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its hidden >>> > on our net. >>> > >>> > Is 10.223.129.2 something internal to route server #2? >>> > >>> > > show route 198.179.154.0 hidden detail >>> > >>> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 holddown, >>> > 1604 hidden) >>> > 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> (3 entries, 1 announced) >>> > BGP >>> > Next hop type: Router, Next hop index: 0 >>> > Address: 0x113614cc >>> > Next-hop reference count: 1 >>> > Source: 206.108.255.2 >>> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected >>> > Session Id: 0x0 >>> > State: <Hidden Ext> >>> > Inactive reason: Unusable path >>> > Local AS: 65400 Peer AS: 53679 >>> > Age: 10:02:05 >>> > Validation State: unverified >>> > Task: BGP_53679.206.108.255.2 >>> > AS path: I >>> > Communities: target:21693:1000 >>> > Router ID: 206.108.255.2 >>> > Hidden reason: protocol nexthop is not on the interface >>> > >>> > >>> > -- >>> > Chris Wopat >>> > Network Engineer, WiscNet >>> > wopat(a)wiscnet.net<mailto:wopat@wiscnet.net> 608-210-3965 >>> >>> >>> >>> -- >>> Jay Hanke, President >>> South Front Networks >>> jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> >>> Phone 612-204-0000 >> >> >> >> -- >> =============================================== >> David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> >> Networking & Telecommunication Services >> Office of Information Technology >> University of Minnesota >> 2218 University Ave SE Phone: 612-626-0815 >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 >> =============================================== >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > >-- >Jay Hanke, President >South Front Networks >jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> >Phone 612-204-0000 > > >-- >=============================================== >David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> >Networking & Telecommunication Services >Office of Information Technology >University of Minnesota >2218 University Ave SE Phone: 612-626-0815 >Minneapolis, MN 55414-3029 Cell: 612-812-9952 >=============================================== > >________________________________ > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 6, 2020