- mice-discuss - lists.micemn.net

Re: Please Welcome Genesis Wireless to MICE!

by Doug McIntyre

On Wed, May 06, 2020 at 09:16:49AM -0500, Jay Hanke wrote: >Update, Genesis requested to use a previously allocated address to >them which had not been used. > >Name: Genesis Technology Communication, LLC >Port: MN VoIP Remote Switch (Port Assignment Pending) >ASN: 30032 >IPv4: 206.108.255.103/24 >IPv6: 2001:504:27:0:0:7550::1/64 >Peering Contact: admin(a)genesiswireless.com This was already setup as 'RevNet'. I guess we are renaming RevNet to Genesis for this one. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 7, 2020

Re: route server down?

by Richard Laager

On 5/6/20 12:09 PM, David Farmer wrote: > BIRD put a malformed BGP message on the wire >From what I'm seeing here, I would agree. Ideally, someone should report this to the BIRD developers. That said, for such a bug report to be useful, we may need more information. If it was just the RFC 1918 next-hop that triggered this, then this should be trivially reproducible. But if it was something else, we may not have enough information to get to the bottom of this. -- Richard

May 7, 2020

Re: route server down?

by Jay Hanke

> IOS XR said "hey look that's bad, and I'm going to ignore it", and Brocade said, "hey that's bad, and you're bad so I'm not going to talk to you", I think the Junipers reacted much as IOS XR did, but that doesn't mean there wasn't a bad message. I can confirm the Juniper reaction, it got marked as "invalid" with unreachable next hop. I don't think it threw an actual error, just put it in the naughty pile and ignored it.

May 6, 2020

Re: route server down?

by David Farmer

My IOS XR router detected a malformed BGP message, logs are in the thread below, but it treated it as a withdrawal per RFC7606. The Brocades saw the error and closed the BGP session, which is what many older BGP implementations do. I'm not sure the reason for the malformed BGP message, it could have something to do the next-hop, or it could be for some completely unrelated reason, but it as far as I can tell, BIRD put a malformed BGP message on the wire. IOS XR said "hey look that's bad, and I'm going to ignore it", and Brocade said, "hey that's bad, and you're bad so I'm not going to talk to you", I think the Junipers reacted much as IOS XR did, but that doesn't mean there wasn't a bad message. Thanks On Wed, May 6, 2020 at 11:49 AM Doug McIntyre <merlyn(a)iphouse.net> wrote: > I think we've come up with some methods for potentially creating > filters for this situation to roll out in the future. > > I don't think its' BIRD place to impose restrictions on what a BGP > peer is advertising, even if it is bogus. > > Clearly its only a bug with Brocade, because other vendors (especially > Juniper) didn't > care that it was announced this way, I just had a bogus route inserted > from them, but > it didn't affect any of my BGP sessions with MICE. > > > > > > On Wed, May 06, 2020 at 04:45:33PM +0000, Frank Bulk wrote: > >Do we need to submit a bug to the developers of BIRD? > > > >Frank > > > >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David > Farmer > >Sent: Thursday, April 30, 2020 8:37 AM > >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > >Subject: Re: [MICE-DISCUSS] route server down? > > > >I think it was more than just an invalid next-hop. If it was simply an > invalid next-hop that shouldn't have created a malformed BGP update. Unless > the invalid next-hop caused BIRD to send out a malformed BGP update. > > > >On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io<mailto: > jayhanke(a)southfront.io>> wrote: > >I emailed xcel about the invalid next-hop address. > > > >We should filter invalid next hops on the route servers. > > > >There also appears to be an issue with how some routers handle the > >invalid next hop. > > > >Are all the peers with the issue of losing the session to RS2 running > Brocade? > > > >On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu<mailto: > farmer(a)umn.edu>> wrote: > >> > >> someone with Access should see what route server 2 sees for that > prefix, and maybe kick it over after look at it. > >> > >> On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io > <mailto:jayhanke@southfront.io>> wrote: > >>> > >>> We're seeing the same with a good next-hop from RS1. > >>> > >>> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net<mailto: > wopat(a)wiscnet.net>> wrote: > >>> > > >>> > On 4/30/20 7:49 AM, David Farmer wrote: > >>> > > >>> > > We're running IOS XR, I found these droppings in our logs; > >>> > > > >>> > > >>> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: > >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received > from > >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, > error > >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF… > > > >>> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: > >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received > from > >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, > error > >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF… > > > >>> > > > >>> > > Maybe try resting you BGP sessions. > >>> > > > >>> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its > hidden > >>> > on our net. > >>> > > >>> > Is 10.223.129.2 something internal to route server #2? > >>> > > >>> > > show route 198.179.154.0 hidden detail > >>> > > >>> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 > holddown, > >>> > 1604 hidden) > >>> > 198.179.154.0/23< > https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> > (3 entries, 1 announced) > >>> > BGP > >>> > Next hop type: Router, Next hop index: 0 > >>> > Address: 0x113614cc > >>> > Next-hop reference count: 1 > >>> > Source: 206.108.255.2 > >>> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected > >>> > Session Id: 0x0 > >>> > State: <Hidden Ext> > >>> > Inactive reason: Unusable path > >>> > Local AS: 65400 Peer AS: 53679 > >>> > Age: 10:02:05 > >>> > Validation State: unverified > >>> > Task: BGP_53679.206.108.255.2 > >>> > AS path: I > >>> > Communities: target:21693:1000 > >>> > Router ID: 206.108.255.2 > >>> > Hidden reason: protocol nexthop is not on the > interface > >>> > > >>> > > >>> > -- > >>> > Chris Wopat > >>> > Network Engineer, WiscNet > >>> > wopat(a)wiscnet.net<mailto:wopat@wiscnet.net> 608-210-3965 > >>> > >>> > >>> > >>> -- > >>> Jay Hanke, President > >>> South Front Networks > >>> jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> > >>> Phone 612-204-0000 > >> > >> > >> > >> -- > >> =============================================== > >> David Farmer Email:farmer@umn.edu<mailto: > Email%3Afarmer(a)umn.edu> > >> Networking & Telecommunication Services > >> Office of Information Technology > >> University of Minnesota > >> 2218 University Ave SE Phone: 612-626-0815 > >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 > >> =============================================== > >> > >> ________________________________ > >> > >> To unsubscribe from the MICE-DISCUSS list, click the following link: > >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1< > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2… > > > > > > > > > >-- > >Jay Hanke, President > >South Front Networks > >jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> > >Phone 612-204-0000 > > > > > >-- > >=============================================== > >David Farmer Email:farmer@umn.edu<mailto: > Email%3Afarmer(a)umn.edu> > >Networking & Telecommunication Services > >Office of Information Technology > >University of Minnesota > >2218 University Ave SE Phone: 612-626-0815 > >Minneapolis, MN 55414-3029 Cell: 612-812-9952 > >=============================================== > > > >________________________________ > > > >To unsubscribe from the MICE-DISCUSS list, click the following link: > >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1< > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2… > > > > -- > Doug McIntyre <merlyn(a)iphouse.net> > ~.~ ipHouse ~.~ > Network Engineer/Provisioning/Jack of all Trades > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

May 6, 2020

Re: route server down?

by Doug McIntyre

I think we've come up with some methods for potentially creating filters for this situation to roll out in the future. I don't think its' BIRD place to impose restrictions on what a BGP peer is advertising, even if it is bogus. Clearly its only a bug with Brocade, because other vendors (especially Juniper) didn't care that it was announced this way, I just had a bogus route inserted from them, but it didn't affect any of my BGP sessions with MICE. On Wed, May 06, 2020 at 04:45:33PM +0000, Frank Bulk wrote: >Do we need to submit a bug to the developers of BIRD? > >Frank > >From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer >Sent: Thursday, April 30, 2020 8:37 AM >To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET >Subject: Re: [MICE-DISCUSS] route server down? > >I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. > >On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: >I emailed xcel about the invalid next-hop address. > >We should filter invalid next hops on the route servers. > >There also appears to be an issue with how some routers handle the >invalid next hop. > >Are all the peers with the issue of losing the session to RS2 running Brocade? > >On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu<mailto:farmer@umn.edu>> wrote: >> >> someone with Access should see what route server 2 sees for that prefix, and maybe kick it over after look at it. >> >> On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: >>> >>> We're seeing the same with a good next-hop from RS1. >>> >>> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net<mailto:wopat@wiscnet.net>> wrote: >>> > >>> > On 4/30/20 7:49 AM, David Farmer wrote: >>> > >>> > > We're running IOS XR, I found these droppings in our logs; >>> > > >>> > >>> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >>> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: >>> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >>> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >>> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >>> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >>> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >>> > > >>> > > Maybe try resting you BGP sessions. >>> > > >>> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its hidden >>> > on our net. >>> > >>> > Is 10.223.129.2 something internal to route server #2? >>> > >>> > > show route 198.179.154.0 hidden detail >>> > >>> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 holddown, >>> > 1604 hidden) >>> > 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> (3 entries, 1 announced) >>> > BGP >>> > Next hop type: Router, Next hop index: 0 >>> > Address: 0x113614cc >>> > Next-hop reference count: 1 >>> > Source: 206.108.255.2 >>> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected >>> > Session Id: 0x0 >>> > State: <Hidden Ext> >>> > Inactive reason: Unusable path >>> > Local AS: 65400 Peer AS: 53679 >>> > Age: 10:02:05 >>> > Validation State: unverified >>> > Task: BGP_53679.206.108.255.2 >>> > AS path: I >>> > Communities: target:21693:1000 >>> > Router ID: 206.108.255.2 >>> > Hidden reason: protocol nexthop is not on the interface >>> > >>> > >>> > -- >>> > Chris Wopat >>> > Network Engineer, WiscNet >>> > wopat(a)wiscnet.net<mailto:wopat@wiscnet.net> 608-210-3965 >>> >>> >>> >>> -- >>> Jay Hanke, President >>> South Front Networks >>> jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> >>> Phone 612-204-0000 >> >> >> >> -- >> =============================================== >> David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> >> Networking & Telecommunication Services >> Office of Information Technology >> University of Minnesota >> 2218 University Ave SE Phone: 612-626-0815 >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 >> =============================================== >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> > > > >-- >Jay Hanke, President >South Front Networks >jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> >Phone 612-204-0000 > > >-- >=============================================== >David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> >Networking & Telecommunication Services >Office of Information Technology >University of Minnesota >2218 University Ave SE Phone: 612-626-0815 >Minneapolis, MN 55414-3029 Cell: 612-812-9952 >=============================================== > >________________________________ > >To unsubscribe from the MICE-DISCUSS list, click the following link: >http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 6, 2020

Re: route server down?

by Frank Bulk

Do we need to submit a bug to the developers of BIRD? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of David Farmer Sent: Thursday, April 30, 2020 8:37 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] route server down? I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: I emailed xcel about the invalid next-hop address. We should filter invalid next hops on the route servers. There also appears to be an issue with how some routers handle the invalid next hop. Are all the peers with the issue of losing the session to RS2 running Brocade? On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu<mailto:farmer@umn.edu>> wrote: > > someone with Access should see what route server 2 sees for that prefix, and maybe kick it over after look at it. > > On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io<mailto:jayhanke@southfront.io>> wrote: >> >> We're seeing the same with a good next-hop from RS1. >> >> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net<mailto:wopat@wiscnet.net>> wrote: >> > >> > On 4/30/20 7:49 AM, David Farmer wrote: >> > >> > > We're running IOS XR, I found these droppings in our logs; >> > > >> > >> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: >> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: >> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from >> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error >> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error >> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), Data >> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> >> > > >> > > Maybe try resting you BGP sessions. >> > > >> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its hidden >> > on our net. >> > >> > Is 10.223.129.2 something internal to route server #2? >> > >> > > show route 198.179.154.0 hidden detail >> > >> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 holddown, >> > 1604 hidden) >> > 198.179.154.0/23<https://urldefense.proofpoint.com/v2/url?u=http-3A__198.179.154.0_23&d=DwMF…> (3 entries, 1 announced) >> > BGP >> > Next hop type: Router, Next hop index: 0 >> > Address: 0x113614cc >> > Next-hop reference count: 1 >> > Source: 206.108.255.2 >> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected >> > Session Id: 0x0 >> > State: <Hidden Ext> >> > Inactive reason: Unusable path >> > Local AS: 65400 Peer AS: 53679 >> > Age: 10:02:05 >> > Validation State: unverified >> > Task: BGP_53679.206.108.255.2 >> > AS path: I >> > Communities: target:21693:1000 >> > Router ID: 206.108.255.2 >> > Hidden reason: protocol nexthop is not on the interface >> > >> > >> > -- >> > Chris Wopat >> > Network Engineer, WiscNet >> > wopat(a)wiscnet.net<mailto:wopat@wiscnet.net> 608-210-3965 >> >> >> >> -- >> Jay Hanke, President >> South Front Networks >> jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> >> Phone 612-204-0000 > > > > -- > =============================================== > David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> > Networking & Telecommunication Services > Office of Information Technology > University of Minnesota > 2218 University Ave SE Phone: 612-626-0815 > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > =============================================== > > ________________________________ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…> -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io<mailto:jayhanke@southfront.io> Phone 612-204-0000 -- =============================================== David Farmer Email:farmer@umn.edu<mailto:Email%3Afarmer@umn.edu> Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 =============================================== ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.iphouse.net_cgi-2…>

May 6, 2020

Re: Please Welcome Genesis Wireless to MICE!

by Jeremy Lumby

Port Assignment for MN VoIP extension is Ethernet 18 -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Wednesday, May 06, 2020 9:17 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Please Welcome Genesis Wireless to MICE! Update, Genesis requested to use a previously allocated address to them which had not been used. Name: Genesis Technology Communication, LLC Port: MN VoIP Remote Switch (Port Assignment Pending) ASN: 30032 IPv4: 206.108.255.103/24 IPv6: 2001:504:27:0:0:7550::1/64 Peering Contact: admin(a)genesiswireless.com Route Servers- YES BFD: No On Wed, May 6, 2020 at 8:44 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > > Name: Genesis Technology Communication, LLC > Port: MN VoIP Remote Switch (Port Assignment Pending) > ASN: 30032 > IPv4: 206.108.255.165/24 > IPv6: 2001:504:27:0:0:7550::1/64 > Peering Contact: admin(a)genesiswireless.com > Route Servers- YES > BFD: No > > Welcome Aboard! > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 6, 2020

Re: Please Welcome Genesis Wireless to MICE!

by Jay Hanke

Update, Genesis requested to use a previously allocated address to them which had not been used. Name: Genesis Technology Communication, LLC Port: MN VoIP Remote Switch (Port Assignment Pending) ASN: 30032 IPv4: 206.108.255.103/24 IPv6: 2001:504:27:0:0:7550::1/64 Peering Contact: admin(a)genesiswireless.com Route Servers- YES BFD: No On Wed, May 6, 2020 at 8:44 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > > Name: Genesis Technology Communication, LLC > Port: MN VoIP Remote Switch (Port Assignment Pending) > ASN: 30032 > IPv4: 206.108.255.165/24 > IPv6: 2001:504:27:0:0:7550::1/64 > Peering Contact: admin(a)genesiswireless.com > Route Servers- YES > BFD: No > > Welcome Aboard! > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 6, 2020

Please Welcome Genesis Wireless to MICE!

by Jay Hanke

Name: Genesis Technology Communication, LLC Port: MN VoIP Remote Switch (Port Assignment Pending) ASN: 30032 IPv4: 206.108.255.165/24 IPv6: 2001:504:27:0:0:7550::1/64 Peering Contact: admin(a)genesiswireless.com Route Servers- YES BFD: No Welcome Aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 6, 2020

Re: [ADV] [MICE-DISCUSS] MICE PTR records missing for a handful of IPs

by Doug McIntyre

Yes, it is a manual process to update the PTR record updates to the DNS, and the forwards are handled elsewhere so I just batch them up every now and then and send off a request to the forward host contact every now-and-then. In the past virtually nobody has cared, so they don't tend to notice when I update PTRs or not. I can get going on these. On Tue, May 05, 2020 at 02:15:05PM +0000, Chris Wopat wrote: >Hi folks, > >This morning we stumbled across a cosmetic issue, PTRs don't seem to >exist for a handful of participants. Note we did not eyeball v6, this >was just noticed when making some changes on our device. > >A few may have custom PTRs, one has an extra n in 'micemn' (micemnn), >the others simply lack PTRs. > >Quick list below, see attachment has more info. > >cat /tmp/mice-without-ptr.txt | grep 206.108 | awk '{print $3}' >100ge49-1.usi-cr04-mpls.u >xe0-0-0.er1.511.msp.as535 >AS393238.micemnn.net >206.108.255.145 >206.108.255.148 >206.108.255.149 >206.108.255.152 >206.108.255.153 >206.108.255.154 >206.108.255.155 >206.108.255.156 >206.108.255.157 >206.108.255.158 >206.108.255.159 >206.108.255.160 >206.108.255.233 > >Cheers, >-- >Chris Wopat >Network Engineer, WiscNet >wopat(a)wiscnet.net 608-210-3965 >wopat@r-minneapolis-isp> show arp interface irb.100 | except micemn.net >MAC Address Address Name Interface Flags >74:83:ef:2f:04:1b 206.108.255.19 100ge49-1.usi-cr04-mpls.u irb.100 [ae0.100] none >9c:8a:cb:9c:0e:03 206.108.255.65 xe0-0-0.er1.511.msp.as535 irb.100 [ae0.100] none >08:96:ad:25:2e:3c 206.108.255.114 AS393238.micemnn.net irb.100 [ae0.100] none >74:a2:e6:c5:77:00 206.108.255.145 206.108.255.145 irb.100 [ae0.100] none >78:fe:3d:ae:15:cc 206.108.255.148 206.108.255.148 irb.100 [ae0.100] none >64:d1:54:f7:47:93 206.108.255.149 206.108.255.149 irb.100 [ae0.100] none >00:1d:71:73:2f:00 206.108.255.152 206.108.255.152 irb.100 [ae0.100] none >00:bc:60:0b:f8:a8 206.108.255.153 206.108.255.153 irb.100 [ae0.100] none >74:8e:f8:62:08:01 206.108.255.154 206.108.255.154 irb.100 [ae0.100] none >5c:5e:ab:de:e5:00 206.108.255.155 206.108.255.155 irb.100 [ae0.100] none >f4:cc:55:79:41:56 206.108.255.156 206.108.255.156 irb.100 [ae0.100] none >d0:dd:49:4a:eb:d2 206.108.255.157 206.108.255.157 irb.100 [ae0.100] none >cc:4e:24:9a:69:00 206.108.255.158 206.108.255.158 irb.100 [ae0.100] none >ec:38:73:ed:a2:67 206.108.255.159 206.108.255.159 irb.100 [ae0.100] none >50:2f:a8:e2:b0:00 206.108.255.160 206.108.255.160 irb.100 [ae0.100] none >b8:c2:53:57:93:c0 206.108.255.233 206.108.255.233 irb.100 [ae0.100] none > -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 5, 2020

MICE PTR records missing for a handful of IPs

by Chris Wopat

Hi folks, This morning we stumbled across a cosmetic issue, PTRs don't seem to exist for a handful of participants. Note we did not eyeball v6, this was just noticed when making some changes on our device. A few may have custom PTRs, one has an extra n in 'micemn' (micemnn), the others simply lack PTRs. Quick list below, see attachment has more info. cat /tmp/mice-without-ptr.txt | grep 206.108 | awk '{print $3}' 100ge49-1.usi-cr04-mpls.u xe0-0-0.er1.511.msp.as535 AS393238.micemnn.net 206.108.255.145 206.108.255.148 206.108.255.149 206.108.255.152 206.108.255.153 206.108.255.154 206.108.255.155 206.108.255.156 206.108.255.157 206.108.255.158 206.108.255.159 206.108.255.160 206.108.255.233 Cheers, -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

May 5, 2020

Re: Additional 100G Card for Arista

by Mike Horwath

On Fri, May 01, 2020 at 01:06:48PM -0500, Richard Laager wrote: > > 2 - DCS7280SR-48C6, 48 SFP, 6 QSFP Switch, SN:SSJ16451365, SN:SSJ16402897 > > 1 - DCS7280QR-C36, 36 QSFP Switch, SN:JPE16362500 > > We don't have a need for these right now, and whatever we take, Arista > wants us to pay for support on. If they're going to go in the garbage, > we'd take them. Otherwise, no thanks. I don't have any Arista in my lab. I should probably look at fixing that sometime in the future. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

May 1, 2020

Re: Additional 100G Card for Arista

by Richard Laager

On 5/1/20 11:59 AM, David Farmer wrote: > I need to get back to completing the paperwork, but before I do that, > there is other equipment we have that we are also willing to transfer > to MICE as well. However, it wasn't clear MICE wanted it all. > > We also have the following; > > 1 - DCS-7504N Chassis, SN:HSH16345632 > 4 - PWR-3KT-AC-RED Power Supplies > 4 - DCS-7500R-FM Fabric Modules, SN:JPE16470459, SN:JPE16470431, > SN:JPE16470537, SN:JPE16470531 > 2 - 7500-SUP2 Supervisor Modules, SN:JPE16305892, SN:JPE16312295 Yes to all this. > 2 - DCS7280SR-48C6, 48 SFP, 6 QSFP Switch, SN:SSJ16451365, SN:SSJ16402897 > 1 - DCS7280QR-C36, 36 QSFP Switch, SN:JPE16362500 We don't have a need for these right now, and whatever we take, Arista wants us to pay for support on. If they're going to go in the garbage, we'd take them. Otherwise, no thanks. As previously mentioned, QCIX is probably interested. Jay would be the one to talk to there. If you are interested in selling them to for-profit entities, there are definitely some that would be interested. > 7 - AOC-10028-20, QSFP100 to QSFP100 Active Optical Cable - 5M > 4 - CAB-Q-Q-100G-3M, QSFP100 to QSFP100 Passive Copper Cable - 3M > 4 - QSFP-100-SR4, QSFP100 SR4 Optical Modules Yes to all this. We would use it to connect the two switches within the same rack. Granted, we wouldn't need all of it, but we see no reason to turn it down. It's not going to affect the support cost. -- Richard

May 1, 2020

Re: Additional 100G Card for Arista

by Steve Howard

May 1, 2020

Re: Additional 100G Card for Arista

by David Farmer

Jeremy has had the line card since the emergency maintenance a couple of weeks ago, it is a 7500R-36QC-LC, it is SN:SSJ16420315. I need to get back to completing the paperwork, but before I do that, there is other equipment we have that we are also willing to transfer to MICE as well. However, it wasn't clear MICE wanted it all. We also have the following; 1 - DCS-7504N Chassis, SN:HSH16345632 4 - PWR-3KT-AC-RED Power Supplies 4 - DCS-7500R-FM Fabric Modules, SN:JPE16470459, SN:JPE16470431, SN:JPE16470537, SN:JPE16470531 2 - 7500-SUP2 Supervisor Modules, SN:JPE16305892, SN:JPE16312295 2 - DCS7280SR-48C6, 48 SFP, 6 QSFP Switch, SN:SSJ16451365, SN:SSJ16402897 1 - DCS7280QR-C36, 36 QSFP Switch, SN:JPE16362500 7 - AOC-10028-20, QSFP100 to QSFP100 Active Optical Cable - 5M 4 - CAB-Q-Q-100G-3M, QSFP100 to QSFP100 Passive Copper Cable - 3M 4- QSFP-100-SR4, QSFP100 SR4 Optical Modules So, besides the 36 port QSFP Line card what does MICE want? On Fri, May 1, 2020 at 10:11 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > Is there an update on the additional 100G card for the Main Switch? > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

May 1, 2020

Re: [ADV] [MICE-DISCUSS] Please Welcome IP Pathways to MICE!

by Doug McIntyre

On Fri, May 01, 2020 at 11:22:26AM -0500, Jay Hanke wrote: >Name: IP Pathways, LLC >Port: Ethernet8 (SFN Remote Switch) >ASN: 393775 >IPv4: 206.108.255.164/24 >IPv6: 2001:504:27:0:6:022f::1/64 >Peering Contact: peering(a)ippathways.com >Route Servers- YES >BFD: Yes I've added them in to the route servers. Welcome! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

May 1, 2020

Please Welcome IP Pathways to MICE!

by Jay Hanke

Name: IP Pathways, LLC Port: Ethernet8 (SFN Remote Switch) ASN: 393775 IPv4: 206.108.255.164/24 IPv6: 2001:504:27:0:6:022f::1/64 Peering Contact: peering(a)ippathways.com Route Servers- YES BFD: Yes Welcome Aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 1, 2020

Additional 100G Card for Arista

by Jay Hanke

Is there an update on the additional 100G card for the Main Switch? -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

May 1, 2020

Re: route server down?

by James Urwiller

Looks like that did it.. back to established! James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 30, 2020 09:07 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] route server down? I received a message from Xcel that they have corrected their RS2 config to match RS1. How are things looking for the brocade folks? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Thursday, April 30, 2020 8:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] route server down? On Thu, Apr 30, 2020 at 8:37 AM David Farmer <farmer(a)umn.edu> wrote: > > I think it was more than just an invalid next-hop. If it was simply an > invalid next-hop that shouldn't have created a malformed BGP update. > Unless the invalid next-hop caused BIRD to send out a malformed BGP > update. The problem left here successfully.... :)

April 30, 2020

Re: route server down?

by Chris Wopat

On 4/30/20 9:11 AM, Steve Howard wrote: > > We are also receiving some routes from Xcel with the bogus ASN 65500 in the path: > > *> 192.234.135.0/24 206.108.255.115 125 0 21693 65500 i > *> 192.234.136.0/24 206.108.255.115 125 0 21693 65500 i > *> 192.234.137.0/24 206.108.255.115 125 0 21693 65500 i > *> 192.234.138.0/24 206.108.255.115 125 0 21693 65500 i > *> 192.234.140.0/24 206.108.255.115 125 0 21693 65500 i > > Although, I don't think that should cause a route server problem. > While they should be removing their private ASN when sending, most vendors have a remove-private-as type of flag that you should be able to set on your end too for additional sanity. -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

April 30, 2020

Re: route server down?

by Steve Howard

April 30, 2020

Re: route server down?

by Jeremy Lumby

I received a message from Xcel that they have corrected their RS2 config to match RS1. How are things looking for the brocade folks? -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jay Hanke Sent: Thursday, April 30, 2020 8:40 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] route server down? On Thu, Apr 30, 2020 at 8:37 AM David Farmer <farmer(a)umn.edu> wrote: > > I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. The problem left here successfully.... :)

April 30, 2020

Re: route server down?

by Jay Hanke

On Thu, Apr 30, 2020 at 8:37 AM David Farmer <farmer(a)umn.edu> wrote: > > I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. The problem left here successfully.... :)

April 30, 2020

Re: route server down?

by David Farmer

I think it was more than just an invalid next-hop. If it was simply an invalid next-hop that shouldn't have created a malformed BGP update. Unless the invalid next-hop caused BIRD to send out a malformed BGP update. On Thu, Apr 30, 2020 at 8:24 AM Jay Hanke <jayhanke(a)southfront.io> wrote: > I emailed xcel about the invalid next-hop address. > > We should filter invalid next hops on the route servers. > > There also appears to be an issue with how some routers handle the > invalid next hop. > > Are all the peers with the issue of losing the session to RS2 running > Brocade? > > On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu> wrote: > > > > someone with Access should see what route server 2 sees for that prefix, > and maybe kick it over after look at it. > > > > On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io> > wrote: > >> > >> We're seeing the same with a good next-hop from RS1. > >> > >> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net> wrote: > >> > > >> > On 4/30/20 7:49 AM, David Farmer wrote: > >> > > >> > > We're running IOS XR, I found these droppings in our logs; > >> > > > >> > > >> > RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: > >> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from > >> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error > >> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23 > >> > RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: > >> > %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from > >> > neighbor 206.108.255.2 (VRF: default) - message length 59 bytes, error > >> > flags 0x00000200, action taken "TreatAsWdr". Error details: "Error > >> > 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 0), > Data > >> > [400200]". NLRIs: [IPv4 Unicast] 198.179.154.0/23 > >> > > > >> > > Maybe try resting you BGP sessions. > >> > > > >> > We're seeing a weird next-hop ip on that prefix (rfc1918) and its > hidden > >> > on our net. > >> > > >> > Is 10.223.129.2 something internal to route server #2? > >> > > >> > > show route 198.179.154.0 hidden detail > >> > > >> > inet.0: 795967 destinations, 2081403 routes (795589 active, 0 > holddown, > >> > 1604 hidden) > >> > 198.179.154.0/23 (3 entries, 1 announced) > >> > BGP > >> > Next hop type: Router, Next hop index: 0 > >> > Address: 0x113614cc > >> > Next-hop reference count: 1 > >> > Source: 206.108.255.2 > >> > Next hop: 10.223.129.2 via xe-0/1/5.300, selected > >> > Session Id: 0x0 > >> > State: <Hidden Ext> > >> > Inactive reason: Unusable path > >> > Local AS: 65400 Peer AS: 53679 > >> > Age: 10:02:05 > >> > Validation State: unverified > >> > Task: BGP_53679.206.108.255.2 > >> > AS path: I > >> > Communities: target:21693:1000 > >> > Router ID: 206.108.255.2 > >> > Hidden reason: protocol nexthop is not on the > interface > >> > > >> > > >> > -- > >> > Chris Wopat > >> > Network Engineer, WiscNet > >> > wopat(a)wiscnet.net 608-210-3965 > >> > >> > >> > >> -- > >> Jay Hanke, President > >> South Front Networks > >> jayhanke(a)southfront.io > >> Phone 612-204-0000 > > > > > > > > -- > > =============================================== > > David Farmer Email:farmer@umn.edu > > Networking & Telecommunication Services > > Office of Information Technology > > University of Minnesota > > 2218 University Ave SE Phone: 612-626-0815 > > Minneapolis, MN 55414-3029 Cell: 612-812-9952 > > =============================================== > > > > ________________________________ > > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 30, 2020

Re: route server down?

by Steve Savoy

I am running Brocade as well. Steve Savoy WCTA Sent from my iPhone > On Apr 30, 2020, at 8:26 AM, James Urwiller <james.urwiller(a)turnkey-telecom.com> wrote: > > I am running brocade, yes. > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jay Hanke > Sent: Thursday, April 30, 2020 08:24 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] route server down? > > I emailed xcel about the invalid next-hop address. > > We should filter invalid next hops on the route servers. > > There also appears to be an issue with how some routers handle the invalid > next hop. > > Are all the peers with the issue of losing the session to RS2 running > Brocade? > >> On Thu, Apr 30, 2020 at 8:17 AM David Farmer <farmer(a)umn.edu> wrote: >> >> someone with Access should see what route server 2 sees for that prefix, >> and maybe kick it over after look at it. >> >>> On Thu, Apr 30, 2020 at 8:04 AM Jay Hanke <jayhanke(a)southfront.io> wrote: >>> >>> We're seeing the same with a good next-hop from RS1. >>> >>> On Thu, Apr 30, 2020 at 7:55 AM Chris Wopat <wopat(a)wiscnet.net> wrote: >>>> >>>> On 4/30/20 7:49 AM, David Farmer wrote: >>>> >>>>> We're running IOS XR, I found these droppings in our logs; >>>>> >>>> >>>> RP/0/RP0/CPU0:Apr 29 21:50:26.798 CDT: bgp[1068]: >>>> %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received >>>> from neighbor 206.108.255.2 (VRF: default) - message length 59 >>>> bytes, error flags 0x00000200, action taken "TreatAsWdr". Error >>>> details: "Error 0x00000200, Field "Attr-data", Attribute 2 (Flags >>>> 0x40, Length 0), Data [400200]". NLRIs: [IPv4 Unicast] >>>> 198.179.154.0/23 RP/0/RP1/CPU0:Apr 29 21:50:26.797 CDT: bgp[1068]: >>>> %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received >>>> from neighbor 206.108.255.2 (VRF: default) - message length 59 >>>> bytes, error flags 0x00000200, action taken "TreatAsWdr". Error >>>> details: "Error 0x00000200, Field "Attr-data", Attribute 2 (Flags >>>> 0x40, Length 0), Data [400200]". NLRIs: [IPv4 Unicast] >>>> 198.179.154.0/23 >>>>> >>>>> Maybe try resting you BGP sessions. >>>>> >>>> We're seeing a weird next-hop ip on that prefix (rfc1918) and its >>>> hidden on our net. >>>> >>>> Is 10.223.129.2 something internal to route server #2? >>>> >>>>> show route 198.179.154.0 hidden detail >>>> >>>> inet.0: 795967 destinations, 2081403 routes (795589 active, 0 >>>> holddown, >>>> 1604 hidden) >>>> 198.179.154.0/23 (3 entries, 1 announced) >>>> BGP >>>> Next hop type: Router, Next hop index: 0 >>>> Address: 0x113614cc >>>> Next-hop reference count: 1 >>>> Source: 206.108.255.2 >>>> Next hop: 10.223.129.2 via xe-0/1/5.300, selected >>>> Session Id: 0x0 >>>> State: <Hidden Ext> >>>> Inactive reason: Unusable path >>>> Local AS: 65400 Peer AS: 53679 >>>> Age: 10:02:05 >>>> Validation State: unverified >>>> Task: BGP_53679.206.108.255.2 >>>> AS path: I >>>> Communities: target:21693:1000 >>>> Router ID: 206.108.255.2 >>>> Hidden reason: protocol nexthop is not on the >>>> interface >>>> >>>> >>>> -- >>>> Chris Wopat >>>> Network Engineer, WiscNet >>>> wopat(a)wiscnet.net 608-210-3965 >>> >>> >>> >>> -- >>> Jay Hanke, President >>> South Front Networks >>> jayhanke(a)southfront.io >>> Phone 612-204-0000 >> >> >> >> -- >> =============================================== >> David Farmer Email:farmer@umn.edu >> Networking & Telecommunication Services Office of Information >> Technology University of Minnesota >> 2218 University Ave SE Phone: 612-626-0815 >> Minneapolis, MN 55414-3029 Cell: 612-812-9952 >> =============================================== >> >> ________________________________ >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > -- > Jay Hanke, President > South Front Networks > jayhanke(a)southfront.io > Phone 612-204-0000 > Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

April 30, 2020