- mice-discuss - lists.micemn.net

Re: Routing of non-IX traffic

by Frank Bulk

Thanks, Matthew, for explaining why ARP might be happening. Now that CNS has its proxy ARP turned off, it’s AS393639 that’s responding: SiouxCenter-Arista-North(s1)#traceroute ip www.yamaha-dealers.com source et 3/24 traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte packets 1 AS393639.micemn.net (206.108.255.47) 13.936 ms 14.004 ms 13.998 ms 2 v415.core1.msp1.he.net (184.105.25.93) 14.146 ms 14.205 ms 14.274 ms 3 100ge13-1.core2.chi1.he.net (184.105.223.177) 22.533 ms 22.369 ms 22.538 ms 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 *^CSiouxCenter-Arista-North(s1)# Can an ACL be created on the Arista that discards in/outbound ARP requests for the non-MICE address space? Frank From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Steve Howard Sent: Thursday, August 16, 2018 9:32 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Routing of non-IX traffic I've disabled proxy arp on the CNS router... Has the behavior changed? On 08/16/2018 05:00 PM, Matthew Beckwell wrote: I'm getting similar behavior as Frank. Like Doug, I only have 45.60.73.0/24<http://45.60.73.0/24> from transit connections. So a traceroute from my MICE interface should ARP and die (I would think).... When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): 00:23:33:c6:a0:c0 206.108.255.50 Cooperative Network Services (CNS) 32609 e4:aa:5d:83:73:06 206.108.255.47 IVDesk 393639 88:43:e1:00:f2:10 206.108.255.18 Consolidated Communications 12042 b0:aa:77:33:7b:03 206.108.255.79 Gigamonster, LLC 31939 3c:08:f6:81:6e:a5 206.108.255.46 OneNetUSA 46131 00:1d:e5:c0:78:c3 206.108.255.5 Implex 21709 54:75:d0:e6:08:30 206.108.255.106 Nuvera Communications 23465 00:11:5d:82:6c:00 206.108.255.80 Future Technologies 26451 Proxy ARP (or something like it)? CNS seems to be consistently coming in first place when I clear my ARP entry. ~Matthew matthewb(a)aitech.net<mailto:matthewb@aitech.net> AS13746 On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com<http://www.yamaha-dealers.com> (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net<http://AS32609.micemn.net> (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net<http://cns70.cnsllc.net> (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net<http://fg30.ips.cnsllc.net> (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net<http://crss2.PaulBunyan.net> (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net<http://cra.PaulBunyan.net> (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net<http://fp233.ips.PaulBunyan.net> (205.149.159.233) 22.669 ms cra.PaulBunyan.net<http://cra.PaulBunyan.net> (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 17, 2018

Re: Routing of non-IX traffic

by AnthonyAnderberg＠nuvera.net

Interesting - today I've learned that although IOS XR has Proxy ARP disabled by default, IOS XE has it enabled by default: nuveragw1#show ip int te0/3/0 | i ARP Proxy ARP is enabled RP/0/RSP0/CPU0:bbvgw1#show ip int te0/0/0/2 | i ARP Proxy ARP is disabled I’d agree that it’s not expected behavior in this context, so will disable it on Nuvera’s XE router. Thanks for making me smarter, anthony [Anthony_Anderberg]<https://www.nuvera.net/> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Thursday, August 16, 2018 5:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Routing of non-IX traffic I'm getting similar behavior as Frank. Like Doug, I only have 45.60.73.0/24<http://45.60.73.0/24> from transit connections. So a traceroute from my MICE interface should ARP and die (I would think).... When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): 00:23:33:c6:a0:c0 206.108.255.50 Cooperative Network Services (CNS) 32609 e4:aa:5d:83:73:06 206.108.255.47 IVDesk 393639 88:43:e1:00:f2:10 206.108.255.18 Consolidated Communications 12042 b0:aa:77:33:7b:03 206.108.255.79 Gigamonster, LLC 31939 3c:08:f6:81:6e:a5 206.108.255.46 OneNetUSA 46131 00:1d:e5:c0:78:c3 206.108.255.5 Implex 21709 54:75:d0:e6:08:30 206.108.255.106 Nuvera Communications 23465 00:11:5d:82:6c:00 206.108.255.80 Future Technologies 26451 Proxy ARP (or something like it)? CNS seems to be consistently coming in first place when I clear my ARP entry. ~Matthew matthewb(a)aitech.net<mailto:matthewb@aitech.net> AS13746 On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com<http://www.yamaha-dealers.com> (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net<http://AS32609.micemn.net> (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net<http://cns70.cnsllc.net> (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net<http://fg30.ips.cnsllc.net> (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net<http://crss2.PaulBunyan.net> (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net<http://cra.PaulBunyan.net> (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net<http://fp233.ips.PaulBunyan.net> (205.149.159.233) 22.669 ms cra.PaulBunyan.net<http://cra.PaulBunyan.net> (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 17, 2018

Re: Routing of non-IX traffic

by Steve Howard

Aug. 17, 2018

Re: Subscription Mail: Re: [MICE-DISCUSS] Routing of non-IX traffic

by David Farmer

At one level I agree with you, especially when you are talking about IXPs. However, facing userland, proxy arp plasters over a myriad of silly misconfigurations and typos. Which from a TAC and help desk point of view is a good thing, many tickets that are never opened, because it just works. On Thu, Aug 16, 2018 at 5:25 PM, Ryan Goldberg <RGoldberg(a)compudyne.com> wrote: > Proxy arp enabled by default is the silliest thing Cisco ever did. > > On Aug 16, 2018, at 5:14 PM, Andrew Hoyos <hoyosa(a)GMAIL.COM> wrote: > > Yeah, was just writing this. > Your router should see that it’s not a directly connected IP, and back up > to routing table/FIB. It may ARP for next hop depending on path. > > For those with this issue, what say your routing tables for this subnet? > And do *you* have proxy arp turned off? > > — > Andrew Hoyos > hoyosa(a)gmail.com > > > > On Aug 16, 2018, at 3:11 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Why would your router ARP for an address that is not on the same subnet as > any of your interfaces? > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Matthew Beckwell > *Sent:* Thursday, August 16, 2018 5:01 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Routing of non-IX traffic > > I'm getting similar behavior as Frank. > > Like Doug, I only have 45.60.73.0/24 from transit connections. > So a traceroute from my MICE interface should ARP and die (I would > think).... > > When I traceroute to 45.60.73.16-- my router sends out an ARP request, as > expected. > But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the > order they came into my interface): > > > 00:23:33:c6:a0:c0 > 206.108.255.50 > Cooperative Network Services (CNS) > 32609 > e4:aa:5d:83:73:06 > 206.108.255.47 > IVDesk > 393639 > 88:43:e1:00:f2:10 > 206.108.255.18 > Consolidated Communications > 12042 > b0:aa:77:33:7b:03 > 206.108.255.79 > Gigamonster, LLC > 31939 > 3c:08:f6:81:6e:a5 > 206.108.255.46 > OneNetUSA > 46131 > 00:1d:e5:c0:78:c3 > 206.108.255.5 > Implex > 21709 > 54:75:d0:e6:08:30 > 206.108.255.106 > Nuvera Communications > 23465 > 00:11:5d:82:6c:00 > 206.108.255.80 > Future Technologies > 26451 > > > Proxy ARP (or something like it)? > CNS seems to be consistently coming in first place when I clear my ARP > entry. > > ~Matthew > matthewb(a)aitech.net > AS13746 > > > > > On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com> > wrote: > When I force a traceroute to originate from our MICE-facing connection, > the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? > > To making things more interesting, Incapsula-destined traffic goes via > Paul Bunyan. Here's just one example: > > traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte > packets > 1 AS32609.micemn.net <http://as32609.micemn.net/> (206.108.255.50) > 14.059 ms 14.084 ms 14.076 ms > 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms > 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms > 4 crss2.PaulBunyan.net <http://crss2.paulbunyan.net/> (205.149.159.197) > 20.527 ms 20.562 ms 20.619 ms > 5 cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) > 23.398 ms fp233.ips.PaulBunyan.net <http://fp233.ips.paulbunyan.net/> (205.149.159.233) > 22.669 ms cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) > 23.393 ms > 6 * * * > 7 * * * > 8 * * * > 9 * * * > 10 * * * > 11 * * * > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * > SiouxCenter-Arista-North(s1) > > The reason I stumbled across this is because we've had more than a dozen > customers over the last month complain about access to Incapsula-protected > sites. Packet captures show TCP RSTs coming from the far side. > > Regards, > > Frank Bulk > AS53347 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

Aug. 16, 2018

Re: Routing of non-IX traffic

by Matthew Beckwell

On some gear, it will ARP for an address that is not directly connected when you force the interface being used. (Basically telling the router "Ignore the forwarding table...use this interface no matter what"). To which the router says "okay, but i'm gonna have to ARP for it, and anyone watching is going to think I don't know what i'm doing...." (And they'd be correct...) Under normal circumstances (not specifying the interface), my traceroutes happily go out the shortest transit path shown in the full BGP feed. No proxy arp here. :) ~Matthew On Thu, Aug 16, 2018 at 5:13 PM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > Yeah, was just writing this. > Your router should see that it’s not a directly connected IP, and back up > to routing table/FIB. It may ARP for next hop depending on path. > > For those with this issue, what say your routing tables for this subnet? > And do *you* have proxy arp turned off? > > — > Andrew Hoyos > hoyosa(a)gmail.com > > > > On Aug 16, 2018, at 3:11 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Why would your router ARP for an address that is not on the same subnet as > any of your interfaces? > > *From:* MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET > <MICE-DISCUSS(a)LISTS.IPHOUSE.NET>] *On Behalf Of *Matthew Beckwell > *Sent:* Thursday, August 16, 2018 5:01 PM > *To:* MICE-DISCUSS(a)LISTS.IPHOUSE.NET > *Subject:* Re: [MICE-DISCUSS] Routing of non-IX traffic > > I'm getting similar behavior as Frank. > > Like Doug, I only have 45.60.73.0/24 from transit connections. > So a traceroute from my MICE interface should ARP and die (I would > think).... > > When I traceroute to 45.60.73.16-- my router sends out an ARP request, as > expected. > But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the > order they came into my interface): > > > 00:23:33:c6:a0:c0 > 206.108.255.50 > Cooperative Network Services (CNS) > 32609 > e4:aa:5d:83:73:06 > 206.108.255.47 > IVDesk > 393639 > 88:43:e1:00:f2:10 > 206.108.255.18 > Consolidated Communications > 12042 > b0:aa:77:33:7b:03 > 206.108.255.79 > Gigamonster, LLC > 31939 > 3c:08:f6:81:6e:a5 > 206.108.255.46 > OneNetUSA > 46131 > 00:1d:e5:c0:78:c3 > 206.108.255.5 > Implex > 21709 > 54:75:d0:e6:08:30 > 206.108.255.106 > Nuvera Communications > 23465 > 00:11:5d:82:6c:00 > 206.108.255.80 > Future Technologies > 26451 > > > Proxy ARP (or something like it)? > CNS seems to be consistently coming in first place when I clear my ARP > entry. > > ~Matthew > matthewb(a)aitech.net > AS13746 > > > > > On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com> > wrote: > When I force a traceroute to originate from our MICE-facing connection, > the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? > > To making things more interesting, Incapsula-destined traffic goes via > Paul Bunyan. Here's just one example: > > traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte > packets > 1 AS32609.micemn.net <http://as32609.micemn.net/> (206.108.255.50) > 14.059 ms 14.084 ms 14.076 ms > 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms > 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms > 4 crss2.PaulBunyan.net <http://crss2.paulbunyan.net/> (205.149.159.197) > 20.527 ms 20.562 ms 20.619 ms > 5 cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) > 23.398 ms fp233.ips.PaulBunyan.net <http://fp233.ips.paulbunyan.net/> (205.149.159.233) > 22.669 ms cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) > 23.393 ms > 6 * * * > 7 * * * > 8 * * * > 9 * * * > 10 * * * > 11 * * * > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * > SiouxCenter-Arista-North(s1) > > The reason I stumbled across this is because we've had more than a dozen > customers over the last month complain about access to Incapsula-protected > sites. Packet captures show TCP RSTs coming from the far side. > > Regards, > > Frank Bulk > AS53347 > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

Aug. 16, 2018

Re: Subscription Mail: Re: [MICE-DISCUSS] Routing of non-IX traffic

by Ryan Goldberg

Proxy arp enabled by default is the silliest thing Cisco ever did. On Aug 16, 2018, at 5:14 PM, Andrew Hoyos <hoyosa(a)GMAIL.COM<mailto:hoyosa@GMAIL.COM>> wrote: Yeah, was just writing this. Your router should see that it’s not a directly connected IP, and back up to routing table/FIB. It may ARP for next hop depending on path. For those with this issue, what say your routing tables for this subnet? And do *you* have proxy arp turned off? — Andrew Hoyos hoyosa(a)gmail.com<mailto:hoyosa@gmail.com> On Aug 16, 2018, at 3:11 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM<mailto:jlumby@MNVOIP.COM>> wrote: Why would your router ARP for an address that is not on the same subnet as any of your interfaces? From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Thursday, August 16, 2018 5:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] Routing of non-IX traffic I'm getting similar behavior as Frank. Like Doug, I only have 45.60.73.0/24<http://45.60.73.0/24> from transit connections. So a traceroute from my MICE interface should ARP and die (I would think).... When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): 00:23:33:c6:a0:c0 206.108.255.50 Cooperative Network Services (CNS) 32609 e4:aa:5d:83:73:06 206.108.255.47 IVDesk 393639 88:43:e1:00:f2:10 206.108.255.18 Consolidated Communications 12042 b0:aa:77:33:7b:03 206.108.255.79 Gigamonster, LLC 31939 3c:08:f6:81:6e:a5 206.108.255.46 OneNetUSA 46131 00:1d:e5:c0:78:c3 206.108.255.5 Implex 21709 54:75:d0:e6:08:30 206.108.255.106 Nuvera Communications 23465 00:11:5d:82:6c:00 206.108.255.80 Future Technologies 26451 Proxy ARP (or something like it)? CNS seems to be consistently coming in first place when I clear my ARP entry. ~Matthew matthewb(a)aitech.net<mailto:matthewb@aitech.net> AS13746 On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com<mailto:fbulk@mypremieronline.com>> wrote: When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com<http://www.yamaha-dealers.com/> (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net<http://as32609.micemn.net/> (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net<http://cns70.cnsllc.net/> (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net<http://fg30.ips.cnsllc.net/> (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net<http://crss2.paulbunyan.net/> (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net<http://cra.paulbunyan.net/> (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net<http://fp233.ips.paulbunyan.net/> (205.149.159.233) 22.669 ms cra.PaulBunyan.net<http://cra.paulbunyan.net/> (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 16, 2018

Re: Routing of non-IX traffic

by Andrew Hoyos

Yeah, was just writing this. Your router should see that it’s not a directly connected IP, and back up to routing table/FIB. It may ARP for next hop depending on path. For those with this issue, what say your routing tables for this subnet? And do *you* have proxy arp turned off? — Andrew Hoyos hoyosa(a)gmail.com > On Aug 16, 2018, at 3:11 PM, Jeremy Lumby <jlumby(a)MNVOIP.COM> wrote: > > Why would your router ARP for an address that is not on the same subnet as any of your interfaces? > > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET>] On Behalf Of Matthew Beckwell > Sent: Thursday, August 16, 2018 5:01 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> > Subject: Re: [MICE-DISCUSS] Routing of non-IX traffic > > I'm getting similar behavior as Frank. > > Like Doug, I only have 45.60.73.0/24 <http://45.60.73.0/24> from transit connections. > So a traceroute from my MICE interface should ARP and die (I would think).... > > When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. > But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): > > > 00:23:33:c6:a0:c0 > 206.108.255.50 > Cooperative Network Services (CNS) > 32609 > e4:aa:5d:83:73:06 > 206.108.255.47 > IVDesk > 393639 > 88:43:e1:00:f2:10 > 206.108.255.18 > Consolidated Communications > 12042 > b0:aa:77:33:7b:03 > 206.108.255.79 > Gigamonster, LLC > 31939 > 3c:08:f6:81:6e:a5 > 206.108.255.46 > OneNetUSA > 46131 > 00:1d:e5:c0:78:c3 > 206.108.255.5 > Implex > 21709 > 54:75:d0:e6:08:30 > 206.108.255.106 > Nuvera Communications > 23465 > 00:11:5d:82:6c:00 > 206.108.255.80 > Future Technologies > 26451 > > > Proxy ARP (or something like it)? > CNS seems to be consistently coming in first place when I clear my ARP entry. > > ~Matthew > matthewb(a)aitech.net <mailto:matthewb@aitech.net> > AS13746 > > > > > On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com <mailto:fbulk@mypremieronline.com>> wrote: > When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? > > To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: > > traceroute to www.yamaha-dealers.com <http://www.yamaha-dealers.com/> (45.60.73.16), 30 hops max, 60 byte packets > 1 AS32609.micemn.net <http://as32609.micemn.net/> (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms > 2 cns70.cnsllc.net <http://cns70.cnsllc.net/> (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms > 3 fg30.ips.cnsllc.net <http://fg30.ips.cnsllc.net/> (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms > 4 crss2.PaulBunyan.net <http://crss2.paulbunyan.net/> (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms > 5 cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net <http://fp233.ips.paulbunyan.net/> (205.149.159.233) 22.669 ms cra.PaulBunyan.net <http://cra.paulbunyan.net/> (205.149.159.181) 23.393 ms > 6 * * * > 7 * * * > 8 * * * > 9 * * * > 10 * * * > 11 * * * > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * > SiouxCenter-Arista-North(s1) > > The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. > > Regards, > > Frank Bulk > AS53347 > > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

Aug. 16, 2018

Re: Routing of non-IX traffic

by Jeremy Lumby

Why would your router ARP for an address that is not on the same subnet as any of your interfaces? From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Matthew Beckwell Sent: Thursday, August 16, 2018 5:01 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Routing of non-IX traffic I'm getting similar behavior as Frank. Like Doug, I only have 45.60.73.0/24 from transit connections. So a traceroute from my MICE interface should ARP and die (I would think).... When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): 00:23:33:c6:a0:c0206.108.255.50Cooperative Network Services (CNS)32609 e4:aa:5d:83:73:06206.108.255.47IVDesk393639 88:43:e1:00:f2:10206.108.255.18Consolidated Communications12042 b0:aa:77:33:7b:03206.108.255.79Gigamonster, LLC31939 3c:08:f6:81:6e:a5206.108.255.46OneNetUSA46131 00:1d:e5:c0:78:c3206.108.255.5Implex21709 54:75:d0:e6:08:30206.108.255.106Nuvera Communications23465 00:11:5d:82:6c:00206.108.255.80Future Technologies26451 Proxy ARP (or something like it)? CNS seems to be consistently coming in first place when I clear my ARP entry. ~Matthew matthewb(a)aitech.net AS13746 On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net (205.149.159.233) 22.669 ms cra.PaulBunyan.net (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

Aug. 16, 2018

Re: Routing of non-IX traffic

by Matthew Beckwell

I'm getting similar behavior as Frank. Like Doug, I only have 45.60.73.0/24 from transit connections. So a traceroute from my MICE interface should ARP and die (I would think).... When I traceroute to 45.60.73.16-- my router sends out an ARP request, as expected. But...I get ARP replies for 45.60.73.16 from these Cisco MACs (in the order they came into my interface): 00:23:33:c6:a0:c0 206.108.255.50 Cooperative Network Services (CNS) 32609 e4:aa:5d:83:73:06 206.108.255.47 IVDesk 393639 88:43:e1:00:f2:10 206.108.255.18 Consolidated Communications 12042 b0:aa:77:33:7b:03 206.108.255.79 Gigamonster, LLC 31939 3c:08:f6:81:6e:a5 206.108.255.46 OneNetUSA 46131 00:1d:e5:c0:78:c3 206.108.255.5 Implex 21709 54:75:d0:e6:08:30 206.108.255.106 Nuvera Communications 23465 00:11:5d:82:6c:00 206.108.255.80 Future Technologies 26451 Proxy ARP (or something like it)? CNS seems to be consistently coming in first place when I clear my ARP entry. ~Matthew matthewb(a)aitech.net AS13746 On Thu, Aug 16, 2018 at 3:25 PM, Frank Bulk <fbulk(a)mypremieronline.com> wrote: > When I force a traceroute to originate from our MICE-facing connection, > the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? > > To making things more interesting, Incapsula-destined traffic goes via > Paul Bunyan. Here's just one example: > > traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte > packets > 1 AS32609.micemn.net (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms > 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms > 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms > 4 crss2.PaulBunyan.net (205.149.159.197) 20.527 ms 20.562 ms 20.619 > ms > 5 cra.PaulBunyan.net (205.149.159.181) 23.398 ms > fp233.ips.PaulBunyan.net (205.149.159.233) 22.669 ms cra.PaulBunyan.net > (205.149.159.181) 23.393 ms > 6 * * * > 7 * * * > 8 * * * > 9 * * * > 10 * * * > 11 * * * > 12 * * * > 13 * * * > 14 * * * > 15 * * * > 16 * * * > 17 * * * > 18 * * * > 19 * * * > 20 * * * > 21 * * * > 22 * * * > 23 * * * > 24 * * * > 25 * * * > 26 * * * > 27 * * * > 28 * * * > 29 * * * > 30 * * * > SiouxCenter-Arista-North(s1) > > The reason I stumbled across this is because we've had more than a dozen > customers over the last month complain about access to Incapsula-protected > sites. Packet captures show TCP RSTs coming from the far side. > > Regards, > > Frank Bulk > AS53347 >

Aug. 16, 2018

Re: Routing of non-IX traffic

by Steve Howard

Aug. 16, 2018

Re: Routing of non-IX traffic

by Doug McIntyre

On Thu, Aug 16, 2018 at 08:25:54PM +0000, Frank Bulk wrote: >When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? > >To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: > >traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte packets > 1 AS32609.micemn.net (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms > 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms > 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms > 4 crss2.PaulBunyan.net (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms > 5 cra.PaulBunyan.net (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net (205.149.159.233) 22.669 ms cra.PaulBunyan.net (205.149.159.181) 23.393 ms ... Hmm, I don't find that route available via MICE. I only see the route from my transit providers as 45.60.73.0/24 originating from AS19551. I do not learn it from MICE. That prefix isn't in the BIRD routing table. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 16, 2018

Re: Routing of non-IX traffic

by Jeremy Lumby

I am surprised that you are getting any traceroute at all. Your MICE facing interface IP should not exist on the public internet. I am wondering if there is a default route in someone's table (CNS/Paul Bunyan) (or they are originating the MICE subnet internally) that is enabling you to get as much of a trace as you are getting. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Frank Bulk Sent: Thursday, August 16, 2018 3:26 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Routing of non-IX traffic When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net (205.149.159.233) 22.669 ms cra.PaulBunyan.net (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347

Aug. 16, 2018

Re: Routing of non-IX traffic

by Steve Howard

Aug. 16, 2018

Routing of non-IX traffic

by Frank Bulk

When I force a traceroute to originate from our MICE-facing connection, the first hop is 206.108.255.50 (AS32609 aka CNS). Any reason why? To making things more interesting, Incapsula-destined traffic goes via Paul Bunyan. Here's just one example: traceroute to www.yamaha-dealers.com (45.60.73.16), 30 hops max, 60 byte packets 1 AS32609.micemn.net (206.108.255.50) 14.059 ms 14.084 ms 14.076 ms 2 cns70.cnsllc.net (205.149.150.9) 18.484 ms 18.434 ms 18.507 ms 3 fg30.ips.cnsllc.net (205.149.150.30) 20.254 ms 20.346 ms 20.267 ms 4 crss2.PaulBunyan.net (205.149.159.197) 20.527 ms 20.562 ms 20.619 ms 5 cra.PaulBunyan.net (205.149.159.181) 23.398 ms fp233.ips.PaulBunyan.net (205.149.159.233) 22.669 ms cra.PaulBunyan.net (205.149.159.181) 23.393 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * SiouxCenter-Arista-North(s1) The reason I stumbled across this is because we've had more than a dozen customers over the last month complain about access to Incapsula-protected sites. Packet captures show TCP RSTs coming from the far side. Regards, Frank Bulk AS53347

Aug. 16, 2018

Compudyne (AS47096) Network Maintenance

by Ryan Nelson

We will be migrating our MICE connection to a new router tomorrow. In preparation for this we will be shutting down all BGP sessions in the early AM and intend to have them back online by early evening. Thanks, and sorry for the short notice, Ryan Nelson | Network Infrastructure Architect Compudyne | 306 W Michigan Street - Suite 200 | Duluth, MN 55802 rnelson(a)compudyne.com<mailto:rnelson@compudyne.com> | 218.336.2220 | compudyne.com<http://www.compudyne.com/> Duluth | Hibbing | Saint Paul | Marquette | Houghton | Anchorage ________________________________ [cid:image002.png@01D0BA36.584BB020]

Aug. 14, 2018

Re: Facebook/Instagram

by Jay Hanke

I've been seeing slow loads from my computer and phone (on wifi) the past couple of days. Of course only a couple times since I'm normally working.... Jay On Thu, Aug 9, 2018 at 3:53 PM, AnthonyAnderberg(a)nuvera.net < AnthonyAnderberg(a)nuvera.net> wrote: > Has anyone else gotten reports of issues with Facebook/Instagram the last > day or two? We’re seeing intermittent problems where Facebook pages will > load normally but then some of the pictures/videos in the page will be slow > loading. I haven’t really narrowed it beyond a handful of source addresses > in Facebook’s network, mostly in the 157.240.2.0/24 range, and haven’t > seen any other service issues or narrowed it to any one upstream carrier. > > > > It doesn’t appear any of the content is coming through MICE, but I’m > curious if anyone is/has seen something similar. We’ve reached out to > Facebook but haven’t gotten a response yet. > > > > Thanks in advance, > > anthony > > > > [image: Anthony_Anderberg] <https://www.nuvera.net/> > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

Aug. 9, 2018

Facebook/Instagram

by AnthonyAnderberg＠nuvera.net

Has anyone else gotten reports of issues with Facebook/Instagram the last day or two? We're seeing intermittent problems where Facebook pages will load normally but then some of the pictures/videos in the page will be slow loading. I haven't really narrowed it beyond a handful of source addresses in Facebook's network, mostly in the 157.240.2.0/24 range, and haven't seen any other service issues or narrowed it to any one upstream carrier. It doesn't appear any of the content is coming through MICE, but I'm curious if anyone is/has seen something similar. We've reached out to Facebook but haven't gotten a response yet. Thanks in advance, anthony [Anthony_Anderberg]<https://www.nuvera.net/>

Aug. 9, 2018

Re: Please welcome Global Net to MICE!

by Doug McIntyre

On Tue, Jul 31, 2018 at 10:01:20PM -0500, Richard Laager wrote: >On 07/30/2018 02:32 PM, Jay Hanke wrote: >> Route Server: No > >I communicated with Global Net off-list. This is actually a "Yes" now. >They _do_ want to be setup on the route servers. I've added them to the route servers now, Welcome aboard! -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

Aug. 2, 2018

UG 29 Election

by Richard Laager

At UG 29 in October, we will be holding an election for a board seat. The board consists of three seats with staggered terms, such that one seat comes up for re-election each year (at the fall meeting). This year, Reid's seat is up for election. Reid _is_ running for re-election. If anyone else is interested in running, please nominate yourself. Voting will be done in person. If you will not be attending in person, you must designate a proxy, in advance, in writing. The bylaws state that you send that to the Chief Manager (me, Richard Laager), and electronic means are acceptable. So a simple email is sufficient. If you are having trouble finding a proxy who will be there in person, ask on the list or I can help you find someone. -- Richard

Aug. 1, 2018

Re: Please welcome Global Net to MICE!

by Richard Laager

On 07/30/2018 02:32 PM, Jay Hanke wrote: > Route Server: No I communicated with Global Net off-list. This is actually a "Yes" now. They _do_ want to be setup on the route servers. -- Richard

Aug. 1, 2018

Please welcome Global Net to MICE!

by Jay Hanke

Name: Global Net Port: Ethernet5/37 (Main Switch) Speed: 10G ASN: 395466 IPv4: 206.108.255.97/24 IPv6: 2001:504:27:0:6:08ca::1/64 Route Server: No BFD Support: No Peering Contact: peering(a)theglobal.net Please provide your Arista compatible optic to Cologix or a member of the tech committee. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 30, 2018

MICE BFD Settings

by AnthonyAnderberg＠nuvera.net

At the last UG meeting it was suggested that we document MICE's route server BFD settings, that is done now: http://www.micemn.net/technical.html Was good seeing everyone again, anthony [Anthony_Anderberg]<https://www.nuvera.net/>

July 25, 2018

Dyn blog post has some recommendations for IXPs who may experience bad actors

by Frank Bulk

The whole blog post I interesting, but the last fifth relates to IXPs: https://dyn.com/blog/shutting-down-the-bgp-hijack-factory/ Food for thought. Frank

July 12, 2018

Re: MICE UG 28

by Mike Horwath

On Fri, Jul 06, 2018 at 07:35:16PM -0500, Jay Hanke wrote: > I'm not going to make this one. Family vacation Cool. I'm in a different boat and fully scheduled up for the 11th :( -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

July 9, 2018

Re: MICE UG 28

by Jay Hanke

I'm not going to make this one. Family vacation On Fri, Jul 6, 2018 at 4:55 PM, Mike Horwath <drechsau(a)geeks.org> wrote: > On Fri, Jul 06, 2018 at 03:52:39PM -0500, Richard Laager wrote: >> Reminder: UG 28 is on Wednesday of next week (July 11) at 3:00pm >> >> Remote Conferencing: 952-562-5400 passcode MICE (6423) >> >> Location Details: >> >> Alerus Bank >> 3137 32nd Ave S >> 2nd Floor Conference Room >> Fargo, ND 58103 > > hey Jay - you driving up? > > -- > Mike Horwath, reachable via drechsau(a)Geeks.ORG -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

July 7, 2018