- mice-discuss - lists.micemn.net

Re: Zayo to Omaha IX down
by Justin Huffman Oct. 23, 2019

Oct. 23, 2019

Our link from Sioux City to Minneapolis went down around 2:10 today. This is getting out of hand. I think we've experienced outages every other week for almost 2 months now along with the outage lasting 16+ hours. Does anyone have recommendations for diverse routes up to 511? Justin Long Lines Broadband

1 0

Re: Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Thanks everyone. Another MICE member reached out to me directly and said they had a similar problem with this Brocade "feature" and advised me how to filter this unwanted traffic. The port now appears to be online and I can re-enable my peering tonight.

2 1

Re: Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Thanks, Jeremy. I was hoping you could provide just that sort of detail. I'm just not sure how there would be multiple coming from my side when this is a P2P link and the switch port is in route-only mode; I may have to work with Brocade on that. I will look into it, but can you tell me if the MAC addresses appear to be all from the same vendor?

2 1

Re: Problem with my connection to MICE
by Jeremy Lumby Oct. 22, 2019

Oct. 22, 2019

We are seeing Port security errors on your port for too many Mac addresses. The industry standard is one Mac address, however currently mice allows up to five. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Keith Anderson <keith.anderson(a)MARCONET.COM> Date: 10/22/19 8:33 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Problem with my connection to MICE _____ >From : Keith Anderson [keith.anderson(a)MARCONET.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, October 22 2019 08:23:08 Good morning all. I have a problem with my MICE connection and I'm wondering if anyone can give me some direction. On September 9th, my predecessor disabled our port facing MICE due to BGP flapping (his description). Since there was chatter on this board about a fiber cut that day, he thought it was related to that. Unfortunately, when I re-enabled the port, I saw similar behavior. BGP seems to flap about every three minutes, and my log indicates that link is actually dropping on that port. Even though light levels never seem to drop below -4.4db, I had Cologix check the cross connect, clean the ends of the fiber, and re-seat the sfp. None of that helped. On Wednesday I replaced the SFP, but when I enabled the port, I still saw the same behavior. Last night I disabled all of my BGP peers and brought the port back online for further layer2 testing, but I can't ping any of my peers and I'm not seeing any arp entries on this port. Anyone have any ideas what could be going on or what I should check next? I've included my port config below for reference. The log entries associated with this problem are: Oct 22 08:17:33:I:SYSTEM: port 2/4 link fault status is cleared(physical link goes down 24) Oct 22 08:17:33:I:SYSTEM: port 2/4 is down( local fault) Oct 22 08:17:33:I:System: Interface ethernet 2/4, state down - local fault Oct 22 08:17:09:I:System: Interface ethernet 2/4, state up Oct 22 08:16:45:W: Latched low RX Power warning, port 2/4 Oct 22 08:16:45:A: Latched low RX Power alarm, port 2/4 interface ethernet 2/4 port-name BGP peering with MICE Arista 5/19 XC23175 load-interval 30 no fdp enable no cdp enable route-only ip address xxx.xxx.xxx.xxx/24 no spanning-tree sflow forwarding !

1 0

Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Good morning all. I have a problem with my MICE connection and I'm wondering if anyone can give me some direction. On September 9th, my predecessor disabled our port facing MICE due to BGP flapping (his description). Since there was chatter on this board about a fiber cut that day, he thought it was related to that. Unfortunately, when I re-enabled the port, I saw similar behavior. BGP seems to flap about every three minutes, and my log indicates that link is actually dropping on that port. Even though light levels never seem to drop below -4.4db, I had Cologix check the cross connect, clean the ends of the fiber, and re-seat the sfp. None of that helped. On Wednesday I replaced the SFP, but when I enabled the port, I still saw the same behavior. Last night I disabled all of my BGP peers and brought the port back online for further layer2 testing, but I can't ping any of my peers and I'm not seeing any arp entries on this port. Anyone have any ideas what could be going on or what I should check next? I've included my port config below for reference. The log entries associated with this problem are: Oct 22 08:17:33:I:SYSTEM: port 2/4 link fault status is cleared(physical link goes down 24) Oct 22 08:17:33:I:SYSTEM: port 2/4 is down( local fault) Oct 22 08:17:33:I:System: Interface ethernet 2/4, state down - local fault Oct 22 08:17:09:I:System: Interface ethernet 2/4, state up Oct 22 08:16:45:W: Latched low RX Power warning, port 2/4 Oct 22 08:16:45:A: Latched low RX Power alarm, port 2/4 interface ethernet 2/4 port-name BGP peering with MICE Arista 5/19 XC23175 load-interval 30 no fdp enable no cdp enable route-only ip address xxx.xxx.xxx.xxx/24 no spanning-tree sflow forwarding !

1 0

2019 Election Results
by AnthonyAnderberg＠nuvera.net Oct. 17, 2019

Oct. 17, 2019

Richard was re-elected unanimously, thanks for your continued service! [cid:60F64B08-B963-4E01-B287-F9A49C3DDDFD-L0-001] Sent from my iPhone

2 1

MICE UG 33
by Richard Laager Oct. 14, 2019

Oct. 14, 2019

Reminder: UG 33 is on Wednesday of this week (October 16) at 3:00pm Remote Conferencing: 952-562-5400 passcode MICE (6423) Location: 511 Building Cafeteria Agenda The agenda is attached. I'm behind on getting this out, so I didn't send a separate call for agenda. If you have something to add, let me know ASAP and I'll send an updated agenda. I'd like to try the modified ordering again. The idea here is to group business (official reports and things that could involve a motion/vote) before open discussion. This way, people who are not interested in the discussion can leave earlier if they wish, confident that they won't miss an official motion/vote. By shortening the "mandatory" part of the meeting, this may make the meetings more accessible. I'd like to give this a try for a couple meetings and see what people think. The UG 32 draft minutes are also attached. -- Richard

1 0

Re: Zayo Maintenance??
by Tim Fjeld Oct. 14, 2019

Oct. 14, 2019

Zayo was in our DC Tuesday afternoon. They were working their way north from Omaha testing fibers and verifying connections in effort to finalize turn up of their new transport equipment. I would imagine the outage might be related to that project, but cannot say with certainty. Tim Fjeld City of Windom

6 10

MN VoIP Remote Upgrade
by Jeremy Lumby Oct. 11, 2019

Oct. 11, 2019

In accordance with the MICE remote switch operator guidelines, MN VoIP will be upgrading their remote switch and uplink to the MICE core on the afternoon of October 11th at approx 2PM to keep uplink utilization under 66% of the link capacity. Prior to the maintenance, BGP will be blocked so traffic can gracefully reroute. During the maintenance connectivity will be down for approximately 15 minutes. A 100G switch will be added between the existing MN VoIP extension and the MICE core. This switch will not only increase the MN VoIP uplink from 40G to 100G, it will also make 32 100G ports available to anyone wishing to connect to MICE via MN VoIP @ 100G. I will send out a notification to the discuss list right before I commence with the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

3 4

Zayo Maintenance??
by James Urwiller Oct. 10, 2019

Oct. 10, 2019

Does anyone know if Zayo is doing maintenance ? I find it strange that my circuit dropped at 00:04 CST. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300]

3 3

MICE 2019 Board Election
by Richard Laager Oct. 4, 2019

Oct. 4, 2019

The MICE board consists of three directors with staggered 3-year terms. One position is up for election each year at our fall meeting. This year's fall meeting, UG 33, will be held on October 16 at the 511 Building. The current seat up for election is the one that I hold. At this time, the board would like to open nominations. If you or someone you know is interested in running, please reply on the MICE-DISCUSS list or contact any board member. Nominations will close September 30th. -- Richard

2 2

Please welcome Land O'Lakes to MICE!
by Jay Hanke Oct. 1, 2019

Oct. 1, 2019

Name: Land O Lakes Port: Ethernet 4/39 Main Switch ASN: 14161 IPv4: 206.108.255.145/24 IPv6: 2001:504:27:0:0:3751::1/64 Peering Contact: kghugill(a)landolakes.com Route Servers- YES BFD: NO Please provide your optic to either a tech committee member (Jeremy) or to Cologix when you place your cross connect order! Also, if you haven't already please join the MICE mailing lists at micemn.net Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 26, 2019

Sept. 26, 2019

latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

4 5

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Current update 9:51 PM: ETR 2-3 hours

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo is responding to our ticket about every hour with the standard general response "Issues are still currently being investigated by the field teams on site"

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo ticket update: Wed 9/25/2019 1:31 PM Current update, Zayo OSP is on site searching for the damage, Splicers will be on site in 1 hour, Underground construction team is on site with dig equipment and Trailer with conduit and fiber expected in 90 minutes We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

3 2

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo Ticket update:Current update, Zayo crews are currently dispatching to County Rd 35 near Roberts Ave Worthington, MN as this looks to be the location of the cut, OSP is en route and splice teams are also being dispatched. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

2 1

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Our Zayo ticket case comments from Zayo: Wed 9/25/2019 9:51AM Your case has been associated to a higher level outage in the Zayo network affecting MN please standby for further updates Wed 9/25/2019 10:51AM Issues are still currently under investigation with our field teams

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

We show on graphing that Zayo is down hard to Omaha from Windom at 8:10am. Put a ticket in with Zayo and offered to OTDR from Windom south towards Omaha. Jeff Dahna Windomnet

3 3

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

I received word from another company on the Zayo Omaha route, that Zayo thinks it is a possible fiber cut in the Worthington, MN area. Zayo fiber crew going to investigate.

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

We cannot see the port at Bigelow, MN Zayo hut. So I'd say it is between Windom and Bigelow. Fiber cut or equipment failure/fat finger. Jeff Dahna Windomnet

1 0

Anyone else see their Zayo-based link to MICE go down?
by Frank Bulk Sept. 25, 2019

Sept. 25, 2019

Our 10G dropped at 8:10 am. Frank

6 6

Anyone else down to Belle Plain since 3:45 pm?
by Frank Bulk Sept. 23, 2019

Sept. 23, 2019

Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

8 19

Zayo Bell Plaine
by Jeremy Lumby Sept. 20, 2019

Sept. 20, 2019

Just an FYI, I have received temperature alarms from the Zayo hut in Belle Plaine again. A ticket has been opened with Zayo Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

BGP Optimizers
by David Farmer Sept. 19, 2019

Sept. 19, 2019

I found an interesting article in my LinkedIn feed last night on BGP Optimizers; https://www.itnews.com.au/news/bgp-optimisers-seem-a-good-idea-until-they-b… ? I'd be interesting if anyone in the MICE community is using a BGP Optimizer? Especially one that generates more specific prefixes in BGP. I don't want to expose anyone to ridicule, so please don't go there if anyone fesses up, even in jest, this needs to be treated seriously. However, if anyone is using a BGP Optimizer, especially one generating more specific prefixes, I think it would behoove the MICE community to put in extra defenses against propagating these more specific prefixes through the exchange and out to the Internet in general or even our own downstream customers. For example we could create a MICE-DROP BGP Community that we can tag any routes that should be dropped if they are (accidentally) announced to the MICE route server or to other MICE peers, such as these more specific routes created by a BGP Optimizer. Basically we would each add something to our routing policy, and on the MICE route servers too, looking for that BGP Community and immediately dropping any routes tag with it. Also, if anyone is using a BGP Optimizer in our community that would be a very good reason to accelerate IRR based router filtering for our exchange. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

5 6