- mice-discuss - lists.micemn.net

Nuvera Port Upgrade
by Jay Hanke Dec. 20, 2019

Dec. 20, 2019

Nuvera will be upgrading to 2x10G LAG on new ports and then removing their existing port. The new port assignments are (Main Switch) Ethernet 4/28 Ethernet 5/28 This port will be vacated Ethernet 4/15 Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

3 2

MICE MAC Address Limit
by Richard Laager Dec. 14, 2019

Dec. 14, 2019

We have discussed this a bit in the past, and this came up at the last UG. I'm looking for feedback from the group before formally proposing this to the board for a vote. I first brought this up to the technical committee, CCing the board. I have heard no objections. Jeremy is "in favor of it 100%". Based on our quick conversation after the UG, I _think_ Anthony is in favor as well; we both made notes to follow up on this. ----- I am proposing that MICE change its MAC address limit from 5 MACs per port to 1 MAC per port. 1 MAC address is enough for normal scenarios and this would further limit the potential for problems on the fabric. This restriction is one that SIX and AMS-IX both have, with the latter being known for their excellent configuration guide for participants: https://www.ams-ix.net/ams/documentation/config-guide To be clear, this is still only for end ports, not ports facing remote switches, of course. The remote switches are responsible for enforcing the current MAC limit, and would be responsible for changing those settings as described here. If someone is swapping their MICE-facing router, the resulting port flap on the MICE/remote switch will clear the limit anyway, if they are directly connected. If they are unable to flap the port (e.g. because they have someone else's layer 2 gear in the middle), they will have to either work with that carrier or their switch operator (MICE/remote) to flap the port. This is a non-zero burden, but I think it's pretty minimal in practice. Additionally, if someone plans to make an equipment swap, we would increase the limit in advance, temporarily, upon request. For reference, SIX goes further and locks you to a particular MAC address in an ACL, so you have to contact them to change your MAC. I am not proposing that. Still, I've been through that a couple of times, and even that isn't really too big of a deal. So I don't think the change to a limit of 1 MAC at MICE will be particularly onerous. There are a couple of participants who have two IP address sets (one IPv4 and one IPv6) on the same port. For those ports, the limit obviously must be at least 2 MACs, but I propose 3 MACs as the limit. If they're using two IPs, they probably are doing so to get some redundancy out of it. For this particular use case, needing to flap the port defeats that redundancy goal, as it breaks the other router. Having a MAC limit of 3 would allow them to swap a device without needing to flap the port. We would grandfather these setups until they are looking for a port-type upgrade; at that point they would need to get to one IP address set per port (by splitting into two ports or reducing to one IP address set) or request an exception as outlined below. The exception process would allow us to better evaluate this use case at that time. There are some members who have multiple MACs showing up now, despite using only one device. For those, I am proposing we set the MAC limit to however many MACs are currently showing up on their port. That is, they would be grandfathered at their current situation for now. At the time of this change, we would encourage them to investigate and fix the multiple MACs issue at their leisure. In the future, if they are doing a equipment swap (especially like-for-like), we would again encourage them to address it, but would otherwise leave their higher limit in place. If they do a port-type upgrade, we would not bring the grandfathering over. If they needed, they could ask for an official exception at that time. There may be cases where people cannot reasonably fix their routers to use only 1 MAC, other scenarios I haven't considered, or things that may come up in the future. For that reason, I think we should allow for the possibility of exceptions upon request to the board. In practice, I would expect the board would confer with the technical committee and/or the technical committee would be bringing these to the board anyway. The goal is to strike a reasonable balance between protecting the fabric without preventing networks from connecting. -- Richard

2 2

NebraskaLink Port Move
by Jay Hanke Dec. 12, 2019

Dec. 12, 2019

Nebraskalink is moving their connection from the Zayo remote switch to the main switch. All IP address/Contact information will remain the same. The new port assignment is Main Switch: Ethernet4/13 Welcome aboard again! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

1 0

MICE graphs - gap from today?
by Andrew Hoyos Dec. 4, 2019

Dec. 4, 2019

Hi folks, Looks like the Cacti graphs are showing a gap from this afternoon. Anyone know what happened? — Andrew Hoyos hoyosa(a)gmail.com <mailto:hoyosa@gmail.com>

2 1

Internet Wether Report
by David Farmer Dec. 3, 2019

Dec. 3, 2019

Anyone notice the somewhat impressive approximately 100G traffic spike on MICE just after 12 noon today, I'm told that is Halo for XBox One. https://www.express.co.uk/entertainment/gaming/1211738/Halo-Reach-Release-D… -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

3 2

Any MICE bottleneck for Google Stadia?
by Jon Westlund Dec. 2, 2019

Dec. 2, 2019

I am "testing" Stadia...just to confirm what our customers will experience :) My Stadia stream is sourced from the 136.112.0.0/12 prefix. There is considerable lag/disconnects during gameplay...I have seen the reports of this being a widespread issue with the new platform, but wanted to confirm that there is no bottleneck with MICE connectivity to Google - 206.108.255.141 AS15169. Can we confirm there is no saturation of their uplinks? Has there been a noticeable spike since Friday when some users received their Stadia system? Thanks, JW

2 1

HBC Port Augment/LAG conversion
by Jay Hanke Nov. 25, 2019

Nov. 25, 2019

Ethernet4/11 has been allocated to HBC. Their existing port (e5/11) will need to be converted to a LAG and then e4/11 added into the group. Welcome aboard (again)! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

packet loss today
by James Urwiller Nov. 23, 2019

Nov. 23, 2019

Greetings all, I have been noticing packet loss today similar to how it was a few weeks back before Zayo replaced a card. I know they had a fiber cut yesterday in Omaha south of where I hop on to get to the exchange, I was unaffected by that but today I just have a lot of packet loss. Anyone else? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300]

4 11

Please Welcome PhirePhly Design to MICE!
by Jay Hanke Nov. 21, 2019

Nov. 21, 2019

Name: PhirePhly Design Port: Ethernet 11 MN Voip Remote Switch ASN: 7034 IPv4: 206.108.255.152/24 IPv6: 2001:504:27:0:0:1b7a::1/64 Peering Contact: kenneth(a)phirephly.design Route Servers- YES BFD: No Welcome Aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Additional WIN 10G Port
by Jay Hanke Nov. 20, 2019

Nov. 20, 2019

Port 4/43 on the main switch has been allocated to WIN. Please add this port to their existing LAG group. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

MICE Limelight Upgrade/Route Server addition
by Jay Hanke Nov. 19, 2019

Nov. 19, 2019

Limelight is moving from their existing 2x10G LAG to 1x100G LAG on main switch port 3/15/1. The existing 10G ports will be vacated. This move was competed this afternoon. Limelight is also adding an additional 100G port to the MN VoIP remote switch with an additional IP assignment. Name: Limelight (New additional) Port: MN VoIP Port 2 (100G Switch) ASN: 22882 IPv4: 206.108.255.151 IPv6: 2001:504:27:0:0:5926::2 Limelight has also requested RS sessions for both of their IP assignments. The following Limelight IPs need RS sessions created or verified IPv4: 206.108.255.43 IPv6: 2001:504:27:0:0:5926::1 Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

3 2

Re: Google Issues?
by Jeff Dahna Nov. 19, 2019

Nov. 19, 2019

Apparently it is a fiber cut on Level3 as per previous message.

2 1

Google Issues?
by Nathan Beard Nov. 19, 2019

Nov. 19, 2019

All, Google peering sessions via MICE have been down for nearly 8 hours at this point for multiple customers including Stellar Association. Is anybody aware of an issue? I don't recall seeing an update for any work being completed but it's possible that I missed something... *Nathan Beard* *7Sigma Systems, Inc.* Network Engineer (218) 820-7636 www.7sigma.com

2 1

Re: Google
by Jeff Dahna Nov. 18, 2019

Nov. 18, 2019

ETA= Estimated Time of Arrival. It is going to be awhile before it is restored. ETR= Estimated Time of Restoration

1 0

Google
by Dean Bahls Nov. 18, 2019

Nov. 18, 2019

Any word on what's going on at Google? I'm guessing someone had a bad day. Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>

5 7

Please Welcome SBT to MICE!
by Jay Hanke Nov. 18, 2019

Nov. 18, 2019

Name: SBT, Int Port: Ethernet 4 MNVoip Remote Switch ASN: 11255 IPv4: 206.108.255.149/24 IPv6: 2001:504:27:0:0:2bf7::1/64 Peering Contact: peering(a)sbt.net Route Servers- YES BFD: YES Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Re: random packet loss since yesterday afternoon anyone?
by Jeremy Lumby Nov. 7, 2019

Nov. 7, 2019

I got the same update. I did see loss last night at Peak utilization, however it was nowhere near as bad as the previous night. I am also waiting on notification for them to swap the cards in the cyan gear Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Date: 11/5/19 9:26 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? _____ >From : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, November 5 2019 09:26:57 Last ticket I received they traced out some other customers path (not mine) and told me there was no common path, but they wanted to swap a cyan transport card anyway, and would schedule the maintenance “soon” However I think its better-ish now?? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com stealth300 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Tuesday, November 5, 2019 09:17 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We also opened up a case with Zayo however they put a loopback on the circuit and did not see any errors apparently. Are you all still seeing loss? If so, what did Zayo come up with? Thanks in advance, Justin From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 11:01 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. The last few updates I had were that they dispatched a tech to Belle Plaine, then they moved the ticket back to the backbone group who engaged the hardware vendor. I still do not have an ETR, however yesterday I did see a little bit of loss at 8AM, however it got much worse as the traffic to MICE increased in the evening. Currently I am seeing loss, however it is much less than Sunday evening. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 10:43 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Only updates I have on my ticket is that a tech is investigating.. it has certainly been sporadic for us. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com stealth300 From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Justin Huffman Sent: Monday, November 4, 2019 10:30 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? We have had a few customers complain about packet loss to google.com today as well and was able to see packet loss from one of our routers up to 511 building. However, I don’t see packet loss anymore, but maybe it is really intermittent. Did you hear any update from Zayo? Justin Long Lines Broadband From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Monday, November 4, 2019 8:36 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. I am seeing it on the Zayo path between Belle Plaine, and Minneapolis. I have a ticket open since yesterday afternoon, and they appear to be struggling to nail it down. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Monday, November 04, 2019 8:33 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? Greetings all, Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com stealth300 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

3 6

random packet loss since yesterday afternoon anyone?
by James Urwiller Nov. 5, 2019

Nov. 5, 2019

Greetings all, Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300]

3 6

Fwd: [arin-announce] Vote Now in the 2019 ARIN Board of Trustees, ARIN Advisory Council, and NRO NC Elections
by David Farmer Oct. 31, 2019

Oct. 31, 2019

Voting has opened for ARIN and will remain open to 6:00 PM ET, Friday, 8 November. Please take the time to vote if you are a designated voting contact for ARIN. I would like to call your attention to a candidate from our community For ARIN Advisory Council; Jeffrey Anderson, Implex.net If you contact me directly, I'm happy to make other recommendations to fill out the slate if you wish. Again Please vote if you are eligible. Thanks ---------- Forwarded message --------- From: ARIN <info(a)arin.net> Date: Thu, Oct 31, 2019 at 3:03 PM Subject: [arin-announce] Vote Now in the 2019 ARIN Board of Trustees, ARIN Advisory Council, and NRO NC Elections To: <arin-announce(a)arin.net> Eligible Voting Contacts from General Members in Good Standing may now cast online ballots in the 2019 ARIN Elections to fill two seats on the ARIN Board of Trustees, five seats on the ARIN Advisory Council, and one seat on the Number Resource Organization Number Council (NRO NC). Online ballots may be cast now through 6:00 PM ET, Friday, 8 November. To vote, simply log in to ARIN Online and look for the “Vote Now” link on your dashboard. To view candidate biographies, please view the ARIN Elections 2019 Voter Guide at: https://www.arin.net/participate/oversight/elections/candidate_bios.pdf You may view video of candidate speeches and the Board Candidate forum that took place at ARIN 44 starting on late Friday afternoon, 1 November, at: https://www.arin.net/participate/oversight/elections/ To view or submit a Statement of Support, please visit the link below. Anyone, regardless of voter status, is eligible to submit a Statement of Support for a candidate. https://www.bigpulse.com/p56129/ During the week of 4 November, all eligible Voting Contacts should receive a personal call from an ARIN representative reminding them to please vote and to answer any election-related questions they may have. Participation in ARIN Elections requires only minutes of a voter’s time and is a key member responsibility. A single cast ballot provides eligible member organizations an opportunity to shape the future of ARIN, our community, and the Internet. For questions about voting, or if you encounter an issue with the election system, please contact a member of the Member Services team immediately via email at members(a)arin.net. You may also submit a question via ARIN Online and direct it to Meetings/Elections. Thank you for your participation in the 2019 ARIN Elections! Regards, Wendy Leedy Member Engagement Specialist American Registry for Internet Numbers (ARIN) _______________________________________________ ARIN-Announce You are receiving this message because you are subscribed to the ARIN Announce Mailing List (ARIN-announce(a)arin.net) Unsubscribe or manage your mailing list subscription at: https://lists.arin.net/mailman/listinfo/arin-announce Please contact info(a)arin.net if you experience any issues. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

1 0

Please welcome Omnitel to MICE!
by Jay Hanke Oct. 31, 2019

Oct. 31, 2019

Name: Omnitel Communications Port: Ethernet 5 SFN Remote Switch ASN: 62886 IPv4: 206.108.255.148/24 IPv6: 2001:504:27:0:0:f5a6::1/64 Peering Contact: it(a)omnitel.biz Route Servers- YES BFD: YES Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Traffic Requirements
by Jeremy Lumby Oct. 29, 2019

Oct. 29, 2019

I am wondering if many MICE members have been denied peering with other networks on MICE because you do not meet their traffic level requirements. I have, and I am brainstorming some options. Let me know off list if you fall into this category, and if you want to explore a potential solution with me. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

Please welcome Apple to MICE!
by Jay Hanke Oct. 25, 2019

Oct. 25, 2019

Name: Apple Ports: Ethernet 3/14/1 (100G LAG #1), Ethernet 4/49/1 (100G LAG#2) ASN: 714 (LAG Group #1) IPv4: 206.108.255.146/24 IPv6: 2001:504:27:0:0:2ca::1/64 (LAG Group #2) IPv4: 206.108.255.147/24 IPv6: 2001:504:27:0:0:2ca::2/64 Peering Contact: peering(a)group.apple.com Route Servers: NO BFD: NO Please note: Ethernet 4/49/1 has been reclaimed from a member that failed to migrate and did not respond. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

3 2

Re: Zayo to Omaha IX down
by Jeff Dahna Oct. 24, 2019

Oct. 24, 2019

288 ct fiber damaged near Worthington, MN. Zayo ticket 0646hrs - Current Update: Crews are finishing boring on one end and should have the fiber tails in the trailer in about an hour. On the other side they have fiber tails in the trailer beginning to prep.

2 1

Zayo to Omaha IX down
by Jeff Dahna Oct. 24, 2019

Oct. 24, 2019

We lost our route to Omaha IX via Zayo around 2:04 PM CST. Got a ticket in with Zayo. Anyone else down to Omaha via Zayo? Jeff Dahna Windomnet

4 4

Re: Zayo to Omaha IX down
by Justin Huffman Oct. 23, 2019

Oct. 23, 2019

Our link from Sioux City to Minneapolis went down around 2:10 today. This is getting out of hand. I think we've experienced outages every other week for almost 2 months now along with the outage lasting 16+ hours. Does anyone have recommendations for diverse routes up to 511? Justin Long Lines Broadband

1 0

Re: Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Thanks everyone. Another MICE member reached out to me directly and said they had a similar problem with this Brocade "feature" and advised me how to filter this unwanted traffic. The port now appears to be online and I can re-enable my peering tonight.

2 1

Re: Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Thanks, Jeremy. I was hoping you could provide just that sort of detail. I'm just not sure how there would be multiple coming from my side when this is a P2P link and the switch port is in route-only mode; I may have to work with Brocade on that. I will look into it, but can you tell me if the MAC addresses appear to be all from the same vendor?

2 1

Re: Problem with my connection to MICE
by Jeremy Lumby Oct. 22, 2019

Oct. 22, 2019

We are seeing Port security errors on your port for too many Mac addresses. The industry standard is one Mac address, however currently mice allows up to five. Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Keith Anderson <keith.anderson(a)MARCONET.COM> Date: 10/22/19 8:33 AM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Problem with my connection to MICE _____ >From : Keith Anderson [keith.anderson(a)MARCONET.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Tuesday, October 22 2019 08:23:08 Good morning all. I have a problem with my MICE connection and I'm wondering if anyone can give me some direction. On September 9th, my predecessor disabled our port facing MICE due to BGP flapping (his description). Since there was chatter on this board about a fiber cut that day, he thought it was related to that. Unfortunately, when I re-enabled the port, I saw similar behavior. BGP seems to flap about every three minutes, and my log indicates that link is actually dropping on that port. Even though light levels never seem to drop below -4.4db, I had Cologix check the cross connect, clean the ends of the fiber, and re-seat the sfp. None of that helped. On Wednesday I replaced the SFP, but when I enabled the port, I still saw the same behavior. Last night I disabled all of my BGP peers and brought the port back online for further layer2 testing, but I can't ping any of my peers and I'm not seeing any arp entries on this port. Anyone have any ideas what could be going on or what I should check next? I've included my port config below for reference. The log entries associated with this problem are: Oct 22 08:17:33:I:SYSTEM: port 2/4 link fault status is cleared(physical link goes down 24) Oct 22 08:17:33:I:SYSTEM: port 2/4 is down( local fault) Oct 22 08:17:33:I:System: Interface ethernet 2/4, state down - local fault Oct 22 08:17:09:I:System: Interface ethernet 2/4, state up Oct 22 08:16:45:W: Latched low RX Power warning, port 2/4 Oct 22 08:16:45:A: Latched low RX Power alarm, port 2/4 interface ethernet 2/4 port-name BGP peering with MICE Arista 5/19 XC23175 load-interval 30 no fdp enable no cdp enable route-only ip address xxx.xxx.xxx.xxx/24 no spanning-tree sflow forwarding !

1 0

Problem with my connection to MICE
by Keith Anderson Oct. 22, 2019

Oct. 22, 2019

Good morning all. I have a problem with my MICE connection and I'm wondering if anyone can give me some direction. On September 9th, my predecessor disabled our port facing MICE due to BGP flapping (his description). Since there was chatter on this board about a fiber cut that day, he thought it was related to that. Unfortunately, when I re-enabled the port, I saw similar behavior. BGP seems to flap about every three minutes, and my log indicates that link is actually dropping on that port. Even though light levels never seem to drop below -4.4db, I had Cologix check the cross connect, clean the ends of the fiber, and re-seat the sfp. None of that helped. On Wednesday I replaced the SFP, but when I enabled the port, I still saw the same behavior. Last night I disabled all of my BGP peers and brought the port back online for further layer2 testing, but I can't ping any of my peers and I'm not seeing any arp entries on this port. Anyone have any ideas what could be going on or what I should check next? I've included my port config below for reference. The log entries associated with this problem are: Oct 22 08:17:33:I:SYSTEM: port 2/4 link fault status is cleared(physical link goes down 24) Oct 22 08:17:33:I:SYSTEM: port 2/4 is down( local fault) Oct 22 08:17:33:I:System: Interface ethernet 2/4, state down - local fault Oct 22 08:17:09:I:System: Interface ethernet 2/4, state up Oct 22 08:16:45:W: Latched low RX Power warning, port 2/4 Oct 22 08:16:45:A: Latched low RX Power alarm, port 2/4 interface ethernet 2/4 port-name BGP peering with MICE Arista 5/19 XC23175 load-interval 30 no fdp enable no cdp enable route-only ip address xxx.xxx.xxx.xxx/24 no spanning-tree sflow forwarding !

1 0

2019 Election Results
by AnthonyAnderberg＠nuvera.net Oct. 17, 2019

Oct. 17, 2019

Richard was re-elected unanimously, thanks for your continued service! [cid:60F64B08-B963-4E01-B287-F9A49C3DDDFD-L0-001] Sent from my iPhone

2 1

MICE UG 33
by Richard Laager Oct. 15, 2019

Oct. 15, 2019

Reminder: UG 33 is on Wednesday of this week (October 16) at 3:00pm Remote Conferencing: 952-562-5400 passcode MICE (6423) Location: 511 Building Cafeteria Agenda The agenda is attached. I'm behind on getting this out, so I didn't send a separate call for agenda. If you have something to add, let me know ASAP and I'll send an updated agenda. I'd like to try the modified ordering again. The idea here is to group business (official reports and things that could involve a motion/vote) before open discussion. This way, people who are not interested in the discussion can leave earlier if they wish, confident that they won't miss an official motion/vote. By shortening the "mandatory" part of the meeting, this may make the meetings more accessible. I'd like to give this a try for a couple meetings and see what people think. The UG 32 draft minutes are also attached. -- Richard

1 0

Re: Zayo Maintenance??
by Tim Fjeld Oct. 14, 2019

Oct. 14, 2019

Zayo was in our DC Tuesday afternoon. They were working their way north from Omaha testing fibers and verifying connections in effort to finalize turn up of their new transport equipment. I would imagine the outage might be related to that project, but cannot say with certainty. Tim Fjeld City of Windom

6 10

MN VoIP Remote Upgrade
by Jeremy Lumby Oct. 11, 2019

Oct. 11, 2019

In accordance with the MICE remote switch operator guidelines, MN VoIP will be upgrading their remote switch and uplink to the MICE core on the afternoon of October 11th at approx 2PM to keep uplink utilization under 66% of the link capacity. Prior to the maintenance, BGP will be blocked so traffic can gracefully reroute. During the maintenance connectivity will be down for approximately 15 minutes. A 100G switch will be added between the existing MN VoIP extension and the MICE core. This switch will not only increase the MN VoIP uplink from 40G to 100G, it will also make 32 100G ports available to anyone wishing to connect to MICE via MN VoIP @ 100G. I will send out a notification to the discuss list right before I commence with the maintenance. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

3 4

Zayo Maintenance??
by James Urwiller Oct. 10, 2019

Oct. 10, 2019

Does anyone know if Zayo is doing maintenance ? I find it strange that my circuit dropped at 00:04 CST. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com [image: stealth300]

3 3

MICE 2019 Board Election
by Richard Laager Oct. 4, 2019

Oct. 4, 2019

The MICE board consists of three directors with staggered 3-year terms. One position is up for election each year at our fall meeting. This year's fall meeting, UG 33, will be held on October 16 at the 511 Building. The current seat up for election is the one that I hold. At this time, the board would like to open nominations. If you or someone you know is interested in running, please reply on the MICE-DISCUSS list or contact any board member. Nominations will close September 30th. -- Richard

2 2

Please welcome Land O'Lakes to MICE!
by Jay Hanke Oct. 1, 2019

Oct. 1, 2019

Name: Land O Lakes Port: Ethernet 4/39 Main Switch ASN: 14161 IPv4: 206.108.255.145/24 IPv6: 2001:504:27:0:0:3751::1/64 Peering Contact: kghugill(a)landolakes.com Route Servers- YES BFD: NO Please provide your optic to either a tech committee member (Jeremy) or to Cologix when you place your cross connect order! Also, if you haven't already please join the MICE mailing lists at micemn.net Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

2 1

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 26, 2019

Sept. 26, 2019

latest Zayo update Wed 9/25/2019 4:16 PM Current update Crews have located the damage and are waiting on 811 to perform emergency locates, Estimating 60 minutes. Once the locates are done crews will be digging up 100 ft on each side of of the fiber. The reason for the extra digging is the cable is near a high pressure gas main and crews must be extra careful. ETR is 6 hours. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

4 5

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 26, 2019

Sept. 26, 2019

Current update 9:51 PM: ETR 2-3 hours

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo is responding to our ticket about every hour with the standard general response "Issues are still currently being investigated by the field teams on site"

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo ticket update: Wed 9/25/2019 1:31 PM Current update, Zayo OSP is on site searching for the damage, Splicers will be on site in 1 hour, Underground construction team is on site with dig equipment and Trailer with conduit and fiber expected in 90 minutes We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

3 2

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Zayo Ticket update:Current update, Zayo crews are currently dispatching to County Rd 35 near Roberts Ave Worthington, MN as this looks to be the location of the cut, OSP is en route and splice teams are also being dispatched. We apologize for any inconveniences and delays along the way and will continue to work towards resolution and provide updates as they are available. Regards Joshua Johnson NCC Technician (Fiber & Facilities) | Zayo Group

2 1

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

Our Zayo ticket case comments from Zayo: Wed 9/25/2019 9:51AM Your case has been associated to a higher level outage in the Zayo network affecting MN please standby for further updates Wed 9/25/2019 10:51AM Issues are still currently under investigation with our field teams

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

We show on graphing that Zayo is down hard to Omaha from Windom at 8:10am. Put a ticket in with Zayo and offered to OTDR from Windom south towards Omaha. Jeff Dahna Windomnet

3 3

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

I received word from another company on the Zayo Omaha route, that Zayo thinks it is a possible fiber cut in the Worthington, MN area. Zayo fiber crew going to investigate.

1 0

Re: Anyone else see their Zayo-based link to MICE go down?
by Jeff Dahna Sept. 25, 2019

Sept. 25, 2019

We cannot see the port at Bigelow, MN Zayo hut. So I'd say it is between Windom and Bigelow. Fiber cut or equipment failure/fat finger. Jeff Dahna Windomnet

1 0

Anyone else see their Zayo-based link to MICE go down?
by Frank Bulk Sept. 25, 2019

Sept. 25, 2019

Our 10G dropped at 8:10 am. Frank

6 6

Anyone else down to Belle Plain since 3:45 pm?
by Frank Bulk Sept. 24, 2019

Sept. 24, 2019

Bounce and then down. Sep 9 15:45:21 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:19 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:37 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Cleared), Link (Failure), Link failure, 2019-09-09 08:45:27 PM (GMT), to Belle Plaine via WIT for MICE Sep 9 15:45:52 Fibercomm, 10G Ethernet Term. ETH-1-1-4, Critical (Raised), Link (Failure), Link failure, 2019-09-09 08:45:49 PM (GMT), to Belle Plaine via WIT for MICE Frank

8 19

Zayo Bell Plaine
by Jeremy Lumby Sept. 20, 2019

Sept. 20, 2019

Just an FYI, I have received temperature alarms from the Zayo hut in Belle Plaine again. A ticket has been opened with Zayo Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

1 0

BGP Optimizers
by David Farmer Sept. 19, 2019

Sept. 19, 2019

I found an interesting article in my LinkedIn feed last night on BGP Optimizers; https://www.itnews.com.au/news/bgp-optimisers-seem-a-good-idea-until-they-b… ? I'd be interesting if anyone in the MICE community is using a BGP Optimizer? Especially one that generates more specific prefixes in BGP. I don't want to expose anyone to ridicule, so please don't go there if anyone fesses up, even in jest, this needs to be treated seriously. However, if anyone is using a BGP Optimizer, especially one generating more specific prefixes, I think it would behoove the MICE community to put in extra defenses against propagating these more specific prefixes through the exchange and out to the Internet in general or even our own downstream customers. For example we could create a MICE-DROP BGP Community that we can tag any routes that should be dropped if they are (accidentally) announced to the MICE route server or to other MICE peers, such as these more specific routes created by a BGP Optimizer. Basically we would each add something to our routing policy, and on the MICE route servers too, looking for that BGP Community and immediately dropping any routes tag with it. Also, if anyone is using a BGP Optimizer in our community that would be a very good reason to accelerate IRR based router filtering for our exchange. Thanks. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

5 6

Could I have a MICE admin look at the route reflectors?
by Frank Bulk Sept. 9, 2019

Sept. 9, 2019

For some reason at 5:38 pm we (AS18883, FiberNet) lost IPv4 and IPv6 peering with both of MICE RR's and Hurricane Electric and it's been unstable since then. All our other peers with MICE are fine. I didn't change any configuration, even though the "bad AS number" note in some of the error messages would suggest that. Any hints would be welcomed. I've shutdown those six sessions for now. Frank Sep 6 17:38:49 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:38:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:21 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:21 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:28 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:39:33 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:39:36 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:39:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:02 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:02 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:40:06 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:40:22 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:28 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:40:48 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:40:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:41:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:41:51 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:41:51 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:00 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:2 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:42:33 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 206.108.255.1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x000049c3 Sep 6 17:42:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.1 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Sep 6 17:42:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:1 (AS 53679) old state Established event RecvNotify new state Idle Sep 6 17:42:44 207.32.15.28 Rib: %BGP-3-NOTIFICATION: received from neighbor 2001:504:27::d1af:0:1 (AS 53679) 2/2 (Open Message Error/bad AS number) 4 bytes 0x0000d063 Sep 6 17:43:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::d1af:0:2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:44:36 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.2 (AS 53679) old state OpenConfirm event RecvKeepAlive new state Established ... Sep 6 17:44:34 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:44:39 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 17:46:15 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:54 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 17:50:57 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:00:44 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:00:49 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:04:53 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:11:03 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:11:06 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:13:05 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:13:13 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:14:37 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:14:41 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:15:35 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:23:11 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:23:16 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:26:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:27:46 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:27:48 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 2001:504:27::1b1b:0:1 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established Sep 6 18:33:58 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state Established event RecvNotify new state Idle Sep 6 18:34:07 207.32.15.28 Rib: %BGP-5-ADJCHANGE: peer 206.108.255.52 (AS 6939) old state OpenConfirm event RecvKeepAlive new state Established

3 7