- mice-discuss - lists.micemn.net

MICE Server Updates
by Cameron Skappel 09 Feb '24

09 Feb '24

This evening I applied server updates to some of the MICE servers including Router Server 2. Let me know if there are issues. -- Cameron

1 1

MICE BFD Errors
by Richard Laager 08 Feb '24

08 Feb '24

BIRD is complaining that various peers are sending BFD packets with an unknown session id. Anyone have thoughts on this? The logs look like this: 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. rs1: Inteliquent (Neutral Tandem) 206.108.255.137 * rs2: les.net 2001:504:27::4813:0:1 Savage Communications 206.108.255.17 * 2001:504:27::5adc:0:1 * Stellar Association 206.108.255.53 2001:504:27::8e16:0:1 * Hoyos Consulting 2001:504:27::d15d:0:1 Eastern Iowa Data Group 206.108.255.168 * 2001:504:27::f7a2:0:1 * Omnitel Communications 206.108.255.148 Here is for the entire month of January: rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c 9 2001:504:27::3823:0:2 2 2001:504:27::8252:0:1 1183836 2001:504:27::b7f8:0:1 4 2001:504:27::d15d:0:1 1183834 206.108.255.101 7498 206.108.255.119 380 206.108.255.121 1463099 206.108.255.137 3 206.108.255.176 3 206.108.255.185 root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c 12 2001:504:27::1e49:0:1 1 2001:504:27::3823:0:2 1 2001:504:27::39:0:1 1457433 2001:504:27::5adc:0:1 3 2001:504:27::6478:0:1 1457369 2001:504:27::8e16:0:1 1 2001:504:27::9dbe:0:1 5 2001:504:27::9dbe:0:2 1179733 2001:504:27::b7f8:0:1 1176975 2001:504:27::d15d:0:1 4 2001:504:27::e4f:0:1 1 2001:504:27::f756:0:1 1331747 2001:504:27::f7a2:0:1 1179735 206.108.255.101 7494 206.108.255.119 1954 206.108.255.120 3 206.108.255.137 3 206.108.255.161 1331745 206.108.255.168 1457437 206.108.255.17 2 206.108.255.176 8 206.108.255.185 11 206.108.255.3 6 206.108.255.46 1457345 206.108.255.53 4 206.108.255.92 -- Richard

2 1

MICE Core Switch Card Upgrade
by Richard Laager 07 Feb '24

07 Feb '24

Date & Time Wednesday, February 14 at 15:00 US/Central Description Jeremy will swap a (now unused) 10G line card for a 100G line card in the Arista core switch at 511. There is no expected impact for most participants. In the (relatively unlikely) event that a fiber cable is physically in the way and he has to unplug it to remove/install the card, he will disable BGP for that participant first and wait (unless the port is part of a LAG with sufficient excess capacity at that time of day). The Fiber Minnesota and Minnesota VoIP remote switches are connected by 3x100G LAG. Jeremy will move one port from each LAG from card 6 to the new card. Rollback Plan If there is some problem, the new 100G card can be removed. The ports in the Fiber Minnesota and Minnesota VoIP remote switches can be moved back to card 6. -- Richard

1 0

MICE Core Switch EOS Upgrade
by Richard Laager 07 Feb '24

07 Feb '24

Date & Time Wednesday, January 31 at 06:00 US/Central Description Jeremy will upgrade software on the Arista core switch at 511. This is necessary to be able to install the (newer) 100G card into the switch. Rollback Plan The previous software image will be available to revert. -- Richard

3 5

Addresses on the list...
by Mike Horwath 31 Jan '24

31 Jan '24

Calling out: From: Tritan Internet LLC (AS393577) <noc(a)as393577.net> Subject: TD#937 - Acknowledgement from Tritan Internet NOC Please subscribe either a user to the list or update things so that autoresponses do not occur. LISTSERV picked it up and did the right thing, thankfully. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

1 0

BGP ADD-PATH capability
by Richard Laager 24 Jan '24

24 Jan '24

I propose that MICE enable BGP ADD-PATH capability (RFC 7911) on the route servers (in TX mode only). This is negotiated BGP capability, so we do not need any per-participant setting for it. ADD-PATH is useful if the best path route is being filtered but another route would not be. We discussed adding this at a UG meeting years ago. There are other IXPs that have this enabled. For example, this one: https://www.bcix.de/ixp/content/0/route-servers In terms of timeline, I propose something like this: * January 9: Enable it on Route Server 1. * January 16: Enable it on Route Server 2. The actual change looks like this for the RS1-only piece, and then the conditional would be removed when we want to deploy to RS2: --- a/roles/ixpmanager/files/mice/api/v4/router/server/bird2/neighbors.foil.php +++ b/roles/ixpmanager/files/mice/api/v4/router/server/bird2/neighbors.foil.php @@ -315,6 +315,9 @@ protocol bgp pb_<?= $int['fvliid'] ?>_as<?= $int['autsys'] ?> from tb_rsclient { import filter f_import_as<?= $int['autsys'] ?>; table t_<?= $int['fvliid'] ?>_as<?= $int['autsys'] ?>; export filter f_export_as<?= $int['autsys'] ?>; +<?php if( $t->router->router_id == "206.108.255.1" ): ?> + add paths tx; +<?php endif; ?> }; <?php if( $int['bgpmd5secret'] && !$t->router->skip_md5 ): ?> password "<?= $int['bgpmd5secret'] ?>"; -- Richard

6 11

MICE Maintenance 2024-01-18 & 2024-01-19
by Richard Laager 08 Jan '24

08 Jan '24

Date & Time January 18, 00:00 - 05:00 US/Central January 19, 00:00 - 05:00 US/Central Description Our underlying fiber provider will be doing maintenance on the connection from 511 to Ridgeview. They are attempting to repair some other fiber issues. We may or may not go down. If we did go down, participants on the 511 side would see a loss of BGP to Route Server 2. -- Richard

1 0

Route servers?
by Miles McCredie 03 Jan '24

03 Jan '24

FWIW, the AS11232 sessions to route server 2 have been active for about 25 minutes. https://ixpmgr.micemn.net/lg/rs2-ipv4 and https://ixpmgr.micemn.net/lg/rs2-ipv6 <https://ixpmgr.micemn.net/lg/rs2-ipv6%20show%20that%20AS11232> show quite a few other ASNs that are not established. https://ixpmgr.micemn.net/lg/rs1-ipv4 and https://ixpmgr.micemn.net/lg/rs1-ipv6 show more not-established sessions than I would expect but fewer than rs2. I haven't looked more closely at the impact to routing yet. Any details about the work in progress? Thanks [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com

4 6

Strange Cacti graphs
by Dean Bahls 21 Dec '23

21 Dec '23

Anyone notice the strange spikes/valleys on the Cacti graphs lately? Looks like a heartbeat every hour on the hour. Thanks, Dean Dean Bahls COO | Fiber Minnesota 14 Main St SW | PO Box J | Menahga, MN 56464 218.564-1321 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)fiberminnesota.com<mailto:dean.bahls@fiberminnesota.com> [cid:image001.png@01DA3298.4466DA60] [cid:image002.png@01DA3298.4466DA60]

2 1

MICE almost hit 2T
by David Farmer 05 Dec '23

05 Dec '23

We had a 1.96T burst last night. [image: image.png] The extra Fortnite Chapter 5 Season 1 release traffic over the weekend can be seen. [image: image.png] -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

5 5

micelg.usinternet.com
by Jeremy Lumby 28 Nov '23

28 Nov '23

I no longer seem to be able to connect to http://micelg.usinternet.com via IPv4 or IPv6 Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com

5 6

OmniTech (AS54821) has disconnected from MICE
by Jay Hanke 27 Nov '23

27 Nov '23

OmniTech AS54821 has disconnected from MICE. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/ <https://southfront.io/path.html>

1 0

Thursday night Prime Football
by Steve Savoy 22 Nov '23

22 Nov '23

Hello everyone, Just curious if anyone else has customers complaining about Prime football on Thursday nights being extremely 'grainy' - like the old VHS days........ Thank you, Steve Savoy Confidentiality Notice: This correspondence is the property of Winnebago Cooperative Telecom Association and is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

8 8

Global Net (AS395466) - Upcoming Maintenance 11/15/2023
by Global Net NOC 18 Nov '23

18 Nov '23

MICEMN Members - We have a maintenance window scheduled for Wednesday, November 15th. BGP sessions will go down during the window. Start: 11/15/23 0701 UTC (00:01 MST) End: 11/15/23 1300 UTC (06:00 MST) GLOBAL NET AS 395466 IPV4 206.108.255.97 IPV6 2001:504:27:0:6:8ca:0:1 Thanks, Global Net NOC noc(a)theglobal.net Global Net Peering peering(a)theglobal.net

1 1

AS394330 LTD Broadband/GigFire Port Move
by Jay Hanke 08 Nov '23

08 Nov '23

LTD Broadband/GigFire has moved from the MN VoIP Remote to port eth9/1 on the SFN remote switch. The MN VoIP port has been disabled. All IP information remains the same. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/ <https://southfront.io/path.html>

1 0

[OT] Calix GigaSpire RG Failures
by Richard Laager 05 Nov '23

05 Nov '23

"Critical: EXOS – GigaSpire RG Systems May Display Amber LED and Be Unreachable" https://community.calix.com/s/feed/0D54u0000A5ugVA "We have confirmed today that *only releases 23.2 and 23.3 are impacted*. EXOS Release 23.3.0.3.57 is now available for use in workflows within Calix Cloud, or for local download from theCalix Software Center <https://www.calix.com/my-calix/software-center.html#>.Please find the release notes for this build here <https://www.calix.com/content/dam/calix/doc-library/systems/prem/support/ex…>, and please reviewService Bulletin SB-23-005 <https://www.calix.com/bin/calix/servlets/docdownload?fileId=/content/dam/ca…>for additional technical detail regarding this matter. With the new release now available,*_we strongly recommend any systems running 23.2 or 23.3 be upgraded to this new release immediately_*. Systems running 23.1 or lower should upgrade to the 23.4 release when it becomes available later this month." -- Richard

1 0

Amazon TNF Bitrate/Stream
by Ben Wiechman 02 Nov '23

02 Nov '23

Has anyone spent any time studying the bitrate per stream this year? What are you seeing on your networks? Our Qwilt cache is reporting 25Mbps/stream average, which doesn't seem realistic by other measures. Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman(a)arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com

4 7

Google to stop advertising to MICE Route Servers
by Jay Hanke 23 Oct '23

23 Oct '23

We have received notice that Google AS15159 will stop sending and receiving routes from the MICE route servers on November 8th, 2023. Google has asked us to share the following notice. Google will keep current BGP bilateral with individual peers on MICE. Google invites other networks that rely on Route-Servers to request a bilateral session. Bilateral session request can be done at: https://isp.google.com/iwantpeering Thanks! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/

2 2

AS57 (NorthernLights/UofM) Maintenance Complete
by Colin Murphy 14 Oct '23

14 Oct '23

University of Minnesota Engineers have completed the software upgrades on the NorthernLights Gigapop routers. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870

1 0

AS57 (NorthernLights/UofM) Maintenance
by Colin Murphy 12 Oct '23

12 Oct '23

University of Minnesota Engineers will be upgrading the software on the NorthernLights Gigapop routers. The expected outage for each router will be approximately 2 hours. BGP sessions will be shutdown prior to the software upgrades. Please let us know if you have any questions. Change window: 10/14/2023 4 AM to 5 PM CDT. Thanks, Colin Murphy Sr. Network Engineer Office of Information Technology University of Minnesota Northern Lights Gigapop Desk: 612-626-0870

1 0

AS11796 Maintenance
by Charles Monson 26 Sep '23

26 Sep '23

AS11796/WIN Technology will be disabling all BGP sessions over the MICE exchange tonight after 2300 (Central) in preparation for maintenance. Please keep your BGP sessions with us active to avoid unnecessary troubleshooting once the link is back up. Thank you, Charles Monson WIN Technology (AS11796/AS26652)

1 0

UG 44 Meeting & Election
by Richard Laager 25 Sep '23

25 Sep '23

The MICE UG 44 meeting will be held Wednesday, October 25, 3:00 pm at UMN. Here are the details (parking, etc.) from last time: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 An updated Zoom link will be coming. *Nominations* We will be holding an election for a board seat. The board consists of three seats with staggered terms, such that one seat comes up for re-election each year (at the fall meeting). This year, Anthony's seat is up for election. As in the past, we will close nominations in advance of the meeting so people have a chance to instruct a proxy appropriately. *Nominations will be open for two weeks (so until the end of October 7).* * * *Proxies * The bylaws state that you send proxies to the Chief Manager (me, Richard Laager), and electronic means are acceptable. So a simple email is sufficient: rlaager(a)wiktel.com or chiefmanager(a)micemn.net. If you send me a /vote/ by email (e.g. "We vote for <X>."), I will treat it as an a proxy appointment, appointing me to vote on your behalf, but restricted to /only/ the election item and to /only/ vote for the candidate you expressed. If you send me a proxy with different instructions (e.g. "We appoint <Y> to vote for us."), obviously that will be honored as written. -- Richard

2 1

Gateway Fiber ribbon cutting
by Michael Horwath 14 Sep '23

14 Sep '23

They had that this moring at 1000 hours in Blaine. I made a joke to my friend that I wonder how many of 3 of us will be there. Well, I got a surprise as there were at least 8 folks who were not affilitated with the event. Talking to a couple of folks - would like to have customers up by end of year in Blaine - my area (Coon Rapids) would come afterwards but could not get them to commit or ‘about 6 months’ - GPON for delivery - using Comlink Solutions for the pulls - 511 is on their radar - mentioned expanding south from Blaine into SLP, Mounds View - mentioned expanding north-west into Andover I mentioned MICE and gave el-presidente my email address to contact me, and the website URL And I did not get a pic of their pricing flyer! But 2Gbps bidirectional is $150/month but I cannot remember the lower speeds cause I didn’t care :) -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

3 2

Global Net (AS395466) - Down
by Global Net NOC 12 Sep '23

12 Sep '23

MICEMN Members - Our circuit to Cologix in Minneapolis is hard down due to an upstream provider issue. Our BGP sessions remain down. GLOBAL NET AS 395466 IPV4 206.108.255.97 IPV6 2001:504:27:0:6:8ca:0:1 Thanks, Global Net NOC noc(a)theglobal.net Global Net Peering Peering peering(a)theglobal.net

1 0

Route Server 2
by Tom Krenn 08 Sep '23

08 Sep '23

Hi all, Anyone else noticing the route server at 206.108.255.2 / 2001:504:27::d1af:0:2 dropping out for a second or so the last few days? I'm seeing similar from both of my routers. LC/0/0/CPU0:Sep 8 07:29:45.717 CDT: bfd_agent[173]: %L2-BFD-6-SESSION_STATE_DOWN : BFD session to neighbor 2001:504:27::d1af:0:2 on interface HundredGigE0/0/0/20 has gone down. Reason: Control timer expired RP/0/RSP0/CPU0:Sep 8 07:29:45.718 CDT: bgp[1065]: %ROUTING-BGP-5-ADJCHANGE : neighbor 2001:504:27::d1af:0:2 Down - BFD (Bidirectional forwarding detection) session down (VRF: default) (AS: 53679) LC/0/0/CPU0:Sep 8 07:29:45.795 CDT: bfd_agent[173]: %L2-BFD-6-SESSION_STATE_DOWN : BFD session to neighbor 206.108.255.2 on interface HundredGigE0/0/0/20 has gone down. Reason: Control timer expired RP/0/RSP0/CPU0:Sep 8 07:29:45.797 CDT: bgp[1065]: %ROUTING-BGP-5-ADJCHANGE : neighbor 206.108.255.2 Down - BFD (Bidirectional forwarding detection) session down (VRF: default) (AS: 53679) LC/0/0/CPU0:Sep 8 07:29:50.384 CDT: bfd_agent[173]: %L2-BFD-6-SESSION_STATE_UP : BFD session to neighbor 2001:504:27::d1af:0:2 on interface HundredGigE0/0/0/20 is up LC/0/0/CPU0:Sep 8 07:29:50.384 CDT: bfd_agent[173]: %L2-BFD-6-SESSION_STATE_UP : BFD session to neighbor 206.108.255.2 on interface HundredGigE0/0/0/20 is up RP/0/RSP0/CPU0:Sep 8 07:30:58.517 CDT: bgp[1065]: %ROUTING-BGP-5-ADJCHANGE : neighbor 206.108.255.2 Up (VRF: default) (AS: 53679) RP/0/RSP0/CPU0:Sep 8 07:31:01.850 CDT: bgp[1065]: %ROUTING-BGP-5-ADJCHANGE : neighbor 2001:504:27::d1af:0:2 Up (VRF: default) (AS: 53679) Tom Krenn Network Architect Enterprise Architecture - Information Technology Office: 612-596-8994 tom.krenn(a)hennepin.us<mailto:tom.krenn@hennepin.us> | hennepin.us<https://hennepin.us/> Hennepin County Government Center 300 South 6th Street, A-190 Minneapolis, MN, 55487 [Hennepin County logo] Connect: Facebook<https://www.facebook.com/hennepin> | Twitter<https://twitter.com/Hennepin> | YouTube<https://www.youtube.com/user/HennepinMN> | LinkedIn<https://www.linkedin.com/company/hennepin-county> | Instagram<https://www.instagram.com/hennepincounty> Disclaimer: If you are not the intended recipient of this message, please immediately notify the sender of the transmission error and then promptly permanently delete this message from your computer system.

4 4