- mice-discuss - lists.micemn.net

outage of some sort last night
by Jeff Wilde June 17, 2024

June 17, 2024

Did anyone experience some type of odd outage in the middle of the night? Like before 5am? I don't know if it was a dns issue or what, but it seems to have resolved itself. [A picture containing icon Description automatically generated] Jeff Wilde * Network Engineer * Park Region Office 218.826.6161 * Direct 218.826.8330 * Fax 218.826.6298

4 3

Link to Ridgeview down
by Jeremy Lumby June 10, 2024

June 10, 2024

An FYI that we are aware that the link between the MICE core switch @ 511, and the MICE core switch @ Ridgeview is down. There is a complete loss of light. I have contacted the dark fiber provider to begin troubleshooting the link. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington MN 55425 Main 612-355-7740 x211 Direct 612-392-6814 Fax 952-873-7425 jlumby(a)mnvoip.com

2 2

Minnesota Repeals Municipal Broadband Restrictions
by David Farmer May 24, 2024

May 24, 2024

https://arstechnica.com/tech-policy/2024/05/another-us-state-repeals-law-th… Minnesota this week eliminated two laws that made it harder for cities and towns to build their own broadband networks. The state-imposed restrictions were repealed in an omnibus commerce policy bill signed on Tuesday by Gov. Tim Walz, a Democrat. .... -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

3 3

Traffic spike starting at 2:00am today
by Dean Bahls May 24, 2024

May 24, 2024

Curious if anyone knows why we are seeing about 25-30% increase in traffic today. Some big updates going on perhaps? Thanks, Dean Dean Bahls COO | Fiber Minnesota & Aurora Fiber Optic Networks 14 Main St SW | PO Box J | Menahga, MN 56464 218.564-1321 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)fiberminnesota.com<mailto:dean.bahls@fiberminnesota.com> [cid:image001.png@01DAADBE.2C9CB570] [cid:image002.jpg@01DAADBE.2C9CB570]

16 19

Cacti vulnerabilities
by Michael Horwath May 14, 2024

May 14, 2024

https://thehackernews.com/2024/05/critical-flaws-in-cacti-framework-could.h… Sharing is caring, and I care about sharing! -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

1 0

Server Updates
by Cameron Skappel May 3, 2024

May 3, 2024

We plan to update MICE servers and firewalls Friday 11:00AM Central. We'll only be doing 1 route server at a time. -- Cameron

1 1

Peering Summit Bus Mankato->Kansas City
by Jay Hanke April 24, 2024

April 24, 2024

The Midwest Peering Summit bus is a go. Here's the schedule: Wednesday 6/12/2024 10:00 AM Leave Mankato, We have free parking available 5:20 PM Arrive at Westin Crown Plaza Kansas City Friday 6/14/2024 4:00 PM Leave Kansas City Westin Crown Plaza Kansas City 11:25 PM Arrive in Mankato Cost is $150.00 round trip per person There is space on the bus to transport down exhibit equipment as well if needed. Email mwps(a)southfront.io to register. -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/

1 0

MWPS 2024 Transportion
by Jay Hanke April 18, 2024

April 18, 2024

If I had a coach bus going from MN to Kansas City Peering Summit, are others interested? -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 507-327-2398 https://southfront.io/

4 4

Fwd: Google - Drain of routes for Route-Servers in your IX
by Richard Laager April 5, 2024

April 5, 2024

I haven't heard this for MICE, but presumably they will be doing the same thing here? -------- Forwarded Message -------- Subject: Google - Drain of routes for Route-Servers in your IX (fwd) Date: Wed, 3 Apr 2024 14:31:55 +0000 (UTC) From: Chris Caputo <ccaputo(a)seattleix.net> To: members(a)seattleix.net FYI from Google... Chris ---------- Forwarded message ---------- Date: Wed, 03 Apr 2024 04:21:02 -0700 From: Google Peering Operations <peering-ops(a)google.com> To: info(a)seattleix.net Subject: Drain of routes for Route-Servers in your IX Dear Internet Exchange, As part of Google's effort to simplify peering operations and increase routing security we have decided to stop advertising and receiving prefix information from Route-Servers in Internet Exchanges. The BGP session with the route servers will remain established. We are planning to implement the change in your IX in the next few weeks. We won’t disconnect from the IX and we will keep current BGP bilateral sessions with individual peers on the IX. We invite other networks that rely on Route-Servers to request a bilateral session. Bilateral session request can be done at: https://isp.google.com/iwantpeering We kindly request you to inform peers in your Internet Exchange of these future changes so they can request bilateral sessions with us if they desire to do so. Sincerely, Google Network Operations

10 11

MICE interface flap
by Chris Wopat March 18, 2024

March 18, 2024

WiscNet's remote switch flapped twice within the last half hour: Feb 14 16:34:16.657 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Feb 14 16:39:29.456 2024 s-minneapolis-hub mib2d[8448]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 540, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/23 Aggregate traffic definitely shows a hit: http://micelg.usinternet.com/export/graph_2625.html Curious if others experienced flaps or could reveal logs from the core Arista? -- Chris Wopat Network Engineer, WiscNet wopat(a)wiscnet.net 608-210-3965

3 6

UG 45 Meeting Reminder
by Anthony Anderberg March 18, 2024

March 18, 2024

The MICE UG 45 meeting will be held Wednesday, March 20, 3:00 pm at the University of Minnesota IT Building. 2218 University Ave SE Minneapolis, MN 55414, Room 185 Across University Ave SE from Huntington (TCF) Bank Stadium Parking, etc. details same as before: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 Zoom Details: https://umn.zoom.us/j/94910223662?pwd=RUluTERkODlsdU93eHlQejR4eFVRdz09 ID: 94910223662, passcode: XdrNr3 651-372-8299 (passcode: 988007) Thanks everyone, Anthony

1 0

Re: MICE Usage Graphs
by Anthony Anderberg March 6, 2024

March 6, 2024

Yes, this had been on my todo list since Jeremy’s port changes - guess I just needed my friends to prod me a bit. :-) I think I’ve got everything cleaned up; I was able to preserve CarverLink’s data history but most of the others are new since the port changes. Cheers, Anthony From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> on behalf of Randy Lehs <rlehs(a)CARVERLINK.COM> Reply-To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Date: Tuesday, March 5, 2024 at 5:26 PM To: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] MICE Usage Graphs I noticed that several ports stopped graphing on the weathermap about a month ago, and just wanted to make sure someone was aware/working on it since I use the graphs all the time. Randy Lehs | CarverLink Fiber Manager CarverLink – A Department of Carver County Public Services 600 East Fourth Street Chaska, MN 55318-2102 www.carverlink.com<http://www.carverlink.com/> | rlehs(a)carverlink.com<mailto:rlehs@carverlink.com> Office: 952-361-1494 | Fax: 952-361-1582 | Cell: 612-719-1656 | Standard and Emergency Support 24/7/365: 952-361-1200 ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

2 1

MICE Usage Graphs
by Randy Lehs - CarverLink March 5, 2024

March 5, 2024

I noticed that several ports stopped graphing on the weathermap about a month ago, and just wanted to make sure someone was aware/working on it since I use the graphs all the time. Randy Lehs | CarverLink Fiber Manager CarverLink - A Department of Carver County Public Services 600 East Fourth Street Chaska, MN 55318-2102 www.carverlink.com<http://www.carverlink.com/> | rlehs(a)carverlink.com<mailto:rlehs@carverlink.com> Office: 952-361-1494 | Fax: 952-361-1582 | Cell: 612-719-1656 | Standard and Emergency Support 24/7/365: 952-361-1200

1 0

IRR Mandatory at MICE / New Route Servers
by Richard Laager Feb. 27, 2024

Feb. 27, 2024

[I plan to send this to MICE-ANNOUNCE too, but I want to see if anyone has corrections.] MICE will soon be deploying new route servers which will require IRR (Internet Routing Registry) records, as is a best practice at IXPs. *What* * You MUST have an as-set object listing your AS and your downstream ASes (if any). o You MUST either list that as-set in PeeringDB or email the name of your as-set to me (off-list to rlaager(a)wiktel.com please). * A route/route6 object MUST exist for each prefix you announce to the route servers (whether originated by you or transited through you) and it must list an Origin AS that is in your as-set. *When* * *If you are a transit AS (i.e. have ASes behind you) and don't have an /as-set/ object, fix this **/now/**.* Without an as-set object, your downstream ASes announcements will be blocked (filtered) immediately when the /first/ new route server is cut in. (Granted, they will still work through the second route server until it is upgraded.) Figure you have 1-2 weeks at most. * Enforcement of the route/route6 objects (for both transit and non-transit ASes) will come later, but not a lot later. So please, start on this now. ** *Where* If you are not sure /where/ to create IRR records, use ARIN (assuming you are in the ARIN region). *How (with ARIN)* 1. Login to ARIN Online. (Go to arin.net and click Login in the top right.) 2. On the left side, expand "Routing Security" and click "IRR". 3. Click "as-set" at the top. 4. Click "Create an Object". 5. Fill in the fields: The "AS Set Name" is what you will list in PeeringDB (or email to me). "Description" is unparsed, but they suggest the location and have a button to "Copy the Address from My Org ID". "Members" is where you list your ASN and downstream ASes (if any). 6. Click "Review". Once ready, click "Submit". 7. Click "route/route6" at the top. 8. Click "Create an Object". 9. Fill in the fields: "Prefix" is the prefix, e.g. 192.0.2.0/24. "Origin" is your ASN. 10. Click "Review". Once ready, click "Submit". 11. Repeat to create additional route objects until all of your announcements are covered. Don't forget IPv6! *Examples* Here is my as-set: https://www.radb.net/query?keywords=AS-WIKTEL Here is one example route: https://www.radb.net/query?keywords=69.89.192.0%2F20 (I created the AS33362 one. The AS19905 one is because another AS can originate this route for DDoS scrubbing reasons.) -- Richard

10 34

Global Net - AS395466 Maintenance
by Global Net NOC Feb. 22, 2024

Feb. 22, 2024

MICEMN Members - Our upstream provider, Zayo, will be undergoing a forced relocation at our datacenter in Montana. Expect our BGP sessions to drop during this time. 2/22 2200 – 2/23 0500 MST GLOBAL NET AS 395466 IPV4 206.108.255.97 IPV6 2001:504:27:0:6:8ca:0:1 Thanks, Global Net NOC noc(a)theglobal.net<mailto:noc@theglobal.net> Global Net Peering peering(a)theglobal.net<mailto:peering@theglobal.net>

1 0

UG 45 Meeting
by Richard Laager Feb. 21, 2024

Feb. 21, 2024

The MICE UG 45 meeting will be held Wednesday, March 20, 3:00 pm at UMN. Parking, etc. details same as before: https://lists.iphouse.net/cgi-bin/wa?A2=MICE-DISCUSS;2c503e19.2209 Zoom Details: https://umn.zoom.us/j/94910223662?pwd=RUluTERkODlsdU93eHlQejR4eFVRdz09 ID: 94910223662, passcode: XdrNr3 651-372-8299 (passcode: 988007) -- Richard

1 1

MICE Server Updates
by Cameron Skappel Feb. 9, 2024

Feb. 9, 2024

This evening I applied server updates to some of the MICE servers including Router Server 2. Let me know if there are issues. -- Cameron

1 1

MICE BFD Errors
by Richard Laager Feb. 8, 2024

Feb. 8, 2024

BIRD is complaining that various peers are sending BFD packets with an unknown session id. Anyone have thoughts on this? The logs look like this: 2024-02-07 23:44:14 <RMT> bfd1: Bad packet from YOUR_IP - unknown session id (SOME_INTEGER) Those marked with an asterisk were doing this yesterday, so prior to the reboot of rs2. rs1: Inteliquent (Neutral Tandem) 206.108.255.137 * rs2: les.net 2001:504:27::4813:0:1 Savage Communications 206.108.255.17 * 2001:504:27::5adc:0:1 * Stellar Association 206.108.255.53 2001:504:27::8e16:0:1 * Hoyos Consulting 2001:504:27::d15d:0:1 Eastern Iowa Data Group 206.108.255.168 * 2001:504:27::f7a2:0:1 * Omnitel Communications 206.108.255.148 Here is for the entire month of January: rlaager@rs1:~$ rg "^2024-01-.*Bad packet from" /var/log/bird/rs1-ipv*.1 | awk '{print $8}' | sort | uniq -c 9 2001:504:27::3823:0:2 2 2001:504:27::8252:0:1 1183836 2001:504:27::b7f8:0:1 4 2001:504:27::d15d:0:1 1183834 206.108.255.101 7498 206.108.255.119 380 206.108.255.121 1463099 206.108.255.137 3 206.108.255.176 3 206.108.255.185 root@rs2:~# rg "^2024-01-.*Bad packet from" /var/log/bird/rs2-ipv*.1 | awk '{print $8}' | sort | uniq -c 12 2001:504:27::1e49:0:1 1 2001:504:27::3823:0:2 1 2001:504:27::39:0:1 1457433 2001:504:27::5adc:0:1 3 2001:504:27::6478:0:1 1457369 2001:504:27::8e16:0:1 1 2001:504:27::9dbe:0:1 5 2001:504:27::9dbe:0:2 1179733 2001:504:27::b7f8:0:1 1176975 2001:504:27::d15d:0:1 4 2001:504:27::e4f:0:1 1 2001:504:27::f756:0:1 1331747 2001:504:27::f7a2:0:1 1179735 206.108.255.101 7494 206.108.255.119 1954 206.108.255.120 3 206.108.255.137 3 206.108.255.161 1331745 206.108.255.168 1457437 206.108.255.17 2 206.108.255.176 8 206.108.255.185 11 206.108.255.3 6 206.108.255.46 1457345 206.108.255.53 4 206.108.255.92 -- Richard

2 1

MICE Core Switch Card Upgrade
by Richard Laager Feb. 7, 2024

Feb. 7, 2024

Date & Time Wednesday, February 14 at 15:00 US/Central Description Jeremy will swap a (now unused) 10G line card for a 100G line card in the Arista core switch at 511. There is no expected impact for most participants. In the (relatively unlikely) event that a fiber cable is physically in the way and he has to unplug it to remove/install the card, he will disable BGP for that participant first and wait (unless the port is part of a LAG with sufficient excess capacity at that time of day). The Fiber Minnesota and Minnesota VoIP remote switches are connected by 3x100G LAG. Jeremy will move one port from each LAG from card 6 to the new card. Rollback Plan If there is some problem, the new 100G card can be removed. The ports in the Fiber Minnesota and Minnesota VoIP remote switches can be moved back to card 6. -- Richard

1 0

MICE Core Switch EOS Upgrade
by Richard Laager Feb. 7, 2024

Feb. 7, 2024

Date & Time Wednesday, January 31 at 06:00 US/Central Description Jeremy will upgrade software on the Arista core switch at 511. This is necessary to be able to install the (newer) 100G card into the switch. Rollback Plan The previous software image will be available to revert. -- Richard

3 5

Addresses on the list...
by Mike Horwath Jan. 31, 2024

Jan. 31, 2024

Calling out: From: Tritan Internet LLC (AS393577) <noc(a)as393577.net> Subject: TD#937 - Acknowledgement from Tritan Internet NOC Please subscribe either a user to the list or update things so that autoresponses do not occur. LISTSERV picked it up and did the right thing, thankfully. -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

1 0

BGP ADD-PATH capability
by Richard Laager Jan. 24, 2024

Jan. 24, 2024

I propose that MICE enable BGP ADD-PATH capability (RFC 7911) on the route servers (in TX mode only). This is negotiated BGP capability, so we do not need any per-participant setting for it. ADD-PATH is useful if the best path route is being filtered but another route would not be. We discussed adding this at a UG meeting years ago. There are other IXPs that have this enabled. For example, this one: https://www.bcix.de/ixp/content/0/route-servers In terms of timeline, I propose something like this: * January 9: Enable it on Route Server 1. * January 16: Enable it on Route Server 2. The actual change looks like this for the RS1-only piece, and then the conditional would be removed when we want to deploy to RS2: --- a/roles/ixpmanager/files/mice/api/v4/router/server/bird2/neighbors.foil.php +++ b/roles/ixpmanager/files/mice/api/v4/router/server/bird2/neighbors.foil.php @@ -315,6 +315,9 @@ protocol bgp pb_<?= $int['fvliid'] ?>_as<?= $int['autsys'] ?> from tb_rsclient { import filter f_import_as<?= $int['autsys'] ?>; table t_<?= $int['fvliid'] ?>_as<?= $int['autsys'] ?>; export filter f_export_as<?= $int['autsys'] ?>; +<?php if( $t->router->router_id == "206.108.255.1" ): ?> + add paths tx; +<?php endif; ?> }; <?php if( $int['bgpmd5secret'] && !$t->router->skip_md5 ): ?> password "<?= $int['bgpmd5secret'] ?>"; -- Richard

6 11

MICE Maintenance 2024-01-18 & 2024-01-19
by Richard Laager Jan. 8, 2024

Jan. 8, 2024

Date & Time January 18, 00:00 - 05:00 US/Central January 19, 00:00 - 05:00 US/Central Description Our underlying fiber provider will be doing maintenance on the connection from 511 to Ridgeview. They are attempting to repair some other fiber issues. We may or may not go down. If we did go down, participants on the 511 side would see a loss of BGP to Route Server 2. -- Richard

1 0

Route servers?
by Miles McCredie Jan. 3, 2024

Jan. 3, 2024

FWIW, the AS11232 sessions to route server 2 have been active for about 25 minutes. https://ixpmgr.micemn.net/lg/rs2-ipv4 and https://ixpmgr.micemn.net/lg/rs2-ipv6 <https://ixpmgr.micemn.net/lg/rs2-ipv6%20show%20that%20AS11232> show quite a few other ASNs that are not established. https://ixpmgr.micemn.net/lg/rs1-ipv4 and https://ixpmgr.micemn.net/lg/rs1-ipv6 show more not-established sessions than I would expect but fewer than rs2. I haven't looked more closely at the impact to routing yet. Any details about the work in progress? Thanks [http://images.midcocomm.com/ES_MidcoLogo.png] Miles McCredie Principal Network Engineer II-Core IP Office: 6052755192 Miles.McCredie(a)midco.com Midco.com

4 6

Strange Cacti graphs
by Dean Bahls Dec. 21, 2023

Dec. 21, 2023

Anyone notice the strange spikes/valleys on the Cacti graphs lately? Looks like a heartbeat every hour on the hour. Thanks, Dean Dean Bahls COO | Fiber Minnesota 14 Main St SW | PO Box J | Menahga, MN 56464 218.564-1321 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)fiberminnesota.com<mailto:dean.bahls@fiberminnesota.com> [cid:image001.png@01DA3298.4466DA60] [cid:image002.png@01DA3298.4466DA60]

2 1