RST Hurricane Congestion
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA. Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss. I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now. -- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
Thu night football killing the link? Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com On Thu, Sep 15, 2022, 20:16 Darin Steffl <darin.steffl@mnwifi.com> wrote:
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA.
Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss.
I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now.
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
It's not football killing us tonight! They finally responded after I called them twice and waited 1 hour, 51 mins for a ticket reply. They said several circuits are down to Rochester and they're trying to balance traffic on the remaining links. It's slowly getting better but we're still down 25% from our normal peak at this time of night. It's better than being down 50% of normal. This seems to be happening more frequently the last 2-3 years or there's more capacity sold, being used, in RST compared to years past when there's been fiber cuts. The slow performance is very noticeable. Less than 2 Mbps download and 10-20% packetloss anywhere outside of RST. Ben, are there any Arvig cuts between RST and Belle Plaine/MSP? I'm pretty sure they have some 100G paths from you guys. On Thu, Sep 15, 2022 at 9:41 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
Thu night football killing the link?
Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com
On Thu, Sep 15, 2022, 20:16 Darin Steffl <darin.steffl@mnwifi.com> wrote:
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA.
Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss.
I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now.
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
We didn't have anything down in that area in our maintenance/outage list. Another carrier has been fighting a cut between Winona and Wabasha in a difficult location. Maybe HE has some services through them as well? It seems out of the way, but... Ben Wiechman Director of Network Strategy and Engineering 320.247.3224 | ben.wiechman@arvig.com Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com On Thu, Sep 15, 2022 at 9:46 PM Darin Steffl <darin.steffl@mnwifi.com> wrote:
It's not football killing us tonight! They finally responded after I called them twice and waited 1 hour, 51 mins for a ticket reply. They said several circuits are down to Rochester and they're trying to balance traffic on the remaining links. It's slowly getting better but we're still down 25% from our normal peak at this time of night. It's better than being down 50% of normal.
This seems to be happening more frequently the last 2-3 years or there's more capacity sold, being used, in RST compared to years past when there's been fiber cuts. The slow performance is very noticeable. Less than 2 Mbps download and 10-20% packetloss anywhere outside of RST.
Ben, are there any Arvig cuts between RST and Belle Plaine/MSP? I'm pretty sure they have some 100G paths from you guys.
On Thu, Sep 15, 2022 at 9:41 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
Thu night football killing the link?
Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com
On Thu, Sep 15, 2022, 20:16 Darin Steffl <darin.steffl@mnwifi.com> wrote:
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA.
Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss.
I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now.
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
We didn't have anything down in that area in our maintenance/outage list. Another carrier has been fighting a cut between Winona and Wabasha in a difficult location. Maybe HE has some services through them as well? It seems out of the way, but...
There's only so many viable hops out of RST that go someplace useful for peering. The stuff heading towards Chicago gets horribly congested during network events. If that cut is on the CP rail between Winona and Wabasha that would remove a good chunk of capacity between MN and Chicago. MICE can help quite a bit with the Chicago problem...
Just to be clear, this was a fiber on some...legacy RST fiber. It has multiple 100s for us, so the reroute wasn't as pretty as it should be. We are actively working on making things a bit better between RST and the MSP area, but its a work in progress. Sorry for any issues, but I think many of us know the problematic history of this run of fiber. Reid On Fri, Sep 16, 2022 at 12:54 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
We didn't have anything down in that area in our maintenance/outage list. Another carrier has been fighting a cut between Winona and Wabasha in a difficult location. Maybe HE has some services through them as well? It seems out of the way, but...
Ben Wiechman
Director of Network Strategy and Engineering
320.247.3224 | ben.wiechman@arvig.com
Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com
On Thu, Sep 15, 2022 at 9:46 PM Darin Steffl <darin.steffl@mnwifi.com> wrote:
It's not football killing us tonight! They finally responded after I called them twice and waited 1 hour, 51 mins for a ticket reply. They said several circuits are down to Rochester and they're trying to balance traffic on the remaining links. It's slowly getting better but we're still down 25% from our normal peak at this time of night. It's better than being down 50% of normal.
This seems to be happening more frequently the last 2-3 years or there's more capacity sold, being used, in RST compared to years past when there's been fiber cuts. The slow performance is very noticeable. Less than 2 Mbps download and 10-20% packetloss anywhere outside of RST.
Ben, are there any Arvig cuts between RST and Belle Plaine/MSP? I'm pretty sure they have some 100G paths from you guys.
On Thu, Sep 15, 2022 at 9:41 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
Thu night football killing the link?
Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com
On Thu, Sep 15, 2022, 20:16 Darin Steffl <darin.steffl@mnwifi.com> wrote:
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA.
Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss.
I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now.
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178
Thanks for the info Reid. I'm not sure if you're referencing Zayo or not. They seem to have plenty of fiber issues near the metro. It seems reconvergence takes longer than it should when there's a cut too. We see 2-4 minutes of slow speeds or lost routes/downtime when I assume there's a cut on a path feeding into RST. At one point, a software update was done and it got much better but now it seems to be troublesome again. Congestion is usually bad when there's a cut and I have reported many events over the life of our account. Hopefully you guys have good luck improving things into RST. Thank you On Sat, Sep 17, 2022, 12:28 AM Reid Fishler < 0000000c9d62e487-dmarc-request@lists.iphouse.net> wrote:
Just to be clear, this was a fiber on some...legacy RST fiber. It has multiple 100s for us, so the reroute wasn't as pretty as it should be. We are actively working on making things a bit better between RST and the MSP area, but its a work in progress. Sorry for any issues, but I think many of us know the problematic history of this run of fiber.
Reid
On Fri, Sep 16, 2022 at 12:54 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
We didn't have anything down in that area in our maintenance/outage list. Another carrier has been fighting a cut between Winona and Wabasha in a difficult location. Maybe HE has some services through them as well? It seems out of the way, but...
Ben Wiechman
Director of Network Strategy and Engineering
320.247.3224 | ben.wiechman@arvig.com
Arvig | 224 East Main Street | Melrose, MN 56352 | arvig.com
On Thu, Sep 15, 2022 at 9:46 PM Darin Steffl <darin.steffl@mnwifi.com> wrote:
It's not football killing us tonight! They finally responded after I called them twice and waited 1 hour, 51 mins for a ticket reply. They said several circuits are down to Rochester and they're trying to balance traffic on the remaining links. It's slowly getting better but we're still down 25% from our normal peak at this time of night. It's better than being down 50% of normal.
This seems to be happening more frequently the last 2-3 years or there's more capacity sold, being used, in RST compared to years past when there's been fiber cuts. The slow performance is very noticeable. Less than 2 Mbps download and 10-20% packetloss anywhere outside of RST.
Ben, are there any Arvig cuts between RST and Belle Plaine/MSP? I'm pretty sure they have some 100G paths from you guys.
On Thu, Sep 15, 2022 at 9:41 PM Ben Wiechman <ben.wiechman@arvig.com> wrote:
Thu night football killing the link?
Ben Wiechman Director of IP Strategy and Engineering Direct: 320.256.0184 Cell: 320.247.3224 ben.wiechman@arvig.com 150 Second Street SW | Perham, MN 56573 | arvig.com
On Thu, Sep 15, 2022, 20:16 Darin Steffl <darin.steffl@mnwifi.com> wrote:
Anyone else connected to Hurricane's RST router noticing major congestion? It looks like the RST-BLP-MSP path is down and traffic is being diverted to CHI first or DVN-OMA.
Any speedtests to ISP's like MiBroadband or KMTelecom that are connected to RST are fine but any tests outside of that like to US Internet or Carleton College, I'm seeing under 2 Mbps download speeds and 20% packet loss.
I reported it to Hurricane at 7:39pm but it's been ongoing longer than that. I called 10 minutes after I emailed their NOC and the lady there wouldn't transfer me to an engineer and just said they'd reply to my ticket after they looked. I stressed the urgency since our traffic levels are only 50% of what they should be right now.
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi Like us on Facebook <http://www.facebook.com/minnesotawifi>
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
-- Reid Fishler Senior Director Hurricane Electric +1-510-580-4178
------------------------------
To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1
participants (4)
-
Ben Wiechman -
Darin Steffl -
Jay Hanke -
Reid Fishler