- mice-discuss - lists.micemn.net

Re: Google Dropped

by Andrew Johnson

Here is what I received as a customer due to latency issues with CL - I believe this is directly related. - Update provided by Centurylink: "Network Operations Center reports that services are impacted by multiple fiber cuts that are impacting the CenturyLink network. Field Operations and all necessary repair personnel have arrived to the first fiber cut location and are currently working to pull in the slack on the fiber span. Approximately 3,000 feet of new fiber will be required to pulled in." On Tue, Apr 21, 2020, 10:55 Andrew Hoyos <hoyosa(a)gmail.com> wrote: > Looks like a fiber cut between Milwaukee, WI and Zion, IL. Guessing it’ll > be a bit! > > Latest update: > > *** CASCADED EXTERNAL NOTES 21-Apr-2020 15:48:56 GMT From CASE: 18572279 - > Event > The Transport NOC reports that one Field Operations team is set to arrive > in Milwaukee, WI momentarily, and the secondary Field Operations team > remains in transit to the gateway facility in Zion, IL. An ETA of 17:00 GMT > remains expected. > > > On Apr 21, 2020, at 9:32 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > There does appear to be an issue between Madison and Chicago on Level3/CL > glass. Google backhauls to Chicago on this. > > On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> wrote: > > > Does anyone know why Google dropped all traffic? Looks like a fiber cut > since it's at 0. > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > <http://www.facebook.com/minnesotawifi> Like us on Facebook > <http://www.facebook.com/minnesotawifi> > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > > > ------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 21, 2020

Re: Google Dropped

by Andrew Hoyos

Looks like a fiber cut between Milwaukee, WI and Zion, IL. Guessing it’ll be a bit! Latest update: *** CASCADED EXTERNAL NOTES 21-Apr-2020 15:48:56 GMT From CASE: 18572279 - Event The Transport NOC reports that one Field Operations team is set to arrive in Milwaukee, WI momentarily, and the secondary Field Operations team remains in transit to the gateway facility in Zion, IL. An ETA of 17:00 GMT remains expected. > On Apr 21, 2020, at 9:32 AM, Andrew Hoyos <hoyosa(a)gmail.com> wrote: > > There does appear to be an issue between Madison and Chicago on Level3/CL glass. Google backhauls to Chicago on this. > >> On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> wrote: >> >> >> Does anyone know why Google dropped all traffic? Looks like a fiber cut since it's at 0. >> >> -- >> Darin Steffl >> Minnesota WiFi >> www.mnwifi.com <http://www.mnwifi.com/> >> 507-634-WiFi >> <http://www.facebook.com/minnesotawifi> Like us on Facebook <http://www.facebook.com/minnesotawifi> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1>

April 21, 2020

Re: Google Dropped

by Andrew Hoyos

> On Apr 21, 2020, at 10:02 AM, James Stahr <0000000d8a27a856-dmarc-request(a)LISTS.IPHOUSE.NET> wrote: > > On 2020-04-21 09:32, Andrew Hoyos wrote: >> There does appear to be an issue between Madison and Chicago on >> Level3/CL glass. Google backhauls to Chicago on this. > > Do you have a time this occurred? Possibly unrelated, but we had a Level3/Centurylink circuit go down (times in CDT): > > Apr 21 2020 08:53:59.515: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/0/5, changed state to down > Apr 21 2020 08:53:59.566: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet2/0/5, link down due to local fault > > This was a MKE circuit, but I believe it was being backhauled to Chicago. Lines up with a few we’re seeing down: Apr 21 08:54:00 xxxxxxxx mib2d[2109]: SNMP_TRAP_LINK_DOWN: ifIndex 538, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-2/0/0 Level3 in MKE also does head down to Chicago. MSP -> MSN -> MKE -> ORD is their primary backbone.

April 21, 2020

Re: Google Dropped

by Jeff Wilde

Make that 8:55 alarm came in a little late. Jeff Wilde Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeff Wilde Sent: Tuesday, April 21, 2020 10:06 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google Dropped Trigger time: 4/21/2020 9:00 AM Triggered by: 206.108.255.141 Alert Definition: BGP Neighbors Escalation: Level 1 Acknowledged by: Not yet... Jeff Wilde Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Stahr Sent: Tuesday, April 21, 2020 10:03 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google Dropped On 2020-04-21 09:32, Andrew Hoyos wrote: > There does appear to be an issue between Madison and Chicago on > Level3/CL glass. Google backhauls to Chicago on this. > Do you have a time this occurred? Possibly unrelated, but we had a Level3/Centurylink circuit go down (times in CDT): Apr 21 2020 08:53:59.515: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/0/5, changed state to down Apr 21 2020 08:53:59.566: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet2/0/5, link down due to local fault This was a MKE circuit, but I believe it was being backhauled to Chicago. -James >> On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> >> wrote: > >> >> Does anyone know why Google dropped all traffic? Looks like a fiber >> cut since it's at 0. >> >> -- >> >> Darin Steffl >> Minnesota WiFi >> www.mnwifi.com [1] >> 507-634-WiFi >> [2] Like us on Facebook [2] >> >> ------------------------- >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------- > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > Links: > ------ > [1] http://www.mnwifi.com/ > [2] http://www.facebook.com/minnesotawifi -- -James

April 21, 2020

Re: Google Dropped

by Jeff Wilde

Trigger time: 4/21/2020 9:00 AM Triggered by: 206.108.255.141 Alert Definition: BGP Neighbors Escalation: Level 1 Acknowledged by: Not yet... Jeff Wilde Park Region Telephone Company P: 218-826-8330 E: jeff.wilde(a)parkregion.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of James Stahr Sent: Tuesday, April 21, 2020 10:03 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Google Dropped On 2020-04-21 09:32, Andrew Hoyos wrote: > There does appear to be an issue between Madison and Chicago on > Level3/CL glass. Google backhauls to Chicago on this. > Do you have a time this occurred? Possibly unrelated, but we had a Level3/Centurylink circuit go down (times in CDT): Apr 21 2020 08:53:59.515: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/0/5, changed state to down Apr 21 2020 08:53:59.566: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet2/0/5, link down due to local fault This was a MKE circuit, but I believe it was being backhauled to Chicago. -James >> On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> >> wrote: > >> >> Does anyone know why Google dropped all traffic? Looks like a fiber >> cut since it's at 0. >> >> -- >> >> Darin Steffl >> Minnesota WiFi >> www.mnwifi.com [1] >> 507-634-WiFi >> [2] Like us on Facebook [2] >> >> ------------------------- >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------- > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > Links: > ------ > [1] http://www.mnwifi.com/ > [2] http://www.facebook.com/minnesotawifi -- -James

April 21, 2020

Re: Google Dropped

by James Stahr

On 2020-04-21 09:32, Andrew Hoyos wrote: > There does appear to be an issue between Madison and Chicago on > Level3/CL glass. Google backhauls to Chicago on this. > Do you have a time this occurred? Possibly unrelated, but we had a Level3/Centurylink circuit go down (times in CDT): Apr 21 2020 08:53:59.515: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/0/5, changed state to down Apr 21 2020 08:53:59.566: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet2/0/5, link down due to local fault This was a MKE circuit, but I believe it was being backhauled to Chicago. -James >> On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> >> wrote: > >> >> Does anyone know why Google dropped all traffic? Looks like a fiber >> cut since it's at 0. >> >> -- >> >> Darin Steffl >> Minnesota WiFi >> www.mnwifi.com [1] >> 507-634-WiFi >> [2] Like us on Facebook [2] >> >> ------------------------- >> >> To unsubscribe from the MICE-DISCUSS list, click the following link: >> http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > ------------------------- > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 > > Links: > ------ > [1] http://www.mnwifi.com/ > [2] http://www.facebook.com/minnesotawifi -- -James

April 21, 2020

Re: Google Dropped

by Andrew Hoyos

There does appear to be an issue between Madison and Chicago on Level3/CL glass. Google backhauls to Chicago on this. > On Apr 21, 2020, at 9:26 AM, Darin Steffl <darin.steffl(a)mnwifi.com> wrote: > > > Does anyone know why Google dropped all traffic? Looks like a fiber cut since it's at 0. > > -- > Darin Steffl > Minnesota WiFi > www.mnwifi.com > 507-634-WiFi > Like us on Facebook > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 21, 2020

Google Dropped

by Darin Steffl

Does anyone know why Google dropped all traffic? Looks like a fiber cut since it's at 0. -- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi <http://www.facebook.com/minnesotawifi> Like us on Facebook <http://www.facebook.com/minnesotawifi>

April 21, 2020

Thanks, Cloudflare!

by Mike Horwath

Without further comment: https://arstechnica.com/?p=1669279 -- Mike Horwath, reachable via drechsau(a)Geeks.ORG

April 19, 2020

OneNetGlobal 10G Upgrade

by Jay Hanke

OneNet Global is upgrading their 1G Connection to 10G. The new 10G port is Ethernet4/27 on the main switch. Welcome aboard! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 18, 2020

Fwd: [arin-announce] Upcoming Web-Based Internet Routing Registry Deployment

by David Farmer

FYI, This should simplify things for many of you to get your prefixes into the IRR. ---------- Forwarded message --------- From: ARIN <info(a)arin.net> Date: Fri, Apr 17, 2020 at 1:31 PM Subject: [arin-announce] Upcoming Web-Based Internet Routing Registry Deployment To: <arin-announce(a)arin.net> On 17 March 2020, we notified our community via a blog post on TeamARIN that we were in the process of developing a new web-based Internet Routing Registry (IRR). That blog post can be read at the following link: https://teamarin.net/2020/03/17/making-progress-on-arins-new-web-based-irr/ Now, we are pleased to announce that the initial deployment of this new web-based IRR will occur on or around 10 June 2020. This deployment will provide core functionality to create, edit, and delete IRR objects via ARIN Online. The existing IRR-email interface will continue to function; however, when your organization begins using the web-based IRR, you will lose access to the email interface. For that reason, if your organization would like to retain access to the email interface, please do not make any changes to IRR objects using ARIN Online. As part of this deployment, ARIN will analyze data in the current IRR before migration and objects will be placed into two categories: * Objects that were created by the authorized organization and meet all ARIN requirements will be migrated to the new system and placed in an authorized dataset (“ARIN-AUTH”). They can be managed in either ARIN Online or via email templates. (Note that using ARIN Online the first time to manage these objects will permanently disallow future changes using email templates.) * Objects that cannot be verified will still be migrated, but placed into a non-authorized (“ARIN-NONAUTH”) dataset. These objects can be managed only by using email templates or by creating a Help Desk Ticket using the Ask ARIN feature in ARIN Online. Users who get routing data from ARIN’s IRR will need to make some changes when connecting to our two Near Real Time Monitoring (NRTM) streams: ARIN-AUTH and ARIN-NONAUTH. Port 43 whois queries to rr.arin.net will still be accepted and return both authorized and unauthorized objects. More information will be provided in later updates. Best regards, John Sweeting Chief Customer Officer American Registry for Internet Numbers (ARIN) _______________________________________________ ARIN-Announce You are receiving this message because you are subscribed to the ARIN Announce Mailing List (ARIN-announce(a)arin.net) Unsubscribe or manage your mailing list subscription at: https://lists.arin.net/mailman/listinfo/arin-announce Please contact info(a)arin.net if you experience any issues. -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 17, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

We did not have any evidence of established traffic being impacted by the line card issue. The issue was more that it had problems that were filling up the storage, and it would not detect new optics. From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Jeff Wilde Sent: Thursday, April 16, 2020 6:20 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Could this possibly have caused traffic anomalies? I’m talking about the bad card, but I’m guessing not. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 5:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE The main reason is because mice is 100% volunteer and it is difficult to get volunteers to come in in the middle of the night Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Ryan Malek <rmalek(a)ROUTER12.NET> Date: 4/16/20 5:43 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >From : Ryan Malek [rmalek(a)ROUTER12.NET] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Thursday, April 16 2020 17:43:42 I hope this isn't a stupid question... but why was this done at 5PM? I found out in a very brutal manner that something is wrong with my BGP set up... Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 4/16/2020 5:29 PM, Jeremy Lumby wrote: The maintenance has completed, and everything appears to be returning to normal. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 4:14 PM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 17, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeff Wilde

Could this possibly have caused traffic anomalies? I’m talking about the bad card, but I’m guessing not. From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 5:47 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE The main reason is because mice is 100% volunteer and it is difficult to get volunteers to come in in the middle of the night Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> -------- Original message -------- From: Ryan Malek <rmalek(a)ROUTER12.NET<mailto:rmalek@ROUTER12.NET>> Date: 4/16/20 5:43 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE ________________________________ From : Ryan Malek [rmalek(a)ROUTER12.NET] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Thursday, April 16 2020 17:43:42 I hope this isn't a stupid question... but why was this done at 5PM? I found out in a very brutal manner that something is wrong with my BGP set up... ________________________________ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 4/16/2020 5:29 PM, Jeremy Lumby wrote: The maintenance has completed, and everything appears to be returning to normal. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 4:14 PM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com<mailto:james.urwiller@stealthbroadband.com> -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET><mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com<mailto:gary.glissendorf@sdncommunications.com> -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET><mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET<mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com<mailto:jlumby@mnvoip.com> ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.*** ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by David Farmer

The GigaPoP and the University are in good shape, I shutdown our peers down about 4:00 and brought them back about 5:40. Traffic levels are back to about where they should be for this time of day. Thanks On Thu, Apr 16, 2020 at 6:03 PM James Urwiller < james.urwiller(a)turnkey-telecom.com> wrote: > Hows it lookin? Looks like my peers are back up. > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Thursday, April 16, 2020 16:14 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Sorry guys, I am running a bit behind, so this will end up being closer to > 5PM anyway... > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Thursday, April 16, 2020 11:10 AM > To: MICE Discuss > Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > It is the MICE core. I would be willing to move it to 5PM CDT, however I > would want the board to give their blessing since it was already announced > for 4PM > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > James Urwiller > Sent: Thursday, April 16, 2020 11:07 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > This is on the mice core? Or the USInternet switch.. are they one in the > same? Im confused. > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Thursday, April 16, 2020 11:01 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Correct, any port number starting with 3/ is on the card. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Gary > Glissendorf > Sent: Thursday, April 16, 2020 10:58 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Jermey which card is the 100G. is it slot 3 > > > Gary Glissendorf | Network Architect > gary.glissendorf(a)sdncommunications.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Wednesday, April 15, 2020 5:56 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > <table style="border:1px; border-style:solid; border-color:#ac6337; > background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This > email originated from outside of SDN Communications. Exercise caution when > opening attachments or clicking links, especially from unknown > senders.</font></td></tr></table><br> > > In troubleshooting the 100G line card, Arista has requested that the line > card be power cycled. Being that there is a possibility that the line card > would not recover from a restart, I am working with Dave @ the UofM to have > a spare available in the event that the card fails to restart. I plan on > blocking BGP to all ports on the card before restarting the card. > Based on the current state of the card, I am unsure that it will block BGP > properly. I plan on restarting the card tomorrow at 4PM CDT. In the past > I > have noticed that the 100G cards take a long time to boot up, so I would > anticipate a minimum of 15 minutes of downtime if everything goes > perfectly. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Wednesday, April 15, 2020 4:18 PM > To: MICE Discuss > Subject: 100G Linecard > > I wanted to let everyone know that we are experiencing an issue with the > 100G linecard on the MICE Arista core switch. The card is currently > outputting errors and not detecting new optics. I am currently working > with > Arista to troubleshoot the issue. At the moment any 100G upgrades would > need to be on hold until the issue is resolved. All established > connections > seem to be working fine. I will report back as soon as I have more > information. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > ***This message and any attachments are solely for the intended recipient. > If you are not the intended recipient, disclosure, copying, use or > distribution of the information included in this message is prohibited -- > Please immediately and permanently delete.*** > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

Everything should be back to normal now Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: James Urwiller <james.urwiller(a)TURNKEY-TELECOM.COM> Date: 4/16/20 6:02 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE _____ >From : James Urwiller [james.urwiller(a)TURNKEY-TELECOM.COM] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Thursday, April 16 2020 17:22:43 Hows it lookin? Looks like my peers are back up. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 16:14 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders. In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Ryan Malek

That's a pretty good reason... I'll fix my BGP. ------------------------------------------------------------------------ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 4/16/2020 5:47 PM, Jeremy Lumby wrote: > The main reason is because mice is 100% volunteer and it is difficult > to get volunteers to come in in the middle of the night > > > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S > Suite 216 > Bloomington MN 55425 > Main 612-355-7740 > Direct 612-392-6814 > EFax 952-873-7425 > jlumby(a)mnvoip.com > > > > -------- Original message -------- > From: Ryan Malek <rmalek(a)ROUTER12.NET> > Date: 4/16/20 5:43 PM (GMT-06:00) > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > ------------------------------------------------------------------------ > From : Ryan Malek [rmalek(a)ROUTER12.NET] > To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] > Date : Thursday, April 16 2020 17:43:42 > > I hope this isn't a stupid question... but why was this done at 5PM? > > I found out in a very brutal manner that something is wrong with my > BGP set up... > > > ------------------------------------------------------------------------ > Ryan Malek - Router12 Networks LLC > Internet, Phone, and Hosted Services > > Ph. 641.420.7180 > On 4/16/2020 5:29 PM, Jeremy Lumby wrote: >> The maintenance has completed, and everything appears to be returning to normal. >> >> -----Original Message----- >> From: Jeremy Lumby [mailto:jlumby@mnvoip.com] >> Sent: Thursday, April 16, 2020 4:14 PM >> To: MICE Discuss >> Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... >> >> -----Original Message----- >> From: Jeremy Lumby [mailto:jlumby@mnvoip.com] >> Sent: Thursday, April 16, 2020 11:10 AM >> To: MICE Discuss >> Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM >> >> -----Original Message----- >> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller >> Sent: Thursday, April 16, 2020 11:07 AM >> To:MICE-DISCUSS@LISTS.IPHOUSE.NET >> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> This is on the mice core? Or the USInternet switch.. are they one in the >> same? Im confused. >> >> James Urwiller >> Network Architect >> Stealth Broadband >> james.urwiller(a)stealthbroadband.com >> >> >> -----Original Message----- >> From: MICE Discuss<MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy >> Lumby >> Sent: Thursday, April 16, 2020 11:01 >> To:MICE-DISCUSS@LISTS.IPHOUSE.NET >> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> Correct, any port number starting with 3/ is on the card. >> >> -----Original Message----- >> From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of >> Gary Glissendorf >> Sent: Thursday, April 16, 2020 10:58 AM >> To:MICE-DISCUSS@LISTS.IPHOUSE.NET >> Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> Jermey which card is the 100G. is it slot 3 >> >> >> Gary Glissendorf | Network Architect >> gary.glissendorf(a)sdncommunications.com >> >> -----Original Message----- >> From: MICE Discuss<MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy >> Lumby >> Sent: Wednesday, April 15, 2020 5:56 PM >> To:MICE-DISCUSS@LISTS.IPHOUSE.NET >> Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE >> >> <table style="border:1px; border-style:solid; border-color:#ac6337; >> background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This >> email originated from outside of SDN Communications. Exercise caution when >> opening attachments or clicking links, especially from unknown >> senders.</font></td></tr></table><br> >> >> In troubleshooting the 100G line card, Arista has requested that the line >> card be power cycled. Being that there is a possibility that the line >> card would not recover from a restart, I am working with Dave @ the UofM >> to have a spare available in the event that the card fails to restart. I >> plan on blocking BGP to all ports on the card before restarting the card. >> Based on the current state of the card, I am unsure that it will block BGP >> properly. I plan on restarting the card tomorrow at 4PM CDT. In the past >> I have noticed that the 100G cards take a long time to boot up, so I would >> anticipate a minimum of 15 minutes of downtime if everything goes >> perfectly. >> >> Jeremy Lumby >> Minnesota VoIP >> 9217 17th Ave S #216 >> Bloomington, MN 55425 >> M: 612-355-7740 >> D: 612-392-6814 >> F: 952-873-7425 >> jlumby(a)mnvoip.com >> >> >> >> -----Original Message----- >> From: Jeremy Lumby [mailto:jlumby@mnvoip.com] >> Sent: Wednesday, April 15, 2020 4:18 PM >> To: MICE Discuss >> Subject: 100G Linecard >> >> I wanted to let everyone know that we are experiencing an issue with the >> 100G linecard on the MICE Arista core switch. The card is currently >> outputting errors and not detecting new optics. I am currently working >> with Arista to troubleshoot the issue. At the moment any 100G upgrades >> would need to be on hold until the issue is resolved. All established >> connections seem to be working fine. I will report back as soon as I have >> more information. >> >> Jeremy Lumby >> Minnesota VoIP >> 9217 17th Ave S #216 >> Bloomington, MN 55425 >> M: 612-355-7740 >> D: 612-392-6814 >> F: 952-873-7425 >> jlumby(a)mnvoip.com >> ***This message and any attachments are solely for the intended recipient. >> If you are not the intended recipient, disclosure, copying, use or >> distribution of the information included in this message is prohibited -- >> Please immediately and permanently delete.*** > > ------------------------------------------------------------------------ > > To unsubscribe from the MICE-DISCUSS list, click the following link: > http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 >

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

The main reason is because mice is 100% volunteer and it is difficult to get volunteers to come in in the middle of the night Jeremy Lumby Minnesota VoIP 9217 17th Ave S Suite 216 Bloomington MN 55425 Main 612-355-7740 Direct 612-392-6814 EFax 952-873-7425 jlumby(a)mnvoip.com -------- Original message -------- From: Ryan Malek <rmalek(a)ROUTER12.NET> Date: 4/16/20 5:43 PM (GMT-06:00) To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE _____ >From : Ryan Malek [rmalek(a)ROUTER12.NET] To : MICE-DISCUSS(a)LISTS.IPHOUSE.NET [MICE-DISCUSS(a)LISTS.IPHOUSE.NET] Date : Thursday, April 16 2020 17:43:42 I hope this isn't a stupid question... but why was this done at 5PM? I found out in a very brutal manner that something is wrong with my BGP set up... _____ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 4/16/2020 5:29 PM, Jeremy Lumby wrote: The maintenance has completed, and everything appears to be returning to normal. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 4:14 PM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Ryan Malek

I hope this isn't a stupid question... but why was this done at 5PM? I found out in a very brutal manner that something is wrong with my BGP set up... ------------------------------------------------------------------------ Ryan Malek - Router12 Networks LLC Internet, Phone, and Hosted Services Ph. 641.420.7180 On 4/16/2020 5:29 PM, Jeremy Lumby wrote: > The maintenance has completed, and everything appears to be returning to normal. > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Thursday, April 16, 2020 4:14 PM > To: MICE Discuss > Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Thursday, April 16, 2020 11:10 AM > To: MICE Discuss > Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller > Sent: Thursday, April 16, 2020 11:07 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > This is on the mice core? Or the USInternet switch.. are they one in the > same? Im confused. > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Thursday, April 16, 2020 11:01 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Correct, any port number starting with 3/ is on the card. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Gary Glissendorf > Sent: Thursday, April 16, 2020 10:58 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Jermey which card is the 100G. is it slot 3 > > > Gary Glissendorf | Network Architect > gary.glissendorf(a)sdncommunications.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Wednesday, April 15, 2020 5:56 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > <table style="border:1px; border-style:solid; border-color:#ac6337; > background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This > email originated from outside of SDN Communications. Exercise caution when > opening attachments or clicking links, especially from unknown > senders.</font></td></tr></table><br> > > In troubleshooting the 100G line card, Arista has requested that the line > card be power cycled. Being that there is a possibility that the line > card would not recover from a restart, I am working with Dave @ the UofM > to have a spare available in the event that the card fails to restart. I > plan on blocking BGP to all ports on the card before restarting the card. > Based on the current state of the card, I am unsure that it will block BGP > properly. I plan on restarting the card tomorrow at 4PM CDT. In the past > I have noticed that the 100G cards take a long time to boot up, so I would > anticipate a minimum of 15 minutes of downtime if everything goes > perfectly. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Wednesday, April 15, 2020 4:18 PM > To: MICE Discuss > Subject: 100G Linecard > > I wanted to let everyone know that we are experiencing an issue with the > 100G linecard on the MICE Arista core switch. The card is currently > outputting errors and not detecting new optics. I am currently working > with Arista to troubleshoot the issue. At the moment any 100G upgrades > would need to be on hold until the issue is resolved. All established > connections seem to be working fine. I will report back as soon as I have > more information. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > ***This message and any attachments are solely for the intended recipient. > If you are not the intended recipient, disclosure, copying, use or > distribution of the information included in this message is prohibited -- > Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by David Farmer

Facebook(Ethernet3/10/1) Graph has bogus data several hundred PetaBytes which is throwing off the aggregate traffic graph as well, someone with cacti access needs to delete the bogus data. On Thu, Apr 16, 2020 at 5:29 PM Jeremy Lumby <jlumby(a)mnvoip.com> wrote: > The maintenance has completed, and everything appears to be returning to > normal. > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Thursday, April 16, 2020 4:14 PM > To: MICE Discuss > Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Sorry guys, I am running a bit behind, so this will end up being closer to > 5PM anyway... > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Thursday, April 16, 2020 11:10 AM > To: MICE Discuss > Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > It is the MICE core. I would be willing to move it to 5PM CDT, however I > would want the board to give their blessing since it was already announced > for 4PM > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > James Urwiller > Sent: Thursday, April 16, 2020 11:07 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > This is on the mice core? Or the USInternet switch.. are they one in the > same? Im confused. > > James Urwiller > Network Architect > Stealth Broadband > james.urwiller(a)stealthbroadband.com > > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Thursday, April 16, 2020 11:01 > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Correct, any port number starting with 3/ is on the card. > > -----Original Message----- > From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of > Gary Glissendorf > Sent: Thursday, April 16, 2020 10:58 AM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > Jermey which card is the 100G. is it slot 3 > > > Gary Glissendorf | Network Architect > gary.glissendorf(a)sdncommunications.com > > -----Original Message----- > From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy > Lumby > Sent: Wednesday, April 15, 2020 5:56 PM > To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET > Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE > > <table style="border:1px; border-style:solid; border-color:#ac6337; > background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This > email originated from outside of SDN Communications. Exercise caution when > opening attachments or clicking links, especially from unknown > senders.</font></td></tr></table><br> > > In troubleshooting the 100G line card, Arista has requested that the line > card be power cycled. Being that there is a possibility that the line > card would not recover from a restart, I am working with Dave @ the UofM > to have a spare available in the event that the card fails to restart. I > plan on blocking BGP to all ports on the card before restarting the card. > Based on the current state of the card, I am unsure that it will block BGP > properly. I plan on restarting the card tomorrow at 4PM CDT. In the past > I have noticed that the 100G cards take a long time to boot up, so I would > anticipate a minimum of 15 minutes of downtime if everything goes > perfectly. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > > > > -----Original Message----- > From: Jeremy Lumby [mailto:jlumby@mnvoip.com] > Sent: Wednesday, April 15, 2020 4:18 PM > To: MICE Discuss > Subject: 100G Linecard > > I wanted to let everyone know that we are experiencing an issue with the > 100G linecard on the MICE Arista core switch. The card is currently > outputting errors and not detecting new optics. I am currently working > with Arista to troubleshoot the issue. At the moment any 100G upgrades > would need to be on hold until the issue is resolved. All established > connections seem to be working fine. I will report back as soon as I have > more information. > > Jeremy Lumby > Minnesota VoIP > 9217 17th Ave S #216 > Bloomington, MN 55425 > M: 612-355-7740 > D: 612-392-6814 > F: 952-873-7425 > jlumby(a)mnvoip.com > ***This message and any attachments are solely for the intended recipient. > If you are not the intended recipient, disclosure, copying, use or > distribution of the information included in this message is prohibited -- > Please immediately and permanently delete.*** > -- =============================================== David Farmer Email:farmer@umn.edu Networking & Telecommunication Services Office of Information Technology University of Minnesota 2218 University Ave SE Phone: 612-626-0815 Minneapolis, MN 55414-3029 Cell: 612-812-9952 ===============================================

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

The maintenance has completed, and everything appears to be returning to normal. -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 4:14 PM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by James Urwiller

Hows it lookin? Looks like my peers are back up. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 16:14 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

Sorry guys, I am running a bit behind, so this will end up being closer to 5PM anyway... -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Thursday, April 16, 2020 11:10 AM To: MICE Discuss Subject: RE: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by James Urwiller

It can stay 4PM CDT if it must.. I will notify my customers. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:10 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

It is the MICE core. I would be willing to move it to 5PM CDT, however I would want the board to give their blessing since it was already announced for 4PM -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of James Urwiller Sent: Thursday, April 16, 2020 11:07 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by James Urwiller

This is on the mice core? Or the USInternet switch.. are they one in the same? Im confused. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by James Urwiller

I would very much not like this to happen at 4PM due to Accessing HE for Transit services as well as exchange traffic. Even after 5 would be better for business sake. James Urwiller Network Architect Stealth Broadband james.urwiller(a)stealthbroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Thursday, April 16, 2020 11:01 To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

Correct, any port number starting with 3/ is on the card. -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Gary Glissendorf Sent: Thursday, April 16, 2020 10:58 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: 100G Linecard EMERGENCY MAINTENANCE

by Gary Glissendorf

Jermey which card is the 100G. is it slot 3 Gary Glissendorf | Network Architect gary.glissendorf(a)sdncommunications.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Wednesday, April 15, 2020 5:56 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] 100G Linecard EMERGENCY MAINTENANCE <table style="border:1px; border-style:solid; border-color:#ac6337; background-color:#ffeb9c"><tr><td><font size=2 face=calibri>CAUTION: This email originated from outside of SDN Communications. Exercise caution when opening attachments or clicking links, especially from unknown senders.</font></td></tr></table><br> In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com ***This message and any attachments are solely for the intended recipient. If you are not the intended recipient, disclosure, copying, use or distribution of the information included in this message is prohibited -- Please immediately and permanently delete.***

April 16, 2020

Re: [ADV] [MICE-DISCUSS] Savage Communications Upgrade

by Jeremy Lumby

4/33 has been added -----Original Message----- From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Mike Danielson Sent: Thursday, April 16, 2020 9:23 AM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [ADV] [MICE-DISCUSS] Savage Communications Upgrade We have cut our traffic over to the new port 5/33 and we are ready for port 4/33 to be put into the LAG group. Thanks, Mike Danielson Savage Communications Inc. Office: 320-384-7442 x717 Cell: 651-343-4241 scibroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Thursday, April 02, 2020 4:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [ADV] [MICE-DISCUSS] Savage Communications Upgrade On Thu, Apr 02, 2020 at 03:03:34PM -0500, Jay Hanke wrote: >Savage Communications is upgrading from 1x10G to 2x10G LAG. > >Please allocate Ethernet 5/33 on the Main switch. This will require a >new LAG and then their existing port (4/33) will be migrated into the LAG. Okay, Port-Channel36 setup, Eth 5/33 put into the LACP lag, and ready to go for them to cut their traffic over to it when they are ready. Once traffic has stopped over 4/33, we can put that into the LAG group and they can continue along when they want. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 16, 2020

Re: [ADV] [MICE-DISCUSS] Savage Communications Upgrade

by Mike Danielson

We have cut our traffic over to the new port 5/33 and we are ready for port 4/33 to be put into the LAG group. Thanks, Mike Danielson Savage Communications Inc. Office: 320-384-7442 x717 Cell: 651-343-4241 scibroadband.com -----Original Message----- From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Doug McIntyre Sent: Thursday, April 02, 2020 4:04 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] [ADV] [MICE-DISCUSS] Savage Communications Upgrade On Thu, Apr 02, 2020 at 03:03:34PM -0500, Jay Hanke wrote: >Savage Communications is upgrading from 1x10G to 2x10G LAG. > >Please allocate Ethernet 5/33 on the Main switch. This will require a >new LAG and then their existing port (4/33) will be migrated into the LAG. Okay, Port-Channel36 setup, Eth 5/33 put into the LACP lag, and ready to go for them to cut their traffic over to it when they are ready. Once traffic has stopped over 4/33, we can put that into the LAG group and they can continue along when they want. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 16, 2020

100G Linecard EMERGENCY MAINTENANCE

by Jeremy Lumby

In troubleshooting the 100G line card, Arista has requested that the line card be power cycled. Being that there is a possibility that the line card would not recover from a restart, I am working with Dave @ the UofM to have a spare available in the event that the card fails to restart. I plan on blocking BGP to all ports on the card before restarting the card. Based on the current state of the card, I am unsure that it will block BGP properly. I plan on restarting the card tomorrow at 4PM CDT. In the past I have noticed that the 100G cards take a long time to boot up, so I would anticipate a minimum of 15 minutes of downtime if everything goes perfectly. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com -----Original Message----- From: Jeremy Lumby [mailto:jlumby@mnvoip.com] Sent: Wednesday, April 15, 2020 4:18 PM To: MICE Discuss Subject: 100G Linecard I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

April 15, 2020

100G Linecard

by Jeremy Lumby

I wanted to let everyone know that we are experiencing an issue with the 100G linecard on the MICE Arista core switch. The card is currently outputting errors and not detecting new optics. I am currently working with Arista to troubleshoot the issue. At the moment any 100G upgrades would need to be on hold until the issue is resolved. All established connections seem to be working fine. I will report back as soon as I have more information. Jeremy Lumby Minnesota VoIP 9217 17th Ave S #216 Bloomington, MN 55425 M: 612-355-7740 D: 612-392-6814 F: 952-873-7425 jlumby(a)mnvoip.com

April 15, 2020

SDN Upgrade

by Jay Hanke

SDN will be upgrading their existing LAG configuration to 1x100G. The new port assignment on the main switch is Ethernet3/23/1. Welcome aboard again! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 15, 2020

Re: Port was down

by Todd Mueller

Thanks for checking Jeremy, I am not sure how that could have happened yet but the MAC address should be helpful in tracking that down. Todd From: MICE Discuss <MICE-DISCUSS(a)LISTS.IPHOUSE.NET> On Behalf Of Jeremy Lumby Sent: Tuesday, April 14, 2020 4:25 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: Re: [MICE-DISCUSS] Port was down Looks like the switch detected a second MAC address, and errordisabled the port: Apr 14 10:06:34 core-sw0 PortSec: %PORTSECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 64:9e:f3:eb:6e:b0 on vlan 1 interface Ethernet4/21. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Todd Mueller Sent: Tuesday, April 14, 2020 4:06 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET <mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET> Subject: [MICE-DISCUSS] Port was down Our (Velocity Telephone) MICE port physically went down at 10:06A and came back up at 2:31P today. Anyone aware of maintenance or anything else? Thanks! Todd _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1 _____ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS <http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1> &A=1

April 14, 2020

Re: Port was down

by Jeremy Lumby

Looks like the switch detected a second MAC address, and errordisabled the port: Apr 14 10:06:34 core-sw0 PortSec: %PORTSECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 64:9e:f3:eb:6e:b0 on vlan 1 interface Ethernet4/21. Jeremy From: MICE Discuss [mailto:MICE-DISCUSS@LISTS.IPHOUSE.NET] On Behalf Of Todd Mueller Sent: Tuesday, April 14, 2020 4:06 PM To: MICE-DISCUSS(a)LISTS.IPHOUSE.NET Subject: [MICE-DISCUSS] Port was down Our (Velocity Telephone) MICE port physically went down at 10:06A and came back up at 2:31P today. Anyone aware of maintenance or anything else? Thanks! Todd To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 14, 2020

Port was down

by Todd Mueller

Our (Velocity Telephone) MICE port physically went down at 10:06A and came back up at 2:31P today. Anyone aware of maintenance or anything else? Thanks! Todd

April 14, 2020

Giga Pop Additional MICE Connection

by Jay Hanke

The UofM Gigapop will be adding an additional connection to MICE via the new WiscNet Remote Switch. Name: University of Minnesota Port: WiscNet Remote Switch ASN: 57 IPv4: 206.108.255.162/24 IPv6: 2001:504:27:0:0:0039::3/64 Route Server: Yes BFD: Yes Please report back the port assignments for our records. Welcome aboard again! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 7, 2020

Re: Blackhole IP if needed?

by Andrew Hoyos

On Apr 6, 2020, at 6:46 PM, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > On 4/6/20 6:38 PM, Andrew Hoyos wrote: >> >> On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager(a)WIKTEL.COM <mailto:rlaager@WIKTEL.COM>> wrote: >>> >>> On 4/6/20 5:17 PM, Andrew Hoyos wrote: >>>> This came up in two different conversations in June and January (per >>>> mice-discuss archives) >>>> I think the concern here is that the route servers aren’t doing IRR >>>> filtering of members yet. >>>> I agree 100% with the approach below, but with one caveat - we need to >>>> be filtering members on the route servers first via IRR. >>>> Otherwise, nothing preventing a member from advertising a /32 tagged >>>> with a blackhole community of 8.8.8.8, or another member’s IP address. >>> >>> Nothing is stopping me from announcing 8.8.8.0/24 right now, right? >> >> I agree, that’s true in the current scenario. Nothing preventing you from doing that towards the route servers. >> >> As it stands right now, though, we wouldn’t necessarily have to accept that route, and we’d have local control to reject/block or filter your ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: operational processes to fix the issue without reliance on MICE intervention, and without completely dropping our route server sessions). >> >> Or perhaps folks are already doing self-generated prefix lists and actions towards route server prefixes already (we, for example, passively keep an eye on ASNs/IRR data vs routes learned from MICE route servers). >> >> In short, we’ve got tools to deal, if you do. > ... >> In the case we enable a RTBH community without strict filtering, we’ve taken the control out of the members hands, and put a powerful tool in any members hands with no regard to security. > I'm still not understanding the difference between someone originating 8.8.8.0/24 and 8.8.8.8/32+65535:666. In either case (with blackholing support added), the route server will accept and propagate that to route server users. Whether they choose to accept that announcement is up to them the same either way. > > Note that announcements to the route server do not directly result in blackholing traffic across the fabric. It simply sets a special next-hop.* If a participant chooses to accept that BGP route and route traffic to that next-hop*, only then will the traffic will be eaten. To put a finer point on this: if someone does not use the route servers, this change does not affect them in any way no matter what anyone announces as a blackhole route to the route servers. Ack, thanks. Perhaps I misunderstood intent of actually dropping in flight traffic on switch fabric - this makes more sense and lessens my concern. Still think we should get IRR filtering in place first.

April 7, 2020

Re: Blackhole IP if needed?

by Richard Laager

On 4/6/20 6:38 PM, Andrew Hoyos wrote: > > On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager(a)WIKTEL.COM > <mailto:rlaager@WIKTEL.COM>> wrote: >> >> On 4/6/20 5:17 PM, Andrew Hoyos wrote: >>> This came up in two different conversations in June and January (per >>> mice-discuss archives) >>> I think the concern here is that the route servers aren’t doing IRR >>> filtering of members yet. >>> I agree 100% with the approach below, but with one caveat - we need to >>> be filtering members on the route servers first via IRR. >>> Otherwise, nothing preventing a member from advertising a /32 tagged >>> with a blackhole community of 8.8.8.8, or another member’s IP address. >> >> Nothing is stopping me from announcing 8.8.8.0/24 right now, right? > > I agree, that’s true in the current scenario. Nothing preventing you > from doing that towards the route servers. > > As it stands right now, though, we wouldn’t necessarily have to accept > that route, and we’d have local control to reject/block or filter your > ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: > operational processes to fix the issue without reliance on MICE > intervention, and without completely dropping our route server sessions). > > Or perhaps folks are already doing self-generated prefix lists and > actions towards route server prefixes already (we, for example, > passively keep an eye on ASNs/IRR data vs routes learned from MICE > route servers). > > In short, we’ve got tools to deal, if you do. ... > In the case we enable a RTBH community without strict filtering, we’ve > taken the control out of the members hands, and put a powerful tool in > any members hands with no regard to security. I'm still not understanding the difference between someone originating 8.8.8.0/24 and 8.8.8.8/32+65535:666. In either case (with blackholing support added), the route server will accept and propagate that to route server users. Whether they choose to accept that announcement is up to them the same either way. Note that announcements to the route server do /not/ directly result in blackholing traffic across the fabric. It simply sets a special next-hop.* If a participant chooses to accept that BGP route and route traffic to that next-hop*, only then will the traffic will be eaten. To put a finer point on this: if someone does not use the route servers, this change does not affect them in any way no matter what anyone announces as a blackhole route to the route servers. * I forgot to mention that we'll need something on the route servers that will respond to ARP/ND requests for the blackhole IPs and return the blackhole MAC. -- Richard

April 6, 2020

Re: Blackhole IP if needed?

by Andrew Hoyos

On Apr 6, 2020, at 5:25 PM, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > On 4/6/20 5:17 PM, Andrew Hoyos wrote: >> This came up in two different conversations in June and January (per >> mice-discuss archives) >> I think the concern here is that the route servers aren’t doing IRR >> filtering of members yet. >> I agree 100% with the approach below, but with one caveat - we need to >> be filtering members on the route servers first via IRR. >> Otherwise, nothing preventing a member from advertising a /32 tagged >> with a blackhole community of 8.8.8.8, or another member’s IP address. > > Nothing is stopping me from announcing 8.8.8.0/24 right now, right? I agree, that’s true in the current scenario. Nothing preventing you from doing that towards the route servers. As it stands right now, though, we wouldn’t necessarily have to accept that route, and we’d have local control to reject/block or filter your ASN, drop RPKI invalid prefixes, etc, and to react as we please (ie: operational processes to fix the issue without reliance on MICE intervention, and without completely dropping our route server sessions). Or perhaps folks are already doing self-generated prefix lists and actions towards route server prefixes already (we, for example, passively keep an eye on ASNs/IRR data vs routes learned from MICE route servers). In short, we’ve got tools to deal, if you do. > I agree that the IRR filtering is important, but is that absolutely > necessary here? In the case we enable a RTBH community without strict filtering, we’ve taken the control out of the members hands, and put a powerful tool in any members hands with no regard to security. Say someone does do something nefarious, or inadvertent that breaks something important? How quickly can it be fixed, who can fix it, etc, etc? Operational processes become a lot more important. The fix on a members side is a lot more intrusive (ie: turn down port), and the impact is much greater (blackholed across entire IX fabric - and potentially also affecting bilateral peering traffic). I think it’d be extremely irresponsible of MICE, and I’m vehemently opposed to enabling RTBH on the route servers or fabric, without first having the proper tools in place to ensure it can be used sanely and securely (IRR based filtering + route server looking glass). My $0.02… — Andrew Hoyos hoyosa(a)gmail.com <mailto:hoyosa@gmail.com>

April 6, 2020

Re: Blackhole IP if needed?

by Richard Laager

On 4/6/20 5:17 PM, Andrew Hoyos wrote: > This came up in two different conversations in June and January (per > mice-discuss archives) > I think the concern here is that the route servers aren’t doing IRR > filtering of members yet. > I agree 100% with the approach below, but with one caveat - we need to > be filtering members on the route servers first via IRR. > Otherwise, nothing preventing a member from advertising a /32 tagged > with a blackhole community of 8.8.8.8, or another member’s IP address. Nothing is stopping me from announcing 8.8.8.0/24 right now, right? I agree that the IRR filtering is important, but is that absolutely necessary here? -- Richard

April 6, 2020

Re: Blackhole IP if needed?

by Andrew Hoyos

This came up in two different conversations in June and January (per mice-discuss archives) I think the concern here is that the route servers aren’t doing IRR filtering of members yet. I agree 100% with the approach below, but with one caveat - we need to be filtering members on the route servers first via IRR. Otherwise, nothing preventing a member from advertising a /32 tagged with a blackhole community of 8.8.8.8, or another member’s IP address. — Andrew Hoyos hoyosa(a)gmail.com <mailto:hoyosa@gmail.com> > On Apr 6, 2020, at 5:12 PM, Richard Laager <rlaager(a)WIKTEL.COM> wrote: > > On 4/6/20 4:50 PM, Frank Bulk wrote: >> One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS >> attack for about 20 minutes and some of the incoming traffic was going >> over our MICE link. >> >> Do the MICE admins have a way to blackhole our IP, if needed? > > No. > > I'd love to see us implement something like this: > https://www.seattleix.net/blackholing > > Let's see if we can get this done. Here's what I see as steps: > > 1) Pick IPs and MACs. > Here's a proposal: > IPv4: 206.108.255.0 > same idea as SIX > IPv6: 2001:504:27:0:0:FFFF::666 > from 65535:666 used in RFC 7999 > MAC: 66:66:de:ad:be:ef > same as SIX > > 2) Jeremy?: Configure MAC ACLs to drop traffic to that MAC on the core > switches. > > 3) Doug?: Configure the route servers to: > accept /32 (IPv4) and /128 (IPv6) > set next-hop to the blackhole IP (see above) > add no-export community > when: > next-hop == blackhole IP > OR > 65535:666 is set > See also the BIRD example on the SIX page. > > 4) Me: Document this on our website & notify the members it's ready. > Ask, but not require, that remote switches do the same. > > -- > Richard

April 6, 2020

Re: Blackhole IP if needed?

by Richard Laager

On 4/6/20 4:50 PM, Frank Bulk wrote: > One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS > attack for about 20 minutes and some of the incoming traffic was going > over our MICE link. > > Do the MICE admins have a way to blackhole our IP, if needed? No. I'd love to see us implement something like this: https://www.seattleix.net/blackholing Let's see if we can get this done. Here's what I see as steps: 1) Pick IPs and MACs. Here's a proposal: IPv4: 206.108.255.0 same idea as SIX IPv6: 2001:504:27:0:0:FFFF::666 from 65535:666 used in RFC 7999 MAC: 66:66:de:ad:be:ef same as SIX 2) Jeremy?: Configure MAC ACLs to drop traffic to that MAC on the core switches. 3) Doug?: Configure the route servers to: accept /32 (IPv4) and /128 (IPv6) set next-hop to the blackhole IP (see above) add no-export community when: next-hop == blackhole IP OR 65535:666 is set See also the BIRD example on the SIX page. 4) Me: Document this on our website & notify the members it's ready. Ask, but not require, that remote switches do the same. -- Richard

April 6, 2020

Blackhole IP if needed?

by Frank Bulk

One our IPs, 96.31.13.225, was on the receiving end of a volumetric DoS attack for about 20 minutes and some of the incoming traffic was going over our MICE link. Do the MICE admins have a way to blackhole our IP, if needed? Frank Premier Communications (AS53347) FiberNet Communications (AS18883)

April 6, 2020

Re: ADV: [MICE-DISCUSS] Zayo outage in the metro?

by Jeff Dahna

We have a A-Link to Plymouth down since 10:14pm yesterday. No maintenance notice either. I put in a ticket with IQ last night. IQ response was Zayo maintenance, no notification and were added to the master ticket. Zayo went past maintenance window and was hands off 9ish. 10:30am - Called IQ to escalate and got the usual response. Still not fixed.... Jeff Dahna General Manager | City of Windom Telecom Dept-Windomnet On Apr 4, 2020 13:58, Dean Bahls <dean.bahls(a)COOPERATIVE-NETWORKS.COM> wrote: Anyone know of an outage that Zayo has been experiencing in the Metro area since about 10:30pm last night? We’ve found that it was a planned maintenance that nobody had been notified about. I don’t know if it affects other systems (like Ethernet or dark fiber), but toll services using Intelliquent (Onvoy) have been down since that time and nobody has gotten any updates about this thing. Thanks for any info you can share! Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com<mailto:dean.bahls@cooperative-networks.com> [cid:image003.jpg@01D60A89.0CBE6550] ________________________________ To unsubscribe from the MICE-DISCUSS list, click the following link: http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1

April 4, 2020

Zayo outage in the metro?

by Dean Bahls

Anyone know of an outage that Zayo has been experiencing in the Metro area since about 10:30pm last night? We've found that it was a planned maintenance that nobody had been notified about. I don't know if it affects other systems (like Ethernet or dark fiber), but toll services using Intelliquent (Onvoy) have been down since that time and nobody has gotten any updates about this thing. Thanks for any info you can share! Dean Dean Bahls Network Services Manager | Cooperative Network Services, LLC 14 Main St SW | PO Box J | Menahga, MN 56464 218.346.3635 (Direct) | 218.564.3000 (Main Office) dean.bahls(a)cooperative-networks.com <mailto:dean.bahls@cooperative-networks.com>

April 4, 2020

Re: [ADV] [MICE-DISCUSS] Wiscnet Remote Switch

by Richard Laager

On 4/3/20 11:05 AM, Doug McIntyre wrote: > I have allocated Port-Channel 37 for Wiscnet. That's what I was assuming it was going to be. So I just pushed the changes I made. The website should be updated, including the equipment diagram and the Order Form. -- Richard

April 3, 2020

Re: [ADV] [MICE-DISCUSS] Wiscnet Remote Switch

by Doug McIntyre

On Fri, Apr 03, 2020 at 08:03:08AM -0500, Jay Hanke wrote: >Please allocate port Ethernet 3/22/1 on the Main Switch for the new Wiscnet >Remote switch. > >Please configure the port as a single member LAG group and the port >security settings for a remote switch. > All set, I have allocated Port-Channel 37 for Wiscnet. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 3, 2020

Wiscnet Remote Switch

by Jay Hanke

Please allocate port Ethernet 3/22/1 on the Main Switch for the new Wiscnet Remote switch. Please configure the port as a single member LAG group and the port security settings for a remote switch. Welcome Aboard Again! -- Jay Hanke, President South Front Networks jayhanke(a)southfront.io Phone 612-204-0000

April 3, 2020

Re: [ADV] [MICE-DISCUSS] Savage Communications Upgrade

by Doug McIntyre

On Thu, Apr 02, 2020 at 03:03:34PM -0500, Jay Hanke wrote: >Savage Communications is upgrading from 1x10G to 2x10G LAG. > >Please allocate Ethernet 5/33 on the Main switch. This will require a new >LAG and then their existing port (4/33) will be migrated into the LAG. Okay, Port-Channel36 setup, Eth 5/33 put into the LACP lag, and ready to go for them to cut their traffic over to it when they are ready. Once traffic has stopped over 4/33, we can put that into the LAG group and they can continue along when they want. -- Doug McIntyre <merlyn(a)iphouse.net> ~.~ ipHouse ~.~ Network Engineer/Provisioning/Jack of all Trades

April 2, 2020