Wireless Water Cooler

Reply
MVP
Posts: 1,110
Registered: ‎10-11-2011

Brutal mistakes and lessons learned

I thought it would be fun to share mistakes made on the job, wireless or not, that caused you grief and what you learned to never do again.  It's often mistakes that we learn the most from and never forget.  They're fun to share so I thought I'd post one of my recent self-inflicted outages:

 

 

My story

 

My most recent experience was this past weekend.  I was remotely upgrading 6 stacks of MAS 3500s and was trying to automate the process so the switches would be upgraded overnight while I was sound asleep.  I uploaded the firmware to the switches and scheduled the switch reboots in Airwave.  I wanted to supress alerts for the switches while they rebooted so I setup a 30 min maintenance window for the Airwave group the switches were in.  I went to bed confident that I'd wake up with upgraded 3500s.

 

I woke up the next morning and discovered that none of the switches were up.  I couldn't figure out what happened so I had to head in to work.  At work, I consoled into one switch stack and found that the switch hadn't been upgraded and it had a config from a different switch on it.  No wonder it was down!  Checked all the other switches and they had the same config as the first.  I went to Airwave, grabbed the backed up configs and started loading them up on the switches.  4 hours later I was back up and running.

 

Lesson learned

 

If you're an Airwave expert (I'm not), you may have spotted my mistake - the maintenance window.  I mistakenly assumed the maintenance window would supress alerts during the window I specified.  I had no idea that this would put my monitor only devices into manage mode and push a config to them.  I called TAC, thinking the scheduled reboot was buggy, but they quickly pointed out my mistake in the way I used the maintenance window.  TAC was great about explaining the way the feature is meant to work and that I should be very careful when using it.  Very embarassing, BUT I learned the hard way how this feature works!

 

 

What lesson did you have to learn the hard way?

 

 

=======================================
If a reply adequately addresses your issue, please click on the "Accept as Solution" and "Give Kudos" button so this information can benefit other users.
Guru Elite
Posts: 8,456
Registered: ‎09-08-2010

Re: Brutal mistakes and lessons learned

[ Edited ]

On a link from an upstream router to a controller: 

 

Using:

switchport trunk allowed vlan 172

 

instead of:

switchport trunk allowed vlan add 172

 

Woops. Right @victorfabian ?

 

Never change configs while eating lunch!


Tim Cappalli | Aruba Security TME
@timcappalli | timcappalli.me | ACMX #367 / ACCX #480
MVP
Posts: 1,110
Registered: ‎10-11-2011

Re: Brutal mistakes and lessons learned

Oh yeah! Definitely feel your pain there.
=======================================
If a reply adequately addresses your issue, please click on the "Accept as Solution" and "Give Kudos" button so this information can benefit other users.
Community Administrator
Posts: 2,254
Registered: ‎12-03-2013

Re: Brutal mistakes and lessons learned

Changing config on wrong port when working remotely.1500 miles away! Had to politely ask to get a console hookup.

CWNA, ACMP, Security +
MVP
Posts: 2,989
Registered: ‎10-25-2011

Re: Brutal mistakes and lessons learned

[ Edited ]

Tim you know that this reminds me when i used to work on an ISP and someone did that? on one switch that had like 600 vlans going through  that trunk? omg suddenly a big part of the clients were down.  The engineer that accidentally did that of course he noticed like woops... and start adding the vlans as fast as he could.

 

Cheers

Carlos

----------------------------------------------------
Product Manager - Aruba Networks
Alternetworks Corp
MVP
Posts: 1,413
Registered: ‎11-30-2011

Re: Brutal mistakes and lessons learned

having your test switch ssh session open next to the production core switch is a bad idea. luckily nothing bad happened but i was close a few times.

 

the worst was when i reloaded the test switch (i thought) and suddenly my connection dropped, scared i had used the wrong window. turned out someone yanked out my network cable by accident :)

MVP
Posts: 1,110
Registered: ‎10-11-2011

Re: Brutal mistakes and lessons learned

That's a good one! I had a coworker that did the same thing with an
internet router. Self inflicted outage!
=======================================
If a reply adequately addresses your issue, please click on the "Accept as Solution" and "Give Kudos" button so this information can benefit other users.
Contributor II
Posts: 46
Registered: ‎04-24-2013

Re: Brutal mistakes and lessons learned

I ran into almost the same situation last year, except we were upgrading 28 Wi-Fi controllers.

 

Having an awfully old config (two year at least by then) being pushed on every single controller and another weirder config on all IAPs made a nightmare out of a relatively good upgrade. It tooks us (2 Wi-Fi admins, our local SE and two TAC engineers) a long 19 hours to recover everything.

 

Lesson learned: no more Maintenance ever. Period.

 

There is still something awkward with this though. I am unaware of any way to suppress any alarms that may be generated by AirWave for anything it monitors. Is there a way, other than shutting it off, to do so?

Simon
Search Airheads
Showing results for 
Search instead for 
Did you mean: