Wireless Water Cooler

I thought it would be fun to share mistakes made on the job, wireless or not, that caused you grief and what you learned to never do again.  It's often mistakes that we learn the most from and never forget.  They're fun to share so I thought I'd post one of my recent self-inflicted outages:

My story

My most recent experience was this past weekend.  I was remotely upgrading 6 stacks of MAS 3500s and was trying to automate the process so the switches would be upgraded overnight while I was sound asleep.  I uploaded the firmware to the switches and scheduled the switch reboots in Airwave.  I wanted to supress alerts for the switches while they rebooted so I setup a 30 min maintenance window for the Airwave group the switches were in.  I went to bed confident that I'd wake up with upgraded 3500s.

I woke up the next morning and discovered that none of the switches were up.  I couldn't figure out what happened so I had to head in to work.  At work, I consoled into one switch stack and found that the switch hadn't been upgraded and it had a config from a different switch on it.  No wonder it was down!  Checked all the other switches and they had the same config as the first.  I went to Airwave, grabbed the backed up configs and started loading them up on the switches.  4 hours later I was back up and running.

Lesson learned

If you're an Airwave expert (I'm not), you may have spotted my mistake - the maintenance window.  I mistakenly assumed the maintenance window would supress alerts during the window I specified.  I had no idea that this would put my monitor only devices into manage mode and push a config to them.  I called TAC, thinking the scheduled reboot was buggy, but they quickly pointed out my mistake in the way I used the maintenance window.  TAC was great about explaining the way the feature is meant to work and that I should be very careful when using it.  Very embarassing, BUT I learned the hard way how this feature works!

What lesson did you have to learn the hard way?

On a link from an upstream router to a controller: 

Using:

switchport trunk allowed vlan 172

instead of:

switchport trunk allowed vlan add 172

Woops. Right @victorfabian ?

Never change configs while eating lunch!

Changing config on wrong port when working remotely.1500 miles away! Had to politely ask to get a console hookup.

Tim you know that this reminds me when i used to work on an ISP and someone did that? on one switch that had like 600 vlans going through  that trunk? omg suddenly a big part of the clients were down.  The engineer that accidentally did that of course he noticed like woops... and start adding the vlans as fast as he could.

Cheers

Carlos

having your test switch ssh session open next to the production core switch is a bad idea. luckily nothing bad happened but i was close a few times.

the worst was when i reloaded the test switch (i thought) and suddenly my connection dropped, scared i had used the wrong window. turned out someone yanked out my network cable by accident :)

I ran into almost the same situation last year, except we were upgrading 28 Wi-Fi controllers.

Having an awfully old config (two year at least by then) being pushed on every single controller and another weirder config on all IAPs made a nightmare out of a relatively good upgrade. It tooks us (2 Wi-Fi admins, our local SE and two TAC engineers) a long 19 hours to recover everything.

Lesson learned: no more Maintenance ever. Period.

There is still something awkward with this though. I am unaware of any way to suppress any alarms that may be generated by AirWave for anything it monitors. Is there a way, other than shutting it off, to do so?