Network Management

Reply
MVP
Posts: 520
Registered: ‎05-11-2011

How to tutorial: Airwave upgrade cause kernel panic and no backup available...

So - I encountered this when upgrading Airwave from 8.0.0 to 8.0.3.

8.0.3 also has a CentOS kernel upgrade and at the end of the upgrade you get a message informing you of how to proceed to do this. During the script either me or the customer (it was a remote session) halted the script. Re-starting the kernel upgrade script just informed us that everything was fine, and proceed with booting the server.

 

After reboot a message like this was displayed with more textinformation below it.

 

-> kernel panic - not syncing vfs unable to mount root fs on unknown-block(0 0)

 

Google told me various things, but I was not able to fix this.

 

Very new installation, but customer had already spent many hours in the VisualRF section creating walls and placing the AP's. Rebuilding this was not something he would like to do again.. So - I was stupid enough to NOT have copied a nightly-backup file from the system before doing the ugprade, and customer informed me that the off-server backup routine wasn't implemented yet. So - now how to get a backupfile from this filesystem..

 

Note! There might be a way to actually add the correct Kernel version and get the system up and running again, but I couldn't find it. If you have a way - please update this post with the instructions :)

 

Note2! When you create a new Airwave see if you can use the same version you had before you tried to upgrade - meaning the same version as your backup is on. That will cause less problems. After my sucessfull restore I had problems I think was caused by version mismatch, but solved it by doing an amp_upgrade to 8.0.3.1...

 

Note3! A big shout out to sreedhar@aruba-tac for helping me solv this issue! 

 

I hope none of you get in this situation, but now atleast you have some information of how to get around it :)

 

So lets get to it!

First try to boot the Airwave, and press a key to enter Boot menu. See if you can try some of the options there to boot the old Kernel. If that fails, then your next viable option is to boot using CentOS LiveCD, get the backup files, and then just start rebuilding..

 

Download the most recent CentOS LiveCD.iso file.

Upload this to your VMWare datastore

Stop the Airwave Virtual Machine

Edit the Airwave Virtual Machine settings

  • Under Hardware mount this ISO file as the CD-ROM
  • Under Options / Boot Options select "Force BIOS setup" for next boot

Boot the Airwave Appliance with Console open and BIOS setup will start

Change the BOOT sequence to CDROM before Harddrive

Let CentoS Live boot

Start a CMD prompt

Now the trick is to find the correct partition/volume, and mount this as a folder to make the files accessible for you.

First you might want to set the correct IP-setup for your server so you can copy files across the LAN to another server.

  • You do this under System -> Preferences -> Network Connections

This might be different depending on the situation you are in, but this is what we had to do:

  • Sudo sh
    • to get the correct permission level
  • fdisk /l
    • This lists the drives
    • Airwave will have a small partition on /dev/sda1 and the bigger most likely as /dev/sda2. This last one is the one your data is on
  • Cd /
  • Mkdir aw-rescue
    • Create the folder you want to mount the Airwave partition to
  • vgscan
    • Run this to check the various volumes
    • In my case /dev/VolGroup00/LogVol00 is the one I want to mount
  • lvchange --available y /dev/VolGroup00/LogVol00
    • To make this volume available for mounting
  • mount -t ext4 /dev/VolGroup00/LogVol00 /aw-rescue/
    • Now you specify what kind of filesystem and mount the volume as /aw-rescue
  • cd /aw-rescue/var/airwave-backup
  • ls -la
    • This lists your files. Find the one named  "pre-upgrade-" something that matches your need.

 

For the next step you will need a SFTP server running on a server you can reach from this crashed Airwave server, either internal on your LAN or on the Internet somewhere. I just googled for a free sftp server and found a Bitvise SSH server that did the trick for me.

 

The command to copy the file to your sftp server:

  • scp from-full-filename username@sftp-server-ip:to-full-filename

 

Now you do the restore

2. Copy the using SCP backupfile.tar.gz file to the /tmp directory in the new AMP.

Use a file transfer client that supports SFTP/SCP for Windows like WinSCP: http://winscp.sourceforge.net/eng/

3. Log onto the new server as root.

4. Change to the scripts directory by typing scripts.

5. Run the restore script by typing ./amp_restore -d /tmp/backupfile.tar.gz.

 

 


Regards
John Solberg

-ACMX #316 :: ACCP-
Intelecom - Norway
----------------------------
Remember to Kudo if a post helped you! || Problem Solved? Click "Accept as Solution" in a post!
Search Airheads
Showing results for 
Search instead for 
Did you mean: