So this morning, I was tasked to upgrade 2 Airwave servers (1 of them acting as the failover).
These Airwave servers contain massive amounts of data, the backup of the primary is >5Gb
This is what happened to me (doing upgrades remotely but they are hosted at our datacenter and we have a fibre connection to the datacenter)
1. Upgrade the backup to 7.6.3 (works)
2. disable the backup (amp_disable now)
3. Upgrade the primary to 7.6.3
Lost internet connectivity in the office, my SSH session during the upgrade dies
4. Upgrade never completes on the server, it is practically missing everything
doesn't understand start_amp_upgrade, amp_version, that directory is empty
5. try to restore the backup on the failover, problem is its already been upgraded to 7.6.2 so backup won't restore
6. Airheads and KB searches later
7. I move amp_enable, start_amp_upgrade, amp_version and chmod a+x the files
8. try the upgrade again, fails, doesn't complete. fails on /bin/chown -R apache.apache /var/airwave/rrd
we have about 1TB of RRD data on the server
9. I see Sujatha's post regarding make, I see another from Rob's (rgin) regarding make, so I try it (root; make)
10. it puts back certain files and allows me to execute the upgrade again and now it is stuck on
STEP 5: Installing upgrade.
tail -f /var/log/upgrade/AMP-7.6.3-upgrade.log
/bin/chown -R apache.apache /var/airwave/rrd
Won't go further than that.
its been 4 hours now, maybe I should have called TAC ......might have to go with the 5Gb backup file to DC to re-install the primary AIrwave from scratch