If possible, I like to upgrade the OS first since it's the more complicated / time consuming procedure. Then I upgrade from 7.5.4 to 7.7.9 directly (as we support skipping 1 release between upgrades). As @pmonardo says -> you have the option to do the intermediate step which would take care of the intermediate database migrations which could possibly speed up the upgrade process, but it's an extra step that's based on your preference.
Here's the practice that I do in the lab:
If you're doing this upgrade and trying to use the same server post upgrade...
1) Take the backup off server -> change the name to explicitly have version and date
-> additional things to note: ifconfig eth0, cat /var/airwave/license/license.txt
2) Reinstall server on same version, double check the OS version post install
3) Restore the backup, check to see if you need to reapply the license key
4) Check the data is recording and working properly (make sure to check VisualRF)
5) Upgrade to 7.7.9
6) Check the data again (make sure to check VisualRF)
If you choose to hit 7.6.5 inbetween, I would add as 2 steps, the upgrade and then the system sanity check.