Network Management

just installed Airwave 7.6.4 on VMware, configured it and can reach it fine with ssh. but httpd is not coming up it seems. last time it also took a while, but now it is has been close to an hour. is this to be expected or ...?

[root@airwave log]# daemons
  PID  PPID  PGID %CPU %MEM   RSS     ELAPSED CMD
 1711     1  1711  0.1  0.1 11604       45:25 Daemon::LowLevelServiceWatcher
 1792     1  1740  0.0  0.0   500       44:55 msgHandler
 1264     1  1264  0.0  0.0  3224       47:34 rrdcached
 1892     1  1892  0.0  2.8 231876      44:22 snmptrapd

[root@airwave log]# w
 10:51:56 up 58 min,  1 user,  load average: 0.88, 0.87, 0.81
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.99.99    10:03    0.00s  0.33s  0.00s w

can't find any other hints except in httpd error log, where lines like below keep getting added.

[Mon May 13 10:44:27 2013] bootstrap.pl(5801): Starting...
[Mon May 13 10:44:47 2013] bootstrap.pl(5822): Starting...
ap_get_server_banner() not available until httpd/2.2.4 falling back to ap_get_server_version() at /opt/airwave/lib64/perl5/5.10.1/x86_64-linux-thread-multi/XSLoader.pm line 94.
ap_get_server_description() not available until httpd/2.2.4 falling back to ap_get_server_version() at /opt/airwave/lib64/perl5/5.10.1/x86_64-linux-thread-multi/XSLoader.pm line 94.
[Mon May 13 10:44:47 2013] bootstrap.pl(5822): Not the last run yet (PPID: 5821)
[Mon May 13 10:45:07 2013] bootstrap.pl(5826): Starting...
[Mon May 13 10:45:27 2013] bootstrap.pl(5847): Starting...
ap_get_server_banner() not available until httpd/2.2.4 falling back to ap_get_server_version() at /opt/airwave/lib64/perl5/5.10.1/x86_64-linux-thread-multi/XSLoader.pm line 94.
ap_get_server_description() not available until httpd/2.2.4 falling back to ap_get_server_version() at /opt/airwave/lib64/perl5/5.10.1/x86_64-linux-thread-multi/XSLoader.pm line 94.
[Mon May 13 10:45:27 2013] bootstrap.pl(5847): Not the last run yet (PPID: 5846)
[Mon May 13 10:45:46 2013] bootstrap.pl(5867): Starting...

amp_disable / amp_enable solved it (probably a box restart would have also), running fine now, still i wonder if this is normal behaviour?

This is not expected behavior.  What kind of resources have been allocated for this VM instance?  If you're using the VM image, then it expects at least 8 GB of RAM.  If you're installing from the ISO, it should be more flexible.

Anything in the low_level_service_watcher or service_watcher logs?

system has 8gig exactly, so shouldnt be an issue, i did find this in the low_level_service_watcher log

Mon May 13 10:01:34 2013: Couldn't find any apache processes; restarting.
Stopping httpd: [FAILED]
Starting httpd: [  OK  ]
Stopping Pound: [FAILED]
Apache uptime (PID: 2997) appears to be negative (-32449 seconds), perhaps the system clock has jumped backwards at /usr/local/airwave/lib/perl/Mercury/Daemon/LowLevelServiceWatcher.pm line 151.
Mon May 13 10:02:13 2013: Couldn't find any apache processes; restarting.
Stopping httpd: [FAILED]
Starting httpd: [  OK  ]
Stopping Pound: [FAILED]
Apache uptime (PID: 3022) appears to be negative (-32449 seconds), perhaps the system clock has jumped backwards at /usr/local/airwave/lib/perl/Mercury/Daemon/LowLevelServiceWatcher.pm line 151.

which continues until the disable / enable

Mon May 13 11:24:47 2013: /etc/amp_disabled exists, not restarting services
Mon May 13 11:25:02 2013: Couldn't find any apache processes; restarting.
Stopping httpd: [FAILED]
Starting httpd: [  OK  ]
Stopping Pound: [FAILED]

and then nothing anymore there.

Is the date on the server correct?  If it's not, try the following:

Check current date:

     [root@rob mercury]# date
     Tue May 14 07:13:12 PDT 2013

Update against an NTP server:

     [root@rob mercury]# ntpdate 10.10.10.200 && hwclock -w
     14 May 07:14:06 ntpdate[11738]: step time server 10.10.10.200 offset 44.292164 sec

** in this particular case, 10.10.10.200 is the IP of an internal NTP server.  You'll want to use a different server that has NTP access, or a public NTP if the server can hit the internet.  Once the time on the VM instance has been updated, give it a restart/reboot.  I would expect the AMP to come up after that.

thanks Rob, but as stated before then AMP already came up after doing the disable / enable. this was just looking back at what could cause have it. if i recall correctly i had some issue with the time during config and might have messed up something there.

ill pay some more attention during the next install.

Yeah, it's difficult to pinpoint the root cause after the issue's already been fixed.  It's my hunch that the server time is the cause (more often with VMs).  Another thing to watch is to make sure that /var/airwave/rrd/rrd_check is an empty file.  This is a nightly check that makes sure that RRDs are in a sane state.  Often times, if the RRD is newer than the check time, you'll end up with an error on the Home -> Overview page stating that you have data from the future (most often this happens after someone has attempted to manually update time using the 'date' command).  I haven't encountered many fresh installs where the AMP doesn't come up, but when I do - it's more so because disk speed is less than 15k RPM.