Network Management

Hello, I have installed Airwave 7.7.8 and it works fine.

but I have continuous alarm : Restarting service RabbitMQ message broker

when I check the Services status, I see that "RabbitMQ message broker" is DOWN !

is it normal ? do I have to enable it ?

Regards,

Vijay.

Hi Vijay,

Rabbitmq is a messageing process and listen to port number 5672. If any other process uses/ recerves that port then rabbitmq service will fail to come up.

I would recommend to kill the pid of services that uses port 5672 which will allow to start rabbitmq process.

Example:

To find pid

# netstat -anp | grep 5672

tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      3107/****

To kill pid

# kill -9 3107

To verify

[root@amp775 mercury]# netstat -anp | grep 5672
tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 4890/beam.smp
[root@amp775 mercury]#

[root@amp775 mercury]# ps -ef | grep rabbitmq
rabbitmq 2121 1 0 07:07 ? 00:00:00 /opt/airwave/lib64/erlang/erts-5.8.5/bin/epmd -daemon
rabbitmq 4890 1 0 07:23 ? 00:00:00 /opt/airwave/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576

Regards,

Sathish Kumar

Hi,

I have done the procedure but it stays down.
and message still comming in Airwave.

Fri Jan 17 16:20:33 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:20:19 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:20:05 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:19:49 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:19:33 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:19:19 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:19:03 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:18:49 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:18:36 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:18:19 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:18:04 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:17:50 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:17:34 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:17:19 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:17:05 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:16:51 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:16:35 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:16:19 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:16:05 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:15:57 2014    System    System    Restarting service Firewall Client Sessions Processor            
Fri Jan 17 16:15:57 2014    System    System    Restarting service PAPI Facade Queue Message Processor            
Fri Jan 17 16:15:52 2014    System    System    Restarting service RabbitMQ Message Broker            
Fri Jan 17 16:15:49 2014    System    System    Restarting service AMON message aggregator            
Fri Jan 17 16:15:35 2014    System    System    Restarting service RabbitMQ Message Broker



[SSH] Server Version OpenSSH_5.3
[SSH] Logged in (password)

Last login: Wed Jan 15 13:08:57 2014
SVN Root: http://svn.corp.airwave.com/usr/local/svnroot/mercury/tags/RELEASE_7_7_8

Daemons:
  PID  PPID  PGID %CPU %MEM   RSS     ELAPSED CMD
 9678     1  9678  0.2  0.1 20552  2-01:19:49 airbus
 5010     1  5010 11.9 13.2 1617936  01:16:33 Daemon::AsyncLoggerClient
 5100     1  5100 11.9 13.1 1610016  01:16:05 Daemon::AsyncLoggerClient
 9628     1  9628  0.0  7.4 911524 2-01:21:16 httpd_
 9674     1  9674  9.0  7.7 951316 2-01:19:50 Daemon::ServiceWatcher
 9680     1  9680  0.0  7.5 917924 2-01:19:49 Daemon::AMONAggregator
 9683     1  9683  0.0  7.4 907676 2-01:19:49 Daemon::PortGrabber
 9686     1  9686  0.0  7.4 907892 2-01:19:48 Daemon::TupleSpaces
 9689     1  9689  0.5  8.8 1085632 2-01:19:48 Daemon::TupleScheduler
 9746     1  9746  0.0  7.4 910968 2-01:19:47 Daemon::SwarmHandler
 9767     1  9767  0.2  7.4 914224 2-01:19:47 Daemon::PerfCollector
 9778     1  9778  0.0  7.4 911964 2-01:19:47 Daemon::AlertD
 9784     1  9784  1.5  7.6 927372 2-01:19:46 Daemon::APListCacher
 9787     1  9787  0.0  7.5 918220 2-01:19:46 Daemon::APScanner
 9790     1  9790  0.1  7.6 938828 2-01:19:46 Daemon::APWatcher
 9793     1  9793  0.0  7.5 917820 2-01:19:45 Daemon::APWatcherPollNow
 9800     1  9800  0.0  7.5 915156 2-01:19:45 Daemon::ConfigPusher
 9803     1  9803  0.0  7.5 917760 2-01:19:44 Daemon::ConfigVerifier
 9806     1  9806  0.0  7.5 924436 2-01:19:44 Daemon::DeviceEvent
 9810     1  9810  0.0  7.5 918496 2-01:19:44 Daemon::DeviceHTTPSHandler
 9813     1  9813  0.2  7.5 920240 2-01:19:44 Daemon::DNSFetcher
 9816     1  9816  0.0  7.5 919120 2-01:19:43 Daemon::ExecUIQuery
 9819     1  9819  0.0  7.8 953604 2-01:19:43 Daemon::PartialConfigPusher
 9822     1  9822  0.0  7.5 926448 2-01:19:43 Daemon::PAPI
 9825     1  9825  0.0  7.5 923720 2-01:19:42 Daemon::PAPIQueueConsumer
 9828     1  9828  0.0  7.5 925292 2-01:19:42 Daemon::PEFProcessor
 9851     1  9851  0.0  7.5 924268 2-01:19:41 Daemon::SNMPEnabler
 9854     1  9854  0.0  7.5 925292 2-01:19:41 Daemon::SNMPFetcher
 9861     1  9861  0.2  7.7 941040 2-01:19:38 Daemon::SNMPV2Fetcher
 9864     1  9864  2.0 10.1 1243392 2-01:19:38 Daemon::SNMPV2Fetcher
 9867     1  9867  0.1  7.6 932120 2-01:19:37 Daemon::SyncD
 9872     1  9872  0.0  7.6 929272 2-01:19:37 Daemon::WepKeySetter
31239     1 31238  1.2  5.5 674976 1-21:09:32 VisualRF
 9660     1  9660  0.1  0.2 28544  2-01:19:52 Daemon::LowLevelServiceWatcher
 9765     1  9674  0.0  0.0   520  2-01:19:47 msgHandler
 9645     1  9645  0.0  0.0   940  2-01:21:14 pound
 9648  9645  9645  0.0  0.0  4944  2-01:21:14 pound
 9843     1  9843  0.0  0.9 118980 2-01:19:41 radiusd
 6246     1  6246  0.0  0.0  2780  2-01:33:34 rrdcached
 9859     1  9859  0.0  0.8 97932  2-01:19:38 snmptrapd
Current Version: RELEASE_7_7_8
[root@localhost mercury]# netstat -anp | grep 5672
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      9729/beam
tcp        0      0 127.0.0.1:24963             127.0.0.1:5672              ESTABLISHED 9825/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:24965             ESTABLISHED 9729/beam
tcp        0      0 127.0.0.1:24957             127.0.0.1:5672              ESTABLISHED 9680/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:24963             ESTABLISHED 9729/beam
tcp        0      0 127.0.0.1:24965             127.0.0.1:5672              ESTABLISHED 9828/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:24957             ESTABLISHED 9729/beam
[root@localhost mercury]# kill -9 9729
[root@localhost mercury]# netstat -anp | grep 5672
[root@localhost mercury]# netstat -anp | grep 5672
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      20861/beam
tcp        0      0 127.0.0.1:35675             127.0.0.1:5672              ESTABLISHED 20904/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35675             ESTABLISHED 20861/beam
tcp        0      0 127.0.0.1:35678             127.0.0.1:5672              ESTABLISHED 20821/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35677             ESTABLISHED 20861/beam
tcp        0      0 127.0.0.1:35677             127.0.0.1:5672              ESTABLISHED 20907/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35678             ESTABLISHED 20861/beam
[root@localhost mercury]#
[root@localhost mercury]#
[root@localhost mercury]# ps -ef | grep rabbitmq
rabbitmq  8653     1  0 Jan15 ?        00:00:04 /opt/airwave/lib64/erlang/erts-5.8.5/bin/epmd -daemon
rabbitmq 20861     1  0 14:46 ?        00:00:00 /opt/airwave/lib64/erlang/erts-5.8.5/bin/beam -W w -K true -A3
0 -P 1048576 -- -root /opt/airwave/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput
-sname rabbit@localhost -boot /var/lib/rabbitmq/mnesia/rabbit@localhost-plugins-expand/rabbit -config /etc/rab
bitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"127.0.0.1",5672}
] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@l
ocalhost.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@localhost-sasl.log"} -os_mon start_cp
u_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbi
t@localhost" -noshell -noinput > /dev/null 2>&1
rabbitmq 20897 20861  0 14:46 ?        00:00:00 inet_gethost 4
rabbitmq 20898 20897  0 14:46 ?        00:00:00 inet_gethost 4
root     21160 20505  0 14:48 pts/0    00:00:00 grep rabbitmq
[root@localhost mercury]#
[root@localhost mercury]#
[root@localhost mercury]#
[root@localhost mercury]# exit
logout


[SSH] INFO: DISCONNECT


[SSH] Server Version OpenSSH_5.3
[SSH] Logged in (password)

Last login: Fri Jan 17 14:45:37 2014 from omnigogs.hq.azdamiaan.be
SVN Root: http://svn.corp.airwave.com/usr/local/svnroot/mercury/tags/RELEASE_7_7_8

Daemons:
  PID  PPID  PGID %CPU %MEM   RSS     ELAPSED CMD
 9678     1  9678  0.2  0.1 20552  2-02:49:26 airbus
 5010     1  5010  8.9 12.8 1570552  02:46:10 Daemon::AsyncLoggerClient
 5100     1  5100 16.7 13.2 1617460  02:45:42 Daemon::AsyncLoggerClient
 9628     1  9628  0.0  7.3 901952 2-02:50:53 httpd_
 9674     1  9674  9.0  7.7 942788 2-02:49:27 Daemon::ServiceWatcher
 9683     1  9683  0.0  7.3 898160 2-02:49:26 Daemon::PortGrabber
 9686     1  9686  0.0  7.3 898372 2-02:49:25 Daemon::TupleSpaces
 9689     1  9689  0.5  8.8 1079932 2-02:49:25 Daemon::TupleScheduler
 9746     1  9746  0.0  7.3 901464 2-02:49:24 Daemon::SwarmHandler
 9767     1  9767  0.2  7.4 904752 2-02:49:24 Daemon::PerfCollector
 9778     1  9778  0.0  7.3 902524 2-02:49:24 Daemon::AlertD
 9784     1  9784  1.5  7.5 917896 2-02:49:23 Daemon::APListCacher
 9787     1  9787  0.0  7.4 908780 2-02:49:23 Daemon::APScanner
 9790     1  9790  0.1  7.6 929488 2-02:49:23 Daemon::APWatcher
 9793     1  9793  0.0  7.6 927768 2-02:49:22 Daemon::APWatcherPollNow
 9800     1  9800  0.0  7.4 905648 2-02:49:22 Daemon::ConfigPusher
 9803     1  9803  0.0  7.4 908248 2-02:49:21 Daemon::ConfigVerifier
 9806     1  9806  0.0  7.5 914988 2-02:49:21 Daemon::DeviceEvent
 9810     1  9810  0.0  7.4 908992 2-02:49:21 Daemon::DeviceHTTPSHandler
 9813     1  9813  0.2  7.4 910740 2-02:49:21 Daemon::DNSFetcher
 9816     1  9816  0.0  7.4 909612 2-02:49:20 Daemon::ExecUIQuery
 9819     1  9819  0.0  7.7 944452 2-02:49:20 Daemon::PartialConfigPusher
 9822     1  9822  0.0  7.5 916940 2-02:49:20 Daemon::PAPI
 9851     1  9851  0.0  7.4 914760 2-02:49:18 Daemon::SNMPEnabler
 9854     1  9854  0.0  7.5 915788 2-02:49:18 Daemon::SNMPFetcher
 9861     1  9861  0.2  7.6 931596 2-02:49:15 Daemon::SNMPV2Fetcher
 9864     1  9864  2.0 10.1 1234592 2-02:49:15 Daemon::SNMPV2Fetcher
 9867     1  9867  0.1  7.5 922672 2-02:49:14 Daemon::SyncD
 9872     1  9872  0.0  7.5 919764 2-02:49:14 Daemon::WepKeySetter
20821     1 20821  0.0  7.7 949356   01:28:28 Daemon::AMONAggregator
20904     1 20904  0.0  7.7 941776   01:28:25 Daemon::PAPIQueueConsumer
20907     1 20907  0.0  7.7 942164   01:28:25 Daemon::PEFProcessor
31239     1 31238  1.2  5.5 675036 1-22:39:09 VisualRF
 9660     1  9660  0.1  0.2 28556  2-02:49:29 Daemon::LowLevelServiceWatcher
 9765     1  9674  0.0  0.0   520  2-02:49:24 msgHandler
 9645     1  9645  0.0  0.0   940  2-02:50:51 pound
 9648  9645  9645  0.0  0.0  5024  2-02:50:51 pound
 9843     1  9843  0.0  0.9 118976 2-02:49:18 radiusd
 6246     1  6246  0.0  0.0  2868  2-03:03:11 rrdcached
 9859     1  9859  0.0  0.8 97924  2-02:49:15 snmptrapd
Current Version: RELEASE_7_7_8
[root@localhost mercury]# netstat -anp | grep 5672
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      20861/beam
tcp        0      0 127.0.0.1:35675             127.0.0.1:5672              ESTABLISHED 20904/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35675             ESTABLISHED 20861/beam
tcp        0      0 127.0.0.1:35678             127.0.0.1:5672              ESTABLISHED 20821/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35677             ESTABLISHED 20861/beam
tcp        0      0 127.0.0.1:35677             127.0.0.1:5672              ESTABLISHED 20907/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:35678             ESTABLISHED 20861/beam
[root@localhost mercury]# kill -9 20861
[root@localhost mercury]# netstat -anp | grep 5672
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      6725/beam
tcp        0      0 127.0.0.1:5672              127.0.0.1:36415             ESTABLISHED 6725/beam
tcp        0      0 127.0.0.1:5672              127.0.0.1:36411             ESTABLISHED 6725/beam
tcp        0      0 127.0.0.1:36413             127.0.0.1:5672              ESTABLISHED 6767/Daemon:
tcp        0      0 127.0.0.1:36411             127.0.0.1:5672              ESTABLISHED 6673/Daemon:
tcp        0      0 127.0.0.1:36415             127.0.0.1:5672              ESTABLISHED 6770/Daemon:
tcp        0      0 127.0.0.1:5672              127.0.0.1:36413             ESTABLISHED 6725/beam
[root@localhost mercury]#
[root@localhost mercury]# ps -ef | grep rabbitmq
rabbitmq  6725     1  0 16:15 ?        00:00:00 /opt/airwave/lib64/erlang/erts-5.8.5/bin/beam -W w -K true -A30 -P 1048576 -- -root /opt/airwave/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@localhost -boot /var/lib/rabbitmq/mnesia/rabbit@localhost-plugins-expand/rabbit -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"127.0.0.1",5672}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@localhost.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@localhost-sasl.log"} -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@localhost" -noshell -noinput > /dev/null 2>&1
rabbitmq  6764  6725  0 16:15 ?        00:00:00 inet_gethost 4
rabbitmq  6765  6764  0 16:15 ?        00:00:00 inet_gethost 4
root      7459  6467  0 16:19 pts/0    00:00:00 grep rabbitmq
rabbitmq  8653     1  0 Jan15 ?        00:00:04 /opt/airwave/lib64/erlang/erts-5.8.5/bin/epmd -daemon
[root@localhost mercury]#

I had the same issue on my Airwave Server.

After analysis with TAC it turned out, that my server had too few CPU core assigned.

After adding another CPU core to my virtual appliance the issue stopped.

So check your installation (Number of devices attached to Airwave) against the recommended hardware recommendations.