Monitoring, Management & Location Tracking

Environment : All versions of Airwave

Large amount of traffic resulting in a network flood which eventually may result in a network down as well.

It would indicate a server infected with malware/virus

Reinstalling the server is sometimes the only option in this case.

Ensure the new server has a complicated root password.

Also, follow the best network security practices to protect the installed Airwave server.

Troubleshooting steps :

1) Take a pcap on Airwave server and check the traffic pattern. We usually see a large number of packets destined to Internet on random ports like 17777,56789 etc
2) Check netstat -anp and ps -ef

Sample outputs :

netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:8558              0.0.0.0:*                   LISTEN      3432/perl           
tcp        0      0 127.0.0.1:8559              0.0.0.0:*                   LISTEN      3532/Daemon:        
tcp        0      0 127.0.0.1:9999              0.0.0.0:*                   LISTEN      3440/Daemon:        
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1342/rpcbind        
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      3550/pound          
tcp        0      0 127.0.0.1:8560              0.0.0.0:*                   LISTEN      3532/Daemon:        
tcp        0      0 127.0.0.1:4369              0.0.0.0:*                   LISTEN      2190/epmd           
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1481/sshd           
tcp        0      0 127.0.0.1:5432              0.0.0.0:*                   LISTEN      1524/postmaster     
tcp        0      0 127.0.0.1:8569              0.0.0.0:*                   LISTEN      3597/Daemon:        
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1622/master         
tcp        0      0 127.0.0.1:8570              0.0.0.0:*                   LISTEN      3597/Daemon:        
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      3550/pound          
tcp        0      0 127.0.0.1:6654              0.0.0.0:*                   LISTEN      3437/Daemon:        
tcp        0      0 127.0.0.1:43327             0.0.0.0:*                   LISTEN      3492/beam.smp       
tcp        0      0 127.0.0.1:6655              0.0.0.0:*                   LISTEN      3437/Daemon:        
tcp        0      0 127.0.0.1:7777              0.0.0.0:*                   LISTEN      3443/Daemon:        
tcp        0      0 127.0.0.1:5672              0.0.0.0:*                   LISTEN      3492/beam.smp       
tcp        0      0 127.0.0.1:8008              0.0.0.0:*                   LISTEN      2689/httpd_________
tcp        0      0 127.0.0.1:7777              127.0.0.1:13342             ESTABLISHED 3443/Daemon:        
tcp        0    401 172.30.101.206:2493         115.231.17.9:56789          ESTABLISHED 13877/smarvtd       
tcp        0    401 172.30.101.206:2769         115.231.17.9:56789          ESTABLISHED 20979/smarvtd       
tcp        0      0 127.0.0.1:5672              127.0.0.1:29839             ESTABLISHED 3492/beam.smp       
tcp        0      0 127.0.0.1:7777              127.0.0.1:13296             ESTABLISHED 3443/Daemon:        
tcp        0      0 127.0.0.1:8558              127.0.0.1:3013              ESTABLISHED 3432/perl           
tcp    10422      0 127.0.0.1:30065             127.0.0.1:5672              ESTABLISHED 30309/Daemon:       
tcp        0      0 127.0.0.1:5672              127.0.0.1:48638             ESTABLISHED 3492/beam.smp       
tcp        0      0 127.0.0.1:7777              127.0.0.1:43835             ESTABLISHED 3443/Daemon:        
tcp        0    401 172.30.101.206:2632         115.231.17.9:56789          ESTABLISHED 12189/smarvtd       
tcp        0    401 172.30.101.206:2682         115.231.17.9:56789          ESTABLISHED 43296/smarvtd       
tcp        0    401 172.30.101.206:2568         115.231.17.9:56789          ESTABLISHED 8830/smarvtd        
tcp        0    401 172.30.101.206:2814         115.231.17.9:56789          ESTABLISHED 6015/smarvtd        
tcp        0    401 172.30.101.206:2535         115.231.17.9:56789          ESTABLISHED 35395/smarvtd       

truncated...

[root@localhost mercury]# ps-  -eadf

root      6283     1  0 Nov27 ?        00:01:51 /tmp/smarvtd
root      6459     1  0 05:00 ?        00:00:11 /tmp/smarvtd
root      6642     1  0 Nov12 ?        00:07:08 /tmp/smarvtd
root      6918 22957  0 Nov07 ?        00:00:00 [ps] <defunct>
root      6987     1  0 Nov29 ?        00:01:06 /tmp/smarvtd
root      7163     1  0 Nov26 ?        00:02:20 /tmp/smarvtd
root      7175     1  0 Nov29 ?        00:01:13 /tmp/smarvtd
postfix   7396  1622  0 17:50 ?        00:00:00 local -t unix
root      7510     1  0 Nov24 ?        00:03:03 /tmp/smarvtd
root      7640     1  0 Nov27 ?        00:01:51 /tmp/smarvtd
root      7775     1  0 Nov19 ?        00:04:49 /tmp/smarvtd
root      8163     1  0 Nov29 ?        00:01:11 /tmp/smarvtd
root      8306     1  0 Nov08 ?        00:08:39 /tmp/smarvtd
root      8439 22968  0 Nov08 ?        00:00:00 [sh] <defunct>
root      8528     1  0 Nov30 ?        00:00:45 /tmp/smarvtd
root      8555     1  0 Nov27 ?        00:02:05 /tmp/smarvtd
apache    8576  3144  0 Nov24 ?        00:00:00 /opt/airwave/sbin/httpd______________________________________
apache    8622  3144  0 Nov24 ?        00:00:01 /opt/airwave/sbin/httpd______________________________________
apache    8623  3144  0 Nov24 ?        00:00:01 /opt/airwave/sbin/httpd______________________________________
root      8750     1  0 Nov29 ?        00:01:02 /tmp/smarvtd
root      8830     1  0 Nov29 ?        00:01:24 /tmp/smarvtd
apache    9061     1  0 Nov24 ?        00:00:00 awrrdtool [8576] -
postfix   9154  1622  0 17:51 ?        00:00:00 bounce -z -t unix -u
postfix   9159  1622  0 17:51 ?        00:00:00 local -t unix
root      9308     1  0 Nov21 ?        00:04:10 /tmp/smarvtd
root      9439     1  0 Nov28 ?        00:01:36 /tmp/smarvtd
root      9608     1  0 Dec01 ?        00:00:26 /tmp/smarvtd
root      9721 22958  0 Nov07 ?        00:00:00 [ps] <defunct>
root      9855     1  0 Nov19 ?        00:04:34 /tmp/smarvtd
root      9896     1  0 Nov27 ?        00:01:51 /tmp/smarvtd
root     10141     1  0 Nov22 ?        00:03:42 /tmp/smarvtd
root     10225 22955  0 17:51 ?        00:00:00 /tmp/.sshdd1415277056
postgres 10257  1524  0 Nov24 ?        00:00:00 postgres: airwave airwave [local] idle                              
root     10786 22963  0 Nov07 ?        00:00:00 [sh] <defunct>
root     10869  1481  0 17:52 ?        00:00:00 sshd: root@pts/0
root     11008     1  0 Nov25 ?        00:02:38 /tmp/smarvtd
root     11055     1  0 Dec01 ?        00:00:20 /tmp/smarvtd
root     11078 22958  0 Nov08 ?        00:00:00 [ps] <defunct>
root     11105     1  0 Nov24 ?        00:02:45 /tmp/smarvtd

truncated...


In one of the environments we could see a large number of  smarvtd processes running. smarvtd is a potential malware based on the references on the web. One of the useful links pointing towards the same is :
http://lists.centos.org/pipermail/centos/2014-October/146566.html