Network Management

I installed Airwave on a new server with 96 gig of memory. The system runs dead slow. VisualRF html 5 doesn't even show heatmaps. I go into system performance and see the memory being used up in less than a day. I talked to TAC reps at Airheads and they said the slowness problem is fixed in 8.0.7. Anyone else seeing this issue.

We just upgraded to 8.0.6.3 from 7.something, after Dan Comfort (Airwave Product Manager) recommended update for possible memory leak. We'll see how stable it is.

8.0.7 is available 

Can you also report top -c values?  I'd be interested in knowing which processes are tying up memory.

top -a:

top - 16:34:00 up 8:14, 1 user, load average: 4.40, 5.67, 4.90
Tasks: 769 total, 18 running, 751 sleeping, 0 stopped, 0 zombie
Cpu(s): 44.0%us, 14.1%sy, 0.0%ni, 41.2%id, 0.1%wa, 0.0%hi, 0.6%si, 0.0%st
Mem: 99055068k total, 98744696k used, 310372k free, 743516k buffers
Swap: 4194296k total, 72k used, 4194224k free, 52674912k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2394 postgres 20 0 4357m 3.5g 3.5g S 0.0 3.8 15:39.81 postmaster
3502 root 20 0 2666m 1.8g 25m S 0.0 1.9 46:50.10 httpd__________
3995 root 20 0 3071m 1.7g 582m R 12.2 1.8 69:50.64 httpd__________
4033 root 20 0 3083m 1.7g 567m R 73.0 1.8 67:54.52 httpd__________
4011 root 20 0 3066m 1.7g 567m R 71.0 1.8 64:55.06 httpd__________
3200 root 20 0 3059m 1.7g 565m R 12.2 1.8 64:11.72 httpd__________
3986 root 20 0 3062m 1.7g 555m R 69.8 1.8 66:01.48 httpd__________
4029 root 20 0 3057m 1.7g 535m S 71.3 1.8 64:20.91 httpd__________
4050 root 20 0 3099m 1.7g 466m S 14.5 1.8 63:31.12 httpd__________
3936 root 20 0 3085m 1.6g 468m S 6.7 1.7 63:19.25 httpd__________
3959 root 20 0 3077m 1.6g 472m S 6.1 1.7 68:44.41 httpd__________
4038 root 20 0 3098m 1.6g 449m S 10.5 1.7 66:53.48 httpd__________
4045 root 20 0 3096m 1.6g 449m S 21.5 1.7 66:31.64 httpd__________
24160 postgres 20 0 4461m 1.6g 1.5g S 2.0 1.7 11:45.49 postmaster
4022 root 20 0 3065m 1.6g 471m R 11.9 1.7 64:53.23 httpd__________
4004 root 20 0 3062m 1.6g 467m R 11.9 1.7 65:02.00 httpd__________
3975 root 20 0 3068m 1.6g 460m R 95.4 1.7 64:28.65 httpd__________
3930 root 20 0 3070m 1.6g 446m S 9.3 1.7 62:25.93 httpd__________
3950 root 20 0 3075m 1.6g 433m R 11.1 1.7 66:12.23 httpd__________
3943 root 20 0 3060m 1.6g 443m S 15.7 1.7 64:19.95 httpd__________
3965 root 20 0 3063m 1.6g 435m S 18.3 1.7 63:42.66 httpd__________
26566 postgres 20 0 4450m 1.5g 1.4g S 4.7 1.6 11:01.51 postmaster
3503 postgres 20 0 4360m 1.5g 1.5g S 0.0 1.6 5:49.13 postmaster
28690 postgres 20 0 4445m 1.5g 1.4g S 1.5 1.6 10:55.67 postmaster
27387 postgres 20 0 4446m 1.5g 1.4g S 4.9 1.5 9:54.20 postmaster
30712 postgres 20 0 4451m 1.4g 1.3g S 2.6 1.5 9:53.57 postmaster
3271 postgres 20 0 4358m 1.3g 1.2g S 0.3 1.3 2:09.89 postmaster
3715 root 20 0 1613m 1.1g 3972 S 63.7 1.2 273:24.93 httpd__________
7711 postgres 20 0 4437m 1.1g 1.0g S 2.6 1.2 5:09.38 postmaster
3189 root 20 0 1594m 1.1g 3900 S 0.0 1.2 21:55.65 httpd__________
3256 root 20 0 1574m 1.1g 2408 S 0.0 1.2 0:43.78 httpd__________
3903 root 20 0 1582m 1.1g 9724 S 0.0 1.2 0:28.34 httpd__________
13497 postgres 20 0 4423m 1.1g 1.0g S 27.3 1.2 3:25.92 postmaster
3131 root 20 0 1560m 1.1g 2208 R 14.8 1.1 77:39.69 httpd__________
3148 root 20 0 1553m 1.1g 3152 S 0.3 1.1 2:37.05 httpd__________
3182 root 20 0 1554m 1.1g 2260 R 71.9 1.1 134:08.37 httpd__________
3267 root 20 0 1545m 1.1g 2548 S 1.7 1.1 5:13.48 httpd__________
3722 root 20 0 1540m 1.1g 3972 S 43.9 1.1 191:21.88 httpd__________
3479 root 20 0 1538m 1.1g 2640 S 0.3 1.1 0:16.20 httpd__________
2402 root 17 -3 1528m 1.1g 8476 S 0.0 1.1 2:01.59 httpd__________
16969 apache 17 -3 1528m 1.1g 4468 S 0.0 1.1 0:02.47 httpd__________
18119 apache 17 -3 1528m 1.1g 4284 S 0.0 1.1 0:01.14 httpd__________
17686 apache 17 -3 1528m 1.1g 4188 S 0.0 1.1 0:01.76 httpd__________

top -c:

top - 16:34:33 up 8:15, 1 user, load average: 5.02, 5.70, 4.93
Tasks: 771 total, 5 running, 766 sleeping, 0 stopped, 0 zombie
Cpu(s): 23.1%us, 10.3%sy, 0.0%ni, 65.1%id, 1.2%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 99055068k total, 98649860k used, 405208k free, 743532k buffers
Swap: 4194296k total, 72k used, 4194224k free, 52581212k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3797 visualrf 20 0 8318m 943m 13m S 172.8 1.0 496:15.42 /usr/java/jre/bin/java -XX:-AllowUserSignalHandlers -Xmx668m -DFCGI_PO
17859 postgres 20 0 4359m 1.0g 1.0g R 100.0 1.1 10:18.23 postgres: airwave airwave [local] SELECT
3965 root 20 0 3063m 1.6g 439m R 50.6 1.7 63:57.84 Daemon::AsyncLoggerClient
19033 upjs 20 0 15556 1780 896 R 50.6 0.0 0:01.01 top -c
3140 redis 20 0 97804 57m 1036 S 37.7 0.1 67:44.49 /opt/airwave/sbin/redis-server /etc/redis.conf
3715 root 20 0 1613m 1.1g 3972 S 37.7 1.2 273:39.05 Daemon::SNMPV2Fetcher
17477 postgres 20 0 4407m 538m 486m R 33.8 0.6 0:46.81 postgres: airwave airwave [local] idle in transaction
3200 root 20 0 3059m 1.7g 565m S 18.9 1.8 64:16.35 Daemon::AsyncLoggerClient
3975 root 20 0 3068m 1.6g 460m S 18.9 1.7 64:33.88 Daemon::AsyncLoggerClient
3722 root 20 0 1540m 1.1g 3972 S 16.9 1.1 191:32.87 Daemon::SNMPV2Fetcher
4038 root 20 0 3098m 1.6g 449m S 15.9 1.7 67:00.85 Daemon::AsyncLoggerClient
3131 root 20 0 1560m 1.1g 2208 S 14.9 1.1 77:46.18 Daemon::TupleScheduler
3930 root 20 0 3070m 1.6g 446m S 14.9 1.7 62:30.73 Daemon::AsyncLoggerClient
3959 root 20 0 3077m 1.6g 472m S 14.9 1.7 68:48.48 Daemon::AsyncLoggerClient
4033 root 20 0 3083m 1.7g 577m S 13.9 1.8 68:14.65 Daemon::AsyncLoggerClient
2398 postgres 20 0 139m 1440 544 S 12.9 0.0 2:51.16 postgres: stats collector process
3995 root 20 0 3080m 1.7g 582m S 12.9 1.8 70:00.28 Daemon::AsyncLoggerClient
3986 root 20 0 3062m 1.7g 561m S 11.9 1.8 66:16.50 Daemon::AsyncLoggerClient
18761 postgres 20 0 4393m 773m 766m S 10.9 0.8 0:12.39 postgres: autovacuum worker process airwave
3950 root 20 0 3075m 1.6g 433m S 8.9 1.7 66:17.68 Daemon::AsyncLoggerClient
3962 postgres 20 0 4363m 153m 146m S 7.0 0.2 31:13.44 postgres: airwave airwave 127.0.0.1(4540) idle
4011 root 20 0 3066m 1.7g 577m S 7.0 1.8 65:15.27 Daemon::AsyncLoggerClient
4029 root 20 0 3057m 1.7g 544m R 7.0 1.8 64:37.03 Daemon::AsyncLoggerClient
3936 root 20 0 3085m 1.6g 468m S 6.0 1.7 63:24.52 Daemon::AsyncLoggerClient
3943 root 20 0 3060m 1.6g 443m S 6.0 1.7 64:25.72 Daemon::AsyncLoggerClient
4045 root 20 0 3096m 1.6g 449m S 6.0 1.7 66:36.49 Daemon::AsyncLoggerClient
4050 root 20 0 3099m 1.7g 467m S 6.0 1.8 63:36.48 Daemon::AsyncLoggerClient
6357 postgres 20 0 4361m 148m 143m S 6.0 0.2 30:00.43 postgres: airwave airwave 127.0.0.1(9783) idle
19035 postgres 20 0 4358m 31m 29m S 5.0 0.0 0:00.05 postgres: autovacuum worker process airwave
4022 root 20 0 3065m 1.6g 472m S 4.0 1.7 64:57.75 Daemon::AsyncLoggerClient
13497 postgres 20 0 4423m 1.1g 1.0g S 4.0 1.2 3:34.11 postgres: airwave airwave [local] idle
24160 postgres 20 0 4461m 1.6g 1.5g S 4.0 1.7 11:46.59 postgres: airwave airwave [local] idle
16064 postgres 20 0 4416m 823m 763m S 3.0 0.9 1:24.02 postgres: airwave airwave [local] idle
18087 postgres 20 0 4391m 658m 622m S 3.0 0.7 1:27.51 postgres: airwave airwave [local] idle
30712 postgres 20 0 4451m 1.4g 1.3g S 3.0 1.5 9:54.72 postgres: airwave airwave [local] idle
1987 root 20 0 2503m 155m 5736 S 2.0 0.2 14:27.48 /opt/bit9/bin/b9daemon
3150 root 20 0 1531m 1.1g 3216 S 2.0 1.1 4:03.52 Daemon::SwarmHandler
4267 root 20 0 518m 69m 68m S 2.0 0.1 2:30.49 awrrdtool [3200] -
4448 root 20 0 518m 70m 69m S 2.0 0.1 2:33.99 awrrdtool [3975] -
6468 root 20 0 518m 69m 68m S 2.0 0.1 2:30.98 awrrdtool [4033] -
7711 postgres 20 0 4437m 1.1g 1.0g S 2.0 1.2 5:11.73 postgres: airwave airwave [local] idle
15166 postgres 20 0 4420m 817m 753m S 2.0 0.8 1:39.29 postgres: airwave airwave [local] idle
16904 postgres 20 0 4406m 890m 840m S 2.0 0.9 2:04.08 postgres: airwave airwave [local] idle

Looks like VisualRF and the ALCs (workers) are the primary memory users.  This is expected when there's a large number of clients.  I would suggest opening a support case since this behavior could definitely use some additional investigation.

We’re going to update to 8.0.7 and see how much that helps, but we may also open a support case to investigate further.  Thanks Rob, and nice to meet you last week!