Clamav-daemon crashes after a while

UCS - Univention Corporate Server
#1

ucs version 4.4-0 errdata51

After a few hours of running, clamav-deamon crashes:

service clamav-daemon status

● clamav-daemon.service - Clam AntiVirus userspace daemon
Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/clamav-daemon.service.d
└─extend.conf
Active: failed (Result: signal) since Wed 2019-04-24 23:40:39 CEST; 12h ago
Docs: man:clamd(8)
man:clamd.conf(5)
https://www.clamav.net/documents/
Main PID: 737 (code=killed, signal=KILL)
CPU: 1min 16.120s

Apr 24 15:37:06 ucs-7413 clamd[737]: Wed Apr 24 15:37:06 2019 -> SelfCheck: Database status OK.
Apr 24 17:08:47 ucs-7413 clamd[737]: Wed Apr 24 17:08:47 2019 -> SelfCheck: Database status OK.
Apr 24 18:19:34 ucs-7413 clamd[737]: Wed Apr 24 18:19:34 2019 -> SelfCheck: Database status OK.
Apr 24 19:30:27 ucs-7413 clamd[737]: Wed Apr 24 19:30:27 2019 -> SelfCheck: Database status OK.
Apr 24 20:30:27 ucs-7413 clamd[737]: Wed Apr 24 20:30:27 2019 -> SelfCheck: Database status OK.
Apr 24 21:34:56 ucs-7413 clamd[737]: Wed Apr 24 21:34:56 2019 -> SelfCheck: Database status OK.
Apr 24 22:34:56 ucs-7413 clamd[737]: Wed Apr 24 22:34:56 2019 -> SelfCheck: Database status OK.
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Main process exited, code=killed, status=9/KILL
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Unit entered failed state.
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Failed with result ‘signal’.

There is nothing to find in the clamav.log and some inconsequential entries in syslog:

Apr 24 22:09:54 ucs-7413 freshclam[699]: Reading CVD header (main.cvd): WARNING: Can’t get information about db.local.clamav.net: Temporary failure in name resolution
Apr 24 22:09:54 ucs-7413 freshclam[699]: WARNING: Can’t read main.cvd header from db.local.clamav.net (IP: )
Apr 24 22:10:31 ucs-7413 freshclam[699]: WARNING: Can’t query current.cvd.clamav.net
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Main process exited, code=killed, status=9/KILL
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Unit entered failed state.
Apr 24 23:40:39 ucs-7413 systemd[1]: clamav-daemon.service: Failed with result ‘signal’.

After increasing the RAM size in the virtual machine (vSphere) to 6 GB, the deamon runs a bit longer.
It seems as if clamav reserves memory but does not release it.

However, this behavior is also in the previous ucs version (2-3 weeks) to determine.
Before no amavisd errors have occurred.
Because the error messages are sent to a special account, I just did not notice before.

Any suggestions?

#2

Hi,

if it gets killed due to to few memory you will see “oom” messages (Out Of Memory) in the log files. If you do not see them (and your swap space is only used a little bit) it is not due to oom.

Could you post the output of
vmstat 1 30
?

/CV

#3

no “oom” messages. no swap defined (because ucss is running in virtual machine).

# vmstat 1 30
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 745944 196148 1119284    0    0   247   250  248  300 13  3 83  1  0
 0  0      0 745820 196148 1119288    0    0     0     0  107  251  1  0 99  0  0
 0  0      0 746676 196172 1119284    0    0     0    48  736 14097 50 18 32  0  0
 0  0      0 745032 196172 1119292    0    0     0     0  729 16926 57 33 10  0  0
 7  0      0 744100 196188 1119292    0    0     0   352  591 17812 58 29 13  0  0
 4  0      0 742388 196220 1119276    0    0     0    88 1914 14884 74 26  0  0  0
 3  0      0 744120 196236 1119288    0    0     0    28 1399 3973 67 15 17  1  0
 4  0      0 743244 196260 1119288    0    0     0    48 1204 4228 83  9  8  0  0
 0  0      0 746044 196268 1119288    0    0     0    16  102  423  1  0 99  0  0
 0  0      0 565252 196284 1119288    0    0     0   384  132  303  1  3 94  2  0
 0  0      0 565252 196292 1119288    0    0     0    16  108  272  0  1 99  0  0
 0  0      0 565260 196292 1119284    0    0     0     0  130  307  1  0 99  0  0
 0  0      0 565468 196308 1119268    0    0     0    32  121  364  0  0 100  0  0
 0  0      0 565508 196308 1119284    0    0     0     0  113  299  0  0 100  0  0
 0  0      0 565324 196324 1119268    0    0     0   288  201  831 26  6 67  1  0
 3  0      0 563920 196332 1119320    0    0     0    16  808 14762 51 22 27  0  0
 4  0      0 561468 196332 1119320    0    0     0     0  778 18457 56 38  6  0  0
 2  0      0 415392 196364 1119296    0    0     0    64 1037 14894 69 28  3  0  0
 0  0      0 415288 196364 1119316    0    0     0     0 1071 16275 63 26 11  0  0
 0  0      0 289356 196380 1119316    0    0     0   348 1147 4113 69 14 17  0  0
 0  0      0 162160 196428 1119312    0    0     0   272 1183 4015 79 12  9  0  0
 0  0      0 234756 195948 984028    0    0     0    16  150  456  3  4 93  0  0
 0  0      0 234964 195972 983752    0    0     0    48  130  407  2  0 97  1  0
 0  0      0 234964 195972 983752    0    0     0     0   94  232  0  0 100  0  0
 0  0      0 235088 195988 983684    0    0     0   384  114  274  0  0 98  2  0
 0  0      0 235296 196004 983676    0    0     0    40  103  255  0  0 100  0  0
 0  0      0 235716 196004 983444    0    0     0     0  105  256  1  0 99  0  0
 0  0      0 235716 196012 983436    0    0     4    20  122  331  0  0 100  0  0
 0  0      0 235716 196020 983416    0    0     0    16   92  226  1  0 99  0  0
 0  0      0 188700 196036 983400    0    0     0   184  114  277  0  3 95  2  0
#4

Well…I do not know where you got the information of not using swap in a virtual makes sense. Whoever told you this I would not listen to any advice this guy is giving you in the future.

You might not have seen any oom message because of looking at wrong places but I am pretty sure an oom happened. Even in your short output by comparing first and last line:

procs -----------memory---------- —swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 745944 196148 1119284 0 0 247 250 248 300 13 3 83 1 0
[...]
0 0 0 188700 196036 983400 0 0 0 184 114 277 0 3 95 2 0

I see a decrease of free memory from 745,944 to 188,700 so around 200M in 30 seconds…

Do you need further prove of the issue being memory related? Yes, there are still buffers and cache available but I could image this will go down, too.

We already had several systems where customer installed a load of apps (including ClamAV) and wondered why the system is so slow or behaving strange.

Sorry guys, give a UCS server enough memory an you will have not troubles. Especially, give it SWAP!

When swap is available, increase your memory (RAM) until under usual load the vmstat shows nearly just “0” in si and so column.

Ignore the swap usage, si and so is important. Check this article.

Oh, did I mention: create swap and increase memory!

/CV

Mastodon