Hello,
when searching something in “All Folder” one of my Mailaccounts causes Timeouts in OX and this kind of logentry in /var/log/dovecot.err:
Jul 4 13:38:43 mail dovecot: imap(user@domain.at): Error: open(/var/spool/dovecot/private/***MAILDIR***_exner/cur/1499011087.M353693P30815.mail,S=16
84667,W=1706830) failed: Too many open files
I do not understand which open-file-limit is meant there or how to find out which configuration could help me to fix this.
This Mailaccount has about 17500 Emails and 10G size and it feels like it is crashing when creating the index because each time i start a search i get a few more results before it dies.
Other Mailaccounts even with bigger Sizes but fewer files/mails do not cause any problems when searching something.
I took a closer look at this Blog-post: https://www.univention.de/2015/07/mailserver-dovecot-als-standard-imappop3-server-in-ucs/ but this seems to more related to open Connections than open files.
root@mail:/var/log# univention-app info
UCS: 4.2-0 errata69
App Center compatibility: 4
Installed: mailserver=11 open-xchange-text=7.8.3-17-ucs1 oxseforucs=7.8.3-17-ucs2
Upgradable:
Are you using solr to index your emails? Maybe it is opening to many messages at once… which would be weird, as the default ulimit -n is 32768: ucr search security/limits
Good Morning,
i switched on mail_debug as you told me and greped ‘test.mail’. I dropped hundreds of similar acls not found lines.
If you need a more detailed logfile i would prefer to send it on a not so public way to you. https://paste.fedoraproject.org/paste/7KBrL8gGoNLuvqSoPcFSOQ
Please check /var/log/syslog for a message similar to this one: dovecot[6312]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Are the mail/dovecot/limits/.* settings changed?
When the error occurs, please run: lsof | grep dovecot | wc -l
root@mail:/var/log# grep ulimit /var/log/syslog.1
Jul 5 12:44:06 mail dovecot[2545]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul 5 12:59:48 mail dovecot[4569]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul 5 19:41:52 mail dovecot[7105]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul 5 19:46:37 mail dovecot[7487]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
root@mail:/var/log# ucr search /limits |grep /limits/
mail/dovecot/limits/anvil/client_limit: <empty>
mail/dovecot/limits/auth/client_limit: <empty>
mail/dovecot/limits/default_client_limit: 1000
mail/dovecot/limits/default_process_limit: <empty>
mail/dovecot/limits/default_vsz_limit: <empty>
mail/dovecot/limits/imap-login/client_limit: <empty>
mail/dovecot/limits/imap-login/process_min_avail: 4
mail/dovecot/limits/imap-login/service_count: 0
mail/dovecot/limits/imap-login/vsz_limit: <empty>
Limit of the imap-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/imap/process_limit: 100
Maximum number of processes that can exist for the IMAP service. If the variable is unset, the value of mail/dovecot/limits/default_process_limit is used.
mail/dovecot/limits/imap/vsz_limit: <empty>
Limit of the imap process' memory usage. If this limit is too low a huge mailbox may not be accessed. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/managesieve-login/process_min_avail: <empty>
mail/dovecot/limits/managesieve-login/service_count: <empty>
mail/dovecot/limits/managesieve-login/vsz_limit: <empty>
Limit of the managesieve-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/managesieve/mail_max_userip_connections: <empty>
mail/dovecot/limits/managesieve/process_limit: <empty>
mail/dovecot/limits/pop3-login/client_limit: <empty>
mail/dovecot/limits/pop3-login/process_min_avail: <empty>
mail/dovecot/limits/pop3-login/service_count: <empty>
mail/dovecot/limits/pop3-login/vsz_limit: <empty>
Limit of the pop3-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/pop3/process_limit: <empty>
Maximum number of processes that can exist for the pop3 service. If the variable is unset, the value of mail/dovecot/limits/default_process_limit is used.
mail/dovecot/limits/pop3/vsz_limit: <empty>
Limit of the pop3 process' memory usage. If this limit is too low a huge mailbox may not be accessed. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
security/limits/default/user/hard/.*: <empty>
security/limits/default/user/hard/nofile: 32768
security/limits/default/user/soft/.*: <empty>
security/limits/default/user/soft/nofile: 32768
security/limits/group/.*/hard/.*: <empty>
security/limits/group/.*/soft/.*: <empty>
security/limits/group/Domain Users/hard/nproc: 1000
security/limits/user/.*/hard/.*: <empty>
security/limits/user/.*/soft/.*: <empty>
root@mail:/var/log# lsof | grep dovecot | wc -l
1470
grep files /proc/$(pgrep -f 'dovecot -F')/limits
Max open files 1024 4096 files
It seems systemd ignores what happens in /etc/default/dovecot and /etc/init.d/dovecot, but a possible solution is:
Edit /etc/systemd/system/multi-user.target.wants/dovecot.service and append LimitNOFILE=20000 to the [Service] section. Then:
Yes, systemd does not use shell scripts by design:
they are ugly
they are error prone
the repeat bad boiler templates
they are slow
they have poor error handling
they have bad concurrency control
systemd only supports reading environment variables from a file (EnvironmentFile=), which uses shell like syntax. Everything else like resource lmits must be specified using the LimitXXX= statements. See man 5 systemd.exec for details.
This is a bad advise:
/etc/systemd/systemd/*.target/ is the equivalent of run-levels from the SystemV-init-system area; it’s a directory containing symbolic links to the service files which should be started for that target. multi-user is the default target.
the symbolic links point to the .service files in /lib/systemd/system/, which are provided by packages - any change gets overwritten on updates!
Instead do one of the following two options:
copy the file from /lib/systemd/system/ to /etc/systemd/system/ and modify the copied file. This has the draw-back that any change to the original file provided through updated packages will not get applied to your copy.
create a directory /etc/systemd/system/dovecor.service.d/ and a file my-overlay.conf (or whatever with suffix .conf) therein with the following content
LimitNOFILE=infinity
```. (See `man 5 systemd.unit` for more details)
Afterwards always run `systemctl daemo-reload` to force systemd to re-read any modified files from the file system.n
root@imap:/etc/systemd/system# systemctl daemon-reload
root@imap:/etc/systemd/system# systemctl cat dovecot.service
# /lib/systemd/system/dovecot.service
# This file is part of Dovecot
#
# If you want to pass additionally command line options to the dovecot
# binary, create the file:
# `/etc/systemd/system/dovecot.service.d/service.conf'.
[Unit]
Description=Dovecot IMAP/POP3 email server
Documentation=man:dovecot(1)
Documentation=http://wiki2.dovecot.org/
After=local-fs.target network.target
[Service]
Type=forking
ExecStart=/usr/sbin/dovecot
PIDFile=/var/run/dovecot/master.pid
ExecReload=/usr/bin/doveadm reload
ExecStop=/usr/bin/doveadm stop
PrivateTmp=true
NonBlocking=yes
# Enable this if your systemd is new enough to support it:
#ProtectSystem=full
# You can add environment variables with e.g.:
#Environment='CORE_OUTOFMEM=1'
# If you have trouble with `Too many open files' you may set:
#LimitNOFILE=8192
# If you want to allow the Dovecot services to produce core dumps, use:
#LimitCORE=infinity
[Install]
WantedBy=multi-user.target
# /lib/systemd/system/dovecot.service.d/10-oboxprestart.conf
[Unit]
ConditionFileNotEmpty=/usr/lib/dovecot/oboxprestartscript
[Service]
ExecStartPre=-/usr/lib/dovecot/oboxprestartscript
# /lib/systemd/system/dovecot.service.d/20-oboxstop.conf
[Unit]
ConditionFileNotEmpty=/usr/lib/dovecot/oboxstopscript
[Service]
ExecStop=-/usr/lib/dovecot/oboxstopscript
# /etc/systemd/system/dovecot.service.d/my-overlay.conf
[Service]
LimitNOFILES=20000
# SAN, 2018-01-04, see https://help.univention.com/t/dovecot-too-many-open-files/6096/12
root@imap:/etc/systemd/system# systemctl stop dovecot.service
root@imap:/etc/systemd/system# systemctl start dovecot.service
root@imap:/etc/systemd/system# grep files /proc/$(pgrep -f '/usr/sbin/dovecot')/limits
Max open files 1024 4096 files
root@imap:/etc/systemd/system#
But the copy way works.
root@imap:/etc/systemd/system# cp /lib/systemd/system/dovecot.service .
root@imap:/etc/systemd/system# vim dovecot.service
root@imap:/etc/systemd/system#
root@imap:/etc/systemd/system# systemctl daemon-reload
root@imap:/etc/systemd/system# systemctl restart dovecot.service
root@imap:/etc/systemd/system# grep files /proc/$(pgrep -f '/usr/sbin/dovecot')/limits
Max open files 81920 81920 files