Dovecot too many open Files

Apps & App Center
, ,
#1

Hello,
when searching something in “All Folder” one of my Mailaccounts causes Timeouts in OX and this kind of logentry in /var/log/dovecot.err:

Jul  4 13:38:43 mail dovecot: imap(user@domain.at): Error: open(/var/spool/dovecot/private/***MAILDIR***_exner/cur/1499011087.M353693P30815.mail,S=16
84667,W=1706830) failed: Too many open files

I do not understand which open-file-limit is meant there or how to find out which configuration could help me to fix this.
This Mailaccount has about 17500 Emails and 10G size and it feels like it is crashing when creating the index because each time i start a search i get a few more results before it dies.
Other Mailaccounts even with bigger Sizes but fewer files/mails do not cause any problems when searching something.
I took a closer look at this Blog-post: https://www.univention.de/2015/07/mailserver-dovecot-als-standard-imappop3-server-in-ucs/ but this seems to more related to open Connections than open files.

root@mail:/var/log# univention-app info
UCS: 4.2-0 errata69
App Center compatibility: 4
Installed: mailserver=11 open-xchange-text=7.8.3-17-ucs1 oxseforucs=7.8.3-17-ucs2
Upgradable:

Greetings
Matthias Hütter

#2

Are you using solr to index your emails? Maybe it is opening to many messages at once… which would be weird, as the default ulimit -n is 32768:
ucr search security/limits

#3

Solr is installed but disabled in dovecot-config because it crashes dovecot (Enable fulltext search in dovecot)

root@mail:/etc/amavis/conf.d# ulimit -n
65536
root@mail:/etc/amavis/conf.d# ucr search security/limits |grep /limits/
security/limits/default/user/hard/.*: <empty>
security/limits/default/user/hard/nofile: 32768
security/limits/default/user/soft/.*: <empty>
security/limits/default/user/soft/nofile: 32768
security/limits/group/.*/hard/.*: <empty>
security/limits/group/.*/soft/.*: <empty>
security/limits/group/Domain Users/hard/nproc: 1000
security/limits/user/.*/hard/.*: <empty>
security/limits/user/.*/soft/.*: <empty>
#4

Please switch on verbose logging and reproduce the error:

ucr set mail/dovecot/logging/mail_debug=yes
systemctl restart dovecot.service
#5

Good Morning,
i switched on mail_debug as you told me and greped ‘test.mail’. I dropped hundreds of similar acls not found lines.
If you need a more detailed logfile i would prefer to send it on a not so public way to you.
https://paste.fedoraproject.org/paste/7KBrL8gGoNLuvqSoPcFSOQ

#6
  • How many folders does this account have in total?
  • find /var/spool/dovecot/private/faschang.at/test.mail/ -type d | wc -l
  • Is this running on bare metal or in a virtualized environment?
#7 
root@mail:/var/log# find /var/spool/dovecot/private/faschang.at/test.mail/ -type d | wc -l
1685

This is running on a vmware ESX5.5 (HA)

#8

Please check /var/log/syslog for a message similar to this one:
dovecot[6312]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }

Are the mail/dovecot/limits/.* settings changed?

When the error occurs, please run: lsof | grep dovecot | wc -l

#9

some /dovecot/limits/ are altered to the “High Performance Mode” from your Blog post (https://www.univention.de/2015/07/mailserver-dovecot-als-standard-imappop3-server-in-ucs/)

root@mail:/var/log# grep ulimit /var/log/syslog.1
Jul  5 12:44:06 mail dovecot[2545]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul  5 12:59:48 mail dovecot[4569]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul  5 19:41:52 mail dovecot[7105]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
Jul  5 19:46:37 mail dovecot[7487]: Warning: fd limit (ulimit -n) is lower than required under max. load (1024 < 2000), because of service auth { client_limit }
root@mail:/var/log# ucr search /limits |grep /limits/
mail/dovecot/limits/anvil/client_limit: <empty>
mail/dovecot/limits/auth/client_limit: <empty>
mail/dovecot/limits/default_client_limit: 1000
mail/dovecot/limits/default_process_limit: <empty>
mail/dovecot/limits/default_vsz_limit: <empty>
mail/dovecot/limits/imap-login/client_limit: <empty>
mail/dovecot/limits/imap-login/process_min_avail: 4
mail/dovecot/limits/imap-login/service_count: 0
mail/dovecot/limits/imap-login/vsz_limit: <empty>
 Limit of the imap-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/imap/process_limit: 100
 Maximum number of processes that can exist for the IMAP service. If the variable is unset, the value of mail/dovecot/limits/default_process_limit is used.
mail/dovecot/limits/imap/vsz_limit: <empty>
 Limit of the imap process' memory usage. If this limit is too low a huge mailbox may not be accessed. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/managesieve-login/process_min_avail: <empty>
mail/dovecot/limits/managesieve-login/service_count: <empty>
mail/dovecot/limits/managesieve-login/vsz_limit: <empty>
 Limit of the managesieve-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/managesieve/mail_max_userip_connections: <empty>
mail/dovecot/limits/managesieve/process_limit: <empty>
mail/dovecot/limits/pop3-login/client_limit: <empty>
mail/dovecot/limits/pop3-login/process_min_avail: <empty>
mail/dovecot/limits/pop3-login/service_count: <empty>
mail/dovecot/limits/pop3-login/vsz_limit: <empty>
 Limit of the pop3-login process' memory usage. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
mail/dovecot/limits/pop3/process_limit: <empty>
 Maximum number of processes that can exist for the pop3 service. If the variable is unset, the value of mail/dovecot/limits/default_process_limit is used.
mail/dovecot/limits/pop3/vsz_limit: <empty>
 Limit of the pop3 process' memory usage. If this limit is too low a huge mailbox may not be accessed. If the variable is unset, the value of mail/dovecot/limits/default_vsz_limit is used.
security/limits/default/user/hard/.*: <empty>
security/limits/default/user/hard/nofile: 32768
security/limits/default/user/soft/.*: <empty>
security/limits/default/user/soft/nofile: 32768
security/limits/group/.*/hard/.*: <empty>
security/limits/group/.*/soft/.*: <empty>
security/limits/group/Domain Users/hard/nproc: 1000
security/limits/user/.*/hard/.*: <empty>
security/limits/user/.*/soft/.*: <empty>

root@mail:/var/log# lsof | grep dovecot | wc -l
1470
#10

I can confirm this:

grep files /proc/$(pgrep -f 'dovecot -F')/limits 

Max open files            1024                 4096                 files

It seems systemd ignores what happens in /etc/default/dovecot and /etc/init.d/dovecot, but a possible solution is:
Edit /etc/systemd/system/multi-user.target.wants/dovecot.service and append LimitNOFILE=20000 to the [Service] section. Then:

systemctl daemon-reload
systemctl restart dovecot.service

grep files /proc/$(pgrep -f 'dovecot -F')/limits

Max open files            20000                20000                files
#11

Thank you very much!
Your possible solution is working for me.
Is this Edit persistent or do i have to watch this in future updates?

Greets
Matthias Hütter

#12

Yes, systemd does not use shell scripts by design:

  • they are ugly
  • they are error prone
  • the repeat bad boiler templates
  • they are slow
  • they have poor error handling
  • they have bad concurrency control

systemd only supports reading environment variables from a file (EnvironmentFile=), which uses shell like syntax. Everything else like resource lmits must be specified using the LimitXXX= statements. See man 5 systemd.exec for details.

This is a bad advise:

  • /etc/systemd/systemd/*.target/ is the equivalent of run-levels from the SystemV-init-system area; it’s a directory containing symbolic links to the service files which should be started for that target. multi-user is the default target.
  • the symbolic links point to the .service files in /lib/systemd/system/, which are provided by packages - any change gets overwritten on updates!

Instead do one of the following two options:

  • copy the file from /lib/systemd/system/ to /etc/systemd/system/ and modify the copied file. This has the draw-back that any change to the original file provided through updated packages will not get applied to your copy.
  • create a directory /etc/systemd/system/dovecor.service.d/ and a file my-overlay.conf (or whatever with suffix .conf) therein with the following content
LimitNOFILE=infinity
```. (See `man 5 systemd.unit` for more details)

Afterwards always run `systemctl daemo-reload` to force systemd to re-read any modified files from the file system.n
#13

Hi,

we tried this.

But it dit not work



root@imap:/etc/systemd/system# systemctl daemon-reload
root@imap:/etc/systemd/system# systemctl cat dovecot.service
# /lib/systemd/system/dovecot.service
# This file is part of Dovecot
#
# If you want to pass additionally command line options to the dovecot
# binary, create the file:
#       `/etc/systemd/system/dovecot.service.d/service.conf'.

[Unit]
Description=Dovecot IMAP/POP3 email server
Documentation=man:dovecot(1)
Documentation=http://wiki2.dovecot.org/
After=local-fs.target network.target

[Service]
Type=forking
ExecStart=/usr/sbin/dovecot
PIDFile=/var/run/dovecot/master.pid
ExecReload=/usr/bin/doveadm reload
ExecStop=/usr/bin/doveadm stop
PrivateTmp=true
NonBlocking=yes
# Enable this if your systemd is new enough to support it:
#ProtectSystem=full

# You can add environment variables with e.g.:
#Environment='CORE_OUTOFMEM=1'
# If you have trouble with `Too many open files' you may set:
#LimitNOFILE=8192
# If you want to allow the Dovecot services to produce core dumps, use:
#LimitCORE=infinity

[Install]
WantedBy=multi-user.target

# /lib/systemd/system/dovecot.service.d/10-oboxprestart.conf
[Unit]
ConditionFileNotEmpty=/usr/lib/dovecot/oboxprestartscript

[Service]
ExecStartPre=-/usr/lib/dovecot/oboxprestartscript


# /lib/systemd/system/dovecot.service.d/20-oboxstop.conf
[Unit]
ConditionFileNotEmpty=/usr/lib/dovecot/oboxstopscript

[Service]
ExecStop=-/usr/lib/dovecot/oboxstopscript


# /etc/systemd/system/dovecot.service.d/my-overlay.conf
[Service]
LimitNOFILES=20000
# SAN, 2018-01-04, see https://help.univention.com/t/dovecot-too-many-open-files/6096/12
root@imap:/etc/systemd/system# systemctl stop dovecot.service
root@imap:/etc/systemd/system# systemctl start dovecot.service
root@imap:/etc/systemd/system# grep files /proc/$(pgrep -f '/usr/sbin/dovecot')/limits
Max open files            1024                 4096                 files
root@imap:/etc/systemd/system#

But the copy way works.

root@imap:/etc/systemd/system# cp /lib/systemd/system/dovecot.service .
root@imap:/etc/systemd/system# vim dovecot.service
root@imap:/etc/systemd/system#
root@imap:/etc/systemd/system# systemctl daemon-reload
root@imap:/etc/systemd/system# systemctl restart dovecot.service
root@imap:/etc/systemd/system# grep files /proc/$(pgrep -f '/usr/sbin/dovecot')/limits
Max open files            81920                81920                files
#14

Because I made an error in my initial post: The property is called “NoLIMIT” without the trailing “S”.
I fixed my original post to contain that fix.

1 Like
Mastodon