Enable fulltext search in dovecot

Apps & App Center
#1

Hello,
inspired by https://documentation.open-xchange.com/latest/middleware/components/search/crossfolder_fts_in_mail.html
i tried to configure fast fulltext search in Dovecot.
I got a working virtual folder for an All-Folder-Search but dovecot-solr and dovecot-lucene cause a panic in dovecot.

Panic: file fts-search-serialize.c: line 63 (fts_search_deserialize_add_idx): assertion failed: (*idx < buf->used)
With disabled fts-solr or fts-lucene the Search is working but as expected painfully slow.

Googling for this leads to the possible Solution using a dovecot-Version bigger than 2.2.13u1 which is the latest available Version in Univention.: https://www.dovecot.org/list/dovecot/2015-March/100305.html

I would appreciate any help to get fts working.
Maybe there could be already a patched version 2.2.13 because debian shows 2.2.13u3 but i have no idea if this Version would work.
Is it possible to use backports? because there would be version 2.2.27 where i guess this problem could be solved.
Maybe i try to go the wrong way and there is an other/better way to get fulltextsearch in univention/dovecot.

Regards,
Matthias

Dovecot too many open Files
UCS + openxchange email search all folders
#2

There seems to be no significant difference regarding search between 2.2.13u1 and 2.2.13u3. UCS will not upgrade to the backports version during 4.2.

You can try to install a Dovecot version from the backports repository. It will probably work, but it will not be supported.

You can install Dovecot Pro. I know of a customer that uses it with UCS for a big installation, and does use solr for cross-folder-search.

Greetings
Daniel

#3

Hello,
installing something out of backports is no option because of the unsupported state.
I can not understand why it does not work as desired, or why dovecot-solr is even provided via “maintained” (https://updates.software-univention.de/4.2/maintained/4.2-0/amd64/) when it is not working.

I wonder if i do something wrong in setting up dovecot-solr or this version is simple in an useless state.

As one of my primary target is tho satisfy my users i wonder why indexed fast full-text search is not configured by default in univention.

Regarding Dovecot-Pro, i am not sure if this is a version for me. They seem to aim at large Enterprises and Telcos. I found no prize for it but i guess it may be unaffordable for a small business with only~80 Mailboxes. Anyway i will contact my OX-Sales-Partner to clarify.
Even dovecot-ee does not seem to be unproblematical when i read this: https://forge.univention.org/bugzilla/show_bug.cgi?id=44559

I really would appreciate any help getting dovecot-solr running with the univention standard-verions.

Greetings
Matthias

#4

while the debian dovecot packages are supportet, the fulltext/cross folder search with solr is currently not. Solr can be used for more then dovecot cross folder search, that may be the reason why it is still in maintained.

#5

Hi everyone,
we have just received a corresponding request from a customer.
What is the current status? Can fts (preferably with the all folder option) now be realized with Univention and Dovecot in a way it would be supported?
thanks
Sascha

#6

Hi Sascha,

it is not yet automatically installed, and the solr component is not supported.
It does however not require you to do any changes that invalidate the UCS-created mail configuration. So while FTS is not supported, the rest of the system is.
In a support case it’d have to be disabled.

To get this working I followed the instructions of OX: https://documentation.open-xchange.com/7.10.0/middleware/dovecot/search/crossfolder_fts_in_mail.html (with minor modifications). Fill the missing parts in below instructions from there.
The changes I made on my personal mail server to get FTS in Dovecot and mailbox-wide search in OX were:

  1. FTS in Dovecot:
univention-install dovecot-solr openjdk-8-jre-headless
wget http://mirror.dkd.de/apache/lucene/solr/7.5.0/solr-7.5.0.tgz
tar xzf solr-7.5.0.tgz solr-7.5.0/bin/install_solr_service.sh --strip-components=2
./install_solr_service.sh solr-7.5.0.tgz -p 8080
/opt/solr/bin/solr config -c dovecot -p 8080 -action set-user-property -property update.autoCreateFields -value false
sudo -u solr /opt/solr/bin/solr create -c dovecot
rm /var/solr/data/dovecot/conf/managed-schema
cp /usr/share/dovecot/solr-schema.xml /var/solr/data/dovecot/conf/schema.xml
chown solr: /var/solr/data/dovecot/conf/schema.xml
vi  /var/solr/data/dovecot/conf/solrconfig.xml
...
systemctl restart solr
vi /etc/dovecot/conf.d/12-solr.conf
...
vi /etc/dovecot/conf.d/16-mailboxes_local.conf
...
mkdir -p /etc/dovecot/virtual/All
chown -R dovemail: /etc/dovecot/virtual/
vi /etc/dovecot/virtual/All/dovecot-virtual
...
systemctl restart dovecot
doveadm -v index -A "*"
  1. FTS over all folders in OX:
vi /opt/open-xchange/etc/findbasic.properties
...
systemctl restart open-xchange

Those instructions are not official! They just worked for me personally.

Greetings
Daniel

#7

Great help Daniel!
Thanks a lot!

#8

Just to get the full picture.
In case the customer would like to purchase dovecot pro licenses (from us) instead, Would fts work out of the box with univention then, just after replacing the repositories where applicable?

#9

I don’t know much about the pro version. But I don’t think so. I think you’d still need solr to do the actual indexing. But better ask your local Dovecot reseller.

#10

Hi Daniel,
we from Systemschmiede (and FUAGO) are the local dovecot pro reseller :wink:
There’s native fts in pro:
https://documentation.open-xchange.com/7.10.0/middleware/dovecot/search/crossfolder_fts_in_mail.html#dovecot-pro-fts

But basically i would presume, it will perfectly do to just add / change the repo for dovecot pro in univention, and then install the dovecot-ee-fts package. Shouldn’t really interact in any UCS matters, would you agree?
thanks Sascha

#11

The dovecot-ee- packages are a supported alternative to the OSS packages and the Debian package dependencies of univention-mail-dovecot reflect that:

Package: univention-mail-dovecot
Architecture: all
Depends: ${misc:Depends},
[..]
 dovecot-imapd (>= 2.2.13) | dovecot-ee-imapd (>=2.2.13),
 dovecot-ldap (>= 2.2.13) | dovecot-ee-ldap (>=2.2.13),
 dovecot-lmtpd (>= 2.2.13) | dovecot-ee-lmtpd (>=2.2.13),
 dovecot-managesieved (>= 2.2.13) | dovecot-ee-managesieved (>=2.2.13),
 dovecot-pop3d (>= 2.2.13) | dovecot-ee-pop3d (>=2.2.13),
 dovecot-sieve (>= 2.2.13) | dovecot-ee-sieve (>=2.2.13),
[..]

Additionally installing dovecot-ee-fts shouldn’t bother the UCS mail system at all.

#12

Great!
that’s all i wanted to make sure.

Cheers to beautyful Bremen and have a great weekend everyone.
Sascha

#13

Hi, we have in our company Univention 4.3.3 and Open-Xchange for 150 users. We are following:
Dovecot Community + FTS
The problem is when we have to configure the folder to index of mail (location = virtual:/etc/dovecot/virtual:INDEX=/var/vmail/%u/virtual).

The sample use vmail, but we have use open-xchange. Thanks.

#14

Hi,
just for the records. I think elastorsa’s issue has been resolved already (by us), so i think, this complete thread can be marked as solved.
best
Sascha

1 Like
#15

The issue has been resolved. It is working perfectly.

#16

great to hear! Thanks
Sascha

Mastodon