Enable fulltext search in dovecot

mhuetter · June 2, 2017, 10:27am

Hello,
inspired by https://documentation.open-xchange.com/latest/middleware/components/search/crossfolder_fts_in_mail.html
i tried to configure fast fulltext search in Dovecot.
I got a working virtual folder for an All-Folder-Search but dovecot-solr and dovecot-lucene cause a panic in dovecot.

Panic: file fts-search-serialize.c: line 63 (fts_search_deserialize_add_idx): assertion failed: (*idx < buf->used)
With disabled fts-solr or fts-lucene the Search is working but as expected painfully slow.

Googling for this leads to the possible Solution using a dovecot-Version bigger than 2.2.13u1 which is the latest available Version in Univention.: https://www.dovecot.org/list/dovecot/2015-March/100305.html

I would appreciate any help to get fts working.
Maybe there could be already a patched version 2.2.13 because debian shows 2.2.13u3 but i have no idea if this Version would work.
Is it possible to use backports? because there would be version 2.2.27 where i guess this problem could be solved.
Maybe i try to go the wrong way and there is an other/better way to get fulltextsearch in univention/dovecot.

Regards,
Matthias

troeder · June 2, 2017, 1:36pm

There seems to be no significant difference regarding search between 2.2.13u1 and 2.2.13u3. UCS will not upgrade to the backports version during 4.2.

You can try to install a Dovecot version from the backports repository. It will probably work, but it will not be supported.

You can install Dovecot Pro. I know of a customer that uses it with UCS for a big installation, and does use solr for cross-folder-search.

Greetings
Daniel

mhuetter · June 6, 2017, 6:22am

Hello,
installing something out of backports is no option because of the unsupported state.
I can not understand why it does not work as desired, or why dovecot-solr is even provided via “maintained” (https://updates.software-univention.de/4.2/maintained/4.2-0/amd64/) when it is not working.

I wonder if i do something wrong in setting up dovecot-solr or this version is simple in an useless state.

As one of my primary target is tho satisfy my users i wonder why indexed fast full-text search is not configured by default in univention.

Regarding Dovecot-Pro, i am not sure if this is a version for me. They seem to aim at large Enterprises and Telcos. I found no prize for it but i guess it may be unaffordable for a small business with only~80 Mailboxes. Anyway i will contact my OX-Sales-Partner to clarify.
Even dovecot-ee does not seem to be unproblematical when i read this: https://forge.univention.org/bugzilla/show_bug.cgi?id=44559

I really would appreciate any help getting dovecot-solr running with the univention standard-verions.

Greetings
Matthias

thorp-hansen · June 22, 2017, 10:00am

while the debian dovecot packages are supportet, the fulltext/cross folder search with solr is currently not. Solr can be used for more then dovecot cross folder search, that may be the reason why it is still in maintained.

tafkaz · November 22, 2018, 2:30pm

Hi everyone,
we have just received a corresponding request from a customer.
What is the current status? Can fts (preferably with the all folder option) now be realized with Univention and Dovecot in a way it would be supported?
thanks
Sascha

troeder · November 22, 2018, 3:36pm

Hi Sascha,

it is not yet automatically installed, and the solr component is not supported.
It does however not require you to do any changes that invalidate the UCS-created mail configuration. So while FTS is not supported, the rest of the system is.
In a support case it’d have to be disabled.

To get this working I followed the instructions of OX: https://documentation.open-xchange.com/7.10.0/middleware/dovecot/search/crossfolder_fts_in_mail.html (with minor modifications). Fill the missing parts in below instructions from there.
The changes I made on my personal mail server to get FTS in Dovecot and mailbox-wide search in OX were:

FTS in Dovecot:

univention-install dovecot-solr openjdk-8-jre-headless
wget http://mirror.dkd.de/apache/lucene/solr/7.5.0/solr-7.5.0.tgz
tar xzf solr-7.5.0.tgz solr-7.5.0/bin/install_solr_service.sh --strip-components=2
./install_solr_service.sh solr-7.5.0.tgz -p 8080
/opt/solr/bin/solr config -c dovecot -p 8080 -action set-user-property -property update.autoCreateFields -value false
sudo -u solr /opt/solr/bin/solr create -c dovecot
rm /var/solr/data/dovecot/conf/managed-schema
cp /usr/share/dovecot/solr-schema.xml /var/solr/data/dovecot/conf/schema.xml
chown solr: /var/solr/data/dovecot/conf/schema.xml
vi  /var/solr/data/dovecot/conf/solrconfig.xml
...
systemctl restart solr
vi /etc/dovecot/conf.d/12-solr.conf
...
vi /etc/dovecot/conf.d/16-mailboxes_local.conf
...
mkdir -p /etc/dovecot/virtual/All
chown -R dovemail: /etc/dovecot/virtual/
vi /etc/dovecot/virtual/All/dovecot-virtual
...
systemctl restart dovecot
doveadm -v index -A "*"

FTS over all folders in OX:

vi /opt/open-xchange/etc/findbasic.properties
...
systemctl restart open-xchange

Those instructions are not official! They just worked for me personally.

Greetings
Daniel

tafkaz · November 22, 2018, 4:47pm

Great help Daniel!
Thanks a lot!

tafkaz · November 22, 2018, 4:49pm

Just to get the full picture.
In case the customer would like to purchase dovecot pro licenses (from us) instead, Would fts work out of the box with univention then, just after replacing the repositories where applicable?

troeder · November 22, 2018, 7:44pm

I don’t know much about the pro version. But I don’t think so. I think you’d still need solr to do the actual indexing. But better ask your local Dovecot reseller.

tafkaz · November 22, 2018, 8:25pm

Hi Daniel,
we from Systemschmiede (and FUAGO) are the local dovecot pro reseller
There’s native fts in pro:
https://documentation.open-xchange.com/7.10.0/middleware/dovecot/search/crossfolder_fts_in_mail.html#dovecot-pro-fts

But basically i would presume, it will perfectly do to just add / change the repo for dovecot pro in univention, and then install the dovecot-ee-fts package. Shouldn’t really interact in any UCS matters, would you agree?
thanks Sascha

troeder · November 23, 2018, 9:41am

The dovecot-ee- packages are a supported alternative to the OSS packages and the Debian package dependencies of univention-mail-dovecot reflect that:

Package: univention-mail-dovecot
Architecture: all
Depends: ${misc:Depends},
[..]
 dovecot-imapd (>= 2.2.13) | dovecot-ee-imapd (>=2.2.13),
 dovecot-ldap (>= 2.2.13) | dovecot-ee-ldap (>=2.2.13),
 dovecot-lmtpd (>= 2.2.13) | dovecot-ee-lmtpd (>=2.2.13),
 dovecot-managesieved (>= 2.2.13) | dovecot-ee-managesieved (>=2.2.13),
 dovecot-pop3d (>= 2.2.13) | dovecot-ee-pop3d (>=2.2.13),
 dovecot-sieve (>= 2.2.13) | dovecot-ee-sieve (>=2.2.13),
[..]

Additionally installing dovecot-ee-fts shouldn’t bother the UCS mail system at all.

tafkaz · November 23, 2018, 9:54am

Great!
that’s all i wanted to make sure.

Cheers to beautyful Bremen and have a great weekend everyone.
Sascha

elastorsa · February 8, 2019, 3:33pm

Hi, we have in our company Univention 4.3.3 and Open-Xchange for 150 users. We are following:
Dovecot Community + FTS
The problem is when we have to configure the folder to index of mail (location = virtual:/etc/dovecot/virtual:INDEX=/var/vmail/%u/virtual).

The sample use vmail, but we have use open-xchange. Thanks.

tafkaz · June 19, 2019, 7:10am

Hi,
just for the records. I think elastorsa’s issue has been resolved already (by us), so i think, this complete thread can be marked as solved.
best
Sascha

elastorsa · June 19, 2019, 7:26am

The issue has been resolved. It is working perfectly.

tafkaz · June 19, 2019, 7:27am

great to hear! Thanks
Sascha