Migration of Cyrus 2.4 to Dovecot

dovecot
cyrus
dovecot2cyrus
mailservice
migration
ucs-4

#1

Disclaimer

Due to the vast variety of mail appliances and differing structures of environments, we recommend to analyze the feasibility of this migration process in your very own environment beforehand. We strongly recommend to try the entire migration in a test environment to check for possible intolerances and incompatibilites with your productive environment. We strongly recommend to create a backup before this migration in any case! There have been some very rare cases, that could not be reproduced until now, where the Cyrus spool was accidentally deleted during the migration process.

Introduction

This article describes the migration of a Cyrus mail spool into a Dovecot mail spool. The migration keeps mail flags and users’ mail folders. Shared folders are also migrated to Dovecot. We utilize the script “Cyrus2Dovecot” by Freie Universität Berlin, because it allows to automate the migration well with several start parameters.

It is important for the migration to change the LDAP attribute univentionMailHomeServer, which defines the server to redirect incoming mails to in the domain. It exists in every user object with a mail address by default. Assuming that we have a machine that we install Dovecot, and thus uninstall Cyrus, on, the listener-notifier mechanism will automatically create new dovecot mailboxes for all users, that have this server set as Mail home server. If we set Mail home server to another machine’s hostname, the mailboxes will be created there as soon as Dovecot is installed.

The users’ mail addresses are obtained with univention-ldapsearch and handed to cyrus2dovecot, which determines where the to-be-migrated files are and where they shall be migrated to with its start parameters.

Note that the migration script does not delete Cyrus’ mail spool. Tbaherefore two mail spools (Cyrus’ and Dovecot’s) exist on the machine after the migration is finished. If this would result in a capacity problem, make according arrangements! This also gives you the possibility to fallback onto the old Cyrus-Appliance.

Important notes

Note that a downtime might occur, depending on your execution of the migration process. If you plan to do the migration on one single machine, a downtime is unavoidable. In an environment with more than one machine there is the possibility to change the mailHomeServer attributes of the affected users with a script in a way that incoming mails are redirected to the old Cyrus mailboxes until the creation of the according new Dovecot mailbox succeeded. The migration will then migrate all mails from the Cyrus mailboxes to the new Dovecot mailboxes. This is also a possibility, if you want to change servers with migrating or if your environment does not allow a downtime.

Additionally it might be required, that users change their usernames in their mail clients (Thunderbird, Outlook, mobile devices, etc.). Dovecot does only allow logins with the full mail address (user1@example.org) while Cyrus also allows only the UID (user1) as login user.

In-place migration (replacing Cyrus with Dovecot on the same server)

During an in place migration all steps are executed as user “root” on the mailserver (the system running Cyrus at the moment). It might be that clients connecting to the mail server find a completely empty mailbox, when the mailbox has been created but mails have not already been migrated into it.

Preparation

Downloading the script

First, you should download the migration script onto the server. The md5 checksum as of the creation of this article is as follows: 506bb048473ccdb44191d88130606f1d

wget -nv https://raw.githubusercontent.com/a-schild/cyrus2dovecot/master/cyrus2dovecot

If you run into a certificate error, execute this command:

wget -nv --no-check-certificate http://raw.githubusercontent.com/a-schild/cyrus2dovecot/master/cyrus2dovecot

The downloaded script has to be made executable:

chmod +x ./cyrus2dovecot

Installing Dovecot

To install Dovecot and uninstall Cyrus at the same time, this command can be executed. (/var/spool/cyrus and /var/lib/cyrus won’t be touched by this):

univention-install univention-mail-dovecot

Checking the creation of Dovecot mailboxes

During the installation of Dovecot, mailboxes are created for all mail users. You have to wait for the completion of this process. This process can take a long time, depending on how many users’ mailboxes have to be created.

To observe the creation of mailboxes by the listener, execute this command:

tail -f /var/log/univention/listener.log

The process is completed once you see the following line in the file “listener.log”:

LISTENER ( WARN ) : finished initializing module dovecot

Migrating the mail files

After all mailboxes have been created in Dovecot, the actual file migration can be started. After this point in time all incoming mails will be redirected to the new Dovecot mailboxes.

Checking the needed directories

The following directories need to be accessible on the system for the migration script: /var/spool/cyrus, /var/lib/cyrus.

They contain all mails as well as the Cyrus database und have to be accessible on the system that you want to migrate to. Thus, take care that mail spool and Cyrus database are accessible.

Executing the script ‘cyrus2dovecot’

Important note: The migration can take A LOT of time. It strongly depends on how many users have to be moved and how big their mailboxes are. In addition all of the users’ mail data is copied from /var/spool/cyrus to /var/spool/dovecot. Take care that you have free space of at least twice the size of the old Cyrus mail spool on the system for this operation, since the old Cyrus mail spool can only be removed after the migration is completed!

Since the paths for the users beginning with a number are located at a different location than all other users in Cyrus, we provide you with two different commands with different parameters. First all users, whose mail address begins with a number, are migrated. Afterwards all other users are migrated. The paths in the command are the default path names used by Cyrus and Dovecot in UCS.
Unless these paths are not at a different place than in the following command, we recommend to use it, since different variables, ensuring a complete, successful migration, are used.

Migrating all users, whose mail addresses begin with a number

Command to migrate all users, whose mail addresses begin with a number:

for number in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(!(objectClass=univentionMailSharedFolder)))" \
    dn mailPrimaryAddress | grep "^mailPrimaryAddress" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${number}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/user/%U \
    -U /var/lib/cyrus/domain/%g/%d/user/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/user/q/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${number}" | tee ./cyrus2dovecot-numbers.log; \
done

Migrating all users, whose mail addresses do not begin with a number

Command to migrate all users, whose mail addresses do not begin with a number:

for char in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(!(objectClass=univentionMailSharedFolder)))" \
    dn mailPrimaryAddress | grep "^mailPrimaryAddress" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${char}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/user/%U \
    -U /var/lib/cyrus/domain/%g/%d/user/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/user/%1u/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${char}" | tee ./cyrus2dovecot.log; \
done

Migrating all private shared folders, whose mail addresses do begin with a number:

Command to migrate all private shared folders, whose mail addresses do begin with a number:

for sharednumber in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${sharednumber}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${sharednumber}" | tee ./cyrus2dovecot_shared-numbers.log; \
done

Migrating all private shared folders, whose mail addresses do not begin with a number

Command to migrate all private shared folders, whose mail addresses do not begin with a number

for sharedchar in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${sharedchar}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/q/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${sharedchar}" | tee ./cyrus2dovecot_shared.log; \
done

Migrating all public shared folders, whose mail addresses do not begin with a number

Command to migrate all public shared folders, whose mail addresses do not begin with a number

for pubsharednumber in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(!(mailPrimaryAddress=*))(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${pubsharednumber}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/q/%U.seen \
    -D /var/spool/dovecot/public/%d/%P -u "${pubsharednumber}" | tee ./cyrus2dovecot_pub-shared-numbers.log; \
done

Migrating all public shared folders, whose mail addresses do not begin with a number

Command to migrate all public shared folders, whose mail addresses do not begin with a number

for pubsharedchar in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(!(mailPrimaryAddress=*))(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${pubsharedchar}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.seen \
    -D /var/spool/dovecot/public/%d/%P -u "${pubsharedchar}" | tee ./cyrus2dovecot_pub-shared.log; \
done

Testing migration and finishing

You can find a detailed view on how many mails of which user have been migrated in which time in the files cyrus2dovecot.log and cyrus2dovecot-numbers.log aswell as the standard output.

Once the migration is completed, we recommend to grant Dovecot the rights on the migrated files:

chown -R dovemail /var/spool/dovecot/

Cleaning up

Once the migration has finished successfully and the new mail stack’s functionality aswell as the existence of all old mails in the mailboxes has been validated, the old Cyrus mail spool can be deleted from the machine. Additionally the Cyrus packages can be cleaned up:

Uninstalling the cyrus packages

Obsolete Cyrus components can be deleted now:

apt-get purge cyrus-admin-2.4 cyrus-common cyrus-common-2.4 cyrus-imapd-2.4 cyrus-pop3d-2.4 libcyrus-imap-perl24
apt-get autoremove

Deleting Cyrus mail data

Cyrus’ old mail data can be deleted by simply deleting /var/spool/cyrus and /var/lib/cyrus:

rm -rf /var/lib/cyrus /var/spool/cyrus

Migrating Cyrus to Dovecot and switching machines

It is possible to perform the migration of Cyrus to Dovecot to a new server. The executable script does not change. We only changed the commands a bit. In addition the folders containing Cyrus mail data have to be mounted on the new system (using NFS shares for example).

In the first step we will prepare the new mail server and create the Dovecot mailboxes for all users on the new server. Afterwards we will describe how to mount Cyrus’ old mail data onto the new server via NFS. Then a migration command has to be executed, that is slightly different to the upper ones in the in-place migration part. All commands have to be executed as user “root” on the NEW mail server.

Preparation

Downloading the script

First the migration script has to be downloaded to the machine:

wget -nv https://raw.githubusercontent.com/a-schild/cyrus2dovecot/master/cyrus2dovecot

If you run into a certificate error, execute this command:

wget -nv --no-check-certificate http://raw.githubusercontent.com/a-schild/cyrus2dovecot/master/cyrus2dovecot

Make the downloaded script executable:

chmod +x ./cyrus2dovecot

Installing Dovecot

Install the app “Mail server” via the App Center on the UCS system. Alternatively, you can do that with the following command on the command line on the new mailserver:

univention-app install mailserver

Additionally, you should execute the new join scripts:

univention-run-join-scripts

Mount needed directories via NFS

The migration script needs the following diretories to operate: /var/spool/cyrus, /var/lib/cyrus.
They contain all mails aswell as the Cyrus database and have to be accessible on the system, you are migrating TO. For the migration onto a new system, make the directories available via a NFS share on the destination host for example. Create a NFS share on the old mailserver via the UMC for /var/spool/cyrus and /var/lib/cyrus.
Important note: Deactivate root squashing (untick the option “Modify user ID for root user (root squashing)” below the option “NFS” in the “Add” dialog). Then you can mount the shares onto the destination host with the following commands:

mkdir /var/spool/cyrus
mkdir /var/lib/cyrus
mount OLD MAILSERVER:/var/spool/cyrus /var/spool/cyrus
mount OLD MAILSERVER:/var/lib/cyrus /var/lib/cyrus/

Creating Dovecot mailboxes / changing univentionMailHomeServer attribute

To create new mailboxes for all mail users in Dovecot on the new server, the mailHomeServer attribute has to be set to the new machine. You can do this by editing the users in the UMC (mass edit) or with the following command.

Execute this command on the UCS master.
Replace FQDN old mailserver, as well as FQDN old mailserver with the FQDNs of your servers.

while read i; do \
  udm users/user modify --dn "${i}" --set mailHomeServer="FQDN new mailserver"; \
done (univention-ldapsearch -LLLo ldif-wrap=no \
  "(&(univentionMailHomeServer=FQDN old mailserver)(mailPrimaryAddress=*))" \
  dn mailPrimaryAddress | grep "^dn" | cut -f2 -d" ")

The command’s execution will take some time depending on how many users have to be changed.

Checking the new Dovecot mailboxes

During the modification of the mailHomeServer attribute new Dovecot mailboxes are created for all mail users on the new mailserver. This process can take a long time, depending on how many users’ mailboxes have to be created. It has to be completed before the migration of the mail data.

To observe the creation of mailboxes by the listener, execute this command:

tail -f /var/log/univention/listener.log

Migrating Cyrus’ mail data

After all mailboxes have been created in Dovecot, you can begin with the actual data migration. After this point in time all incoming mails will be redirected into the new Dovecot mailboxes. Execute the following commands on the new mailserver:

Executing the script cyrus2dovecot

Important note: The migration can take A LOT of time. It strongly depends on how many users have to be moved and how big their mailboxes are. In addition all of the users’ mail data is copied from /var/spool/cyrus to /var/spool/dovecot. Take care that you have enough free space on the system for this operation!

Since the paths for the users beginning with a number are located at a different location than all other users in Cyrus, we provide you with two different commands with different parameters. First all users, whose mail address begins with a number, are migrated. Afterwards all other users are migrated. The paths in the command are the default path names used by Cyrus and Dovecot in UCS.
Unless these paths are not at a different place than in the following command, we recommend to use it, since different variables, ensuring a complete, successful migration, are used.

We recommend to change the “host basename” of the mails during the migration. The commands come with according start parameters.
Note that it is just about the “normal” hostname (host2) and NOT the FQDN (host2.domain.org).

Migrating all users, whose mail addresses begin with a number

Command to migrate all users, whose mail addresses begin with a number:

for number in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(!(objectClass=univentionMailSharedFolder)))" \
    dn mailPrimaryAddress | grep "^mailPrimaryAddress" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${number}" ] && ./cyrus2dovecot --dovecot-host=$(hostname) \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/user/%U \
    -U /var/lib/cyrus/domain/%g/%d/user/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/user/q/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${number}" | tee ./cyrus2dovecot-numbers.log; \
done

Migrating all users, whose mail addresses do not begin with a number

Command to migrate all users, whose mail addresses do not begin with a number:

for char in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(!(objectClass=univentionMailSharedFolder)))" \
    dn mailPrimaryAddress | grep "^mailPrimaryAddress" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${char}" ] && ./cyrus2dovecot --dovecot-host=$(hostname) \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/user/%U \
    -U /var/lib/cyrus/domain/%g/%d/user/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/user/%1u/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${char}" | tee ./cyrus2dovecot.log; \
done

Migrating all private shared folders, whose mail addresses do begin with a number:

Command to migrate all private shared folders, whose mail addresses do begin with a number:

for sharednumber in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${sharednumber}" ] && ./cyrus2dovecot --dovecot-host=$(hostname) \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${sharednumber}" | tee ./cyrus2dovecot_shared-numbers.log; \
done

Migrating all private shared folders, whose mail addresses do not begin with a number

Command to migrate all private shared folders, whose mail addresses do not begin with a number

for sharedchar in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(mailPrimaryAddress=*)(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${sharedchar}" ] && ./cyrus2dovecot --dovecot-host=$(hostname) \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/q/%U.seen \
    -D /var/spool/dovecot/private/%d/%P/Maildir -u "${sharedchar}" | tee ./cyrus2dovecot_shared.log; \
done

Migrating all public shared folders, whose mail addresses do not begin with a number

Command to migrate all public shared folders, whose mail addresses do not begin with a number

for pubsharednumber in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(!(mailPrimaryAddress=*))(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[0-9]"); do \
  [ -n "${pubsharednumber}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/q/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/q/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/q/%U.seen \
    -D /var/spool/dovecot/public/%d/%P -u "${pubsharednumber}" | tee ./cyrus2dovecot_pub-shared-numbers.log; \
done

Migrating all public shared folders, whose mail addresses do not begin with a number

Command to migrate all public shared folders, whose mail addresses do not begin with a number

for pubsharedchar in $(univention-ldapsearch -LLLo ldif-wrap=no \
    "(&(univentionMailHomeServer=$(hostname -f))(!(mailPrimaryAddress=*))(objectClass=univentionMailSharedFolder))" \
    dn cn | grep "^cn" | cut -f2 -d" " | grep "^[^0-9]"); do \
  [ -n "${pubsharedchar}" ] && ./cyrus2dovecot \
    -C /var/spool/cyrus/mail/domain/%g/%d/%1u/shared/%U \
    -U /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.sub \
    -S /var/lib/cyrus/domain/%g/%d/shared/%1u/%U.seen \
    -D /var/spool/dovecot/public/%d/%P -u "${pubsharedchar}" | tee ./cyrus2dovecot_pub-shared.log; \
done

Testing migration and finishing

You can find a detailed view on how many mails of which user have been migrated in which time in the files ‘cyrus2dovecot*.log’ aswell as the standard output.

Once the migration is completed, we recommend to grant Dovecot the rights on the migrated files:

chown -R dovemail /var/spool/dovecot/

The NFS share can be removed via UMC and unmounted:

umount /var/lib/cyrus /var/spool/cyrus

Cleaning up

Once the migration has been completed successfully and the functionality of the new mail stack, aswell as the existence of all old mails in the new mailboxes has been validated, the old Cyrus’ mail spool can deleted from the machine. Additionally, the Cyrus packages can be removed:

To enable mail users to access their mails/mailboxes, the new mailserver has to be configured in their clients and web apps. The DNS name of the old mailserver might be pointed to IP address of the new mailserver, to simplify this process.

Uninstallig Cyrus on the old mailserver

On the old mailserver, you can remove Cyrus be uninstalling the app “Mail server” via the App Center or by executing this command on the command line:

univention-app remove mailserver

Deleting Cyrus mail data

Cyrus’ old mail data can be deleted by simply deleting /var/spool/cyrus and /var/lib/cyrus:

rm -rf /var/lib/cyrus /var/spool/cyrus

Cyrus/imap No or ambigous results found: 0
Migration von Cyrus 2.4 nach Dovecot
How to remove Cyrus integration for update to UCS 4.3