This article provides a recommendation on how to perform a System Backup of a UCS system or environment and instructions on how to restore said system from these backups.
Disclaimer:
There are innumerable ways in which UCS environments can differ from each other and backup strategies have to be different for each of them. This article does not claim completeness and covers only Setups that are sufficiently close to an artificial default usage scenario.
Note:
There is sometimes confusion about the names nubus and UCS. Nubus is a name for the core functionality of the IAM. It can be run in a kubernetes cluster (nubus for kubernetes) or as part of the linux distribution UCS. This article only deals with the linux distribution UCS and by extension with nubus for UCS.
VM Snapshots and Backups
If UCS is run as a VM on a hypervisor, then the most common and easiest backup strategy is to use the hypervisors tools for a full VM backup or snapshot. This process is not the subject of this article.
It is still advisable to save the important files on a separate site, so please read the rest of the article as well.
Backup Order
If you run multiple UCS servers in your domain, then you will need to respect the following order when creating the snapshots.
- Managed Nodes
- Replica Nodes
- Backup Nodes
- Primary Node
The reason is that the LDAP is replicated between the systems. Since the primary is the only one that can write to the backup, it is important that it always stays the most up to date. Therefore it should be the last to be switched of, and the first to be switched on. In some setups replica nodes replicate from backup nodes instead of the Primary. Therefore the backup nodes have to be more up to date then the replicas.
Managed nodes have no copy of the LDAP. Their exact position in the order depends heavily on their individual purpose. It is generally save to create their backups first.
If you make backups always take backups of all systems.
Backup frequency
It depends on how the UCS environment is used, how often one should create a backup of it. If yous system is used daily and on every day changes could be performed, then we would also advise to perform a daily backup to minimize potential data loss. We also advise to perform backups before (and potentially after) bigger changes like user imports or upgrades.
The default backup processes on UCS run every day. However only the 10 most recent copies are kept.
Practise disaster recovery
None of the steps described here are a guarantee for a successful recovery in case of a disaster. Each environment is different and one has to practise the disaster recovery from time to time. One way to do this is to set up a completely new environment which is in a separated network from the production system.
- Install a UCS
- During the installation select the same settings (host name, domain name, LDAP-Base, IP) as you did for the productions system.
- Perform a backup of the production system
- Copy the backup to the recovery test system.
- Follow the recovery steps described here or your own process.
- Reboot the system(s)
- Perform smoke tests on the recovery test environment to ensure all services are running again.
Automatic Backups
UCS creates by default backups of some components automatically. Every night a dump of the UCR Variables and the LDAP is created and saved to /var/univention-backup. These have to be copied to a separate site by the user.
These automatic backups can be configured with the UCR variables
backup/clean/max_age
Automatic deletion of backup-files in /var/univention-backup/, of files older than backup/clean/max_age. If unset, no files are deleted. If less than backup/clean/min_backups backup-files exist, no files are deleted.
Default: 365
backup/clean/min_backups
Automatic deletion of backup-files in /var/univention-backup/: if backup/clean/max_age is set and less than backup/clean/min_backups backup-files exist, no files are deleted.
Default: 10
samba4/backup/cron/options
This variable allows the configuration of options for the univention-samba4-backup started by cron.
samba4/backup/cron
This variable configures the time/interval when to start the backup of the samba provision directory. The format is documented under ‘man 5 crontab’. If the variable is unset, no backup is performed.
Default: 0 3 * * *
slapd/backup/cron
Cron interval for the slapd database backup.
Default: 0 0 * * *
slapd/backup/group
Group for the slapd database backup file.
Default: root
slapd/backup/owner
Owner for the slapd database backup file.
Default: root
slapd/backup/permissions
File permissions for the slapd database backup file.
Default: 0600
slapd/backup
Activates/deactivates the slapd database backup.
Default: true
ucr/backup/enabled
If this option is activated, a daily cron job performs a backup of all Univention Configuration Registry data tor ‘/var/univention-backup/ucr-backup_*.tgz’. If the variable is unset, no backup is performed.
Default: yes
Univention Apps
Some services may be installed with the app center. One should back up a list of these apps.
To get a list of installed apps and their versions use
univention-app info
To get each apps configuration use
univention-app configure --list $APPNAME
The following script can be used to do this in one step
bkg_dir="/var/univention-backup/apps_$(date +'%Y%m%d')"
mkdir -p "$bkg_dir"
univention-app info > "${bkg_dir}/info"
apps=$(cat "${bkg_dir}/info" | sed -n 's/^Installed://p' | sed 's/ /\n/g' | cut -d'=' -f1 | sed 's|^.*/||g')
for app in $(echo $apps); do univention-app configure --list "${app}" > "${bkg_dir}/${app}.conf"; done
To restore the apps first install the apps from the list in the info file. Then apply the configuration for each app.
univention-app configure APP --set VARIABLE=VALUE
Additional to this some apps have additional state data.
Backup OX-Connector
The OX connector app saves its configuration and state files to /var/lib/univention-appcenter/apps/ox-connector. Therefore this directory should also be backed up if you use this app.
Restore OX-Connector
It is sufficient to place the backed up data to /var/lib/univention-appcenter/apps/ox-connector again.
Backup Keycloak
The keycloak app data can be exported with
univention-app shell keycloak /opt/keycloak/bin/kc.sh export \
--db=$(ucr get kc/db/kind) \
--db-driver=$(ucr get kc/db/driver) \
--transaction-xa-enabled=$(ucr get kc/db/xa) \
--dir /var/lib/univention-appcenter/apps/keycloak/data/export_$(date +"%Y%m%d")
This data contains information about realms, clients, groups, and roles.
Source: 4. Configuration — Univention Keycloak app manual 26.x
Restore Keycloak
univention-app shell keycloak /opt/keycloak/bin/kc.sh import \
--db=$(ucr get kc/db/kind) \
--db-driver=$(ucr get kc/db/driver) \
--transaction-xa-enabled=$(ucr get kc/db/xa) \
--dir /var/lib/univention-appcenter/apps/keycloak/data/export_$(date +"%Y%m%d")
Source: 4. Configuration — Univention Keycloak app manual 26.x
Debian package list
To be able to install the same software again, one should save the list of installed packages.
dpkg --get-selections > /var/univention-backup/dpkg_selections_$(date +'%Y%m%d')
While a list is automatically created at /var/backups by /etc/cron.daily/dpkg, it is less conveniently formatted.
OpenLDAP
The LDAP data is automatically saved every night to /var/univention-backup. It is advisable to save these files to a different site, instead of interrupting the LDAP server to create your own snapshot.
Since the LDAP is replicated anyways, it is sufficient to save the backup from the primary to an offsite place.
Recover OpenLDAP
To recover a previous state of the OpenLDAP from a backup you need to perform the following steps on the UCR primary.
# Stop ldap related service
systemctl stop slapd.service
systemctl stop univention-directory-listener.service
systemctl stop univention-directory-notifier.service
# remove the old LDAP database files
rm -r /var/lib/univention-ldap/ldap/*
# write data from the backup into the LDAP
gzip -d /var/univention-backup/ldap-backup_<DATE>.ldif.gz
slapadd -l /var/univention-backup/ldap-backup_<DATE>.ldif
# start the ldap service
systemctl start slapd.service
# Remove listener state files
rm -r /var/lib/univention-directory-listener/*
# start listener and notifier
systemctl start univention-directory-listener.service
systemctl start univention-directory-notifier.service
# ensure join scrips have run and config files are up to date
univention-run-join-scripts
ucr commit
SAMBA
The SAMBA directory is automatically saved every night to /var/univention-backup. It is advisable to save these files to a different site, instead of interrupting the SAMBA server to create your own snapshot.
If you save the samba settings then you should also save the home directories.
Restore SAMBA
cd /etc/samba
tar -xvf /var/univention-backup/samba/samba-backup-<DATE>.tar.bz2
System wide configuration files in /etc/*
The etc directory is the place for system wide configuration files on most linux systems. Not all configuration files in /etc are created from UCR templates. Instead of checking every one, it is advisable to backup the entire directory.
Among the most important files in /etc belong the .secret files. They contain passwords that are necessary for some services. Your list may be different.
- /etc/backup-join.secret
- /etc/idp-ldap-user.secret
- /etc/ldap-backup.secret
- /etc/ldap.secret
- /etc/machine.secret
- /etc/prometheus_ldap.secret
- /etc/self-service-db.secret
- /etc/slave-join.secret
A simple solution is to use borg backup.
borg init --encryption=none /var/univention-backup/etc
borg create /var/univention-backup/etc::manual ~/etc
You may choose to give the backup a timestamp as a name.
Restore /etc backup
If the backup was created as is described above then one can use the following steps to restore from this backup
cd /
borg extract /var/univention-backup/etc::manual
Home directories
It may be surprising, but in certain cases users of UCS systems get home directories. For example a user that has been used on a windows client which is connected to a UCS Samba domain will have some related files in his home directory.
One should check for his environment if home directories have been created. If yes then one should back them up as well.
One way to perform this backup is with borg backup.
# only once, and not for every backup
mkdir -p /var/univention-backup/home
borg init --encryption=none /var/univention-backup/home
# for every backup
borg create /var/univention-backup/home::manual ~/home
You may choose to give the backup a timestamp as a name.
When selecting a backup solution it is important to select one that also saves the POSIX-ACLs. If your selected solution doesn’t do this then you can use
getfacl -R /home > /var/univention-backup/home_acl_$(date +'%Y%m%d')
to save them separately.
If you need Windows-ACLs because you have SAMBA installed then it may also be necessary to save these separately.
getfattr -RP -d -m -- /home > /var/univention-backup/home_attr_$(date +'%Y%m%d')
Borg backup does not need this.
Warning:
The directory /var/univention-backup may be placed on the same partition as /var, the same as the ldap DB and the log files. To prevent issues it may be advisable to save the backups in another place instead.
Restore /home
Extract the backup and overwrite /home
cd /
borg extract /var/univention-backup/home::manual
If one does not know what save points there are in the backup then one can use
borg list /var/univention-backup/home/
to list them.
UCR Variables
UCS systems are configured for the most part via UCR variables (univention configuration registry). To save the current configuration one has to also save the UCR variables. This is already automatically done with the automatic backup job of univention-backup. They are saved to /var/univention-backup. It is advisable to save these files to a different site.
Restore UCR Variables
To restore the variables it is sufficient to overwrite the key value store files with the backup. An additional commit will then regenerate all the config files from the UCR variables.
cd /etc/univention
tar zxvf /var/univention-backup/ucr-backup_<DATE>.tgz
ucr commit