Cool Solution user-group-sync fails

jester · March 23, 2020, 10:08am

Hey Valentin,
Thanks for your reply. The output of those commands looks all right (see further below), but the log /var/log/univention/listener.log is showing some related errors i think:

22.03.20 19:49:11.257  LISTENER    ( WARN    ) : received signal 15
22.03.20 19:49:16.546  DEBUG_INIT
22.03.20 19:49:16.553  LISTENER    ( WARN    ) : Notifier/LDAP server is ucs.mydomain.loc:7389
22.03.20 19:49:16.553  LDAP        ( PROCESS ) : connecting to ldap://ucs.mydomain.loc:7389
22.03.20 19:49:17.554  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py failed
Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py", line 74, in <module>
    owning_user_number = pwd.getpwnam('ucs-sync').pw_uid
KeyError: 'getpwnam(): name not found: ucs-sync'
22.03.20 19:49:17.554  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py failed in module_import()
UNIVENTION_DEBUG_BEGIN  : uldap.__open host=ucs.mydomain.loc port=7389 base=dc=mydomain,dc=loc
UNIVENTION_DEBUG_END    : uldap.__open host=ucs.mydomain.loc port=7389 base=dc=mydomain,dc=loc

The requested output of univention-check-join-status and univention-run-join-scripts:

root:~# univention-run-join-scripts
univention-run-join-scripts: runs all join scripts existing on local computer.
copyright (c) 2001-2019 Univention GmbH, Germany

Running pre-joinscripts hook(s):                           done
Running 01univention-ldap-server-init.inst                 skipped (already executed)
Running 02univention-directory-notifier.inst               skipped (already executed)
Running 03univention-directory-listener.inst               skipped (already executed)
Running 04univention-ldap-client.inst                      skipped (already executed)
Running 05univention-bind.inst                             skipped (already executed)
Running 08univention-apache.inst                           skipped (already executed)
Running 10univention-ldap-server.inst                      skipped (already executed)
Running 11univention-heimdal-init.inst                     skipped (already executed)
Running 11univention-pam.inst                              skipped (already executed)
Running 15univention-directory-notifier-post.inst          skipped (already executed)
Running 15univention-heimdal-kdc.inst                      skipped (already executed)
Running 18python-univention-directory-manager.inst         skipped (already executed)
Running 20univention-directory-policy.inst                 skipped (already executed)
Running 20univention-join.inst                             skipped (already executed)
Running 20univention-ldap-config-master.inst               skipped (already executed)
Running 22univention-directory-manager-rest.inst           skipped (already executed)
Running 26univention-nagios-common.inst                    skipped (already executed)
Running 26univention-samba.inst                            skipped (already executed)
Running 30univention-appcenter.inst                        skipped (already executed)
Running 30univention-nagios-client.inst                    skipped (already executed)
Running 31univention-nagios-ad-connector.inst              skipped (already executed)
Running 33univention-portal.inst                           skipped (already executed)
Running 34univention-management-console-server.inst        skipped (already executed)
Running 35univention-appcenter-docker.inst                 skipped (already executed)
Running 35univention-management-console-module-adconnector.skipped (already executed)
Running 35univention-management-console-module-appcenter.inskipped (already executed)
Running 35univention-management-console-module-diagnostic.iskipped (already executed)
Running 35univention-management-console-module-ipchange.insskipped (already executed)
Running 35univention-management-console-module-join.inst   skipped (already executed)
Running 35univention-management-console-module-lib.inst    skipped (already executed)
Running 35univention-management-console-module-mrtg.inst   skipped (already executed)
Running 35univention-management-console-module-quota.inst  skipped (already executed)
Running 35univention-management-console-module-reboot.inst skipped (already executed)
Running 35univention-management-console-module-services.insskipped (already executed)
Running 35univention-management-console-module-setup.inst  skipped (already executed)
Running 35univention-management-console-module-sysinfo.instskipped (already executed)
Running 35univention-management-console-module-top.inst    skipped (already executed)
Running 35univention-management-console-module-ucr.inst    skipped (already executed)
Running 35univention-management-console-module-udm.inst    skipped (already executed)
Running 35univention-management-console-module-updater.instskipped (already executed)
Running 35univention-server-overview.inst                  skipped (already executed)
Running 36univention-management-console-module-apps.inst   skipped (already executed)
Running 40univention-virtual-machine-manager-schema.inst   skipped (already executed)
Running 81univention-ad-connector.inst                     skipped (already executed)
Running 81univention-nfs-server.inst                       skipped (already executed)
Running 90univention-bind-post.inst                        skipped (already executed)
Running 91univention-saml.inst                             skipped (already executed)
Running 92univention-management-console-web-server.inst    skipped (already executed)
Running 98univention-pkgdb-tools.inst                      skipped (already executed)
Running 99univention-user-group-sync-source_add-ucs-sync.inskipped (already executed)
Running post-joinscripts hook(s):                          done

root:~# univention-check-join-status
Joined successfully

Kind regards.

heidelberger · March 23, 2020, 10:44am

Hi jester,

it seems the user does not exist at least on the local system. I have an idea why but to confirm that I need the server role of your system. You can get it like so: ucr get server/role

Also does the following command give anything back?
univention-ldapsearch -LLL -b “uid=ucs-sync,cn=users,$(ucr get ldap/base)”

Best regards,
Valentin

jester · March 23, 2020, 10:52am

Hi Valentin,

The requested output:

root:~# ucr get server/role
domaincontroller_master

And:

root:~# univention-ldapsearch -LLL -b "uid=ucs-sync,cn=users,$(ucr get ldap/base)"
No such object (32)
Matched DN: cn=users,dc=mydomain,dc=loc

Kind regards.

NOTE: this ‘source’ server is joined to a Windows AD and is an Windows-compatible Memberserver

heidelberger · March 23, 2020, 11:17am

Hi jester,

thanks for the note! I think that’s the problem. The join script creates a new user, which is required for managing the user/group data files created by the sync. It seems that this does not work in your scenario.
You can try to create the user manually.

User name: ucs-sync
unixhome: /var/lib/univention-user-group-sync
uidNumber: 342

After the user has been created generate the SSH key
su - ucs-sync -c “ssh-keygen -q -f ~/.ssh/id_rsa -t rsa -N ‘’”

Lastly, restart the listener
/etc/init.d/univention-directory-listener restart
and see if the error in /var/log/univention/listener.log persists.

Best regards,
Valentin

jester · March 23, 2020, 12:28pm

Hi Valentin,

We’re only getting some different errors now, but no users/groups on the target system (see below). Any other logs i can check or some debug setting i can adjust to get some insight?

23.03.20 13:01:02.232  LISTENER    ( WARN    ) : initializing module univention_user_group_sync_source_generate
23.03.20 13:01:11.472  LISTENER    ( WARN    ) : finished initializing module univention_user_group_sync_source_generate with rv=0

Kind regards

heidelberger · March 23, 2020, 12:37pm

Hi jester,

those messages mean that the module was initialized successfully, so it should be working.
Some things you can check now:

Are files being generated below /var/lib/univention-user-group-sync when you change a user?
Are those files successfully being transferred to the destination system?
Does univention_user_group_sync_dest.py put out any errors when importing these files?
Also check /var/log/univention/user-group-sync.log on the destination for that

Make sure that the objects you’re trying to sync are matched by the default filter mentioned in the docs: Cool Solution - Sync Users and Groups into a second Domain
Search for the paragraph " Filter the LDAP objects to be synchronized".

If you have set a custom filter with ldap/sync/filter, make sure that the objects you’re trying to transfer are actually matched by that filter (use univention-ldapsearch to test both).

Best regards,
Valentin

jester · March 23, 2020, 3:15pm

Hi Valentin,

Yes, files are being generated (like: 01584964871.2601819). But these files do not appear on the ‘destination’ server.

Kind regards.

jester · March 23, 2020, 3:41pm

Hi Valentin,

I’m guessing the files are being transferred with scp to the ‘destination’, so i logged in as the ucs-sync user and tried it manually:

root:# su - ucs-sync
ucs-sync:$ cd ~
ucs-sync:$ pwd
/var/lib/univention-user-group-sync

ucs-sync:$ scp -i ~/.ssh/id_rsa.pub /var/lib/univention-user-group-sync/test.txt ucs-sync@192.168.0.16:/var/lib/univention-user-group-sync/
Enter passphrase for key '/var/lib/univention-user-group-sync/.ssh/id_rsa.pub':
Password:

This seems odd. I’ve re-created the rsa keys as root with your command from a few posts back and also as the ucs-sync user and made sure no password was entered… Result is the same.

This weem weird to me.

heidelberger · March 23, 2020, 3:55pm

Did you copy the ssh key to the destination system? Your output looks like you’re being asked for the (empty) password for the key first and then for the password of ucs-sync@192.168.0.16. That hints at the ssh key not being present at the destination.

Please execute the ssh-copy-id from the cool solution again.

jester · March 23, 2020, 4:06pm

Done that, even checked if the key was added to the /var/lib/univention-user-group-sync/.ssh/authorized_keys on the destination system.

heidelberger · March 23, 2020, 4:34pm

In that case I think somethings wrong with the key. Maybe it does have a password set for some reason?

jester · March 24, 2020, 9:47am

Hi Valentin,
Will try a clean install of both vm’s and re-do the setup to see if it makes a difference. Will report back my findings.

Stay safe & kind regards.

jester · March 24, 2020, 5:23pm

@heidelberger:

2 cleanly installed UCS4.4-4 vm’s.
‘source’ server is a domain member (of Windows domain).

Installing the univention-user-group-sync-source package on the ‘source’ server i straight away get 2 lines of text in red (full output at the end of post):

Warning: cron.service changed on disk. Run 'systemctl daemon-reload' to reload units.
0 univention_user_group_sync_source_generate /usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py

The first one does not seem to upsetting because there is a Reloading systemd daemon below it. But could the second one be the source of my trouble?

How can i help to troubleshoot this further?

root@ucs:~# univention-install univention-user-group-sync-source
Ign:1 https://updates.software-univention.de/4.0/maintained 4.0-0/all/ InRelease

...

Get:202 https://updates.software-univention.de/4.0/unmaintained/component cool-solutions/amd64/ Packages [2,499 B]
Fetched 49.0 kB in 7s (6,667 B/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  univention-user-group-sync-source
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.7 kB of archives.
After this operation, 73.7 kB of additional disk space will be used.
Get:1 https://updates.software-univention.de/4.4/unmaintained/component cool-solutions/all/ univention-user-group-sync-source 2.1.0-23A~4.4.0.202002251544 [12.7 kB]
Preconfiguring packages ...
Fetched 12.7 kB in 0s (77.8 kB/s)
                                 Selecting previously unselected package univention-user-group-sync-source.
(Reading database ... 86911 files and directories currently installed.)
Preparing to unpack .../univention-user-group-sync-source_2.1.0-23A~4.4.0.202002251544_all.deb ...
Unpacking univention-user-group-sync-source (2.1.0-23A~4.4.0.202002251544) ...
Setting up univention-user-group-sync-source (2.1.0-23A~4.4.0.202002251544) ...
Script: /etc/univention/templates/scripts/restart-listener
Calling joinscript 99univention-user-group-sync-source_add-ucs-sync.inst ...
2020-03-24 18:00:13.449564124+01:00 (in joinscript_init)
Creating local service user
Object created: uid=ucs-sync,cn=users,dc=mydomain,dc=loc
Local service user ucs-sync exists now.
No passwd entry for user 'ucs-sync'
Done!
2020-03-24 18:00:14.480297173+01:00 (in joinscript_save_current_version)
Joinscript 99univention-user-group-sync-source_add-ucs-sync.inst finished with exitcode 0
Creating a synchronization cron job
Create cron/ldap-sync-src/command
Create cron/ldap-sync-src/time
Create cron/ldap-sync-src/user
Create cron/ldap-sync-src/description
File: /etc/cron.d/univention-ucr-cronjobs
Restarting univention-directory-listener Service
Restarting cron Service
Warning: cron.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Reloading systemd daemon
Modules:
3       ad-connector    /usr/lib/univention-directory-listener/system/ad-connector.py
3       app_attributes  /usr/lib/univention-directory-listener/system/app_attributes.py
3       bind    /usr/lib/univention-directory-listener/system/bind.py
3       faillog /usr/lib/univention-directory-listener/system/faillog.py
3       gencertificate  /usr/lib/univention-directory-listener/system/gencertificate.py
3       hosteddomains   /usr/lib/univention-directory-listener/system/hosteddomains.py
3       keytab-member   /usr/lib/univention-directory-listener/system/keytab-member.py
3       keytab  /usr/lib/univention-directory-listener/system/keytab.py
3       ldap_extension  /usr/lib/univention-directory-listener/system/ldap_extension.py
3       ldap_server     /usr/lib/univention-directory-listener/system/ldap_server.py
3       license_uuid    /usr/lib/univention-directory-listener/system/license_uuid.py
3       nagios-client   /usr/lib/univention-directory-listener/system/nagios-client.py
3       nfs-homes       /usr/lib/univention-directory-listener/system/nfs-homes.py
3       nfs-shares      /usr/lib/univention-directory-listener/system/nfs-shares.py
3       nscd_update     /usr/lib/univention-directory-listener/system/nscd.py
3       nss     /usr/lib/univention-directory-listener/system/nss.py
3       pkgdb-watch     /usr/lib/univention-directory-listener/system/pkgdb-watch.py
3       portal_groups   /usr/lib/univention-directory-listener/system/portal_groups.py
3       portal_server   /usr/lib/univention-directory-listener/system/portal_server.py
3       quota   /usr/lib/univention-directory-listener/system/quota.py
3       samba-privileges        /usr/lib/univention-directory-listener/system/samba-privileges.py
3       samba-shares    /usr/lib/univention-directory-listener/system/samba-shares.py
3       udm_extension   /usr/lib/univention-directory-listener/system/udm_extension.py
3       umc-service-providers   /usr/lib/univention-directory-listener/system/umc-service-providers.py
3       univention-admin-diary-backend  /usr/lib/univention-directory-listener/system/univention-admin-diary-backend.py
3       univention-saml-idp-config      /usr/lib/univention-directory-listener/system/univention-saml-idp-config.py
3       univention-saml-servers /usr/lib/univention-directory-listener/system/univention-saml-servers.py
3       univention-saml-simplesamlphp-configuration     /usr/lib/univention-directory-listener/system/univention-saml-simplesamlphp-configuration.py
0       univention_user_group_sync_source_generate      /usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py
3       well-known-sid-name-mapping     /usr/lib/univention-directory-listener/system/well-known-sid-name-mapping.py
Processing triggers for univention-config (14.0.0-12A~4.4.0.201909041712) ...
dpkg-query: no packages found matching ldapacl_66univention-appcenter_app.acl

Kind regards.

jester · March 24, 2020, 5:30pm

Also the /var/log/univention/listener.log log is showing this error again:

24.03.20 18:00:21.373  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py failed
Traceback (most recent call last):
  File "/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py", line 74, in <module>
    owning_user_number = pwd.getpwnam('ucs-sync').pw_uid
KeyError: 'getpwnam(): name not found: ucs-sync'
24.03.20 18:00:21.373  LISTENER    ( ERROR   ) : import of filename=/usr/lib/univention-directory-listener/system/univention_user_group_sync_source_generate.py failed in module_import()

The ucs-sync user exists in the WebGUI. And a univention-check-join-status displays: Joined successfully

heidelberger · March 25, 2020, 8:48am

This output means that the initialization of the listener module failed. Very likely due to the error you posted in your last comment.
The user creation works and the user does appear in the local passwd db but a su ucs-sync does not work. I assume this is due to your AD Member Setup. I haven’t seen this cool solution work in such an environment - only in UCS domains, with and without a connected AD.
Depending on what you intend to use UCS for in your scenario, it might be feasible to install UCS not as a member but as an independent DC master and install the AD Connector from the App Center on it to get users and groups from AD. In that scenario the cool solution should work.
We can also modify the cool solution to work in such a scenario in the professional service. Feel free to drop me a message for that. I can get you in touch with sales then.

heidelberger · March 26, 2020, 11:28am

Hi @jester,
I might have another interesting finding here. By coincidence we were able to observe the problem on another standard UCS domain as well yesterday and I’ve implemented a fix. Package updates should be available soon. It seems to be a timing problem between restarting the listener, it noticing that the import of the listener module which generates the files succeeds and triggering a resync of that module.

jester · March 26, 2020, 7:36pm

Hi Valentin,
Sorry for the lack of response, awkward times at the moment.
Hope to find some time tomorrow to see if your fix will help my case… would be great.

Thanks for your perseverance.
Kind regards.

heidelberger · April 3, 2020, 3:30pm

Hi Jester,

we have just released an update for the sync packages, that should fix the issue that occurs during the installation of the source package.

I suggest you clean up just to make sure, then update the package:

eval "$(ucr shell)"

udm users/user remove --ignore_not_exists --dn "uid=ucs-sync,cn=users,$ldap_base"

rm -rf /var/lib/univention-user-group-sync

# This will put out an error from postrm, you can ignore that
apt remove univention-user-group-sync-source

univention-install univention-user-group-sync-source

Best regards,
Valentin

jester · April 6, 2020, 5:21pm

Hi Valentin,
Finally found some time to give the updated package a test.

The files in /var/lib/univention-user-group-sync are now being transferred to the ‘destination’ system, so that part is working now!

On the destination system some of the users from the source system have been imported, but only a small part. When running the command univention_user_group_sync_dest.py i get:

Reading file /var/lib/univention-user-group-sync/01586186643.9837890
W: Different objectClasses detected NEW ['krb5KDCEntry', 'person', 'automount', 'top', 'inetOrgPerson', 'krb5Principal', 'organizationalPerson', 'univentionPWHistory', 'univentionMail', 'univentionObject', 'shadowAccount', 'sambaSamAccount', 'posixAccount'] vs. OLD ['krb5KDCEntry', 'organizationalPerson', 'automount', 'top', 'inetOrgPerson', 'krb5Principal', 'person', 'univentionPWHistory', 'univentionMail', 'univentionSAMLEnabled', 'shadowAccount', 'sambaSamAccount', 'nextcloudUser', 'posixAccount', 'univentionObject']
W: Ignoring difference in objectClass as specified in ldap/sync/ignore_error/objectClass_difference : univentionSAMLEnabled
E: During User.modify_ldap: Traceback (most recent call last):
  File "/usr/bin/univention_user_group_sync_dest.py", line 440, in _direct_update
    lo.modify(user.position.getDn(), modlist)
  File "/usr/lib/python2.7/dist-packages/univention/admin/uldap.py", line 902, in modify
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Object class violation: attribute 'nextcloudEnabled' not allowed

And in /var/log/univention/user-group-sync.log i see a similar error repeat:

Apr 06 18:50:02: Reading file /var/lib/univention-user-group-sync/01586186643.9837890
Apr 06 18:50:02: Modify User: 'uid=myuser,cn=users,dc=mydomain,dc=loc'
Apr 06 18:50:02: W: Different objectClasses detected NEW ['krb5KDCEntry', 'person', 'automount', 'top', 'inetOrgPerson', 'krb5Principal', 'organizationalPerson', 'univentionPWHistory', 'univentionMail', 'univentionObject', 'shadowAccount', 'sambaSamAccount', 'posixAccount'] vs. OLD ['krb5KDCEntry', 'organizationalPerson', 'automount', 'top', 'inetOrgPerson', 'krb5Principal', 'person', 'univentionPWHistory', 'univentionMail', 'univentionSAMLEnabled', 'shadowAccount', 'sambaSamAccount', 'nextcloudUser', 'posixAccount', 'univentionObject']
Apr 06 18:50:02: W: Ignoring difference in objectClass as specified in ldap/sync/ignore_error/objectClass_difference : univentionSAMLEnabled
Apr 06 18:50:02: E: During User.modify_ldap: Traceback (most recent call last):
  File "/usr/bin/univention_user_group_sync_dest.py", line 440, in _direct_update
    lo.modify(user.position.getDn(), modlist)
  File "/usr/lib/python2.7/dist-packages/univention/admin/uldap.py", line 902, in modify
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Object class violation: attribute 'nextcloudEnabled' not allowed

I’m guessing the warnings can be ignored (possibly resolved by installing Active Directory-compatible Domain Controller component), but the error looks more serious.

Kind regards.

heidelberger · April 16, 2020, 1:48pm

Hi jester,

please refer to the paragraph " Remove attributes/objectClasses before sync" in the docs on how to deal with attributes or objectClasses which are not present in both domains.

Best regards,
Valentin