Server password change on DC master fails reproducible

[SirTux]

Hi,

the server password change failed last night night. Also the manual trigger fails:

Q&A: How can I change the machine password from my ucs master?

Question: How can I change the machine password from may ucs master. For example if you get ldap_bind: Invalid credentials (49) from a ldapsearch command. Answer: You can try the server-password change routine: But in some cases this does not work anymore, if the password ist already wrong and the ldapaccess with the old machine.secret does not work, anymore you have to set the password manually. You can use an old one, they are saved in /etc/machine.secret.old or set a new one: echo …

I can only get a working status setting the machine password to the old value.

[2023-03-09 16:34:04.027000415] Starting server password change
[2023-03-09 16:34:04.346111935] Proceeding with regular server password change scheduled for today
run-parts: executing /usr/lib/univention-server/server_password_change.d/50univention-mail-server prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/portal-server-password-rotate prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-admin-diary prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-bind prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-dhcp prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-directory-manager-rest prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-libnss-ldap prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-mail-dovecot prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-node-exporter prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-nscd prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-postgresql-password prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-radius prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-s4-connector prechange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-samba4 prechange
Object modified: cn=ucsmaster,cn=dc,cn=computers,dc=top2,dc=top1
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
ldap_bind: Invalid credentials (49)
authentication error: Authentication failed
run-parts: executing /usr/lib/univention-server/server_password_change.d/50univention-mail-server nochange
File: /etc/listfilter.secret
Multifile: /etc/postfix/ldap.distlist
Multifile: /etc/postfix/ldap.groups
Multifile: /etc/postfix/ldap.external_aliases
Multifile: /etc/postfix/ldap.sharedfolderlocal
Multifile: /etc/postfix/ldap.virtualwithcanonical
Multifile: /etc/postfix/ldap.virtual_mailbox
Multifile: /etc/postfix/ldap.sharedfolderremote
Multifile: /etc/postfix/ldap.sharedfolderlocal_aliases
Multifile: /etc/postfix/ldap.virtual
Multifile: /etc/postfix/ldap.canonicalrecipient
Multifile: /etc/postfix/ldap.transport
Multifile: /etc/postfix/ldap.canonicalsender
Multifile: /etc/postfix/ldap.saslusermapping
Multifile: /etc/postfix/ldap.virtualdomains
run-parts: executing /usr/lib/univention-server/server_password_change.d/portal-server-password-rotate nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-admin-diary nochange
e755799b-e536-414a-8014-be657b69a1dd
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-bind nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-dhcp nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-directory-manager-rest nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-libnss-ldap nochange
File: /etc/libnss-ldap.conf
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-mail-dovecot nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-node-exporter nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-nscd nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-postgresql-password nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-radius nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-s4-connector nochange
run-parts: executing /usr/lib/univention-server/server_password_change.d/univention-samba4 nochange
[2023-03-09 16:35:27.887435336] resetting old server password for cn=ucsmaster,cn=dc,cn=computers,dc=top2,dc=top1, because access to LDAP master did not work with the new password

Has anyone a idea how I can fix this?

Best,
Stefan

EDIT: The password change on a DC slave worked without problems last night

[SirTux]

After Patching UCS 4.4-7 to latest errata 966: "failed to contact LDAP server"

Following @DirkS 's hint via e-mail (many thanks and sorry Mr. Schnick, I first had to awkwardly extract the credentials to our email provider, as we’ve been using fetchmail for years and manage the passwords for direct access in a password manager, which was also no longer accessible…): “Have you configured faillog or ppolicy lockout?” I was sent in the right direction and followed the howto to disable the lockouts and now at least “univention-ldapsearch” works again… This is already the s…

[The_Preacher]

Hello and welcome to the club!

Sadly, the old how-to to enable the lockout is still online and has never been updated to include a warning…

BR,
TP

[gulden]

Hello @The_Preacher ,

@The_Preacher

Sadly, the old how-to to enable the lockout is still online and has never been updated to include a warning…

Can you please be more specific about which how-to you mean and what kind of update you wish to see there? I need at least a link to the how-to to have the right context.

Thank you.

Best regards,
Nico

[The_Preacher]

Hello @gulden and sorry for the delayed answer,

it has been quite a stressful week…

It’s about THIS howto:
In combination (as far as I could understand) with the automated server password change, it occasionally triggers the system to lock itself out.
This happens overnight and of course the next morning you run into successively bigger and bigger problems as you can only do work to fix the lockouts (LDAP etc.) as a root user in the shell.

When we first stumbled upon this, the research was rather difficult.

By means of THIS GUIDE, one solves the problem permanently, but I guess the point would be if the lockouts could be limited to user accounts instead of covering all accounts including system accounts?

Best regards,
TP

[DirkS]

Hey @The_Preacher ,

you are not alone

Look on the bright side: Lockout works
My clear advice is: Do not use lockout and server password change together!
Since ?2020? you can deactivate the server password change without getting trouble with f.e. BSI Grundschutz.

The problem with these two mechanism together is, that during the server password change running services like DHCP still using the old one. It takes only miliseconds to lockout a UCS server running f.e. DHCP during the server password change.

But deactivating server password change will not cure the long list of bugs of the lockout mechanism. My newest list is this one, but maybe there are more: Bug List

This feature has a lot potential (to become better)

Keep smiling DirkS

[The_Preacher]

Hello @DirkS,

I agree: there’s a lot of potential…

I didn’t even know that it is possible to disable the server-password-change! This must have been configured ages ago or was standard configuration back then. You know, I’ve started with the c’t-Edition after SBS 2011…

Anyway, we’re going to wait for the potential to be “exploited” before enabling auto-lockout again!

Best regards,
TP

[talleyrand]

There are a few race conditions that DDOS the pw change, it can also happen on other apps.
and I am sure I have seen an issue where the AD crosses a time zone… that is to say, the time comparison & setting my have an exploit related to TZ, if i remember correctly… unless it got fixed…

I think it relates to users & doing a forced lockout or account deactivation,( maybe related to the user has to change the PW on initial log account setup.)

the other thing that needs sorting out, it that UCS and samba requirements are NOT synced… as per pw requirements , length etc…

[gulden]

The observed behavior has already been reported in Bug 53062 – Server password change can cause ppolicy lockout .