Kopano no caching of kerberos authentication - billion of kerberos requets per day, DOSING OUR SAMBA

Apps & App Center
#1

Hi,

we’re using kopano on a UCS server and have an additional UCS-Server as DC.

As kopano is doing a kerberos requests almost for every action a user take in the webapp, we have nearly a billion kerberos requests per day. We only have around 100 users!

As this renders our samba server almost unusable, i’m seeking for help here.
Any ideas?

iowait is all the time at ~30% on samba server due to the high amount of ongoing requests against ldap/kerberos service.

top - 07:43:18 up 2 days, 18:58,  1 user,  load average: 1,16, 1,04, 0,99
Tasks: 216 total,   2 running, 214 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3,4 us,  0,0 sy,  0,0 ni, 96,6 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu1  :  3,2 us,  0,0 sy,  0,0 ni, 96,8 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu2  :  0,0 us,  0,0 sy,  0,0 ni,100,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu3  :  0,0 us,  0,0 sy,  0,0 ni,100,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu4  : 51,7 us, 13,8 sy,  0,0 ni,  3,4 id, 27,6 wa,  0,0 hi,  3,4 si,  0,0 st
%Cpu5  :  0,0 us,  0,0 sy,  0,0 ni,100,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu6  :  3,4 us,  0,0 sy,  0,0 ni, 93,1 id,  0,0 wa,  0,0 hi,  3,4 si,  0,0 st
%Cpu7  :  3,4 us,  3,4 sy,  0,0 ni, 93,1 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem : 16049028 total,  8481872 free,  5749512 used,  1817644 buff/cache
KiB Swap:   962556 total,   962556 free,        0 used.  9938888 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                 
 1996 root      20   0 5191572 4,280g  17988 R  66,7 28,0   1361:38 samba

Total DISK READ:         0.00 B/s | Total DISK WRITE:         2.54 M/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       2.62 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                      
 1996 be/4 root        0.00 B/s    2.53 M/s  0.00 % 24.40 % samba: conn[kdc_tcp] c[ipv4:172.16.2.4:37328] s[ipv4:172.16.2.2:88] server_id[1996.46]
  481 be/3 root        0.00 B/s    6.96 K/s  0.00 %  0.43 % [jbd2/sda1-8]
 2560 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % samba: conn[ldap] c[ipv4:172.16.2.4:55068] s[ipv4:172.16.2.2:389] server_id[2560]

172.16.2.2 is UCS/samba
172.16.2.4 is Kopano.

Any help is greatly appreciated.

#2

Hi @siegmarb,

the part that does not really make sense for me is “why Kerberos/Samba?” For good reason kopano-server connects directly towards OpenLDAP on UCS. The only case that I am aware of where Kerberos is used on UCS is when there is an external MS AD that UCS binds to.

To reduce the amount of ldap binds a while ago OIDC login was implemented in WebApp. There are instructions for setting it up in https://github.com/Kopano-dev/ucs-oidc-webapp

#3

Hi @fbartels In many cases, kopano is just a member server of a domain. So it is in our setup.

The AD is another UCS-Server. Please implement a caching of credentials.

OIDC is another puzzle that does not integrate easy and simply into UCS/Kopano. We do not want to inject code (git …) and have another piece, that can break.

#4

But if the auth is going over kerberos instead of the locally replicated ldap then it does not sounds like the system is properly part of the same domain.

The Kerberos Auth is coming from the UCS base system, I suggest to talk to their support if you need caching there.

PS: since you are also discussing this directly with the Kopano support as KS-47322 I will no longer reply here to not avoid duplicate answers.

#5

I checked several other Installations i have. All with a UCS system with kopano as part of a UCS or Microsoft domain.

On all systems, AD/DC data is synced to local LDAP-Database, but Kopano-UCS-System generates kerberos-traffic to AD/DC servers on each click in webapp.

I have the feeling, that maybe the passwords or something is not synced to the local LDAP or kopano does not like anything else in the local ldap that triggers kerberos-traffic.

From kopano manual¹:

grafik

@fbartels - can you please elaborate on this?

That is exactly our problem as UCS is using samba4.

¹ https://documentation.kopano.io/kopanocore_administrator_manual/user_management.html#user-management-with-ldap-or-active-directory

#6

I think I’m getting closer to the real problem:

UCS does not sync passwords by default in AD-membermode.

" In AD member mode, in the default setting, the UCS AD Connector exports object data from the AD with the authorizations of the master domain controller’s machine account. These authorizations are not sufficient for exporting encrypted password data.".

https://docs.software-univention.de/manual-4.4.html#ad-connector:ad-connector-einrichtung

HOORAY. That’s it. no kerberos traffic anymore!

I had to do the following:

create AD-user with domain-admin membership (kopano-ad-sync)

ucr set connector/ad/ldap/binddn=kopano-ad-sync
ucr set connector/ad/ldap/bindpw=/etc/univention/connector/password
touch /etc/univention/connector/password
chmod 600 /etc/univention/connector/password
echo -n “password” > /etc/univention/connector/password
ucr set connector/ad/mapping/user/password/kinit=false

and change a single field in each users account (removed “-” from phone-number) to trigger another sync.

after that, UCS synced the user password to local ldap and all is good.

Mastodon