NFSv4 only half as fast as SMB

UCS - Univention Corporate Server
#1

Hi@all,

The UCS 5.0.4 provides all shares via Samba. Except /home because on the Linux clients the desktop does not work well if the /home is mounted via SMB. The shares are on the same partition on the UCS. RAID5 hardware RAID with SSD’s

UUID=84ed7072-b664-4c74-b0fc-9f932bacbf9a       /data01 xfs     defaults        0       0
# /home was on /dev/sda3 during installation
UUID=b0470c13-04d8-4dcb-a11d-a79270986628       /home   xfs     defaults        0       0

The SMB shares are all located below /data01. As an example, the share “Accounting”:

[Buchhaltung]
path = /data01/Buchhaltung
msdfs root = no
writeable = yes
browseable = yes
public = no
dos filemode = yes
hide unreadable = no
create mode = 0660
directory mode = 0770
force create mode = 00
force directory mode = 00
locking = 1
strict locking = Auto
oplocks = 1
level2 oplocks = 1
fake oplocks = 0
csc policy = manual
force user = Administrator
force group = Domain Users
nt acl support = 1
inherit acls = 1
vfs objects = acl_xattr acl_xattr
inherit owner = no
inherit permissions = no
map acl inherit = yes

The corresponding mount on the client (output ‘mount’) looks like this:

//srv01.lan.example.de/Buchhaltung on /data01/Buchhaltung type cifs (rw,relatime,vers=3.1.1,cache=loose,username=s.gehr,uid=1381401120,noforceuid,gid=1381400513,noforcegid,addr=192.168.83.5,file_mode=0755,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

This gives me a fairly constant rate (in both directions) of ~120MB/sec. I rate that as good.

The export of /home

#
"/home" -rw,root_squash,async,subtree_check,sec=krb5i * # LDAP:cn=home,cn=shares,dc=lan,dc=example,dc=de

And the output of 'mount
srv01.lan.example.de:/home/s.gehr on /home/s.gehr type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=krb5i,clientaddr=192.168.83.140,fsc,local_lock=none,addr=192.168.83.5)

On /home I have a constant bandwidth of ~60MB/sec (in both directions). So just half of SMB.

Can the NFSv4 still be optimized?

with best
sven

#2

nfsstat

nfsiostat

see if it’s errors or something else.

#3

Server:

root ✗ srv01:~ nfsstat 
Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
134691     0          0          0          0       

Server nfs v4:
null             compound         
3         0%     134688   99%     

Server nfs v4 operations:
op0-unused       op1-unused       op2-future       access           close            
0         0%     0         0%     0         0%     13376     2%     8899      1%     
commit           create           delegpurge       delegreturn      getattr          
935       0%     259       0%     0         0%     2868      0%     80943    16%     
getfh            link             lock             lockt            locku            
9269      1%     53        0%     2633      0%     4         0%     2595      0%     
lookup           lookup_root      nverify          open             openattr         
7707      1%     0         0%     0         0%     9833      2%     0         0%     
open_conf        open_dgrd        putfh            putpubfh         putrootfh        
0         0%     7         0%     134190   27%     0         0%     6         0%     
read             readdir          readlink         remove           rename           
35354     7%     759       0%     1         0%     2485      0%     1511      0%     
renew            restorefh        savefh           secinfo          setattr          
0         0%     53        0%     1564      0%     6         0%     2267      0%     
setcltid         setcltidconf     verify           write            rellockowner     
0         0%     0         0%     0         0%     27114     5%     0         0%     
bc_ctl           bind_conn        exchange_id      create_ses       destroy_ses      
0         0%     0         0%     12        0%     3         0%     2         0%     
free_stateid     getdirdeleg      getdevinfo       getdevlist       layoutcommit     
2001      0%     0         0%     0         0%     0         0%     0         0%     
layoutget        layoutreturn     secinfononam     sequence         set_ssv          
0         0%     0         0%     3         0%     134658   27%     0         0%     
test_stateid     want_deleg       destroy_clid     reclaim_comp     allocate         
0         0%     0         0%     2         0%     3         0%     59        0%     
copy             copy_notify      deallocate       ioadvise         layouterror      
0         0%     0         0%     0         0%     0         0%     0         0%     
layoutstats      offloadcancel    offloadstatus    readplus         seek             
0         0%     0         0%     0         0%     0         0%     0         0%     
write_same       
0         0%

Client:

root ✗ pc001:/home/s.gehr nfsiostat

srv01.lan.example.de:/home/s.gehr mounted on /home/s.gehr:

           ops/s       rpc bklog
           7.451           0.000

read:              ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)          errors
                   2.068          16.226           7.846        0 (0.0%)           0.518           0.644           0.040        0 (0.0%)
write:             ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)          errors
                   1.584         273.956         172.923        0 (0.0%)          16.259        5672.595        5656.306        0 (0.0%)

I can not see any errors

#4

looks like the writes are all to cock…
The queue time looks not right…
the avg ext is like 5 seconds… and the queue is mad , but the RTT is short.

the RTT is from time to send to some response 16ms is a little too long
but the exe is from time to receive to time to complete, its nearly 5.6 seconds… which is insane…
something is bottle necked…

Mastodon