Slow NFS writes with sec=krb5

[Florian Oppermann]

Hi Ubuntu users,

I observe a really strange phenomenom with my NFS mount:

I used to automount it with sec=sys and had pretty good r/w performance
(~100 MB/s over GBit Ethernet). If I unmount the file system, configure
the automounter to use sec=krb5 and remount it the performance remains
good. If I then reboot I only get ~4–5 MB/s for sequential writes (reads
are still fast). If I again change the automounter configuration and
remount with sec=sys the speed remains low until I reboot again.

It seems like the first mount decides the write speed. (Note: This seems
to be independent from the automounter.)

I managed to find a difference in the network traffic via tshark:

In the slow situation the write traffic consists of many blocks similar
to the following (130.75.25.1 is the NFS server):

> 10:58:14.611344 130.75.25.96 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 10:58:14.611375 130.75.25.96 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 10:58:14.611392 130.75.25.96 -> 130.75.25.1  NFS 1470 V4 Call WRITE StateID:0x763e Offset:0 Len:4096
> 10:58:14.611754  130.75.25.1 -> 130.75.25.96 TCP 66 nfs > 858 [ACK] Seq=3501 Ack=8861 Win=32873 Len=0 TSval=4830212 TSecr=15299083
> 10:58:14.612367  130.75.25.1 -> 130.75.25.96 NFS 230 V4 Reply (Call In 90) WRITE

In the fast situation the client sends many data packets without
subsequent WRITE calls and replies:

> 11:14:51.822138 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822145 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822149 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822154 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822158 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822163 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822168 130.75.25.98 -> 130.75.25.1  TCP 1514 [TCP segment of a reassembled PDU]
> 11:14:51.822172  130.75.25.1 -> 130.75.25.98 TCP 66 nfs > 766 [ACK] Seq=1509 Ack=15485 Win=32865 Len=0 TSval=5079515 TSecr=18891543

Strangely on two of my ~60 machines the kerberized NFS mount is fast. I
did a diff of the /etc directory between a fast and a similar but slow
machine and couldn’t find any relevant difference.

It has to be a client configuration (my laptop with Debian testing was
able to mount the file system fast) but I’m running out of ideas where
to look.

We use Ubuntu 12.04 (precise) or 14.04 (trusty) on all machines. The NFS
server is on trusty, the two fast machines are on precise.

Do you have any recommendations what configurations I can check?

Best regards,
Florian