Once upon a time, Karli Sjöberg <karli(a)inparadise.se> said:
Lastly network, are you sure you activated jumbo frames, all the way
from the storage to the hosts? That makes a huge difference on 10 Gb
ethernet.
BTW: just wanted to mention a good way to check that you are really
getting jumbo frames is to use "ss" to see the MSS on the established
TCP sessions. For example, if you are using iSCSI:
# ss -ti dport = :iscsi-target
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.21.0.201:60180 10.21.0.21:iscsi-target
cubic wscale:5,2 rto:201 rtt:0.737/0.563 ato:40 mss:8948 cwnd:10 ssthresh:16
bytes_acked:1263988453 bytes_received:734634344892 segs_out:55848414 segs_in:90177540 send
971.3Mbps lastsnd:1570 lastrcv:1568 lastack:1568 pacing_rate 1941.6Mbps retrans:0/55
rcv_rtt:2.875 rcv_space:587020
ESTAB 0 0 10.21.0.201:55098 10.21.0.20:iscsi-target
cubic wscale:5,2 rto:201 rtt:0.443/0.119 ato:40 mss:8948 cwnd:19 ssthresh:18
bytes_acked:256605372465 bytes_received:3117834587228 segs_out:286956736 segs_in:442234458
send 3070.2Mbps lastsnd:1445 lastrcv:1444 lastack:1444 pacing_rate 6138.7Mbps retrans:0/9
reordering:5 rcv_rtt:6.75 rcv_space:459048
NFS is a little different (since it doesn't use a single fixed port),
but you can use "ss -ti dst [NFS server IP]" instead.
--
Chris Adams <cma(a)cmadams.net>