Once upon a time, Karli Sjöberg <karli@inparadise.se> said:
Lastly network, are you sure you activated jumbo frames, all the way from the storage to the hosts? That makes a huge difference on 10 Gb ethernet.
BTW: just wanted to mention a good way to check that you are really getting jumbo frames is to use "ss" to see the MSS on the established TCP sessions. For example, if you are using iSCSI: # ss -ti dport = :iscsi-target State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 10.21.0.201:60180 10.21.0.21:iscsi-target cubic wscale:5,2 rto:201 rtt:0.737/0.563 ato:40 mss:8948 cwnd:10 ssthresh:16 bytes_acked:1263988453 bytes_received:734634344892 segs_out:55848414 segs_in:90177540 send 971.3Mbps lastsnd:1570 lastrcv:1568 lastack:1568 pacing_rate 1941.6Mbps retrans:0/55 rcv_rtt:2.875 rcv_space:587020 ESTAB 0 0 10.21.0.201:55098 10.21.0.20:iscsi-target cubic wscale:5,2 rto:201 rtt:0.443/0.119 ato:40 mss:8948 cwnd:19 ssthresh:18 bytes_acked:256605372465 bytes_received:3117834587228 segs_out:286956736 segs_in:442234458 send 3070.2Mbps lastsnd:1445 lastrcv:1444 lastack:1444 pacing_rate 6138.7Mbps retrans:0/9 reordering:5 rcv_rtt:6.75 rcv_space:459048 NFS is a little different (since it doesn't use a single fixed port), but you can use "ss -ti dst [NFS server IP]" instead. -- Chris Adams <cma@cmadams.net>