Disk latency is very high, time taken to copy 1M file is > 10s

hi, I am using HDD : Toshiba 7200 RPM, Data transfer Rate 150MB/s, Interface 6Gb/s. But Hyper-converged configuration stopped with error msg: Disk latency is very high, time taken to copy 1M file is > 10s Please help me to stop this error. ________________________________________________________________________________________ TASK [gluster.features/roles/gluster_hci : Check if time taken to copy 1M file (512B chunks) is < 10s] *** failed: [ov-node-1.hci.com -> ov-node-1.hci.com] (item=11.9508) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.966178", "end": "2019-07-03 15:28:00.162101", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/engine", "rc": 0, "start": "2019-07-03 15:27:48.195923", "stderr": "", "stderr_lines": [], "stdout": "11.9508", "stdout_lines": ["11.9508"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-1.hci.com -> ov-node-1.hci.com] (item=11.5604) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.574648", "end": "2019-07-03 15:28:14.926805", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/data", "rc": 0, "start": "2019-07-03 15:28:03.352157", "stderr": "", "stderr_lines": [], "stdout": "11.5604", "stdout_lines": ["11.5604"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-1.hci.com -> ov-node-1.hci.com] (item=10.9126) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:10.927547", "end": "2019-07-03 15:28:29.053149", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/vmstore", "rc": 0, "start": "2019-07-03 15:28:18.125602", "stderr": "", "stderr_lines": [], "stdout": "10.9126", "stdout_lines": ["10.9126"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-3.hci.com -> ov-node-1.hci.com] (item=12.7177) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:12.805771", "end": "2019-07-03 15:28:01.029631", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/engine", "rc": 0, "start": "2019-07-03 15:27:48.223860", "stderr": "", "stderr_lines": [], "stdout": "12.7177", "stdout_lines": ["12.7177"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-3.hci.com -> ov-node-1.hci.com] (item=11.8554) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.883952", "end": "2019-07-03 15:28:15.810562", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/data", "rc": 0, "start": "2019-07-03 15:28:03.926610", "stderr": "", "stderr_lines": [], "stdout": "11.8554", "stdout_lines": ["11.8554"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-3.hci.com -> ov-node-1.hci.com] (item=11.6409) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.679377", "end": "2019-07-03 15:28:30.376413", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/vmstore", "rc": 0, "start": "2019-07-03 15:28:18.697036", "stderr": "", "stderr_lines": [], "stdout": "11.6409", "stdout_lines": ["11.6409"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-2.hci.com -> ov-node-1.hci.com] (item=12.607) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:12.699951", "end": "2019-07-03 15:28:00.918459", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/engine", "rc": 0, "start": "2019-07-03 15:27:48.218508", "stderr": "", "stderr_lines": [], "stdout": "12.607", "stdout_lines": ["12.607"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-2.hci.com -> ov-node-1.hci.com] (item=11.9233) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.961081", "end": "2019-07-03 15:28:15.837654", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/data/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/data", "rc": 0, "start": "2019-07-03 15:28:03.876573", "stderr": "", "stderr_lines": [], "stdout": "11.9233", "stdout_lines": ["11.9233"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} failed: [ov-node-2.hci.com -> ov-node-1.hci.com] (item=11.5126) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.556528", "end": "2019-07-03 15:28:30.520190", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/vmstore/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/vmstore", "rc": 0, "start": "2019-07-03 15:28:18.963662", "stderr": "", "stderr_lines": [], "stdout": "11.5126", "stdout_lines": ["11.5126"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"} ________________________________________________________________________________________

Hi, On Wed, Jul 3, 2019 at 3:59 PM PS Kazi <faruk.apsara@gmail.com> wrote:
hi, I am using HDD : Toshiba 7200 RPM, Data transfer Rate 150MB/s, Interface 6Gb/s. But Hyper-converged configuration stopped with error msg: Disk latency is very high, time taken to copy 1M file is > 10s Please help me to stop this error.
________________________________________________________________________________________ TASK [gluster.features/roles/gluster_hci : Check if time taken to copy 1M file (512B chunks) is < 10s] *** failed: [ov-node-1.hci.com -> ov-node-1.hci.com] (item=11.9508) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.966178", "end": "2019-07-03 15:28:00.162101", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/engine", "rc": 0, "start": "2019-07-03 15:27:48.195923", "stderr": "", "stderr_lines": [], "stdout": "11.9508", "stdout_lines": ["11.9508"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"}
Firstly solution. Please skip this test with --skip-tags latencycheck For example: ansible-playbook -i <inventory-file> <playbook> --skip-tags latencycheck This check was added to ensure that disklatency would be tested. I have removed this check in this PR: https://github.com/gluster/gluster-ansible-features/pull/33/commits/3e52de0c... Will make it available once it is reviewed and merged. -sac

Hi, On Wed, Jul 3, 2019 at 6:22 PM Sachidananda URS <surs@redhat.com> wrote:
Hi,
On Wed, Jul 3, 2019 at 3:59 PM PS Kazi <faruk.apsara@gmail.com> wrote:
hi, I am using HDD : Toshiba 7200 RPM, Data transfer Rate 150MB/s, Interface 6Gb/s. But Hyper-converged configuration stopped with error msg: Disk latency is very high, time taken to copy 1M file is > 10s Please help me to stop this error.
________________________________________________________________________________________ TASK [gluster.features/roles/gluster_hci : Check if time taken to copy 1M file (512B chunks) is < 10s] *** failed: [ov-node-1.hci.com -> ov-node-1.hci.com] (item=11.9508) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "delta": "0:00:11.966178", "end": "2019-07-03 15:28:00.162101", "failed": false, "invocation": {"module_args": {"_raw_params": "dd if=/dev/zero of=/mnt/tmp/engine/small.file bs=512 count=2050 oflag=direct 2>&1 |\n awk '/copied/{print $(NF-3)}'\n", "_uses_shell": true, "argv": null, "chdir": null, "creates": null, "executable": null, "removes": null, "stdin": null, "stdin_add_newline": true, "strip_empty_ends": true, "warn": true}}, "item": "/mnt/tmp/engine", "rc": 0, "start": "2019-07-03 15:27:48.195923", "stderr": "", "stderr_lines": [], "stdout": "11.9508", "stdout_lines": ["11.9508"]}, "msg": "Disk latency is very high, time taken to copy 1M file is > 10s"}
Firstly solution. Please skip this test with --skip-tags latencycheck For example: ansible-playbook -i <inventory-file> <playbook> --skip-tags latencycheck
This check was added to ensure that disklatency would be tested.
I have removed this check in this PR: https://github.com/gluster/gluster-ansible-features/pull/33/commits/3e52de0c... Will make it available once it is reviewed and merged.
I have done a new build. Please update the gluster-ansible-features rpm from: https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/build/956485/ and please try? -sac
participants (2)
-
PS Kazi
-
Sachidananda URS