[ovirt-users] vdsm command failed connection reset by peer

[Nir Soffer]

On Sat, Dec 12, 2015 at 7:17 PM, alireza sadeh seighalan
<seighalani at gmail.com> wrote:
> hi everyone
>
>
> today i saw an error in events tab of ovirt3.6. it shows me this error:
>
> VDSM HV07   command failed connection reset by peer
>
> my vdsm and libvirt services status is:
>
>
> [root at hv07 ~]# systemctl status vdsmd
> vdsmd.service - Virtual Desktop Server Manager
>    Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
>    Active: active (running) since Thu 2015-12-10 17:28:10 IRST; 2 days ago
>   Process: 17321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
> --post-stop (code=exited, status=0/SUCCESS)
>   Process: 17324 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
> --pre-start (code=exited, status=0/SUCCESS)
>  Main PID: 17417 (vdsm)
>    CGroup: /system.slice/vdsmd.service
>            ├─ 2407 /usr/libexec/ioprocess --read-pipe-fd 151 --write-pipe-fd
> 147 --max-threads 10 --max-queued-requests 10
>            ├─ 4072 /usr/libexec/ioprocess --read-pipe-fd 159 --write-pipe-fd
> 156 --max-threads 10 --max-queued-requests 10
>            ├─ 4190 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd
> 180 --max-threads 10 --max-queued-requests 10
>            ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 153 --write-pipe-fd
> 150 --max-threads 10 --max-queued-requests 10
>            ├─ 4234 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd
> 46 --max-threads 10 --max-queued-requests 10
>            ├─14269 /usr/libexec/ioprocess --read-pipe-fd 35 --write-pipe-fd
> 34 --max-threads 10 --max-queued-requests 10
>            ├─14280 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
> 45 --max-threads 10 --max-queued-requests 10
>            ├─14289 /usr/libexec/ioprocess --read-pipe-fd 56 --write-pipe-fd
> 55 --max-threads 10 --max-queued-requests 10
>            ├─14296 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
> 64 --max-threads 10 --max-queued-requests 10
>            ├─14303 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
> 72 --max-threads 10 --max-queued-requests 10
>            ├─14310 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd
> 80 --max-threads 10 --max-queued-requests 10
>            ├─14317 /usr/libexec/ioprocess --read-pipe-fd 90 --write-pipe-fd
> 89 --max-threads 10 --max-queued-requests 10
>            ├─14324 /usr/libexec/ioprocess --read-pipe-fd 97 --write-pipe-fd
> 96 --max-threads 10 --max-queued-requests 10
>            ├─14331 /usr/libexec/ioprocess --read-pipe-fd 105 --write-pipe-fd
> 104 --max-threads 10 --max-queued-requests 10
>            ├─14338 /usr/libexec/ioprocess --read-pipe-fd 117 --write-pipe-fd
> 113 --max-threads 10 --max-queued-requests 10
>            ├─17417 /usr/bin/python /usr/share/vdsm/vdsm
>            ├─17692 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd
> 130 --max-threads 10 --max-queued-requests 10
>            ├─21151 /usr/libexec/ioprocess --read-pipe-fd 126 --write-pipe-fd
> 124 --max-threads 10 --max-queued-requests 10
>            ├─21215 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd
> 139 --max-threads 10 --max-queued-requests 10
>            ├─21446 /usr/libexec/ioprocess --read-pipe-fd 170 --write-pipe-fd
> 169 --max-threads 10 --max-queued-requests 10
>            └─21511 /usr/libexec/ioprocess --read-pipe-fd 178 --write-pipe-fd
> 177 --max-threads 10 --max-queued-requests 10
>
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (589) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (589) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (590) Request queue full
> Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (590) Request queue full
> Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (591) Request queue full
> Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (591) Request queue full
> Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (592) Request queue full
> Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (592) Request queue full
> Dec 11 16:54:07 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (593) Request queue full
> Dec 11 16:54:10 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING
> (594) Request queue full

Looks like some of your storage is very slow or not accessible, so
IOProcess request queue becomes full.

To understand this issue, we need vdsm and engine logs. The info
available provided by systemctl not enough.

Please open a bug and attach both vdsm.log and engine.log showing the
time when you had this issue.

Adding Yeela.

Nir