vdsm command failed connection reset by peer

hi everyone today i saw an error in events tab of ovirt3.6. it shows me this error: VDSM HV07 command failed connection reset by peer my vdsm and libvirt services status is: [root@hv07 ~]# systemctl status vdsmd vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled) Active: active (running) since Thu 2015-12-10 17:28:10 IRST; 2 days ago Process: 17321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 17324 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 17417 (vdsm) CGroup: /system.slice/vdsmd.service ├─ 2407 /usr/libexec/ioprocess --read-pipe-fd 151 --write-pipe-fd 147 --max-threads 10 --max-queued-requests 10 ├─ 4072 /usr/libexec/ioprocess --read-pipe-fd 159 --write-pipe-fd 156 --max-threads 10 --max-queued-requests 10 ├─ 4190 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd 180 --max-threads 10 --max-queued-requests 10 ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 153 --write-pipe-fd 150 --max-threads 10 --max-queued-requests 10 ├─ 4234 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd 46 --max-threads 10 --max-queued-requests 10 ├─14269 /usr/libexec/ioprocess --read-pipe-fd 35 --write-pipe-fd 34 --max-threads 10 --max-queued-requests 10 ├─14280 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 45 --max-threads 10 --max-queued-requests 10 ├─14289 /usr/libexec/ioprocess --read-pipe-fd 56 --write-pipe-fd 55 --max-threads 10 --max-queued-requests 10 ├─14296 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 64 --max-threads 10 --max-queued-requests 10 ├─14303 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd 72 --max-threads 10 --max-queued-requests 10 ├─14310 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd 80 --max-threads 10 --max-queued-requests 10 ├─14317 /usr/libexec/ioprocess --read-pipe-fd 90 --write-pipe-fd 89 --max-threads 10 --max-queued-requests 10 ├─14324 /usr/libexec/ioprocess --read-pipe-fd 97 --write-pipe-fd 96 --max-threads 10 --max-queued-requests 10 ├─14331 /usr/libexec/ioprocess --read-pipe-fd 105 --write-pipe-fd 104 --max-threads 10 --max-queued-requests 10 ├─14338 /usr/libexec/ioprocess --read-pipe-fd 117 --write-pipe-fd 113 --max-threads 10 --max-queued-requests 10 ├─17417 /usr/bin/python /usr/share/vdsm/vdsm ├─17692 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd 130 --max-threads 10 --max-queued-requests 10 ├─21151 /usr/libexec/ioprocess --read-pipe-fd 126 --write-pipe-fd 124 --max-threads 10 --max-queued-requests 10 ├─21215 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd 139 --max-threads 10 --max-queued-requests 10 ├─21446 /usr/libexec/ioprocess --read-pipe-fd 170 --write-pipe-fd 169 --max-threads 10 --max-queued-requests 10 └─21511 /usr/libexec/ioprocess --read-pipe-fd 178 --write-pipe-fd 177 --max-threads 10 --max-queued-requests 10 Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (589) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (589) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (590) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (590) Request queue full Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (591) Request queue full Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (591) Request queue full Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (592) Request queue full Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (592) Request queue full Dec 11 16:54:07 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (593) Request queue full Dec 11 16:54:10 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (594) Request queue full [root@hv07 ~]# systemctl status libvirtd libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─unlimited-core.conf Active: active (running) since Tue 2015-12-01 09:11:30 IRST; 1 weeks 4 days ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 4523 (libvirtd) CGroup: /system.slice/libvirtd.service └─4523 /usr/sbin/libvirtd --listen Dec 01 09:11:29 hv07.election.local libvirtd[4523]: libvirt version: 1.2.8, package: 16.el7_1.5 (CentOS BuildSystem <http....org) Dec 01 09:11:29 hv07.election.local libvirtd[4523]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not...sible Dec 01 09:11:30 hv07.election.local systemd[1]: Started Virtualization daemon. Dec 01 09:39:59 hv07.election.local libvirtd[4523]: End of file while reading data: Input/output error Dec 10 17:28:09 hv07.election.local libvirtd[4523]: End of file while reading data: Input/output error Dec 10 17:28:09 hv07.election.local libvirtd[4523]: End of file while reading data: Input/output error Hint: Some lines were ellipsized, use -l to show in full. what should i do? thanks in advance

On Sat, Dec 12, 2015 at 7:17 PM, alireza sadeh seighalan <seighalani@gmail.com> wrote:
hi everyone
today i saw an error in events tab of ovirt3.6. it shows me this error:
VDSM HV07 command failed connection reset by peer
my vdsm and libvirt services status is:
[root@hv07 ~]# systemctl status vdsmd vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled) Active: active (running) since Thu 2015-12-10 17:28:10 IRST; 2 days ago Process: 17321 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 17324 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 17417 (vdsm) CGroup: /system.slice/vdsmd.service ├─ 2407 /usr/libexec/ioprocess --read-pipe-fd 151 --write-pipe-fd 147 --max-threads 10 --max-queued-requests 10 ├─ 4072 /usr/libexec/ioprocess --read-pipe-fd 159 --write-pipe-fd 156 --max-threads 10 --max-queued-requests 10 ├─ 4190 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd 180 --max-threads 10 --max-queued-requests 10 ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 153 --write-pipe-fd 150 --max-threads 10 --max-queued-requests 10 ├─ 4234 /usr/libexec/ioprocess --read-pipe-fd 49 --write-pipe-fd 46 --max-threads 10 --max-queued-requests 10 ├─14269 /usr/libexec/ioprocess --read-pipe-fd 35 --write-pipe-fd 34 --max-threads 10 --max-queued-requests 10 ├─14280 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 45 --max-threads 10 --max-queued-requests 10 ├─14289 /usr/libexec/ioprocess --read-pipe-fd 56 --write-pipe-fd 55 --max-threads 10 --max-queued-requests 10 ├─14296 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 64 --max-threads 10 --max-queued-requests 10 ├─14303 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd 72 --max-threads 10 --max-queued-requests 10 ├─14310 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd 80 --max-threads 10 --max-queued-requests 10 ├─14317 /usr/libexec/ioprocess --read-pipe-fd 90 --write-pipe-fd 89 --max-threads 10 --max-queued-requests 10 ├─14324 /usr/libexec/ioprocess --read-pipe-fd 97 --write-pipe-fd 96 --max-threads 10 --max-queued-requests 10 ├─14331 /usr/libexec/ioprocess --read-pipe-fd 105 --write-pipe-fd 104 --max-threads 10 --max-queued-requests 10 ├─14338 /usr/libexec/ioprocess --read-pipe-fd 117 --write-pipe-fd 113 --max-threads 10 --max-queued-requests 10 ├─17417 /usr/bin/python /usr/share/vdsm/vdsm ├─17692 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd 130 --max-threads 10 --max-queued-requests 10 ├─21151 /usr/libexec/ioprocess --read-pipe-fd 126 --write-pipe-fd 124 --max-threads 10 --max-queued-requests 10 ├─21215 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd 139 --max-threads 10 --max-queued-requests 10 ├─21446 /usr/libexec/ioprocess --read-pipe-fd 170 --write-pipe-fd 169 --max-threads 10 --max-queued-requests 10 └─21511 /usr/libexec/ioprocess --read-pipe-fd 178 --write-pipe-fd 177 --max-threads 10 --max-queued-requests 10
Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (589) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (589) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (590) Request queue full Dec 11 16:53:50 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (590) Request queue full Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (591) Request queue full Dec 11 16:53:57 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (591) Request queue full Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (592) Request queue full Dec 11 16:54:00 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (592) Request queue full Dec 11 16:54:07 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (593) Request queue full Dec 11 16:54:10 hv07.election.local vdsm[17417]: vdsm IOProcess WARNING (594) Request queue full
Looks like some of your storage is very slow or not accessible, so IOProcess request queue becomes full. To understand this issue, we need vdsm and engine logs. The info available provided by systemctl not enough. Please open a bug and attach both vdsm.log and engine.log showing the time when you had this issue. Adding Yeela. Nir
participants (2)
-
alireza sadeh seighalan
-
Nir Soffer