vdsm IOProcessClient WARNING Timeout waiting for communication thread for client
We have an ovirt system with 3 clusters, all running centos7. ovirt engine is running on separate host, ovirt-engine-3.6.4.1-1.el7.centos.noarch 2 of the clusters are running newer version of ovirt, 3 nodes each, ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64, vdsm-4.18.11-1.el7.centos.x86_64. 1 cluster is still running the older version, ovirt-engine-3.6.4.1-1.el7.centos.noarch. Yes we are in the process of upgrading the whole system to ovirt4.0, but takes time.... One of the 2 clusters running ovirt4 is complaining of timeouts, vdsm talking to gluster. No warnings on the 2 other clusters. Thread-720062::DEBUG::2017-01-13 07:29:46,814::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/glusterSD/ovirt1-gl.dmz.p rod.j2noc.com:_gv1 Thread-720062::INFO::2017-01-13 07:29:46,814::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,814::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-5874 *Thread-720062::WARNING::2017-01-13 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) Timeout waiting for communication thread for client ioproc**ess-5874* [2017-01-12 07:27:58.685680] I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req The message "I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req" repeated 31 times between [2017-01-12 07:27:58.685680] and [2017-01-12 07:29:46.971939] attached logs: engine.log supervdsm.log vdsm.log etc-glusterfs-glusterd.vol.log cli.log
resending without logs, except vdsm.log since list limit is too small. On 1/13/17 8:50 AM, Bill James wrote:
We have an ovirt system with 3 clusters, all running centos7. ovirt engine is running on separate host, ovirt-engine-3.6.4.1-1.el7.centos.noarch 2 of the clusters are running newer version of ovirt, 3 nodes each, ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64, vdsm-4.18.11-1.el7.centos.x86_64. 1 cluster is still running the older version, ovirt-engine-3.6.4.1-1.el7.centos.noarch.
Yes we are in the process of upgrading the whole system to ovirt4.0, but takes time....
One of the 2 clusters running ovirt4 is complaining of timeouts, vdsm talking to gluster. No warnings on the 2 other clusters.
Thread-720062::DEBUG::2017-01-13 07:29:46,814::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/glusterSD/ovirt1-gl.dmz.p rod.j2noc.com:_gv1 Thread-720062::INFO::2017-01-13 07:29:46,814::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,814::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-5874 *Thread-720062::WARNING::2017-01-13 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) Timeout waiting for communication thread for client ioproc**ess-5874*
[2017-01-12 07:27:58.685680] I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req The message "I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req" repeated 31 times between [2017-01-12 07:27:58.685680] and [2017-01-12 07:29:46.971939]
attached logs: engine.log supervdsm.log vdsm.log etc-glusterfs-glusterd.vol.log cli.log
On Fri, Jan 13, 2017 at 7:41 PM, Bill James <bill.james@j2.com> wrote:
resending without logs, except vdsm.log since list limit is too small.
On 1/13/17 8:50 AM, Bill James wrote:
We have an ovirt system with 3 clusters, all running centos7. ovirt engine is running on separate host, ovirt-engine-3.6.4.1-1.el7.centos.noarch 2 of the clusters are running newer version of ovirt, 3 nodes each, ovirt-engine-4.0.3-1.el7.centos.noarch, glusterfs-3.7.16-1.el7.x86_64, vdsm-4.18.11-1.el7.centos.x86_64. 1 cluster is still running the older version, ovirt-engine-3.6.4.1-1.el7.centos.noarch.
Which ioprocess version?
Yes we are in the process of upgrading the whole system to ovirt4.0, but takes time....
One of the 2 clusters running ovirt4 is complaining of timeouts, vdsm talking to gluster. No warnings on the 2 other clusters.
Thread-720062::DEBUG::2017-01-13 07:29:46,814::outOfProcess::87::Storage.oop::(getProcessPool) Creating ioprocess /rhev/data-center/mnt/glusterSD/ovirt1-gl.dmz.p rod.j2noc.com:_gv1 Thread-720062::INFO::2017-01-13 07:29:46,814::__init__::325::IOProcessClient::(__init__) Starting client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,814::__init__::334::IOProcessClient::(_run) Starting ioprocess for client ioprocess-5874 Thread-720062::DEBUG::2017-01-13 07:29:46,832::__init__::386::IOProcessClient::(_startCommunication) Starting communication thread for client ioprocess-5874 Thread-720062::WARNING::2017-01-13 07:29:46,847::__init__::401::IOProcessClient::(_startCommunication) Timeout waiting for communication thread for client ioprocess-5874
This warning is harmless, it means that ioprocess thread did not start in 1 second. This probably means that the host is overloaded, typically new threads start instantly. Anyway I think we are using too short timeout. Can you open an ioprocess bug for this?
[2017-01-12 07:27:58.685680] I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req The message "I [MSGID: 106488] [glusterd-handler.c:1533:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req" repeated 31 times between [2017-01-12 07:27:58.685680] and [2017-01-12 07:29:46.971939]
attached logs: engine.log supervdsm.log vdsm.log etc-glusterfs-glusterd.vol.log cli.log
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Bill James -
Nir Soffer