[ovirt-users] vdsm vds.dispatcher ERROR SSL

Fri Oct 21 09:43:48 UTC 2016

Hi all,

I think we have the same or similar problem than Gianluca describe in this thread [1]. I've applied patches [2] and [3] to improve logging on vdsm side. It seems that the "vdsm vds.dispatcher ERROR SSL error" occurs only on the SPM node as we noticed that these logs disappears when another host is selected as SPM, and we can find these errors on the new SPM host. We have multiple DCs, one with the hosted engine, and others with only simple hosts. The problem don't occurs on the hosted engine cluster in our deployment. This problem seems related to instability on this cluster, the SPM host fall in unresponsive state mostly every 4 minutes and comeback during the grace period most of the time (but was fenced 1 or 2 times last 2 weeks).

Information about our setup:

* All hosts, including engine, are CentOS 7 x86_64
* All hosts share the same ntp server and are in sync
* Firewalls are disabled on all hosts
* All storage domains are on ISCSI backends
* All DC are located in the same room. We created multiple DC because we have multiple storage backends on separated SANs and we didn't notice any network issues between the engine and the SPM hosts of the problematic cluster.

Engine packages:
ovirt-image-uploader-4.0.1-1.el7.centos.noarch
ovirt-engine-tools-backup-4.0.4.4-1.el7.centos.noarch
ovirt-engine-extension-aaa-misc-1.0.1-1.el7.noarch
ovirt-engine-extension-aaa-jdbc-1.1.0-1.el7.noarch
ovirt-engine-dashboard-1.0.3-1.el7.centos.noarch
ovirt-engine-dwh-4.0.2-1.el7.centos.noarch
ovirt-host-deploy-java-1.5.2-1.el7.centos.noarch
ovirt-engine-userportal-4.0.4.4-1.el7.centos.noarch
ovirt-host-deploy-1.5.2-1.el7.centos.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.0.4.4-1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7.centos.noarch
ovirt-engine-tools-4.0.4.4-1.el7.centos.noarch
ovirt-engine-4.0.4.4-1.el7.centos.noarch
ovirt-guest-agent-common-1.0.12-3.el7.noarch
ovirt-vmconsole-proxy-1.0.4-1.el7.centos.noarch
ovirt-engine-cli-3.6.8.1-1.el7.centos.noarch
ovirt-engine-extension-aaa-ldap-1.2.1-1.el7.noarch
ovirt-engine-wildfly-10.0.0-1.el7.x86_64
ovirt-engine-lib-4.0.4.4-1.el7.centos.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.4.4-1.el7.centos.noarch
ovirt-engine-vmconsole-proxy-helper-4.0.4.4-1.el7.centos.noarch
ovirt-iso-uploader-4.0.1-1.el7.centos.noarch
ovirt-engine-webadmin-portal-4.0.4.4-1.el7.centos.noarch
ovirt-engine-restapi-4.0.4.4-1.el7.centos.noarch
ovirt-engine-jboss-as-7.1.1-1.el7.centos.x86_64
ovirt-engine-dwh-setup-4.0.2-1.el7.centos.noarch
ovirt-guest-tools-iso-4.0-1.fc23.noarch
ovirt-engine-wildfly-overlay-10.0.0-1.el7.noarch
python-ovirt-engine-sdk4-4.0.1-1.el7.centos.x86_64
ovirt-engine-setup-plugin-ovirt-engine-common-4.0.4.4-1.el7.centos.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.0.4.4-1.el7.centos.noarch
ovirt-engine-websocket-proxy-4.0.4.4-1.el7.centos.noarch
ovirt-engine-setup-4.0.4.4-1.el7.centos.noarch
ovirt-engine-extensions-api-impl-4.0.4.4-1.el7.centos.noarch
ovirt-engine-backend-4.0.4.4-1.el7.centos.noarch
ovirt-release35-005-1.noarch
ovirt-setup-lib-1.0.2-1.el7.centos.noarch
ovirt-imageio-common-0.4.0-1.el7.noarch
ovirt-imageio-proxy-setup-0.4.0-0.201608310602.gita9b573b.el7.centos.noarch
ovirt-release40-4.0.4-1.noarch
ovirt-engine-dbscripts-4.0.4.4-1.el7.centos.noarch
ovirt-vmconsole-1.0.4-1.el7.centos.noarch
ovirt-release36-3.6.7-1.noarch
ovirt-engine-extension-aaa-ldap-setup-1.2.1-1.el7.noarch
ovirt-engine-setup-base-4.0.4.4-1.el7.centos.noarch
ovirt-imageio-proxy-0.4.0-0.201608310602.gita9b573b.el7.centos.noarch

Virtualization host packages:
ovirt-vmconsole-1.0.4-1.el7.centos.noarch
vdsm-4.18.13-1.el7.centos.x86_64
vdsm-infra-4.18.13-1.el7.centos.noarch
vdsm-xmlrpc-4.18.13-1.el7.centos.noarch
ovirt-release40-4.0.4-1.noarch
vdsm-jsonrpc-4.18.13-1.el7.centos.noarch
vdsm-api-4.18.13-1.el7.centos.noarch
vdsm-python-4.18.13-1.el7.centos.noarch
ovirt-imageio-common-0.4.0-1.el7.noarch
vdsm-hook-vmfex-dev-4.18.13-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.4-1.el7.centos.noarch
vdsm-yajsonrpc-4.18.13-1.el7.centos.noarch
vdsm-cli-4.18.13-1.el7.centos.noarch
ovirt-imageio-daemon-0.4.0-1.el7.noarch

In the logs: 
oVirt engine: engine.foo.example.com, 192.168.1.10
virtualization host: virt03.foo.example.com, 192.168.1.11

Attached files:
/var/log/ovirt-engine/engine.log
/var/log/vdsm/vdsm.log
/var/log/messages (for virtualization host)

Have a nice day.

Regards

[1] http://lists.ovirt.org/pipermail/users/2016-October/043355.html
[2] https://gerrit.ovirt.org/#/c/61303/
[3] https://gerrit.ovirt.org/#/c/61367/
-- 
Baptiste AGASSE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.log
Type: text/x-log
Size: 885593 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161021/55b24497/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine.log
Type: text/x-log
Size: 23658 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161021/55b24497/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: messages
Type: application/octet-stream
Size: 9776 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161021/55b24497/attachment-0001.obj>