9:55 p.m.
vdsmd 4.4.0 throws an exception in asyncore.py while updating OVF data. First exception
occured on December 14th and happens every hour since then.
An hour before the first exception, we updated engine's and host's rpms packages.
We did not run engine-backup so we cannot restore oVirt database if it got corrupted.
Currently, both engine (RHEL 7.7) and host (RHEL 8.1) have the most recent packages (and
were rebooted). Neither the updates, nor reboots resolve the issue.
In general, oVirt is working. We can run virtual machines with disks in the storage
domain. We can create new virtual machines. But some functionality is not available, for
example, we cannot move disks between domains.
The storage domain in question is on NFS. After the issue appeared, we successfully
created Glusterfs domain but its OVF update also failed. Interesting that it was possible
to move disks to it before OVF failure.
Here is the vdsmd traceback for your convinience
2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python
exception, closing channel
<yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:172.20.1.142', 38002,
0, 0) at 0x7fbda865ed30> (
<class 'TypeError'>:object of type 'NoneType' has no len()
[/usr/lib64/python3.6/asyncore.py|readwrite|108]
[/usr/lib64/python3.6/asyncore.py|handle_read_event|423]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115]
)
(betterAsyncore:179)
I appreciate your help,
Alexander Murashkin
------ Current Versions (that have the exception) ------
Engine
ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch
Host
vdsm-4.40.0-1363.gitf6a1ba0a0.el8.x86_64
vdsm-python-4.40.0-1363.gitf6a1ba0a0.el8.noarch
vdsm-yajsonrpc-4.40.0-1363.gitf6a1ba0a0.el8.noarch
python3-libs-3.6.8-15.1.el8.x86_64
------ December 13th Versions (that did not have the exception) ------
Engine
ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch --- not sure, but
probably the same as now
Host
vdsm-4.40.0-1360.git821afbbc2.el8.x86_64
vdsm-python-4.40.0-1360.git821afbbc2.el8.noarch
vdsm-yajsonrpc-4.40.0-1360.git821afbbc2.el8.noarch
python3-libs-3.6.8-15.1.el8.x86_64
------ vdsm.log ------
2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python
exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher connected
('::ffff:172.20.1.142', 38002, 0, 0) at 0x7fbda865ed30> (<class
'TypeError'>:object of type 'NoneType' has no len()
[/usr/lib64/python3.6/asyncore.py|readwrite|108]
[/usr/lib64/python3.6/asyncore.py|handle_read_event|423]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115])
(betterAsyncore:179)
----- engine.log ------
2019-12-17 16:36:58,395-06 ERROR
[org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [6c787bf3]
Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed:
EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
java.net.SocketException: Connection reset (Failed with error VDS_NETWORK_ERROR and code
5022)
------ Web Interface Events ------
Storage Domain Events
Dec 17, 2019, 4:36:58 PM
Failed to update VMs/Templates OVF data for Storage Domain storedom3 in Data Center
Default.
36b41d9c
oVirt
Dec 17, 2019, 4:36:58 PM
Failed to update OVF disks fcc661df-b2e3-4625-be40-52b65033c6d7, OVF data isn't
updated on those OVF stores (Data Center Default, Storage Domain storedom3).
36b41d9c
oVirt
Host Events
Dec 17, 2019, 4:38:20 PM
Status of host poplar was set to Up.
399a2181
oVirt
Dec 17, 2019, 4:36:58 PM
Host poplar is not responding. Host cannot be fenced automatically because power
management for the host is disabled.
oVirt