
vdsmd 4.4.0 throws an exception in asyncore.py while updating OVF data. First exception occured on December 14th and happens every hour since then. An hour before the first exception, we updated engine's and host's rpms packages. We did not run engine-backup so we cannot restore oVirt database if it got corrupted. Currently, both engine (RHEL 7.7) and host (RHEL 8.1) have the most recent packages (and were rebooted). Neither the updates, nor reboots resolve the issue. In general, oVirt is working. We can run virtual machines with disks in the storage domain. We can create new virtual machines. But some functionality is not available, for example, we cannot move disks between domains. The storage domain in question is on NFS. After the issue appeared, we successfully created Glusterfs domain but its OVF update also failed. Interesting that it was possible to move disks to it before OVF failure. Here is the vdsmd traceback for your convinience 2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:172.20.1.142', 38002, 0, 0) at 0x7fbda865ed30> ( <class 'TypeError'>:object of type 'NoneType' has no len() [/usr/lib64/python3.6/asyncore.py|readwrite|108] [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115] ) (betterAsyncore:179) I appreciate your help, Alexander Murashkin ------ Current Versions (that have the exception) ------ Engine ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch Host vdsm-4.40.0-1363.gitf6a1ba0a0.el8.x86_64 vdsm-python-4.40.0-1363.gitf6a1ba0a0.el8.noarch vdsm-yajsonrpc-4.40.0-1363.gitf6a1ba0a0.el8.noarch python3-libs-3.6.8-15.1.el8.x86_64 ------ December 13th Versions (that did not have the exception) ------ Engine ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch --- not sure, but probably the same as now Host vdsm-4.40.0-1360.git821afbbc2.el8.x86_64 vdsm-python-4.40.0-1360.git821afbbc2.el8.noarch vdsm-yajsonrpc-4.40.0-1360.git821afbbc2.el8.noarch python3-libs-3.6.8-15.1.el8.x86_64 ------ vdsm.log ------ 2019-12-17 16:36:58,393-0600 ERROR (Reactor thread) [vds.dispatcher] uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher connected ('::ffff:172.20.1.142', 38002, 0, 0) at 0x7fbda865ed30> (<class 'TypeError'>:object of type 'NoneType' has no len() [/usr/lib64/python3.6/asyncore.py|readwrite|108] [/usr/lib64/python3.6/asyncore.py|handle_read_event|423] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_read|71] [/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py|handle_read|115]) (betterAsyncore:179) ----- engine.log ------ 2019-12-17 16:36:58,395-06 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [6c787bf3] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketException: Connection reset (Failed with error VDS_NETWORK_ERROR and code 5022) ------ Web Interface Events ------ Storage Domain Events Dec 17, 2019, 4:36:58 PM Failed to update VMs/Templates OVF data for Storage Domain storedom3 in Data Center Default. 36b41d9c oVirt Dec 17, 2019, 4:36:58 PM Failed to update OVF disks fcc661df-b2e3-4625-be40-52b65033c6d7, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain storedom3). 36b41d9c oVirt Host Events Dec 17, 2019, 4:38:20 PM Status of host poplar was set to Up. 399a2181 oVirt Dec 17, 2019, 4:36:58 PM Host poplar is not responding. Host cannot be fenced automatically because power management for the host is disabled. oVirt