Hello,

I'm having a problem with oVirt - we're running 4.3 on CentOS 7:

[root@ovirt ~]# rpm -q ovirt-engine; uname -r
ovirt-engine-4.3.4.3-1.el7.noarch
3.10.0-957.21.2.el7.x86_64

Currently the Data Center alternates between Non responsive and contending status, and SPM selection fails.
The error in the events tab is:

VDSM compute5.domain command HSMGetTaskStatusVDS failed: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',)

Full error:

2019-08-21 13:53:53,507-0700 ERROR (tasks/8) [storage.TaskManager.Task] (Task='3076bb8c-7462-4573-a832-337da478ae0e') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm
    self._upgradePool(expectedDomVersion, __securityOverride=True)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in _upgradePool
    str(targetDomVersion))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, in _convertDomain
    targetFormat)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 447, in convert
    converter(repoPath, hostId, imageRepo, isMsd)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 405, in v5DomainConverter
    domain.convert_volumes_metadata(target_version)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 813, in convert_volumes_metadata
    for vol in self.iter_volumes():
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in iter_volumes
    yield self.produceVolume(img_id, vol_id)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in produceVolume
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py", line 45, in __init__
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 817, in __init__
    self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__
    self.validate()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate
    self.validateVolumePath()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath
    raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',)

Most of our VMs are still running, but we can't start or restart any VMs. All of the domains show as down since SPM selection fails. Any thoughts?

Thanks,
 -Matthew