I am in a severe pinch here. A while back I upgraded from 4.2.8 to 4.3.3
and only had one step remaining and that was to set the cluster compat
level to 4.3 (from 4.2). When I tried this it gave the usual warning that
each VM would have to be rebooted to complete, but then I got my first
unusual piece when it then told me next that this could not be completed
until each host was in maintenance mode. Quirky I thought, but I stopped
all VMs and put both hosts into maintenance mode. I then set the cluster
to 4.3. Things didn't want to become active again and I eventually noticed
that I was being told the DC needed to be 4.3 as well. Don't remember that
from before, but oh well that was easy.
However, the DC and SD remains down. The hosts are non-op. I've powered
everything off and started fresh but still wind up in the same state.
Hosts will look like their active for a bit (green triangle) but then go
non-op after about a minute. It appears that my iSCSI sessions are
active/logged in. The one glaring thing I see in the logs is this in
vdsm.log:
2019-04-05 12:03:30,225-0400 ERROR (monitor/07bb1bf) [storage.Monitor]
Setting up monitor for 07bb1bf8-3b3e-4dc0-bc43-375b09e06683 failed
(monitor:329)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
326, in _setupLoop
self._setupMonitor()
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
348, in _setupMonitor
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 158, in
wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line
366, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'07bb1bf8-3b3e-4dc0-bc43-375b09e06683',)
How do I proceed to get back operational?