[ovirt-users] Failed to activate Storage Domain vms (Data Center Default)

[Gonzalo Faramiñan]

Hello everyone,

I'm brand new to oVirt, but anyway I'm trying to solve a problem
because of staff changes. Any help is really appreciated.

Context:
 - oVirt Engine Version: 3.1.0-3.19.el6 running on Centos 6.7
 - two nodes running Centos 6.7 connected with Gluster 3.3.1.15.el6,
replicate configured
 - six VMs with high-availability on six separated virtual disk
created into de Default volume.

The Data Center was working fine for a few years and after a
maintenance (step-by-step well done) reboot,  the Default volume lost
GBs of data. Just 23MB of replicated data remained :_(
Now the Master Storage Domain seems to be broken and we cannot get it
up and running in any way. Whatever we try, we end up facing this
message: "No valid Data Storage Domains are available in Data Center
Default (please check your storage infrastructure)."

Over this list we saw same log messages, but couldn't solve the problem.

I include what I think are the main error messages

Thanks in advance!

Webgui:

2016-Aug-12, 14:37:11

Host nodo2 cannot access one of the Storage Domains attached to it, or
the Data Center object. Setting Host state to Non-Operational.

2016-Aug-12, 14:37:11

Failed to connect Host nodo2 to Storage Pool Default

2016-Aug-12, 14:40:02

Failed to activate Storage Domain vms (Data Center Default) by admin at internal

Cluster's admin machine:
$ tail /var/log/ovirt-engine/engine.log

2016-08-12 14:12:12,285 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
(QuartzScheduler_Worker-62) Vds: nodo1

2016-08-12 14:12:12,285 ERROR
[org.ovirt.engine.core.vdsbroker.VDSCommandBase]
(QuartzScheduler_Worker-62) Command ConnectStoragePoolVDS execution
failed. Exception: IRSNoMasterDomainException: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: 'spUUID=6c820dd3-24dc-472f-9b9b-b366da23b22f,
msdUUID=fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf'

2016-08-12 14:12:12,286 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(QuartzScheduler_Worker-62) FINISH, ConnectStoragePoolVDSCommand, log
id: 125fb199

2016-08-12 14:12:12,287 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS

2016-08-12 14:12:12,287 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) Exception: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: 'spUUID=6c820dd3-24dc-472f-9b9b-b366da23b22f,
msdUUID=fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf'

2016-08-12 14:12:12,302 WARN
[org.ovirt.engine.core.dal.job.ExecutionMessageDirector]
(QuartzScheduler_Worker-62) [3165a3b2] The message key
ReconstructMasterDomain is missing from bundles/ExecutionMessages

2016-08-12 14:12:12,350 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(QuartzScheduler_Worker-62) [3165a3b2] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected :
ID: fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf Type: Storage

2016-08-12 14:12:12,397 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand]
(QuartzScheduler_Worker-62) [3165a3b2] START,
MarkPoolInReconstructModeVDSCommand(storagePoolId =
6c820dd3-24dc-472f-9b9b-b366da23b22f, ignoreFailoverLimit = false,
compatabilityVersion = null, reconstructMarkAction = ClearCache), log
id: 40a6ee33

2016-08-12 14:12:12,397 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) [3165a3b2] clearing cache for problematic
entities in pool 6c820dd3-24dc-472f-9b9b-b366da23b22f

2016-08-12 14:12:12,398 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand]
(QuartzScheduler_Worker-62) [3165a3b2] FINISH,
MarkPoolInReconstructModeVDSCommand, log id: 40a6ee33