8:28 p.m.
Hello everyone,
I'm brand new to oVirt, but anyway I'm trying to solve a problem
because of staff changes. Any help is really appreciated.
Context:
- oVirt Engine Version: 3.1.0-3.19.el6 running on Centos 6.7
- two nodes running Centos 6.7 connected with Gluster 3.3.1.15.el6,
replicate configured
- six VMs with high-availability on six separated virtual disk
created into de Default volume.
The Data Center was working fine for a few years and after a
maintenance (step-by-step well done) reboot, the Default volume lost
GBs of data. Just 23MB of replicated data remained :_(
Now the Master Storage Domain seems to be broken and we cannot get it
up and running in any way. Whatever we try, we end up facing this
message: "No valid Data Storage Domains are available in Data Center
Default (please check your storage infrastructure)."
Over this list we saw same log messages, but couldn't solve the problem.
I include what I think are the main error messages
Thanks in advance!
Webgui:
2016-Aug-12, 14:37:11
Host nodo2 cannot access one of the Storage Domains attached to it, or
the Data Center object. Setting Host state to Non-Operational.
2016-Aug-12, 14:37:11
Failed to connect Host nodo2 to Storage Pool Default
2016-Aug-12, 14:40:02
Failed to activate Storage Domain vms (Data Center Default) by admin@internal
Cluster's admin machine:
$ tail /var/log/ovirt-engine/engine.log
2016-08-12 14:12:12,285 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]
(QuartzScheduler_Worker-62) Vds: nodo1
2016-08-12 14:12:12,285 ERROR
[org.ovirt.engine.core.vdsbroker.VDSCommandBase]
(QuartzScheduler_Worker-62) Command ConnectStoragePoolVDS execution
failed. Exception: IRSNoMasterDomainException: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: 'spUUID=6c820dd3-24dc-472f-9b9b-b366da23b22f,
msdUUID=fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf'
2016-08-12 14:12:12,286 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(QuartzScheduler_Worker-62) FINISH, ConnectStoragePoolVDSCommand, log
id: 125fb199
2016-08-12 14:12:12,287 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS
2016-08-12 14:12:12,287 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) Exception: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: 'spUUID=6c820dd3-24dc-472f-9b9b-b366da23b22f,
msdUUID=fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf'
2016-08-12 14:12:12,302 WARN
[org.ovirt.engine.core.dal.job.ExecutionMessageDirector]
(QuartzScheduler_Worker-62) [3165a3b2] The message key
ReconstructMasterDomain is missing from bundles/ExecutionMessages
2016-08-12 14:12:12,350 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(QuartzScheduler_Worker-62) [3165a3b2] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected :
ID: fe3dad77-6f27-40cd-8be9-b9e9cd1fb9bf Type: Storage
2016-08-12 14:12:12,397 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand]
(QuartzScheduler_Worker-62) [3165a3b2] START,
MarkPoolInReconstructModeVDSCommand(storagePoolId =
6c820dd3-24dc-472f-9b9b-b366da23b22f, ignoreFailoverLimit = false,
compatabilityVersion = null, reconstructMarkAction = ClearCache), log
id: 40a6ee33
2016-08-12 14:12:12,397 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(QuartzScheduler_Worker-62) [3165a3b2] clearing cache for problematic
entities in pool 6c820dd3-24dc-472f-9b9b-b366da23b22f
2016-08-12 14:12:12,398 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand]
(QuartzScheduler_Worker-62) [3165a3b2] FINISH,
MarkPoolInReconstructModeVDSCommand, log id: 40a6ee33