[Users] Problems with missing master domains after rebooto f

Wed Feb 26 00:11:17 UTC 2014

Hi Guys,

I've been using an FC18-based ovirt setup for a while which has been
updated regularly (about each week). Last  week I tried to setup the
reports module, and ran ovirt-engine-dwh-setup which failed for some
reason (didn't preserve the logging).

After a reboot of the machine I found this in /var/log/ovirt/engine.log:
> 2014-02-26 00:50:42,014 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-3) Could not parse option AutoRecoveryAllowedTypes value.
> 2014-02-26 00:50:46,196 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_ERROR not exist in string table
> [root at leiden ovirt-engine]# grep ERR engine.log  | grep "00:51"
> 2014-02-26 00:51:02,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [7278cc5e] Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:02,046 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [7278cc5e] IrsBroker::Failed::GetStoragePoolInfoVDS
> 2014-02-26 00:51:02,047 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [7278cc5e] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:02,212 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (pool-6-thread-19) [25bc9b42] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
> 2014-02-26 00:51:02,282 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19) [25bc9b42] Failed in ReconstructMasterVDS method
> 2014-02-26 00:51:02,282 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19) [25bc9b42] Error code StorageDomainDoesNotExist and error message VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:02,284 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19) [25bc9b42] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:02,285 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-19) [25bc9b42] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error StorageDomainDoesNotExist and code 358)
> 2014-02-26 00:51:21,192 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-25) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:21,193 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) IrsBroker::Failed::GetStoragePoolInfoVDS
> 2014-02-26 00:51:21,193 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-25) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:21,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35) [37b66fe] Failed in ReconstructMasterVDS method
> 2014-02-26 00:51:21,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35) [37b66fe] Error code StorageDomainDoesNotExist and error message VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:21,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35) [37b66fe] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:21,311 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-35) [37b66fe] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error StorageDomainDoesNotExist and code 358)
> 2014-02-26 00:51:31,242 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
> 2014-02-26 00:51:40,355 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:40,356 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS
> 2014-02-26 00:51:40,356 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:40,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [4e565f79] Failed in ReconstructMasterVDS method
> 2014-02-26 00:51:40,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [4e565f79] Error code StorageDomainDoesNotExist and error message VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:40,479 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [4e565f79] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:40,480 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-50) [4e565f79] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error StorageDomainDoesNotExist and code 358)
> 2014-02-26 00:51:50,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
> 2014-02-26 00:51:59,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-68) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:59,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) IrsBroker::Failed::GetStoragePoolInfoVDS
> 2014-02-26 00:51:59,525 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
> 2014-02-26 00:51:59,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [36e19f0f] Failed in ReconstructMasterVDS method
> 2014-02-26 00:51:59,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [36e19f0f] Error code StorageDomainDoesNotExist and error message VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:59,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50) [36e19f0f] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
> 2014-02-26 00:51:59,620 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-50) [36e19f0f] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error StorageDomainDoesNotExist and code 358)

I once had this before in which case the NFS volumes weren't mounted.
 Everything is hosted on this single machine (NFS-server, NFS-client,
ovirt-engine and webinterface):

Looks great:
> [root at leiden ovirt-engine]# df Filesystem 1K-blocks       Used
> Available Use% Mounted on /dev/mapper/fedora_leiden-root
> 117795148   18486352 93302012  17% / devtmpfs
> 4030616 0    4030616   0% /dev tmpfs 4037360         80    4037280
> 1% /dev/shm tmpfs 4037360        828    4036532   1% /run tmpfs 
> 4037360          0    4037360   0% /sys/fs/cgroup tmpfs 4037360
> 44    4037316   1% /tmp /dev/sdb1 5814235920 4274205592 1540013944
> 74% /raid 192.168.1.44:/raid/ovirt/iso           5814236160
> 4274205696 1540014080  74%
> /rhev/data-center/mnt/192.168.1.44:_raid_ovirt_iso 
> 192.168.1.44:/raid/ovirt/import_export 5814236160 4274205696 
> 1540014080  74% 
> /rhev/data-center/mnt/192.168.1.44:_raid_ovirt_import__export 
> 192.168.1.44:/raid/ovirt/data          5814236160 4274205696 
> 1540014080  74% 
> /rhev/data-center/mnt/192.168.1.44:_raid_ovirt_data

Furthermore /raid/ovirt/data looks fine too:> [root at leiden ovirt]# find
| grep meta
> ./iso/d2676b04-e2ff-420f-8f29-36dafc2df47b/dom_md/metadata
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/8483a1b6-4d79-4675-8433-eb03dcd5f53d/829348f3-0f63-4275-92e1-1e84681a422b.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/907f3071-69eb-4167-8e34-22d3985f63cf/76749503-4a8b-4e8f-a2e4-9d89e0de0d71.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/d718155e-eaf2-41e0-aebd-799a31af18bc/0ad131d7-2619-42a2-899f-d25c33969dc6.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/666b1602-6979-4b17-a7fe-6524a1bf603b/182ce48c-59d0-4883-8265-0269247d22e0.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/8616828d-e3f3-4e55-863f-387df2110ebc/e66b18a7-e2c5-4f6c-9884-03e5c7477e3d.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/_remoMSGKTD/924e5ba6-913e-4591-a15f-3b61eb66a2e1.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/eb32f782-7b29-4f54-bd3e-a736ae8c5476/1d304cb5-67bd-4e21-aa2c-2470c19af885.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/26628bb8-a057-4fab-af65-20258f083ab0/c46bb1c0-dad9-490c-95b4-b74b25b80129.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/b5a5411b-b234-4340-8338-1f2b860e4265/4d583a7a-8399-4299-9799-dec33913c20a.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/40909eb8-be4c-4280-b9ef-51f3fa36340e/aca392f5-8395-46fe-9111-8a3c4812ff72.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/e504ad89-7a37-4a8d-a084-52e862c5a78f/b2c5d2d5-636c-408b-b52f-b7f5558c0f7f.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/fcf7a685-41dc-45d6-bc5c-d8ad53fcd900/e1c098fe-4b5d-4728-81d0-7edfdd3d0ec8.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/cf5371aa-2cbe-4edc-a876-2b503208b0e6/cadcce7f-53ff-4735-b5ff-4d8fd1991d51.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/5e28342e-2e90-491b-a6c3-49b2443092fd/f613aa23-4831-4aba-806e-fb7dcdcd704d.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/85d7448a-ad17-4a77-86a9-6a55c1baf1a6/179ad90d-ed46-467d-ad75-aea6e3ea115e.meta
> ./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/dom_md/metadata
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/6055a0a2-a6e9-4466-b0eb-3928c5c84d99/9e5be41b-c512-4f22-9d7c-81090d62dc31.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/d72c41ff-2e34-474d-86be-3c11181b2128/0dd635c7-ef77-4e0f-962c-90b66085ba93.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/df485d87-7dda-4cee-8fd0-5dc8b00d44c6/988f90f6-a37d-4dfd-8477-70aa5d2db5b6.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/df485d87-7dda-4cee-8fd0-5dc8b00d44c6/38eee7d5-9fd1-44b0-876c-b24e4bc0085b.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/898c01ea-d7b5-4d59-8329-c9140f3e55c1/8b511fc2-4ec5-4c82-9faf-93da8490adc9.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/1f6b7b10-736c-4a6c-9743-a628f370ff2f/8633fb9b-9c08-406b-925e-7d5955912165.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/c866dd6c-c7e5-419a-85e4-af49228be5a2/5e56a396-8deb-4c04-9897-0e4f6582abcc.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/a33a673d-751f-4287-a655-e84dfcfcd005/2cd8d3dc-e92f-4be5-88fa-923076aba287.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/07b94c8d-8195-449b-b5e0-873bde6f85fd/efc46a9a-6fcd-4e48-a197-e6bdf1e655bf.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/72dba8d6-4303-4db7-8a32-aafa0a3165a5/caecf666-302d-426c-8a32-65eda8d9e5df.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/7fd446ce-bfb5-4706-9eb8-4133fcfbc00d/88a7d07b-b4a3-497d-b2e5-3e6ebc85d83e.meta
> ./data/1979444d-b79a-494c-8c1a-bcc132e31a04/dom_md/metadata

But no metadata with uuid 5849b030-626e-47cb-ad90-3ce782d831b3 .
(there's a lot more files but I filtered the find in order to keep
this email as concise as possible :) )

Last time I was able to 'fix' this by running engine-setup which
somehow reconfigured my ovirt installation and retained all VMs etc
etc etc.

This time this doesn't work anymore unfortunately.

I read:

https://access.redhat.com/site/solutions/62137
http://lists.ovirt.org/pipermail/users/2013-April/014061.html

But these don't seem to have an answer to my problems unfortunately.

How should I start debugging this?

Thanks for this wonderful product!

Boudewijn Ector