6:11 p.m.
Hi Guys,
I've been using an FC18-based ovirt setup for a while which has been
updated regularly (about each week). Last week I tried to setup the
reports module, and ran ovirt-engine-dwh-setup which failed for some
reason (didn't preserve the logging).
After a reboot of the machine I found this in /var/log/ovirt/engine.log:
2014-02-26 00:50:42,014 ERROR
[org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (MSC service thread 1-3) Could
not parse option AutoRecoveryAllowedTypes value.
2014-02-26 00:50:46,196 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_ERROR not exist in string table
[root@leiden ovirt-engine]# grep ERR engine.log | grep "00:51"
2014-02-26 00:51:02,045 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-9) [7278cc5e] Command ConnectStoragePoolVDS execution
failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:02,046 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-9) [7278cc5e] IrsBroker::Failed::GetStoragePoolInfoVDS
2014-02-26 00:51:02,047 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-9) [7278cc5e] Exception: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:02,212 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(pool-6-thread-19) [25bc9b42] Command HSMGetAllTasksStatusesVDS execution failed.
Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-02-26 00:51:02,282 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19)
[25bc9b42] Failed in ReconstructMasterVDS method
2014-02-26 00:51:02,282 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19)
[25bc9b42] Error code StorageDomainDoesNotExist and error message VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:02,284 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-19)
[25bc9b42] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage
domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:02,285 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-19)
[25bc9b42] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw
Vdc Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error
StorageDomainDoesNotExist and code 358)
2014-02-26 00:51:21,192 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-25) Command ConnectStoragePoolVDS execution failed.
Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:21,193 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-25) IrsBroker::Failed::GetStoragePoolInfoVDS
2014-02-26 00:51:21,193 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-25) Exception: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:21,309 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35)
[37b66fe] Failed in ReconstructMasterVDS method
2014-02-26 00:51:21,309 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35)
[37b66fe] Error code StorageDomainDoesNotExist and error message VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:21,311 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-35)
[37b66fe] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage
domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:21,311 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-35)
[37b66fe] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw
Vdc Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error
StorageDomainDoesNotExist and code 358)
2014-02-26 00:51:31,242 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-44) Command HSMGetAllTasksStatusesVDS execution failed.
Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-02-26 00:51:40,355 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-44) Command ConnectStoragePoolVDS execution failed.
Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:40,356 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS
2014-02-26 00:51:40,356 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-44) Exception: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:40,478 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[4e565f79] Failed in ReconstructMasterVDS method
2014-02-26 00:51:40,478 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[4e565f79] Error code StorageDomainDoesNotExist and error message VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:40,479 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[4e565f79] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage
domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:40,480 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-50)
[4e565f79] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw
Vdc Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error
StorageDomainDoesNotExist and code 358)
2014-02-26 00:51:50,407 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-68) Command HSMGetAllTasksStatusesVDS execution failed.
Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-02-26 00:51:59,523 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-68) Command ConnectStoragePoolVDS execution failed.
Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:59,525 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-68) IrsBroker::Failed::GetStoragePoolInfoVDS
2014-02-26 00:51:59,525 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-68) Exception: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3,
msdUUID=1979444d-b79a-494c-8c1a-bcc132e31a04'
2014-02-26 00:51:59,618 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[36e19f0f] Failed in ReconstructMasterVDS method
2014-02-26 00:51:59,618 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[36e19f0f] Error code StorageDomainDoesNotExist and error message VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:59,620 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (pool-6-thread-50)
[36e19f0f] Command ReconstructMasterVDS execution failed. Exception: VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Storage
domain does not exist: ('dafe25c2-1ce7-4979-9d8d-a35688da207a',)
2014-02-26 00:51:59,620 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-6-thread-50)
[36e19f0f] Command org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand throw
Vdc Bll exception. With error message VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Storage domain does not exist:
('dafe25c2-1ce7-4979-9d8d-a35688da207a',) (Failed with error
StorageDomainDoesNotExist and code 358)
I once had this before in which case the NFS volumes weren't mounted.
Everything is hosted on this single machine (NFS-server, NFS-client,
ovirt-engine and webinterface):
Looks great:
[root@leiden ovirt-engine]# df Filesystem 1K-blocks Used
Available Use% Mounted on /dev/mapper/fedora_leiden-root
117795148 18486352 93302012 17% / devtmpfs
4030616 0 4030616 0% /dev tmpfs 4037360 80 4037280
1% /dev/shm tmpfs 4037360 828 4036532 1% /run tmpfs
4037360 0 4037360 0% /sys/fs/cgroup tmpfs 4037360
44 4037316 1% /tmp /dev/sdb1 5814235920 4274205592 1540013944
74% /raid 192.168.1.44:/raid/ovirt/iso 5814236160
4274205696 1540014080 74%
/rhev/data-center/mnt/192.168.1.44:_raid_ovirt_iso
192.168.1.44:/raid/ovirt/import_export 5814236160 4274205696
1540014080 74%
/rhev/data-center/mnt/192.168.1.44:_raid_ovirt_import__export
192.168.1.44:/raid/ovirt/data 5814236160 4274205696
1540014080 74%
/rhev/data-center/mnt/192.168.1.44:_raid_ovirt_data
Furthermore /raid/ovirt/data looks fine too:> [root@leiden ovirt]# find
| grep meta
./iso/d2676b04-e2ff-420f-8f29-36dafc2df47b/dom_md/metadata
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/8483a1b6-4d79-4675-8433-eb03dcd5f53d/829348f3-0f63-4275-92e1-1e84681a422b.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/907f3071-69eb-4167-8e34-22d3985f63cf/76749503-4a8b-4e8f-a2e4-9d89e0de0d71.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/d718155e-eaf2-41e0-aebd-799a31af18bc/0ad131d7-2619-42a2-899f-d25c33969dc6.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/666b1602-6979-4b17-a7fe-6524a1bf603b/182ce48c-59d0-4883-8265-0269247d22e0.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/8616828d-e3f3-4e55-863f-387df2110ebc/e66b18a7-e2c5-4f6c-9884-03e5c7477e3d.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/_remoMSGKTD/924e5ba6-913e-4591-a15f-3b61eb66a2e1.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/eb32f782-7b29-4f54-bd3e-a736ae8c5476/1d304cb5-67bd-4e21-aa2c-2470c19af885.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/26628bb8-a057-4fab-af65-20258f083ab0/c46bb1c0-dad9-490c-95b4-b74b25b80129.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/b5a5411b-b234-4340-8338-1f2b860e4265/4d583a7a-8399-4299-9799-dec33913c20a.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/40909eb8-be4c-4280-b9ef-51f3fa36340e/aca392f5-8395-46fe-9111-8a3c4812ff72.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/e504ad89-7a37-4a8d-a084-52e862c5a78f/b2c5d2d5-636c-408b-b52f-b7f5558c0f7f.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/fcf7a685-41dc-45d6-bc5c-d8ad53fcd900/e1c098fe-4b5d-4728-81d0-7edfdd3d0ec8.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/cf5371aa-2cbe-4edc-a876-2b503208b0e6/cadcce7f-53ff-4735-b5ff-4d8fd1991d51.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/5e28342e-2e90-491b-a6c3-49b2443092fd/f613aa23-4831-4aba-806e-fb7dcdcd704d.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/images/85d7448a-ad17-4a77-86a9-6a55c1baf1a6/179ad90d-ed46-467d-ad75-aea6e3ea115e.meta
./import_export/dafe25c2-1ce7-4979-9d8d-a35688da207a/dom_md/metadata
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/6055a0a2-a6e9-4466-b0eb-3928c5c84d99/9e5be41b-c512-4f22-9d7c-81090d62dc31.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/d72c41ff-2e34-474d-86be-3c11181b2128/0dd635c7-ef77-4e0f-962c-90b66085ba93.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/df485d87-7dda-4cee-8fd0-5dc8b00d44c6/988f90f6-a37d-4dfd-8477-70aa5d2db5b6.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/df485d87-7dda-4cee-8fd0-5dc8b00d44c6/38eee7d5-9fd1-44b0-876c-b24e4bc0085b.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/898c01ea-d7b5-4d59-8329-c9140f3e55c1/8b511fc2-4ec5-4c82-9faf-93da8490adc9.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/1f6b7b10-736c-4a6c-9743-a628f370ff2f/8633fb9b-9c08-406b-925e-7d5955912165.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/c866dd6c-c7e5-419a-85e4-af49228be5a2/5e56a396-8deb-4c04-9897-0e4f6582abcc.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/a33a673d-751f-4287-a655-e84dfcfcd005/2cd8d3dc-e92f-4be5-88fa-923076aba287.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/07b94c8d-8195-449b-b5e0-873bde6f85fd/efc46a9a-6fcd-4e48-a197-e6bdf1e655bf.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/72dba8d6-4303-4db7-8a32-aafa0a3165a5/caecf666-302d-426c-8a32-65eda8d9e5df.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/images/7fd446ce-bfb5-4706-9eb8-4133fcfbc00d/88a7d07b-b4a3-497d-b2e5-3e6ebc85d83e.meta
./data/1979444d-b79a-494c-8c1a-bcc132e31a04/dom_md/metadata
But no metadata with uuid 5849b030-626e-47cb-ad90-3ce782d831b3 .
(there's a lot more files but I filtered the find in order to keep
this email as concise as possible :) )
Last time I was able to 'fix' this by running engine-setup which
somehow reconfigured my ovirt installation and retained all VMs etc
etc etc.
This time this doesn't work anymore unfortunately.
I read:
https://access.redhat.com/site/solutions/62137
http://lists.ovirt.org/pipermail/users/2013-April/014061.html
But these don't seem to have an answer to my problems unfortunately.
How should I start debugging this?
Thanks for this wonderful product!
Boudewijn Ector