Good Day To all;
oVirt 4.3.4.3-1.el7
6 Dell R710 oVirt Nodes
10 gig Infiniband iSCSI
Standalone Engine
After an error during scheduled automated VM backkup, the iSCSi datastore that the VMs
beinng backed up lived on suddenly went inactive and refuse to activate.
The backup job did not finish as the source datastore went inactive but the task did clear
from the console after i restarted the ovirt-engine service.
I have tried:
1. Restart the ovirt-engine service
2. Restarted the engine physical server.
3. Tried to migrate the VM disks to a different datastore (failure)
The virtual machines that are currently living on the inactive datastore are still running
and accessible.
Trying to reboot one of those VM as a test fails as the VM will not come up due to the
system believing that the datastore is inactive.
From engin.log:
2019-09-15 17:09:08,787-05 INFO
[org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Removing
'Wildix4-PBX_snapshot_metadata' disk
2019-09-15 17:09:13,399-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] EVENT_ID:
IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: General
Storage Exception: ('5 [\' Logical volume
d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\']
[\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining
activation change with other commands is not advised.\', \'
/dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset
19198423553536\', " Couldn\'t read volume group metadata from
/dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on
/dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for
VG.\', \' Failed to read metadata summary from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group
"d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process
volume group
d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\':
ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'],
parent=\'00000000-0000-0000-0000-000000000000\')}',)
2019-09-15 17:09:13,399-05 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7]
IrsBroker::Failed::DeleteImageGroupVDS: IRSGenericException: IRSErrorException: Failed to
DeleteImageGroupVDS, error = General Storage Exception: ('5 [\' Logical volume
d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\']
[\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining
activation change with other commands is not advised.\', \'
/dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset
19198423553536\', " Couldn\'t read volume group metadata from
/dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on
/dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for
VG.\', \' Failed to read metadata summary from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group
"d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process
volume group
d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\':
ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'],
parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200
2019-09-15 17:09:13,463-05 ERROR
[org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand]
(EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Command
'org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand' failed:
EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException:
IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = General
Storage Exception: ('5 [\' Logical volume
d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\']
[\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining
activation change with other commands is not advised.\', \'
/dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset
19198423553536\', " Couldn\'t read volume group metadata from
/dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on
/dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for
VG.\', \' Failed to read metadata summary from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from
/dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group
"d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process
volume group
d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\':
ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'],
parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200 (Failed with
error StorageException and code 200)
<metadata
xmlns:ns0="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0">
Any help or input would be greaatly appreciated.