[ovirt-users] Cannot Activate iSCSI Storage Domain After Errors

Good Day To all; oVirt 4.3.4.3-1.el7 6 Dell R710 oVirt Nodes 10 gig Infiniband iSCSI Standalone Engine After an error during scheduled automated VM backkup, the iSCSi datastore that the VMs beinng backed up lived on suddenly went inactive and refuse to activate. The backup job did not finish as the source datastore went inactive but the task did clear from the console after i restarted the ovirt-engine service. I have tried: 1. Restart the ovirt-engine service 2. Restarted the engine physical server. 3. Tried to migrate the VM disks to a different datastore (failure) The virtual machines that are currently living on the inactive datastore are still running and accessible. Trying to reboot one of those VM as a test fails as the VM will not come up due to the system believing that the datastore is inactive. From engin.log: 2019-09-15 17:09:08,787-05 INFO [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Removing 'Wildix4-PBX_snapshot_metadata' disk 2019-09-15 17:09:13,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',) 2019-09-15 17:09:13,399-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] IrsBroker::Failed::DeleteImageGroupVDS: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200 2019-09-15 17:09:13,463-05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Command 'org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200 (Failed with error StorageException and code 200) <metadata xmlns:ns0="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> Any help or input would be greaatly appreciated.