URGENT - ovirt-engine master is failing for the past 9 days

Hi, We have been failing CQ master for project ovirt-engine for the past 9 days. this was reported last week and we have not yet seen a fix for this issue. the patch reported by CQ is https://gerrit.ovirt.org/#/c/101913/10 - core: Change CPU config to secure/insecure concept logs for latest failure can be found here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15638/artif... Error: 2019-09-01 06:31:17,248-04 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-1) [174bc463-9a98-441f-ad14-4d97fefde4fa] Running command: UpdateClusterCommand internal: false. Entities affected : ID: 0407bc06-4bef-4c47-99e1-0e42f9ced996 Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN 2019-09-01 06:31:17,263-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [174bc463-9a98-441f-ad14-4d97fefde4fa] EVENT_ID: CLUSTER_UPDATE_CPU_WHEN_DEPRECATED(9,029), Modified the CPU Type to Intel Haswell-noTSX Family when upgrading the Compatibility Version of Cluster test-cluster because the previous CPU Type was deprecated. 2019-09-01 06:31:17,319-04 WARN [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [18fd7817] Validation of action 'UpdateVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_OS_TYPE_IS_NOT_SUPPORTED_BY_ARCHITECTURE_TYPE 2019-09-01 06:31:17,329-04 WARN [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [62694a6a] Validation of action 'UpdateVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_OS_TYPE_IS_NOT_SUPPORTED_BY_ARCHITECTURE_TYPE 2019-09-01 06:31:17,334-04 WARN [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-1) [10e41cd3] Validation of action 'UpdateVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_ILLEGAL_OS_TYPE_IS_NOT_SUPPORTED_BY_ARCHITECTURE_TYPE 2019-09-01 06:31:17,345-04 WARN [org.ovirt.engine.core.bll.UpdateVmTemplateCommand] (default task-1) [26a471dd] Validation of action 'UpdateVmTemplate' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM_TEMPLATE,VMT_CANNOT_UPDATE_ILLEGAL_FIELD 2019-09-01 06:31:17,352-04 WARN [org.ovirt.engine.core.bll.UpdateVmTemplateCommand] (default task-1) [32fd89b5] Validation of action 'UpdateVmTemplate' failed for user admin@internal-authz. Reasons: VAR__ACTION__UPDATE,VAR__TYPE__VM_TEMPLATE,VMT_CANNOT_UPDATE_ILLEGAL_FIELD 2019-09-01 06:31:17,363-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32fd89b5] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Cannot update compatibility version of Vm/Template: [vm-with-iface], Message: [No Message] 2019-09-01 06:31:17,371-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32fd89b5] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Cannot update compatibility version of Vm/Template: [vm-with-iface-template], Message: [No Message] 2019-09-01 06:31:17,380-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32fd89b5] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Cannot update compatibility version of Vm/Template: [vm0], Message: [No Message] 2019-09-01 06:31:17,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32fd89b5] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Cannot update compatibility version of Vm/Template: [vm1], Message: [No Message] 2019-09-01 06:31:17,405-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [32fd89b5] Command [id=50898f00-4956-484a-94ff-f273adc2e93e]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.Cluster; snapshot: Cluster [test-cluster]. 2019-09-01 06:31:17,422-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [32fd89b5] EVENT_ID: USER_UPDATE_CLUSTER_FAILED(812), Failed to update Host cluster (User: admin@internal-authz) 2019-09-01 06:31:17,422-04 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-1) [32fd89b5] Lock freed to object 'EngineLock:{exclusiveLocks='[5d3b2f0f-f05f-41c6-b61a-c517704f79fd=TEMPLATE, 728f5530-4e34-4c92-beef-ca494ec104b9=TEMPLATE]', sharedLocks='[a107316a-a961-403f-adb2-d01f22f0b8f1=VM, dc3a2ad8-6019-4e00-85e5-d9fba7390d4f=VM, 6348722f-61ae-40be-a2b8-bd9d086b06dc=VM]'}' 2019-09-01 06:31:17,424-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Update of cluster compatibility version failed because there are VMs/Templates [vm-with-iface, vm-with-iface-template, vm0, vm1] with incorrect configuration. To fix the issue, please go to each of them, edit, change the Custom Compatibility Version of the VM/Template to the cluster level you want to update the cluster to and press OK. If the save does not pass, fix the dialog validation. After successful cluster update, you can revert your Custom Compatibility Version change.] (END)