[ovirt-devel] vdsm.storage.exception.UnknownTask: Task id unknown (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1641 - Still Failing!)

On Wed, Jun 17, 2020 at 6:28 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: This one failed while trying to create the disk image for the hosted-egnine VM: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... : 2020-06-16 23:03:20,527-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Add HE disks'} ... 2020-06-16 23:14:12,702-0400 DEBUG var changed: host "localhost" var "add_disks" type "<class 'dict'>" value: "{ ... "msg": "Timeout exceed while waiting on result state of the entity." https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... : 2020-06-16 23:03:22,612-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] CommandMultiAsyncTasks::attachTask: Attaching task '6b2a7648-748c-430b-94b6-5e3f719df2ac' to command 'fa81759d-c57a-4237-81e0-beb210faa64d'. 2020-06-16 23:03:22,659-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] Adding task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2020-06-16 23:03:22,699-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-1) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::startPollingTask: Starting to poll task '6b2a7648-748c-430b-94b6-5e3f719df2ac'. ... 2020-06-16 23:03:25,835-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] SPMAsyncTask::PollTask: Polling task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2020-06-16 23:03:25,863-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] BaseAsyncTask::onTaskEndSuccess: Task '6b2a7648-748c-430b-94b6-5e3f719df2ac' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. But then: 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddImageFromScratch' succeeded, clearing tasks. 2020-06-16 23:03:25,897-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '6b2a7648-748c-430b-94b6-5e3f719df2ac' 2020-06-16 23:03:25,899-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='3bcde3b4-b044-11ea-bbb6-5452c0a8c863', ignoreFailoverLimit='false', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 481c2d3d 2020-06-16 23:03:25,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] START, HSMClearTaskVDSCommand(HostName = lago-he-basic-suite-master-host-0, HSMTaskGuidBaseVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd', taskId='6b2a7648-748c-430b-94b6-5e3f719df2ac'}), log id: 17360b3d ... 2020-06-16 23:03:26,054-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-29) [16c24599-0048-44eb-a410-d39b7ce98712] BaseAsyncTask::removeTaskFromDB: Removed task '6b2a7648-748c-430b-94b6-5e3f719df2ac' from DataBase But then: 2020-06-16 23:03:26,315-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'UploadStreamVDSCommand(HostName = lago-he-basic-suite-master-host-0, UploadStreamVDSCommandParameters:{hostId='85ecc51c-f2cb-46a1-9452-fd487399d8dd'})' execution failed: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] 2020-06-16 23:03:26,315-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] FINISH, UploadStreamVDSCommand, return: , log id: 7e3a3e80 2020-06-16 23:03:26,316-04 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [7fe7b467] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <lago-he-basic-suite-master-host-0.lago.local> doesn't match any of the subject alternative names: [lago-he-basic-suite-master-host-0.lago.local] (Failed with error VDS_NETWORK_ERROR and code 5022) Any idea why? Anything changed in how we check the certificate? Perhaps related to upgrade to CentOS 8.2? And, how come it failed only this late? Don't we check the certificate earlier? Anyway, this left the host in "not responding" state, so: 2020-06-16 23:03:29,994-04 ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [16c24599-0048-44eb-a410-d39b7ce98712] Failed to get volume info: org.ovirt.engine.core.common.errors.EngineException: EngineException: No host was found to perform the operation (Failed with error RESOURCE_MANAGER_VDS_NOT_FOUND and code 5004) And perhaps due to an unrelated issue, also: 2020-06-16 23:03:31,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMRevertTaskVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [16c24599-0048-44eb-a410-d39b7ce98712] Trying to revert unknown task '6b2a7648-748c-430b-94b6-5e3f719df2ac' I looked a bit also at: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... and see there some relevant stuff, but nothing I can spot about the root cause (e.g. the word "cert" does not appear there). Can anyone please have a look? Thanks. -- Didi