Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.3 - Build # 472 - Failure!
On Sat, Jun 13, 2020 at 5:01 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/472/
This failed in: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/test... Fault reason is "Operation Failed". Fault detail is "[Cannot activate Host. Related operation is currently in progress. Please try again later.]". HTTP response code is 409. engine.log has: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/arti... 2020-06-15 00:00:58,330-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: e75bc41d-c044-40f6-a645-9a86f2ef2536 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-15 00:00:58,331-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Before acquiring lock in order to prevent monitoring for host 'lago-he-basic-suite-4-3-host-0' from data-center 'Default' 2020-06-15 00:00:58,332-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Clearing cache of pool: 'e186881e-aeb0-11ea-8e5b-5452c0a8c863' for problematic entities of VDS: 'lago-he-basic-suite-4-3-host-0'. 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing vds '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from the domain in maintenance cache 2020-06-15 00:00:58,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing host(s) '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from hosts unseen domain report cache 2020-06-15 00:00:58,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-he-basic-suite-4-3-host-0, DisconnectStoragePoolVDSCommandParameters:{hostId='e75bc41d-c044-40f6-a645-9a86f2ef2536', storagePoolId='e186881e-aeb0-11ea-8e5b-5452c0a8c863', vds_spm_id='1'}), log id: e9ba207 2020-06-15 00:00:58,498-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:01,657-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:03,556-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS]', sharedLocks=''}' 2020-06-15 00:01:03,585-04 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Validation of action 'ActivateVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED 2020-06-15 00:01:03,864-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot activate Host. Related operation is currently in progress. Please try again later.] Sandro reported about a similar case in the thread: [ovirt-devel] ovirt-system-tests_he-basic-suite-4.3 fails on storage domain unreachable About: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/427/ Can't see there any response. Can someone please have a look? Thanks. Best regards, -- Didi
Il giorno lun 15 giu 2020 alle ore 09:15 Yedidyah Bar David <didi@redhat.com> ha scritto:
On Sat, Jun 13, 2020 at 5:01 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/472/
This failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/test...
Fault reason is "Operation Failed". Fault detail is "[Cannot activate Host. Related operation is currently in progress. Please try again later.]". HTTP response code is 409.
engine.log has:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/arti...
2020-06-15 00:00:58,330-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: e75bc41d-c044-40f6-a645-9a86f2ef2536 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-15 00:00:58,331-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Before acquiring lock in order to prevent monitoring for host 'lago-he-basic-suite-4-3-host-0' from data-center 'Default' 2020-06-15 00:00:58,332-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Clearing cache of pool: 'e186881e-aeb0-11ea-8e5b-5452c0a8c863' for problematic entities of VDS: 'lago-he-basic-suite-4-3-host-0'. 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing vds '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from the domain in maintenance cache 2020-06-15 00:00:58,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing host(s) '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from hosts unseen domain report cache 2020-06-15 00:00:58,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-he-basic-suite-4-3-host-0,
DisconnectStoragePoolVDSCommandParameters:{hostId='e75bc41d-c044-40f6-a645-9a86f2ef2536', storagePoolId='e186881e-aeb0-11ea-8e5b-5452c0a8c863', vds_spm_id='1'}), log id: e9ba207 2020-06-15 00:00:58,498-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:01,657-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:03,556-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS]', sharedLocks=''}' 2020-06-15 00:01:03,585-04 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Validation of action 'ActivateVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED 2020-06-15 00:01:03,864-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot activate Host. Related operation is currently in progress. Please try again later.]
Sandro reported about a similar case in the thread:
[ovirt-devel] ovirt-system-tests_he-basic-suite-4.3 fails on storage domain unreachable
About:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/427/
Can't see there any response.
Can someone please have a look? Thanks.
I haven't seen any reaction on this. Anyone from storage team can have a look here? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
it is very unlikely anything has changed in 4.3 suite TBH I wouldn’t bother and rather invest time to get the 4.4 he-basic-suite on par with basic-suite-master. It needs python3, pytest, ost-images… Thanks, michal
On 25 Jun 2020, at 09:56, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno lun 15 giu 2020 alle ore 09:15 Yedidyah Bar David <didi@redhat.com <mailto:didi@redhat.com>> ha scritto: On Sat, Jun 13, 2020 at 5:01 AM <jenkins@jenkins.phx.ovirt.org <mailto:jenkins@jenkins.phx.ovirt.org>> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/ <https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/> Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/472/ <https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/472/>
This failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/test... <https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/testReport/junit/(root)/012_local_maintenance_sdk/local_maintenance/>
Fault reason is "Operation Failed". Fault detail is "[Cannot activate Host. Related operation is currently in progress. Please try again later.]". HTTP response code is 409.
engine.log has:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/arti... <https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/474/artifact/exported-artifacts/test_logs/he-basic-suite-4.3/post-012_local_maintenance_sdk.py/lago-he-basic-suite-4-3-engine/_var_log/ovirt-engine/engine.log>
2020-06-15 00:00:58,330-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Running command: RefreshHostCapabilitiesCommand internal: true. Entities affected : ID: e75bc41d-c044-40f6-a645-9a86f2ef2536 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2020-06-15 00:00:58,331-04 INFO [org.ovirt.engine.core.bll.RefreshHostCapabilitiesCommand] (ForkJoinPool-1-worker-8) [3715b680] Before acquiring lock in order to prevent monitoring for host 'lago-he-basic-suite-4-3-host-0' from data-center 'Default' 2020-06-15 00:00:58,332-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Clearing cache of pool: 'e186881e-aeb0-11ea-8e5b-5452c0a8c863' for problematic entities of VDS: 'lago-he-basic-suite-4-3-host-0'. 2020-06-15 00:00:58,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing vds '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from the domain in maintenance cache 2020-06-15 00:00:58,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-281) [] Removing host(s) '[e75bc41d-c044-40f6-a645-9a86f2ef2536]' from hosts unseen domain report cache 2020-06-15 00:00:58,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [] START, DisconnectStoragePoolVDSCommand(HostName = lago-he-basic-suite-4-3-host-0, DisconnectStoragePoolVDSCommandParameters:{hostId='e75bc41d-c044-40f6-a645-9a86f2ef2536', storagePoolId='e186881e-aeb0-11ea-8e5b-5452c0a8c863', vds_spm_id='1'}), log id: e9ba207 2020-06-15 00:00:58,498-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:01,657-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (ForkJoinPool-1-worker-8) [3715b680] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS_INIT]', sharedLocks=''}' 2020-06-15 00:01:03,556-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[e75bc41d-c044-40f6-a645-9a86f2ef2536=VDS]', sharedLocks=''}' 2020-06-15 00:01:03,585-04 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-1) [e4c7e809-7883-47dc-b7da-3236cc754ebb] Validation of action 'ActivateVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED 2020-06-15 00:01:03,864-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot activate Host. Related operation is currently in progress. Please try again later.]
Sandro reported about a similar case in the thread:
[ovirt-devel] ovirt-system-tests_he-basic-suite-4.3 fails on storage domain unreachable
About:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/427/ <https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/427/>
Can't see there any response.
Can someone please have a look? Thanks.
I haven't seen any reaction on this. Anyone from storage team can have a look here?
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <mailto:sbonazzo@redhat.com> <https://www.redhat.com/> Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>
participants (3)
-
Michal Skrivanek -
Sandro Bonazzola -
Yedidyah Bar David