Well, issue was solved by installing latest async update 4.2.5...
Probably helped this part of upgrade process...
...
[ INFO ] Cleaning async tasks and compensations
[ INFO ] Unlocking existing entities
Right, that was an in-memory lock that blocked you so you probably had to remove the relevant command that acquired the lock from the command_entities table.
...
Cheers,
Jiri
On 08/10/2018 11:03 AM, Jiří Sléžka wrote:
> On 08/10/2018 10:52 AM, Raz Tamir wrote:
>>
>>
>> On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka@slu.cz
>> <mailto:jiri.slezka@slu.cz>> wrote:
>>
>> Hi,
>>
>> On 08/09/2018 11:25 PM, Raz Tamir wrote:
>> > You can try using vdsm-client to check if there are running tasks on the
>> > host using this command:
>> >
>> > vdsm-client Host getAllTasksInfo
>>
>> thanks for support, I tried this on affected host
>>
>> [root@blade01 ~]# vdsm-client Host getAllTasksInfo
>> vdsm-client: Command Host.getAllTasksInfo with args {} failed:
>> (code=654, message=Not SPM: ())
>>
>> so I moved into SPM host (blade03 at this time)
>>
>> [root@blade03 ~]# vdsm-client Host getAllTasksInfo
>> {}
>>
>> I tried also
>>
>> [root@blade01 ~]# vdsm-client Host getAllTasks
>> {}
>> [root@blade01 ~]# vdsm-client Host getJobs
>> {}
>>
>> [root@blade03 ~]# vdsm-client Host getAllTasks
>> {}
>> [root@blade03 ~]# vdsm-client Host getJobs
>> {}
>>
>> It looks like there are no tasks/jobs running on hosts. It looks like
>> something is stucked in engine db.
>>
>> At the beginning of this issue, there were some race condition where
>> blade01 had two running tasks/jobs which cannot be finished (network
>> configuration and activating of host). I deleted this jobs manually in
>> db via
>>
>> SELECT * FROM job ORDER BY start_time DESC;
>> SELECT DeleteJob('...job_id...');
>> SELECT DeleteJob('...job_id...');
>>
>> Maybe it was not wise... but what can I do now to solve this? :-)
>>
>> Yes it can be bad to mess with the DB.
>> My best suugestion at the momet is to try and re-add this host.
>
> I would like to but I cannot even remove this host (Cannot remove Host.
> Related operation is currently in progress. Please try again later.) :-(
> But maybe I can try to add it as brand new one?
>
> But I am curious what checks do manager and why we cannot see the
> tasks/jobs/locks/anything in db...
>
> Cheers, Jiri
>
>
>
>>
>>
>>
>> Cheers, Jiri
>>
>>
>> >
>> > If there are running tasks, you can stop them:
>> >
>> > vdsm-client Task stop taskID=xxx-yyy
>> >
>> > If there are finished tasks, you can clear them:
>> >
>> > vdsm-client Task clear taskID=xxx-yyy
>> >
>> >
>> > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>
>> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote:
>> >
>> > Hello,
>> >
>> > still no luck with solving this issue.
>> >
>> > I cannot even remove this host.
>> >
>> > engine log is now spammed with this messages
>> >
>> > 2018-08-09 15:03:08,410+02 INFO
>> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> > (EE-ManagedThreadFactory-engine-Thread-1057)
>> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>> and wait
>> > lock
>> >
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>> > sharedLocks=''}'
>> > 2018-08-09 15:03:08,446+02 INFO
>> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> > (EE-ManagedThreadFactory-engine-Thread-1057)
>> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>> and wait
>> > lock
>> >
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>> > sharedLocks=''}'
>> > 2018-08-09 15:03:08,451+02 INFO
>> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> > (EE-ManagedThreadFactory-engine-Thread-1057)
>> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>> and wait
>> > lock
>> >
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>> > sharedLocks=''}'
>> > 2018-08-09 15:03:08,486+02 INFO
>> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> > (EE-ManagedThreadFactory-engine-Thread-1057)
>> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock
>> and wait
>> > lock
>> >
>> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
>> > sharedLocks=''}'
>> >
>> > btw. oVirt 4.2.5.2-1.el7/CentOS7
>> >
>> > Any tips how to resolve this and at least remove this host
>> from ovirt?
>> >
>> > Cheers, Jiri
>> >
>> >
>> > On 08/06/2018 12:52 PM, Gobinda Das wrote:
>> > > Can you please post vdsm log which is inside
>> /var/log/vdsm/vdsm.log ?
>> > >
>> > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka
>> <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>
>> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>
>> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>
>> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > no one can help?
>> > >
>> > > I still cannot activate this host - error is "Cannot
>> activate
>> > Host.
>> > > Related operation is currently in progress. Please try again
>> > later."
>> > >
>> > > I believe relevant log entrieas are
>> > >
>> > > 2018-08-06 12:15:50,398+02 INFO
>> > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
>> task-426)
>> > > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire
>> Lock
>> > to object
>> > >
>> >
>> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
>> > > sharedLocks=''}'
>> > > 2018-08-06 12:15:50,398+02 WARN
>> > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
>> task-426)
>> > > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action
>> > > 'ActivateVds' failed for user ***my_username***. Reasons:
>> > >
>> >
>> VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
>> > >
>> > >
>> > > when I tried to "reinstall host" I got entry in Events
>> "Failed
>> > to update
>> > > Host blade01" and correlation id. Relevant log entries are
>> > >
>> > >
>> > > cat /var/log/ovirt-engine/engine.log | grep
>> > > d2b332da-9050-419c-8188-8fdda5e9d807
>> > >
>> > > 2018-08-06 12:08:57,451+02 INFO
>> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>> (default
>> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running
>> command:
>> > > InstallVdsCommand internal: false. Entities affected : ID:
>> > > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group
>> > > EDIT_HOST_CONFIGURATION with role type ADMIN
>> > > 2018-08-06 12:08:57,476+02 INFO
>> > >
>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
>> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>> > Failed to
>> > > Acquire Lock to object
>> > >
>> >
>> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
>> > > sharedLocks=''}'
>> > > 2018-08-06 12:08:57,476+02 WARN
>> > >
>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
>> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>> > Validation of
>> > > action 'InstallVdsInternal' failed for user
>> ***my_username***.
>> > Reasons:
>> > > ACTION_TYPE_FAILED_OBJECT_LOCKED
>> > > 2018-08-06 12:08:57,479+02 ERROR
>> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>> (default
>> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>> > Installation/upgrade of
>> > > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01'
>> failed:
>> > Cannot
>> > > ${action} ${type}. Related operation is currently in
>> progress.
>> > Please
>> > > try again later.
>> > > 2018-08-06 12:08:57,482+02 INFO
>> > > [org.ovirt.engine.core.bll.CommandCompensator] (default
>> task-425)
>> > > [d2b332da-9050-419c-8188-8fdda5e9d807] Command
>> > > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating
>> > > DELETED_OR_UPDATED_ENTITY of
>> > > org.ovirt.engine.core.common.businessentities.VdsStatic;
>> snapshot:
>> > > id=786646cd-c9ef-49a8-8aea-56e858dcf202.
>> > > 2018-08-06 12:08:57,486+02 ERROR
>> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
>> (default
>> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction
>> > rolled-back
>> > > for command
>> > 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'.
>> > > 2018-08-06 12:08:57,491+02 ERROR
>> > >
>> >
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
>> > EVENT_ID:
>> > > USER_FAILED_UPDATE_VDS(105), Failed to update Host
>> blade01 (User:
>> > > ***my_username***).
>> > >
>> > > How can I unlock this entries?
>> > >
>> > > Thanks in advance,
>> > >
>> > > Jiri Slezka
>> > >
>> > >
>> > >
>> > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote:
>> > > > Hi,
>> > > >
>> > > > I have one interesting issue. I was asked by colleague to
>> > repair his
>> > > > cluster which is part of our oVirt installation. He
>> tried to
>> > > upgrade his
>> > > > host to gain 4.2 compatibility level but it fails in some
>> > cases. Never
>> > > > mind, I checked it, repaired, installed upgrades and
>> > activate his
>> > > hosts
>> > > > with one exception.
>> > > >
>> > > > One of hosts was stucked in finishing two tasks. One was
>> > changing
>> > > > network configuration, the second was activating of
>> this host. I
>> > > have to
>> > > > manually clear this tasks in database by
>> > > >
>> > > > SELECT * FROM job ORDER BY start_time DESC;
>> > > > SELECT DeleteJob('...job_id...');
>> > > > SELECT DeleteJob('...job_id...');
>> > > >
>> > > > Now this tasks are gone but still cannot activate this
>> host.
>> > > >
>> > > >
>> > > > Error while executing action:
>> > > >
>> > > > blade01:
>> > > >
>> > > > Cannot activate Host. Related operation is
>> currently in
>> > progress.
>> > > > Please try again later.
>> > > >
>> > > > Host is updatet (latest 4.2 repo), rebooted, looks fine...
>> > > >
>> > > > In tasks in UI are no tasks found (the same in db)...
>> > > >
>> > > > What can I do now?
>> > > >
>> > > > Cheers, Jiri
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > _______________________________________________
>> > > > Users mailing list -- users@ovirt.org
>> <mailto:users@ovirt.org>
>> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>
>> <mailto:users@ovirt.org <mailto:users@ovirt.org>
>> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>>
>> > > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org>
>> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>
>> > > <mailto:users-leave@ovirt.org
>> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org
>> <mailto:users-leave@ovirt.org>>>
>> > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>
>> > > <https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>>
>> > > > oVirt Code of Conduct:
>> > > https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>
>> > > <https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>>
>> > > > List Archives:
>> > >
>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>
>> > >
>> > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>>
>> > > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org
>> <mailto:users@ovirt.org>>
>> > <mailto:users@ovirt.org <mailto:users@ovirt.org>
>> <mailto:users@ovirt.org <mailto:users@ovirt.org>>>
>> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org>
>> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>
>> > > <mailto:users-leave@ovirt.org
>> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org
>> <mailto:users-leave@ovirt.org>>>
>> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>
>> > > <https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>>
>> > > oVirt Code of Conduct:
>> > > https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>
>> > > <https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>>
>> > > List Archives:
>> > >
>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>
>> > >
>> > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>>
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks,
>> > > Gobinda
>> >
>> >
>> > _______________________________________________
>> > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org
>> <mailto:users@ovirt.org>>
>> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org>
>> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>
>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> <https://www.ovirt.org/site/privacy-policy/>
>> > oVirt Code of Conduct:
>> > https://www.ovirt.org/community/about/community-guidelines/
>> <https://www.ovirt.org/community/about/community-guidelines/>
>> > List Archives:
>> >
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/
>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/>
>> >
>>
>>
>>
>>
>>
>> --
>>
>>
>> Raz Tamir
>> Manager, RHV QE
>
>
>
>
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HGS5KF22YQD3XSNXHSQARGZHUVDXXZAQ/
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOTD2GXB7XXQVQE67CNGU2HN5TCZPJUZ/