8:16 a.m.
Hello,
I have a task that seems stuck in webadmin gui, in the sens tha I have
"Tasks(1)" listed
The task is
Restoring VM Snapshot Active VM before the preview of VM snaptest
and the VM is powered down.
Screenshot of expanded steps of task, that actually seem all completed, is
here:
https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?us...
Any hint on what to do to clean things? I'm on oVirt 4.1.6.2-1.el7.centos
and I would like to clean before upgrading to 4.1.7.
Thanks
Gianluca
8:34 a.m.
oVirt upgrade to 4.1.7 will probably cleanup this stale task. However,
if you want to do it before upgrading, run this command:
PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh
-t all -u engine
Note that unlock_entity.sh has many flags and this is just an example
(should clean all stale tasks).
You can find the PGPASSWORD value in the
/etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of 4.2
you won't need to supply credentials anymore [1].
Regards,
Nicolás
[1]: https://gerrit.ovirt.org/82615
El 2017-11-10 14:16, Gianluca Cecchi escribió:
Hello,
I have a task that seems stuck in webadmin gui, in the sens tha I have
"Tasks(1)" listed
The task is Restoring VM Snapshot Active VM before the preview of
VM snaptest
and the VM is powered down.
Screenshot of expanded steps of task, that actually seem all
completed, is here:
https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?us...
[1]
Any hint on what to do to clean things? I'm on oVirt
4.1.6.2-1.el7.centos and I would like to clean before upgrading to
4.1.7.
Thanks
Gianluca
Links:
------
[1]
https://drive.google.com/file/d/1bfl_gEfVotIrxGC9TDzPHPCeRub41mUa/view?us...
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:41 a.m.
On Fri, Nov 10, 2017 at 3:34 PM, <nicolas(a)devels.es> wrote:
oVirt upgrade to 4.1.7 will probably cleanup this stale task.
However, if
you want to do it before upgrading, run this command:
PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh
-t all -u engine
Note that unlock_entity.sh has many flags and this is just an example
(should clean all stale tasks).
You can find the PGPASSWORD value in the
/etc/ovirt-engine/engine.conf.d/10-setup-database.conf
file. As of 4.2 you won't need to supply credentials anymore [1].
Regards,
Nicolás
It seems it didn't work as expected.
I got this at command line output
"
select fn_db_unlock_all();
INSERT 0 1
unlock all completed successfully.
"
But the task remains in webadmin gui and I got an alert message in alert
section, of this type
"
/usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user root
run manually unlock_entity script on entity [type,id] [all,] with db user
engine
"
Gianluca
8:48 a.m.
El 2017-11-10 14:41, Gianluca Cecchi escribió:
On Fri, Nov 10, 2017 at 3:34 PM, <nicolas(a)devels.es> wrote:
> oVirt upgrade to 4.1.7 will probably cleanup this stale task.
> However, if you want to do it before upgrading, run this command:
>
> PGPASSWORD=...
> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u
> engine
>
> Note that unlock_entity.sh has many flags and this is just an
> example (should clean all stale tasks).
>
> You can find the PGPASSWORD value in the
> /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of
> 4.2 you won't need to supply credentials anymore [1].
>
> Regards,
>
> Nicolás
It seems it didn't work as expected.
I got this at command line output
"
select fn_db_unlock_all();
INSERT 0 1
unlock all completed successfully.
"
This is expected.
But the task remains in webadmin gui and I got an alert message in
alert section, of this type
"
/usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user
root run manually unlock_entity script on entity [type,id] [all,] with
db user engine
"
I've seen this behavior too. IIRC the stale cleaning was not instant, it
took some time to be applied.
Regards.
Gianluca
9:02 a.m.
You could also go database diving. I had an issue where I tried to import
a VM from my export domain and it just got hung. I tried running the
unlock_entity script but it just kept failing. It sat there for months
stuck, and found
http://lists.ovirt.org/pipermail/users/2015-April/032346.html
Of course deleting something from your database is quite permanent. I would
wait and upgrade to 4.1.7, but something like the below should work. But
probably not recommended
Drop into postgres
psql -d engine -U postgres
List your tasks and grab the job_id
select * from job order by start_time desc;
select DeleteJob('8424f7a9-2a4c-4567-b528-45bbc1c2534f');
Where the string here is the job ID
On Fri, Nov 10, 2017 at 9:48 AM, <nicolas(a)devels.es> wrote:
El 2017-11-10 14:41, Gianluca Cecchi escribió:
> On Fri, Nov 10, 2017 at 3:34 PM, <nicolas(a)devels.es> wrote:
>
> oVirt upgrade to 4.1.7 will probably cleanup this stale task.
>> However, if you want to do it before upgrading, run this command:
>>
>> PGPASSWORD=...
>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -u
>> engine
>>
>> Note that unlock_entity.sh has many flags and this is just an
>> example (should clean all stale tasks).
>>
>> You can find the PGPASSWORD value in the
>> /etc/ovirt-engine/engine.conf.d/10-setup-database.conf file. As of
>> 4.2 you won't need to supply credentials anymore [1].
>>
>> Regards,
>>
>> Nicolás
>>
>
> It seems it didn't work as expected.
> I got this at command line output
>
> "
>
> select fn_db_unlock_all();
>
>
> INSERT 0 1
> unlock all completed successfully.
> "
>
>
This is expected.
But the task remains in webadmin gui and I got an alert message in
> alert section, of this type
> "
> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh : System user
> root run manually unlock_entity script on entity [type,id] [all,] with
> db user engine
> "
>
>
I've seen this behavior too. IIRC the stale cleaning was not instant, it
took some time to be applied.
Regards.
Gianluca
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:01 a.m.
On Fri, Nov 10, 2017 at 3:48 PM, <nicolas(a)devels.es> wrote:
>
I've seen this behavior too. IIRC the stale cleaning was not instant, it
took some time to be applied.
Regards.
Gianluca
>
Confirmed.
Quite soon after the command I saw that the status of the "Current"
snapshot line changed from Locked (it was so since 8/11) to OK, but the
task remained at least for half an hour.
Now, after about one hour and a half I connected again to the web admin gui
and I see 0 Tasks, so the problem has been resolved.
Thanks again,
Gianluca
10:01 a.m.
Hello,
it seems the same lock (with the same date) has come again in the list of
tasks.
Now I have two things in "supposed running" tasks:
- the "Restoring VM Snapshot Active VM before the preview of VM snaptest"
of 8/11
- the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex" (now
running since half an hour for a vm that has only one 90Gb disk on SAN FC
storage domain)
It was not me to run the "create snapshot" command today, so I'm not sure
if it was the snapshot creation job responsible of letting come out again
the former hung task... or if it was already there before...
Any further hint?
How can I manually verify the status of the snapshot creation task?
Gianluca
On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Fri, Nov 10, 2017 at 3:48 PM, <nicolas(a)devels.es> wrote:
>
>>
> I've seen this behavior too. IIRC the stale cleaning was not instant, it
> took some time to be applied.
>
> Regards.
>
> Gianluca
>>
>
Confirmed.
Quite soon after the command I saw that the status of the "Current"
snapshot line changed from Locked (it was so since 8/11) to OK, but the
task remained at least for half an hour.
Now, after about one hour and a half I connected again to the web admin
gui and I see 0 Tasks, so the problem has been resolved.
Thanks again,
Gianluca
10:28 a.m.
Is the VM showing as locked? You could try to manually unlocking it. I
did this in a test environment a while back with some success after a vm
was stuck in a "locked" state. Of course the gui and engine should handle
most of this for you, manually mucking around the internal DB can cause
some pretty serious issues if you are not careful...
sudo su postgres
psql -d engine -U postgres
SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*';
This should return a string such as: "0ec20854-e1ca-4e49-be87-a6cd36d40c18"
Reset the lock:
update vm_dynamic SET status=0 where vm_guid='
0ec20854-e1ca-4e49-be87-a6cd36d40c18';
On Mon, Nov 13, 2017 at 11:01 AM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com
wrote:
> Hello,
>
> it seems the same lock (with the same date) has come again in the list of
> tasks.
> Now I have two things in "supposed running" tasks:
> - the "Restoring VM Snapshot Active VM before the preview of VM snaptest"
> of 8/11
> - the "Creating VM Snapshot 13-11-2017-16-30-Danilo for VM apex" (now
> running since half an hour for a vm that has only one 90Gb disk on SAN FC
> storage domain)
>
> It was not me to run the "create snapshot" command today, so I'm not
sure
> if it was the snapshot creation job responsible of letting come out again
> the former hung task... or if it was already there before...
> Any further hint?
> How can I manually verify the status of the snapshot creation task?
>
> Gianluca
>
>
> On Fri, Nov 10, 2017 at 5:01 PM, Gianluca Cecchi <
> gianluca.cecchi(a)gmail.com wrote:
>
>> On Fri, Nov 10, 2017 at 3:48 PM, <nicolas(a)devels.es wrote:
>>
>>>
>>>>
>>> I've seen this behavior too. IIRC the stale cleaning was not instant, it
>>> took some time to be applied.
>>>
>>> Regards.
>>>
>>> Gianluca
>>>>
>>>
>> Confirmed.
>> Quite soon after the command I saw that the status of the "Current"
>> snapshot line changed from Locked (it was so since 8/11) to OK, but the
>> task remained at least for half an hour.
>> Now, after about one hour and a half I connected again to the web admin
>> gui and I see 0 Tasks, so the problem has been resolved.
>>
>> Thanks again,
>> Gianluca
>>
>>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
12:01 p.m.
On Mon, Nov 13, 2017 at 5:28 PM, Wesley Stewart <wstewart3(a)gmail.com> wrote:
Is the VM showing as locked? You could try to manually unlocking it.
I
did this in a test environment a while back with some success after a vm
was stuck in a "locked" state. Of course the gui and engine should handle
most of this for you, manually mucking around the internal DB can cause
some pretty serious issues if you are not careful...
sudo su postgres
psql -d engine -U postgres
SELECT vm_guid, vm_name FROM vm_static WHERE vm_name='*VM_Name_Here*';
This should return a string such as: "0ec20854-e1ca-4e49-be87-
a6cd36d40c18"
Reset the lock:
update vm_dynamic SET status=0 where vm_guid='0ec20854-e1ca-4e49-
be87-a6cd36d40c18';
I'd prefer to avoid direct manage of the rdbms...
As it happened on Friday, after running again the command
PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
all -u engine
now the gui after some time shows no active task
but the vm still shows as locked and in the event pane I only have the
message
Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated
by
user1@internal-authz.
but no message about its correct completion
This is my disks' subtab situation for the vm, where it seems all ok:
https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7cTG/view?us...
And this the snapshots' subtab where it seems all ok:
https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsYM4mAa/view?us...
But the VM has the lock symbol in side of itself.
Any guidance appreciated, both to solve the problem and to understand if
the snapshot is consistent, as a user should to changes to the VM with a
chance to have to come back
Gianluca
12:06 p.m.
On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
I'd prefer to avoid direct manage of the rdbms...
As it happened on Friday, after running again the command
PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
all -u engine
now the gui after some time shows no active task
but the vm still shows as locked and in the event pane I only have the
message
Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was initiated
by
user1@internal-authz.
but no message about its correct completion
This is my disks' subtab situation for the vm, where it seems all ok:
https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmUR7
cTG/view?usp=sharing
And this the snapshots' subtab where it seems all ok:
https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_
HsYM4mAa/view?usp=sharing
But the VM has the lock symbol in side of itself.
Any guidance appreciated, both to solve the problem and to understand if
the snapshot is consistent, as a user should to changes to the VM with a
chance to have to come back
Gianluca
I have just verified that if I run a new task (in my case has been the
removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the
deletion task completing with success, while the 2 stale tasks appearing
again in the task pane where they disappeared
see here:
https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_ElsIY/view?us...
1:53 p.m.
For cleaning these "Tasks" entries (which are actually Jobs in the DB), you
will need to use taskcleaner.sh.
The unlock_entity.sh is for removing locks on entities.
Before running the task cleaner, you need to be sure you actually don't
have any tasks running in the SPM.
Regarding the snapshot creation on 'apex", can you provide the logs around
the time of the operation?
On Mon, Nov 13, 2017 at 8:06 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Mon, Nov 13, 2017 at 7:01 PM, Gianluca Cecchi <
gianluca.cecchi(a)gmail.com> wrote:
>
>
>
> I'd prefer to avoid direct manage of the rdbms...
>
> As it happened on Friday, after running again the command
>
> PGPASSWORD=... /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t
> all -u engine
>
> now the gui after some time shows no active task
>
> but the vm still shows as locked and in the event pane I only have the
> message
> Snapshot '13-11-2017-16-30-Danilo' creation for VM 'apex' was
initiated
> by user1@internal-authz.
> but no message about its correct completion
>
> This is my disks' subtab situation for the vm, where it seems all ok:
> https://drive.google.com/file/d/1tF3VdA8gzMVvtn2x4HTKOmltRmU
> R7cTG/view?usp=sharing
>
> And this the snapshots' subtab where it seems all ok:
> https://drive.google.com/file/d/1qbEhdS6eOZL-f8y5xCnuc_4_HsY
> M4mAa/view?usp=sharing
>
> But the VM has the lock symbol in side of itself.
> Any guidance appreciated, both to solve the problem and to understand if
> the snapshot is consistent, as a user should to changes to the VM with a
> chance to have to come back
>
> Gianluca
>
I have just verified that if I run a new task (in my case has been the
removal of the snaptest VM), suddenly I see from 0 to 3 tasks, with the
deletion task completing with success, while the 2 stale tasks appearing
again in the task pane where they disappeared
see here:
https://drive.google.com/file/d/1xcbJsI02iSEAHYEiFYkRWvz9SW_
ElsIY/view?usp=sharing
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:37 p.m.
On Mon, Nov 13, 2017 at 8:53 PM, Fred Rolland <frolland(a)redhat.com> wrote:
For cleaning these "Tasks" entries (which are actually Jobs
in the DB),
you will need to use taskcleaner.sh.
The unlock_entity.sh is for removing locks on entities.
ok. Thanks for clarifying the concept.
But from the inline help of taskcleaner.sh it is not so clear to me which
option makes it display vs remove...
Perhaps without options it only display? or with "-T" it first displays
asking if I want to remove "all comands that have running tasks"?
How can I only display?
Before running the task cleaner, you need to be sure you actually
don't
have any tasks running in the SPM.
How can I check this?
Is still valid this command I found on a previous (around 2014) thread and
executed on the SPM (as shown in web admin portal) host right now?
[root@ov300 vdsm]# vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
[root@ov300 vdsm]#
Does it mean no task?
Regarding the snapshot creation on 'apex", can you provide
the logs around
the time of the operation?
engine.log is here:
https://drive.google.com/file/d/1nD0p90ZvwphrEafgKZkCCquNu_
vVY8p8/view?usp=sharing
vdsm log of the SPM host in .xz format here:
https://drive.google.com/file/d/1Zzk04W6400wPh1sM8O4WyFDAGQaqa
UZK/view?usp=sharing
Thanks,
Gianluca
5 a.m.
In the mean time, as I had to give an answer for the snapshotted VM, I
decided to follow one of the suggestions to run engine-setup and so also to
pass my engine from 4.1.6 to 4.1.7.
And indeed the 2 stale tasks have been cleaned.
The lock symbol has gone away from the apex VM too.
Probably the steps solving the problems were these during engine-setup:
[ INFO ] Cleaning async tasks and compensations
[ INFO ] Unlocking existing entities
Does this mean that in general I can also run engine-setup without
upgrading at all? Is the clean part run also in that case or only during
effective upgrades?
I initiated a clone of the taken snapshot on apex VM and it seems to go
correctly and in task pane I see only that task and no more.
In SPM now I have indeed
[root@ov300 ~]# vdsClient -s 0 getAllTasksStatuses
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus':
{'20fa401f-b6f8-43f5-b0fd-6767d46e2335': {'message': 'running job 1 of
1',
'code': 0, 'taskID': '20fa401f-b6f8-43f5-b0fd-6767d46e2335',
'taskResult':
'', 'taskState': 'running'}}}
[root@ov300 ~]#
It should take about half an hour to complete and I will see.
Anyway in my opinion it would be nice to have some more in deep
documentation about how to run taskcleaner.sh or simply officially say to
leave it to developers if this is the intended case (or to Red Hat support
in case of RHEV usage)
Cheers,
Gianluca
2568
days inactive
2572
days old
12 comments
4 participants
participants (4)
-
Fred Rolland -
Gianluca Cecchi -
nicolas＠devels.es -
Wesley Stewart