Hi all, I have a problem deleting a template from admin portal. In file /var/log/vdsm/vdsm.log (on SPM hypervisor) I got: jsonrpc.Executor/4::ERROR::2016-04-27 10:19:57,122::hsm::1518::Storage.HSM::(deleteImage) Empty or not found image <disk uuid> in SD <storage domain UUID> [...] Looking in the (data) storage domain the disk with that UUID doesn't exists. It seems I reached an inconsistent state between engine database and images on disk. Is there a (safe) way to rebuild a consistent situation? Maybe deleting entries from database? My setup is based: manager RHEV 3.5.8-0.1.el6ev hypervisors: RHEV Hypervisor - 7.2 - 20160328.0.el7ev Thanx in advance, Giulio
Hi, Can you please attach the engine and vdsm logs? Also, can you describe your setup and the steps that reproduced this error? Thanks, Idan On Wed, Apr 27, 2016 at 1:47 PM, Giulio Casella <giulio@di.unimi.it> wrote:
Hi all, I have a problem deleting a template from admin portal. In file /var/log/vdsm/vdsm.log (on SPM hypervisor) I got:
jsonrpc.Executor/4::ERROR::2016-04-27 10:19:57,122::hsm::1518::Storage.HSM::(deleteImage) Empty or not found image <disk uuid> in SD <storage domain UUID> [...]
Looking in the (data) storage domain the disk with that UUID doesn't exists. It seems I reached an inconsistent state between engine database and images on disk.
Is there a (safe) way to rebuild a consistent situation? Maybe deleting entries from database?
My setup is based: manager RHEV 3.5.8-0.1.el6ev hypervisors: RHEV Hypervisor - 7.2 - 20160328.0.el7ev
Thanx in advance, Giulio
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Idan, you can find attached some snippet of my logs (vdsm.log from SPM host and engine.log from manager). Anyway I think logs are clear: there's no template on disk, only on engine database. My setup is composed of 3 datacenters (with clusters of 4, 4, and 6 hosts respectively). I have about a hundred templates, and about 1500 virtual machines. The step to reproduce is quite simple: in admin portal right click on the "damaged" template, remove. Other templates deletion work fine. Thanks, Giulio Il 01/05/2016 08:37, Idan Shaby ha scritto:
Hi,
Can you please attach the engine and vdsm logs? Also, can you describe your setup and the steps that reproduced this error?
Thanks, Idan
On Wed, Apr 27, 2016 at 1:47 PM, Giulio Casella <giulio@di.unimi.it <mailto:giulio@di.unimi.it>> wrote:
Hi all, I have a problem deleting a template from admin portal. In file /var/log/vdsm/vdsm.log (on SPM hypervisor) I got:
jsonrpc.Executor/4::ERROR::2016-04-27 10:19:57,122::hsm::1518::Storage.HSM::(deleteImage) Empty or not found image <disk uuid> in SD <storage domain UUID> [...]
Looking in the (data) storage domain the disk with that UUID doesn't exists. It seems I reached an inconsistent state between engine database and images on disk.
Is there a (safe) way to rebuild a consistent situation? Maybe deleting entries from database?
My setup is based: manager RHEV 3.5.8-0.1.el6ev hypervisors: RHEV Hypervisor - 7.2 - 20160328.0.el7ev
Thanx in advance, Giulio
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
-- Giulio Casella giulio at di.unimi.it System and network manager Computer Science Dept. - University of Milano
Hi Giulio, Indeed it seems that there's an inconsistency between the storage and the database. Somehow the template's disk was removed from the storage. Your logs don't tell how and when. This is what needs to be done in order to get rid of the stale entries - run in the database: Delete from vm_device where device_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from images where image_group_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from base_disks where disk_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from vm_static where vm_guid = '6ec76d42-98a6-4094-a76c-af1b639c5b30'; If it happens again, please file a BZ with the full logs of engine and vdsm, so we can investigate this issue. Feel free to ask anything if you got further questions. Regards Idan On Mon, May 2, 2016 at 2:36 PM, Giulio Casella <giulio@di.unimi.it> wrote:
Hi Idan, you can find attached some snippet of my logs (vdsm.log from SPM host and engine.log from manager). Anyway I think logs are clear: there's no template on disk, only on engine database.
My setup is composed of 3 datacenters (with clusters of 4, 4, and 6 hosts respectively). I have about a hundred templates, and about 1500 virtual machines.
The step to reproduce is quite simple: in admin portal right click on the "damaged" template, remove. Other templates deletion work fine.
Thanks, Giulio
Il 01/05/2016 08:37, Idan Shaby ha scritto:
Hi,
Can you please attach the engine and vdsm logs? Also, can you describe your setup and the steps that reproduced this error?
Thanks, Idan
On Wed, Apr 27, 2016 at 1:47 PM, Giulio Casella <giulio@di.unimi.it <mailto:giulio@di.unimi.it>> wrote:
Hi all, I have a problem deleting a template from admin portal. In file /var/log/vdsm/vdsm.log (on SPM hypervisor) I got:
jsonrpc.Executor/4::ERROR::2016-04-27 10:19:57,122::hsm::1518::Storage.HSM::(deleteImage) Empty or not found image <disk uuid> in SD <storage domain UUID> [...]
Looking in the (data) storage domain the disk with that UUID doesn't exists. It seems I reached an inconsistent state between engine database and images on disk.
Is there a (safe) way to rebuild a consistent situation? Maybe deleting entries from database?
My setup is based: manager RHEV 3.5.8-0.1.el6ev hypervisors: RHEV Hypervisor - 7.2 - 20160328.0.el7ev
Thanx in advance, Giulio
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
-- Giulio Casella giulio at di.unimi.it System and network manager Computer Science Dept. - University of Milano
Thank you Idan, those query solved my problem. I really don't know when it happened, and with thousand VM running it's not so easy to dig into logs. Thank you, Giulio Il 02/05/2016 15:39, Idan Shaby ha scritto:
Hi Giulio,
Indeed it seems that there's an inconsistency between the storage and the database. Somehow the template's disk was removed from the storage. Your logs don't tell how and when.
This is what needs to be done in order to get rid of the stale entries - run in the database:
Delete from vm_device where device_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from images where image_group_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from base_disks where disk_id = '6949d47a-0d38-468e-ad4e-670372841174'; Delete from vm_static where vm_guid = '6ec76d42-98a6-4094-a76c-af1b639c5b30';
If it happens again, please file a BZ with the full logs of engine and vdsm, so we can investigate this issue. Feel free to ask anything if you got further questions.
Regards Idan
On Mon, May 2, 2016 at 2:36 PM, Giulio Casella <giulio@di.unimi.it <mailto:giulio@di.unimi.it>> wrote:
Hi Idan, you can find attached some snippet of my logs (vdsm.log from SPM host and engine.log from manager). Anyway I think logs are clear: there's no template on disk, only on engine database.
My setup is composed of 3 datacenters (with clusters of 4, 4, and 6 hosts respectively). I have about a hundred templates, and about 1500 virtual machines.
The step to reproduce is quite simple: in admin portal right click on the "damaged" template, remove. Other templates deletion work fine.
Thanks, Giulio
Il 01/05/2016 08:37, Idan Shaby ha scritto:
Hi,
Can you please attach the engine and vdsm logs? Also, can you describe your setup and the steps that reproduced this error?
Thanks, Idan
On Wed, Apr 27, 2016 at 1:47 PM, Giulio Casella <giulio@di.unimi.it <mailto:giulio@di.unimi.it> <mailto:giulio@di.unimi.it <mailto:giulio@di.unimi.it>>> wrote:
Hi all, I have a problem deleting a template from admin portal. In file /var/log/vdsm/vdsm.log (on SPM hypervisor) I got:
jsonrpc.Executor/4::ERROR::2016-04-27 10:19:57,122::hsm::1518::Storage.HSM::(deleteImage) Empty or not found image <disk uuid> in SD <storage domain UUID> [...]
Looking in the (data) storage domain the disk with that UUID doesn't exists. It seems I reached an inconsistent state between engine database and images on disk.
Is there a (safe) way to rebuild a consistent situation? Maybe deleting entries from database?
My setup is based: manager RHEV 3.5.8-0.1.el6ev hypervisors: RHEV Hypervisor - 7.2 - 20160328.0.el7ev
Thanx in advance, Giulio
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> http://lists.ovirt.org/mailman/listinfo/users
-- Giulio Casella giulio at di.unimi.it <http://di.unimi.it> System and network manager Computer Science Dept. - University of Milano
participants (2)
-
Giulio Casella -
Idan Shaby