On Wed, Apr 20, 2016 at 10:42 PM, Clint Boggio <clint@theboggios.com> wrote:
I grepped out the engine logs until i found reference to the illegal disk in question. The log indicates that the image has been flagged illegal because the original disk is no longer present. So it is very possible that the backup script, somehow through the miracle of 1's and 0's deleted the base VM disks.
###################### # BEGIN ######################
2016-03-27 18:57:41,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] START, CreateSnapshotVDSCommand( CreateSnapshotVDSCommandParameters:{runAsync='true', storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6', ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c- 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageSizeInBytes='268435456000', volumeFormat='COW', newImageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6', newImageDescription='', imageInitialSizeInBytes='0', imageId='d538e0ef- 2f55-4c74-b8f1-8900fd6b814b', sourceImageGroupId='ad486d26-4594-4d16- a402-68b45d82078a'}), log id: 7648bbd2 2016-03-27 18:57:42,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] FINISH, CreateSnapshotVDSCommand, return: e87e0c7c-4f6f-45e9-90ca-cf34617da3f6, log id: 7648bbd2 2016-03-27 18:58:24,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand ] (org.ovirt.thread.pool-8-thread-20) [30680dce] START, GetImageInfoVDSCommand( GetImageInfoVDSCommandParameters:{runAsync='true', storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6', ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c- 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'}), log id: 6d2d19f6 2016-03-28 14:14:49,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-3) [718f57] START, MergeVDSCommand(HostName = KVM04, MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446- a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232', storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6', storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='e87e0c7c- 4f6f-45e9-90ca-cf34617da3f6', baseImageId='6e008200-3c21-4285-96b8- 07c29c0cb72c', topImageId='d538e0ef-2f55-4c74-b8f1-8900fd6b814b', bandwidth='0'}), log id: 2cc2db4 2016-03-28 17:01:22,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma nd] (default task-77) [410b6a44] START, CreateSnapshotVDSCommand( CreateSnapshotVDSCommandParameters:{runAsync='true', storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6', ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c- 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageSizeInBytes='268435456000', volumeFormat='COW', newImageId='919d6991-43e4-4f26-868e-031a01011191', newImageDescription='', imageInitialSizeInBytes='0', imageId='e87e0c7c- 4f6f-45e9-90ca-cf34617da3f6', sourceImageGroupId='ad486d26-4594-4d16- a402-68b45d82078a'}), log id: 4ed3e9ca 2016-03-28 18:36:28,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-1) [6911a44f] START, MergeVDSCommand(HostName = KVM04, MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446- a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232', storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6', storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='919d6991- 43e4-4f26-868e-031a01011191', baseImageId='e87e0c7c-4f6f-45e9-90ca- cf34617da3f6', topImageId='919d6991-43e4-4f26-868e-031a01011191', bandwidth='0'}), log id: d09cb70 2016-03-28 18:39:53,773 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-99) [6911a44f] Merge command has completed for images 'e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'..'919d6991- 43e4-4f26-868e-031a01011191' 2016-03-28 18:41:23,003 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-44) [a00e3a8] Merging of snapshot 'a1b3c247-2c6f-4731-9e62-c15f5cfb9a72' images 'e87e0c7c-4f6f-45e9-90ca- cf34617da3f6'..'919d6991-43e4-4f26-868e-031a01011191' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation.
This is live merge failure - we have a similar bug causing this, and I have reproduced similar failure today. This may be the same bug, we must inspect the logs to be sure. Typically the merge succeeds in vdsm side, but from some reason the engine fail to detect the merge success and mark the volumes as illegal.
################## # END ##################
If that's the case, then why (how) are the afflicted machines that have not been rebooted still running without thier backing disks ?
It is possible to unlink a file while it is being used by another process. The directory entry is removed so another process cannot access the file, but processes that already opened the file are not affected. But this looks like the live merge issue, not like your backup script trying too hard.
I can upload the logs and a copy of the backup script. Do you all have a repository you'd like meto upload to ? Let me know and i'll upload them right now.
Please file a bug and attach the files there. Nir
On Wed, 2016-04-20 at 13:33 -0400, users-request@ovirt.org wrote:
Send Users mailing list submissions to users@ovirt.org
To subscribe or unsubscribe via the World Wide Web, visit http://lists.ovirt.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-request@ovirt.org
You can reach the person managing the list at users-owner@ovirt.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..."
Today's Topics:
1. Re: vhostmd vdsm-hook (Ars?ne Gschwind) 2. Re: Disks Illegal State (Nir Soffer)
------------------------------------------------------------------- ---
Message: 1 Date: Wed, 20 Apr 2016 19:09:39 +0200 From: Ars?ne Gschwind <arsene.gschwind@unibas.ch> To: Simon Barrett <Simon.Barrett@tradingscreen.com>, "users@ov irt.org" <users@ovirt.org> Subject: Re: [ovirt-users] vhostmd vdsm-hook Message-ID: <5717B7D3.2070502@unibas.ch> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
I've never tried with 2 disks but I will assume that the next free available disk will be used by the vdsm hook and the vm-dump-metrics cmd will check the kind of disk. Let me know if you give a try....
thanks, Ars?ne
On 04/19/2016 02:43 PM, Simon Barrett wrote:
Thanks again but how does that work when a VM is configured to have more than one disk?
If I have a VM with a /dev/vda disk and a /dev/vdb disk, when I turn the vhostmd hook on the vm metric device gets created as /dev/vdb and the original /dev/vdb disk gets bumped to /dev/vdc.
Is that expected behavior? Will that not cause problems?
Thanks,
Simon
*From:*Ars?ne Gschwind [mailto:arsene.gschwind@unibas.ch] *Sent:* Tuesday, 19 April, 2016 13:06 *To:* Simon Barrett <Simon.Barrett@tradingscreen.com>; users@ovirt. org *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
The metric information are available on this additional disk /dev/vdb. You may install the package vm-dump-metrics and use the command vm-dump-metrics which will display all metrics in an xml format.
Ars?ne
On 04/19/2016 10:48 AM, Simon Barrett wrote:
Thanks Ars?ne,
I have vhostmd running on the ovirt node and have set the sap_agent to true on the VM configuration. I also stopped and started the VM to ensure that the config change took effect.
On the oVirt node I see the vhostmd running and see the following entry in the qemu-kvm output:
drive file=/dev/shm/vhostmd0,if=none,id=drive-virtio- disk701,readonly=on,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio- disk701,id=virtio-disk701
The part I wasn?t quite understanding was how this presented itself on the VM but I now see a new disk device ?/dev/vdb?. If I cat the contents of /dev/vdb I now see the information that is provided from the ovirt node, which is great news and very useful.
Thanks for your help.
Simon
*From:*users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> [mailto:users-bounces@ovirt.org] *On Behalf Of *Ars?ne Gschwind *Sent:* Monday, 18 April, 2016 16:03 *To:* users@ovirt.org <mailto:users@ovirt.org> *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
Hi Simon,
You will need to have vhostmd running on the oVirt node and set the "sap_agent" custom property for the vm as you may see on the screenshot.
sap_agent
Ars?ne
On 04/15/2016 12:15 PM, Simon Barrett wrote:
I?m trying to use the vhostmd vdsm host to access ovirt node metrics from within a VM. Vhostmd is running and updating the /dev/shm/vhostmd0 on the ovirt node.
The part I?m stuck on is: ?This disk image is exported read-only to guests. Guests can read the disk image to see metrics? from http://www.ovirt.org/develop/developer-guide/vdsm/hook/vhos tmd/
Does the hook do this by default? I don?t see any new read-only device mounted in the guest. Is there additional work I need to do to mount this and access the data from within the guest?
Many thanks,
Simon
_______________________________________________
Users mailing list
Users@ovirt.org <mailto:Users@ovirt.org>