[ovirt-users] Users Digest, Vol 55, Issue 155
Clint Boggio
clint at theboggios.com
Wed Apr 20 22:10:46 UTC 2016
Bug is filed.
[Bug 1329000] Snapshot Images Flagged as "ILLEGAL" After Backup Script Is Run
Is there a technique for recovery from this condition or should I back up the data on the VM's that are afflicted and still running and start over ?
Thank you all for your help.
> On Apr 20, 2016, at 3:19 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>
>> On Wed, Apr 20, 2016 at 10:42 PM, Clint Boggio <clint at theboggios.com> wrote:
>> I grepped out the engine logs until i found reference to the illegal
>> disk in question. The log indicates that the image has been flagged
>> illegal because the original disk is no longer present. So it is very
>> possible that the backup script, somehow through the miracle of 1's and
>> 0's deleted the base VM disks.
>>
>> ######################
>> # BEGIN
>> ######################
>>
>> 2016-03-27 18:57:41,769
>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>> nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] START,
>> CreateSnapshotVDSCommand(
>> CreateSnapshotVDSCommandParameters:{runAsync='true',
>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>> imageSizeInBytes='268435456000', volumeFormat='COW',
>> newImageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6',
>> newImageDescription='', imageInitialSizeInBytes='0', imageId='d538e0ef-
>> 2f55-4c74-b8f1-8900fd6b814b', sourceImageGroupId='ad486d26-4594-4d16-
>> a402-68b45d82078a'}), log id: 7648bbd2
>> 2016-03-27 18:57:42,835
>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>> nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] FINISH,
>> CreateSnapshotVDSCommand, return: e87e0c7c-4f6f-45e9-90ca-cf34617da3f6,
>> log id: 7648bbd2
>> 2016-03-27 18:58:24,395
>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand
>> ] (org.ovirt.thread.pool-8-thread-20) [30680dce] START,
>> GetImageInfoVDSCommand(
>> GetImageInfoVDSCommandParameters:{runAsync='true',
>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>> imageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'}), log id: 6d2d19f6
>> 2016-03-28 14:14:49,454
>> INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
>> (pool-7-thread-3) [718f57] START, MergeVDSCommand(HostName = KVM04,
>> MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446-
>> a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232',
>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>> storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f',
>> imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='e87e0c7c-
>> 4f6f-45e9-90ca-cf34617da3f6', baseImageId='6e008200-3c21-4285-96b8-
>> 07c29c0cb72c', topImageId='d538e0ef-2f55-4c74-b8f1-8900fd6b814b',
>> bandwidth='0'}), log id: 2cc2db4
>> 2016-03-28 17:01:22,368
>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>> nd] (default task-77) [410b6a44] START, CreateSnapshotVDSCommand(
>> CreateSnapshotVDSCommandParameters:{runAsync='true',
>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>> imageSizeInBytes='268435456000', volumeFormat='COW',
>> newImageId='919d6991-43e4-4f26-868e-031a01011191',
>> newImageDescription='', imageInitialSizeInBytes='0', imageId='e87e0c7c-
>> 4f6f-45e9-90ca-cf34617da3f6', sourceImageGroupId='ad486d26-4594-4d16-
>> a402-68b45d82078a'}), log id: 4ed3e9ca
>> 2016-03-28 18:36:28,404
>> INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
>> (pool-7-thread-1) [6911a44f] START, MergeVDSCommand(HostName = KVM04,
>> MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446-
>> a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232',
>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>> storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f',
>> imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='919d6991-
>> 43e4-4f26-868e-031a01011191', baseImageId='e87e0c7c-4f6f-45e9-90ca-
>> cf34617da3f6', topImageId='919d6991-43e4-4f26-868e-031a01011191',
>> bandwidth='0'}), log id: d09cb70
>> 2016-03-28 18:39:53,773
>> INFO [org.ovirt.engine.core.bll.MergeCommandCallback]
>> (DefaultQuartzScheduler_Worker-99) [6911a44f] Merge command has
>> completed for images 'e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'..'919d6991-
>> 43e4-4f26-868e-031a01011191'
>> 2016-03-28 18:41:23,003 ERROR
>> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
>> (DefaultQuartzScheduler_Worker-44) [a00e3a8] Merging of snapshot
>> 'a1b3c247-2c6f-4731-9e62-c15f5cfb9a72' images 'e87e0c7c-4f6f-45e9-90ca-
>> cf34617da3f6'..'919d6991-43e4-4f26-868e-031a01011191' failed. Images
>> have been marked illegal and can no longer be previewed or reverted to.
>> Please retry Live Merge on the snapshot to complete the operation.
>
> This is live merge failure - we have a similar bug causing this, and I have
> reproduced similar failure today. This may be the same bug, we must inspect
> the logs to be sure.
>
> Typically the merge succeeds in vdsm side, but from some reason the engine
> fail to detect the merge success and mark the volumes as illegal.
>
>>
>> ##################
>> # END
>> ##################
>>
>> If that's the case, then why (how) are the afflicted machines that have
>> not been rebooted still running without thier backing disks ?
>
> It is possible to unlink a file while it is being used by another process.
> The directory entry is removed so another process cannot access the file,
> but processes that already opened the file are not affected.
>
> But this looks like the live merge issue, not like your backup script trying
> too hard.
>
>>
>> I can upload the logs and a copy of the backup script. Do you all have
>> a repository you'd like meto upload to ? Let me know and i'll upload
>> them right now.
>
> Please file a bug and attach the files there.
>
> Nir
>
>>
>>
>>
>>> On Wed, 2016-04-20 at 13:33 -0400, users-request at ovirt.org wrote:
>>> Send Users mailing list submissions to
>>> users at ovirt.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://lists.ovirt.org/mailman/listinfo/users
>>> or, via email, send a message with subject or body 'help' to
>>> users-request at ovirt.org
>>>
>>> You can reach the person managing the list at
>>> users-owner at ovirt.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Re: vhostmd vdsm-hook (Ars?ne Gschwind)
>>> 2. Re: Disks Illegal State (Nir Soffer)
>>>
>>>
>>> -------------------------------------------------------------------
>>> ---
>>>
>>> Message: 1
>>> Date: Wed, 20 Apr 2016 19:09:39 +0200
>>> From: Ars?ne Gschwind <arsene.gschwind at unibas.ch>
>>> To: Simon Barrett <Simon.Barrett at tradingscreen.com>, "users at ov
>>> irt.org"
>>> <users at ovirt.org>
>>> Subject: Re: [ovirt-users] vhostmd vdsm-hook
>>> Message-ID: <5717B7D3.2070502 at unibas.ch>
>>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>>
>>> I've never tried with 2 disks but I will assume that the next free
>>> available disk will be used by the vdsm hook and the vm-dump-metrics
>>> cmd
>>> will check the kind of disk.
>>> Let me know if you give a try....
>>>
>>> thanks,
>>> Ars?ne
>>>
>>>> On 04/19/2016 02:43 PM, Simon Barrett wrote:
>>>>
>>>>
>>>> Thanks again but how does that work when a VM is configured to
>>>> have
>>>> more than one disk?
>>>>
>>>> If I have a VM with a /dev/vda disk and a /dev/vdb disk, when I
>>>> turn
>>>> the vhostmd hook on the vm metric device gets created as /dev/vdb
>>>> and
>>>> the original /dev/vdb disk gets bumped to /dev/vdc.
>>>>
>>>> Is that expected behavior? Will that not cause problems?
>>>>
>>>> Thanks,
>>>>
>>>> Simon
>>>>
>>>> *From:*Ars?ne Gschwind [mailto:arsene.gschwind at unibas.ch]
>>>> *Sent:* Tuesday, 19 April, 2016 13:06
>>>> *To:* Simon Barrett <Simon.Barrett at tradingscreen.com>; users at ovirt.
>>>> org
>>>> *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
>>>>
>>>> The metric information are available on this additional disk
>>>> /dev/vdb.
>>>> You may install the package vm-dump-metrics and use the command
>>>> vm-dump-metrics which will display all metrics in an xml format.
>>>>
>>>> Ars?ne
>>>>
>>>> On 04/19/2016 10:48 AM, Simon Barrett wrote:
>>>>
>>>> Thanks Ars?ne,
>>>>
>>>> I have vhostmd running on the ovirt node and have set the
>>>> sap_agent to true on the VM configuration. I also stopped and
>>>> started the VM to ensure that the config change took effect.
>>>>
>>>> On the oVirt node I see the vhostmd running and see the
>>>> following
>>>> entry in the qemu-kvm output:
>>>>
>>>> drive
>>>> file=/dev/shm/vhostmd0,if=none,id=drive-virtio-
>>>> disk701,readonly=on,format=raw
>>>> -device
>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-
>>>> disk701,id=virtio-disk701
>>>>
>>>> The part I wasn?t quite understanding was how this presented
>>>> itself on the VM but I now see a new disk device ?/dev/vdb?. If
>>>> I
>>>> cat the contents of /dev/vdb I now see the information that is
>>>> provided from the ovirt node, which is great news and very
>>>> useful.
>>>>
>>>> Thanks for your help.
>>>>
>>>> Simon
>>>>
>>>> *From:*users-bounces at ovirt.org <mailto:users-bounces at ovirt.org>
>>>> [mailto:users-bounces at ovirt.org] *On Behalf Of *Ars?ne Gschwind
>>>> *Sent:* Monday, 18 April, 2016 16:03
>>>> *To:* users at ovirt.org <mailto:users at ovirt.org>
>>>> *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
>>>>
>>>> Hi Simon,
>>>>
>>>> You will need to have vhostmd running on the oVirt node and set
>>>> the "sap_agent" custom property for the vm as you may see on
>>>> the
>>>> screenshot.
>>>>
>>>> sap_agent
>>>>
>>>> Ars?ne
>>>>
>>>> On 04/15/2016 12:15 PM, Simon Barrett wrote:
>>>>
>>>> I?m trying to use the vhostmd vdsm host to access ovirt
>>>> node
>>>> metrics from within a VM. Vhostmd is running and updating
>>>> the
>>>> /dev/shm/vhostmd0 on the ovirt node.
>>>>
>>>> The part I?m stuck on is: ?This disk image is exported
>>>> read-only to guests. Guests can read the disk image to see
>>>> metrics? from
>>>> http://www.ovirt.org/develop/developer-guide/vdsm/hook/vhos
>>>> tmd/
>>>>
>>>> Does the hook do this by default? I don?t see any new
>>>> read-only device mounted in the guest. Is there additional
>>>> work I need to do to mount this and access the data from
>>>> within the guest?
>>>>
>>>> Many thanks,
>>>>
>>>> Simon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> Users mailing list
>>>>
>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <http://lists.ovirt.org/pipermail/users/attachments/20160420/d5d
>>> 7d06b/attachment-0001.html>
>>> -------------- next part --------------
>>> A non-text attachment was scrubbed...
>>> Name: not available
>>> Type: image/png
>>> Size: 6941 bytes
>>> Desc: not available
>>> URL: <http://lists.ovirt.org/pipermail/users/attachments/20160420/d5d
>>> 7d06b/attachment-0001.png>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Wed, 20 Apr 2016 20:33:06 +0300
>>> From: Nir Soffer <nsoffer at redhat.com>
>>> To: Clint Boggio <clint at theboggios.com>, Ala Hino <ahino at redhat.com>,
>>> Adam Litke <alitke at redhat.com>
>>> Cc: users <users at ovirt.org>
>>> Subject: Re: [ovirt-users] Disks Illegal State
>>> Message-ID:
>>> <CAMRbyyvfxT4h_T_qgyXNNm+OwQF7xPuwk_qOeiLP+xQ-JkkWdg at mail.gmail
>>> .com>
>>> Content-Type: text/plain; charset=UTF-8
>>>
>>> On Wed, Apr 20, 2016 at 5:34 PM, Clint Boggio <clint at theboggios.com>
>>> wrote:
>>>>
>>>> The "vdsm-tool dump-volume-chains" command on the iSCSI storage
>>>> domain
>>>> shows one disk in "ILLEGAL" state while the gui shows 8 disk images
>>>> in
>>>> the same state.
>>> Interesting - it would be useful to find the missing volume ids in
>>> engine log and
>>> understand wahy they are marked as illegal.
>>>
>>>>
>>>>
>>>> ###########################################
>>>> # BEGIN COMMAND OUTPUT
>>>> ###########################################
>>>>
>>>>
>>>>
>>>> [root at KVM01 ~]# vdsm-tool dump-volume-chains 045c7fda-ab98-4905-
>>>> 876c-
>>>> 00b5413a619f
>>>>
>>>> Images volume chains (base volume first)
>>>>
>>>> image: 477e73af-e7db-4914-81ed-89b3fbc876f7
>>>>
>>>> - c8320522-f839-472e-9707-a75f6fbe5cb6
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 882c73fc-a833-4e2e-8e6a-f714d80c0f0d
>>>>
>>>> - 689220c0-70f8-475f-98b2-6059e735cd1f
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 0ca8c49f-452e-4f61-a3fc-c4bf2711e200
>>>>
>>>> - dac06a5c-c5a8-4f82-aa8d-5c7a382da0b3
>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>> LEGAL,
>>>> type: PREALLOCATED
>>>>
>>>>
>>>> image: 0ca0b8f8-8802-46ae-a9f8-45d5647feeb7
>>>>
>>>> - 51a6de7b-b505-4c46-ae2a-25fb9faad810
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: ae6d2c62-cfbb-4765-930f-c0a0e3bc07d0
>>>>
>>>> - b2d39c7d-5b9b-498d-a955-0e99c9bd5f3c
>>>> status: OK, voltype: INTERNAL, format: COW,
>>>> legality:
>>>> LEGAL, type: SPARSE
>>>>
>>>> - bf962809-3de7-4264-8c68-6ac12d65c151
>>>> status: ILLEGAL, voltype: LEAF, format: COW,
>>>> legality:
>>>> ILLEGAL, type: SPARSE
>>> Lets check vdsm and engine log, and find when and why this disk
>>> became
>>> illegal.
>>>
>>> If this was a result of a live merge that failed while finalizing the
>>> merge
>>> on the engine side, we can safely delete the illegal volume.
>>>
>>> If this is the case, we should find a live merge for volume
>>> bf962809-3de7-4264-8c68-6ac12d65c151, and the live merge
>>> should be successful on vdsm side. At this point, vdsm set the old
>>> volume state to illegal. Engine should ask to delete this volume
>>> later in this flow.
>>>
>>> Adding Ala and Adam to look at this case.
>>>
>>>>
>>>>
>>>>
>>>> image: ff8c64c4-d52b-4812-b541-7f291f98d961
>>>>
>>>> - 85f77cd5-2f86-49a9-a411-8539114d3035
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 70fc19a2-75da-41bd-a1f6-eb857ed2f18f
>>>>
>>>> - a8f27397-395f-4b62-93c4-52699f59ea4b
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 2b315278-65f5-45e8-a51e-02b9bc84dcee
>>>>
>>>> - a6e2150b-57fa-46eb-b205-017fe01b0e4b
>>>> status: OK, voltype: INTERNAL, format: COW,
>>>> legality:
>>>> LEGAL, type: SPARSE
>>>>
>>>> - 2d8e5c14-c923-49ac-8660-8e57b801e329
>>>> status: OK, voltype: INTERNAL, format: COW,
>>>> legality:
>>>> LEGAL, type: SPARSE
>>>>
>>>> - 43100548-b849-4762-bfc5-18a0f281df2e
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: bf4594b0-242e-4823-abfd-9398ce5e31b7
>>>>
>>>> - 4608ce2e-f288-40da-b4e5-2a5e7f3bf837
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 00efca9d-932a-45b3-92c3-80065c1a40ce
>>>>
>>>> - a0bb00bc-cefa-4031-9b59-3cddc3a53a0a
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 5ce704eb-3508-4c36-b0ce-444ebdd27e66
>>>>
>>>> - e41f2c2d-0a79-49f1-8911-1535a82bd735
>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>> LEGAL,
>>>> type: PREALLOCATED
>>>>
>>>>
>>>> image: 11288fa5-0019-4ac0-8a7d-1d455e5e1549
>>>>
>>>> - 5df31efc-14dd-427c-b575-c0d81f47c6d8
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: a091f7df-5c64-4b6b-a806-f4bf3aad53bc
>>>>
>>>> - 38138111-2724-44a4-bde1-1fd9d60a1f63
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: c0b302c4-4b9d-4759-bb80-de1e865ecd58
>>>>
>>>> - d4db9ba7-1b39-4b48-b319-013ebc1d71ce
>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>> LEGAL,
>>>> type: PREALLOCATED
>>>>
>>>>
>>>> image: 21123edb-f74f-440b-9c42-4c16ba06a2b7
>>>>
>>>> - f3cc17aa-4336-4542-9ab0-9df27032be0b
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: ad486d26-4594-4d16-a402-68b45d82078a
>>>>
>>>> - e87e0c7c-4f6f-45e9-90ca-cf34617da3f6
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: c30c7f11-7818-4592-97ca-9d5be46e2d8e
>>>>
>>>> - cb53ad06-65e8-474d-94c3-9acf044d5a09
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 998ac54a-0d91-431f-8929-fe62f5d7290a
>>>>
>>>> - d11aa0ee-d793-4830-9120-3b118ca44b6c
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: a1e69838-0bdf-42f3-95a4-56e4084510a9
>>>>
>>>> - f687c727-ec06-49f1-9762-b0195e0b549a
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: a29598fe-f94e-4215-8508-19ac24b082c8
>>>>
>>>> - 29b9ff26-2386-4fb5-832e-b7129307ceb4
>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>> LEGAL,
>>>> type: PREALLOCATED
>>>>
>>>>
>>>> image: b151d4d7-d7fc-43ff-8bb2-75cf947ed626
>>>>
>>>> - 34676d55-695a-4d2a-a7fa-546971067829
>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>> LEGAL,
>>>> type: SPARSE
>>>>
>>>>
>>>> image: 352a3a9a-4e1a-41bf-af86-717e374a7562
>>>>
>>>> - adcc7655-9586-48c1-90d2-1dc9a851bbe1
>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>> LEGAL,
>>>> type: PREALLOCATED
>>>>
>>>>
>>>> ###########################################
>>>> # END COMMAND OUTPUT
>>>> ########
>>>> ###################################
>>>>
>>>>
>>>> ###########################################
>>>> # BEGIN LOG OUTPUT FROM ENGINE
>>>> ###########################################
>>>>
>>>> The below output is an excerpt from the engine.log while attemtping
>>>> to start one of the afflicted VM's. Following the storage chain out
>>>> i have discovered that "919d6991-43e4-4f26-868e-031a01011191" does
>>>> not exist. This is likely due to a failure of the backup python
>>>> script that the client is using. I can provide that script if you
>>>> all would like.
>>> How is this related to backup? did you restore the files using your
>>> backup script? or maybe
>>> the backup script deleted files by mistake?
>>>
>>>>
>>>>
>>>> 2016-04-20 08:56:58,285
>>>> INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLog
>>>> Director] (org.ovirt.thread.pool-8-thread-8) [] Correlation ID:
>>>> abd1342, Job ID: 1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack:
>>>> null, Custom Event ID: -1, Message: VM Bill-V was started by admin@
>>>> internal (Host: KVM03).
>>>> 2016-04-20 08:57:00,392 ERROR
>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirect
>>>> or] (ForkJoinPool-1-worker-1) [] Correlation ID: null, Call Stack:
>>>> null, Custom Event ID: -1, Message: VM Bill-V is down with error.
>>>> Exit message: Unable to get volume size for domain 045c7fda-ab98-
>>>> 4905-876c- 00b5413a619f volume 919d6991-43e4-4f26-868e-
>>>> 031a01011191.
>>>> 2016-04-20 08:57:00,393
>>>> INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-
>>>> worker-1) [] VM '6ef30172-b010-46fa-9482-accd30682232(Bill-V) is
>>>> running in db and not running in VDS 'KVM03'
>>>> 2016-04-20 08:57:00,498
>>>> WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLog
>>>> Director] (org.ovirt.thread.pool-8-thread-5) [] Correlation ID:
>>>> abd1342, Job ID: 1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack:
>>>> null, Custom Event ID: -1, Message: Failed to run VM Bill-V on Host
>>>> KVM03.
>>> We need the entire engine to investigate.
>>>
>>>>
>>>>
>>>> ###########################################
>>>> # END LOG OUTPUT FROM ENGINE
>>>> ###########################################
>>>>
>>>> I have followed the storage chain out to where the UUID'ed
>>>> snapshots live, and discovered that all of the "ILLEGAL" snapshots
>>>> show to be broken symbolic links.
>>>>
>>>> Attached is a screenshot of the snapshots as they appear in the
>>>> GUI. ALL of the UUID's illustrated show as broken symbolic links in
>>>> the storage domains.
>>> Unless you think that the issue is your backup script deleting files,
>>> I think the best way to proceeed would be to file a bug:
>>> https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-engine
>>>
>>> Use:
>>> oVirt Team: storage
>>> Severity: high
>>>
>>> Please include the information in this mail, and complete vdsm
>>> and engine logs.
>>>
>>> Nir
>>>
>>>>
>>>>> On Tue, 2016-04-19 at 21:28 +0300, Nir Soffer wrote:
>>>>>
>>>>> On Mon, Apr 18, 2016 at 3:16 PM, Clint Boggio <clint at theboggios.c
>>>>> om>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> OVirt 3.6, 4 node cluster with dedicated engine. Main storage
>>>>>> domain is iscsi, ISO and Export domains are NFS.
>>>>>>
>>>>>> Several of my VM snapshot disks show to be in an "illegal
>>>>>> state".
>>>>>> The system will not allow me to manipulate the snapshots in any
>>>>>> way, nor clone the active system, or create a new snapshot.
>>>>>>
>>>>>> In the logs I see that the system complains about not being
>>>>>> able to
>>>>>> "get volume size for xxx", and also that the system appears to
>>>>>> believe that the image is "locked" and is currently in the
>>>>>> snapshot
>>>>>> process.
>>>>>>
>>>>>> Of the VM's with this status, one rebooted and was lost due to
>>>>>> "cannot get volume size for domain xxx".
>>>>> Can you share the vdsm log showing these errors?
>>>>>
>>>>> Also it may helpful to get the output of this command:
>>>>>
>>>>> vdsm-tool dump-volume-chains SDUUID
>>>>>
>>>>> Nir
>>>>>
>>>>>>
>>>>>>
>>>>>> I fear that in this current condition, should any of the other
>>>>>> machine reboot, they too will be lost.
>>>>>>
>>>>>> How can I troubleshoot this problem further, and hopefully
>>>>>> alleviate the condition ?
>>>>>>
>>>>>> Thank you for your help.
>>>>>>
>>>>>> Clint
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> End of Users Digest, Vol 55, Issue 155
>>> **************************************
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
More information about the Users
mailing list