[ovirt-users] Users Digest, Vol 55, Issue 155
Nir Soffer
nsoffer at redhat.com
Wed Apr 20 23:26:53 UTC 2016
On Thu, Apr 21, 2016 at 1:10 AM, Clint Boggio <clint at theboggios.com> wrote:
> Bug is filed.
>
> [Bug 1329000] Snapshot Images Flagged as "ILLEGAL" After Backup Script Is Run
Thanks
>
>
> Is there a technique for recovery from this condition or should I back up the data on the VM's that are afflicted and still running and start over ?
I'm not sure what is the condition.
It may be good chain on host side, with missing volumes on engine side
which are not
needed and can be deleted, or missing volumes on host side.
To check if there are missing volumes on host side, you can inspect
the real chain
used by libvirt using virsh:
# virsh
virsh # list
Please enter your authentication name: vdsm at ovirt
Please enter your password: shibboleth
Id Name State
----------------------------------------------------
2 lsm-test running
virsh # dumpxml 2
<domain type='kvm' id='2'>
[...]
<devices>
[...]
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='qcow2' cache='none'
error_policy='stop' io='native'/>
<source dev='/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/1e999a77-8fbb-4792-9224-0693be3242b9/images/bb26f6eb-d54d-43f3-8d18-e260efb1df7e/4786bb86-da94-44af-b012-51d899cc7225'/>
<backingStore type='block' index='1'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/1e999a77-8fbb-4792-9224-0693be3242b9/images/bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/a07c0fec-242f-444b-8892-e4a0b22e08a7'/>
<backingStore type='block' index='2'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/1e999a77-8fbb-4792-9224-0693be3242b9/images/bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/c76a763a-8208-4fa6-ab60-6bdab15b6159'/>
<backingStore type='block' index='3'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/f9374c0e-ae24-4bc1-a596-f61d5f05bc5f/1e999a77-8fbb-4792-9224-0693be3242b9/images/bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/../bb26f6eb-d54d-43f3-8d18-e260efb1df7e/7c0d5c23-710d-445b-868c-9add6219436d'/>
<backingStore/>
</backingStore>
</backingStore>
</backingStore>
<target dev='vda' bus='virtio'/>
<serial>bb26f6eb-d54d-43f3-8d18-e260efb1df7e</serial>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</disk>
[...]
Here we can see the real chain (removing everything but the volume id)
1.4786bb86-da94-44af-b012-51d899cc7225
2. a07c0fec-242f-444b-8892-e4a0b22e08a7
3. c76a763a-8208-4fa6-ab60-6bdab15b6159
4. 7c0d5c23-710d-445b-868c-9add6219436d
If engine complains about a snapshot which is not part of this chain,
the problem is in
engine database and we can safely remove the snapshot from the database.
If engine complains about a volume which is in this chain, and the volume is
missing on disk, this is an issue on the host side. I'm not sure it is
possible to
restore such missing file unless you have a backup.
It would be useful if you dump the xml of the vms with this issue and
attach it to
the bug.
Nir
>
> Thank you all for your help.
>
>> On Apr 20, 2016, at 3:19 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>>> On Wed, Apr 20, 2016 at 10:42 PM, Clint Boggio <clint at theboggios.com> wrote:
>>> I grepped out the engine logs until i found reference to the illegal
>>> disk in question. The log indicates that the image has been flagged
>>> illegal because the original disk is no longer present. So it is very
>>> possible that the backup script, somehow through the miracle of 1's and
>>> 0's deleted the base VM disks.
>>>
>>> ######################
>>> # BEGIN
>>> ######################
>>>
>>> 2016-03-27 18:57:41,769
>>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>>> nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] START,
>>> CreateSnapshotVDSCommand(
>>> CreateSnapshotVDSCommandParameters:{runAsync='true',
>>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>>> imageSizeInBytes='268435456000', volumeFormat='COW',
>>> newImageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6',
>>> newImageDescription='', imageInitialSizeInBytes='0', imageId='d538e0ef-
>>> 2f55-4c74-b8f1-8900fd6b814b', sourceImageGroupId='ad486d26-4594-4d16-
>>> a402-68b45d82078a'}), log id: 7648bbd2
>>> 2016-03-27 18:57:42,835
>>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>>> nd] (org.ovirt.thread.pool-8-thread-11) [30680dce] FINISH,
>>> CreateSnapshotVDSCommand, return: e87e0c7c-4f6f-45e9-90ca-cf34617da3f6,
>>> log id: 7648bbd2
>>> 2016-03-27 18:58:24,395
>>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand
>>> ] (org.ovirt.thread.pool-8-thread-20) [30680dce] START,
>>> GetImageInfoVDSCommand(
>>> GetImageInfoVDSCommandParameters:{runAsync='true',
>>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>>> imageId='e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'}), log id: 6d2d19f6
>>> 2016-03-28 14:14:49,454
>>> INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
>>> (pool-7-thread-3) [718f57] START, MergeVDSCommand(HostName = KVM04,
>>> MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446-
>>> a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232',
>>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>>> storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f',
>>> imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='e87e0c7c-
>>> 4f6f-45e9-90ca-cf34617da3f6', baseImageId='6e008200-3c21-4285-96b8-
>>> 07c29c0cb72c', topImageId='d538e0ef-2f55-4c74-b8f1-8900fd6b814b',
>>> bandwidth='0'}), log id: 2cc2db4
>>> 2016-03-28 17:01:22,368
>>> INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSComma
>>> nd] (default task-77) [410b6a44] START, CreateSnapshotVDSCommand(
>>> CreateSnapshotVDSCommandParameters:{runAsync='true',
>>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>>> ignoreFailoverLimit='false', storageDomainId='045c7fda-ab98-4905-876c-
>>> 00b5413a619f', imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a',
>>> imageSizeInBytes='268435456000', volumeFormat='COW',
>>> newImageId='919d6991-43e4-4f26-868e-031a01011191',
>>> newImageDescription='', imageInitialSizeInBytes='0', imageId='e87e0c7c-
>>> 4f6f-45e9-90ca-cf34617da3f6', sourceImageGroupId='ad486d26-4594-4d16-
>>> a402-68b45d82078a'}), log id: 4ed3e9ca
>>> 2016-03-28 18:36:28,404
>>> INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
>>> (pool-7-thread-1) [6911a44f] START, MergeVDSCommand(HostName = KVM04,
>>> MergeVDSCommandParameters:{runAsync='true', hostId='b51933a3-9201-4446-
>>> a3e3-906a2ec1b467', vmId='6ef30172-b010-46fa-9482-accd30682232',
>>> storagePoolId='85a72afd-7bde-4065-a1bc-7fc6e22e6bf6',
>>> storageDomainId='045c7fda-ab98-4905-876c-00b5413a619f',
>>> imageGroupId='ad486d26-4594-4d16-a402-68b45d82078a', imageId='919d6991-
>>> 43e4-4f26-868e-031a01011191', baseImageId='e87e0c7c-4f6f-45e9-90ca-
>>> cf34617da3f6', topImageId='919d6991-43e4-4f26-868e-031a01011191',
>>> bandwidth='0'}), log id: d09cb70
>>> 2016-03-28 18:39:53,773
>>> INFO [org.ovirt.engine.core.bll.MergeCommandCallback]
>>> (DefaultQuartzScheduler_Worker-99) [6911a44f] Merge command has
>>> completed for images 'e87e0c7c-4f6f-45e9-90ca-cf34617da3f6'..'919d6991-
>>> 43e4-4f26-868e-031a01011191'
>>> 2016-03-28 18:41:23,003 ERROR
>>> [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand]
>>> (DefaultQuartzScheduler_Worker-44) [a00e3a8] Merging of snapshot
>>> 'a1b3c247-2c6f-4731-9e62-c15f5cfb9a72' images 'e87e0c7c-4f6f-45e9-90ca-
>>> cf34617da3f6'..'919d6991-43e4-4f26-868e-031a01011191' failed. Images
>>> have been marked illegal and can no longer be previewed or reverted to.
>>> Please retry Live Merge on the snapshot to complete the operation.
>>
>> This is live merge failure - we have a similar bug causing this, and I have
>> reproduced similar failure today. This may be the same bug, we must inspect
>> the logs to be sure.
>>
>> Typically the merge succeeds in vdsm side, but from some reason the engine
>> fail to detect the merge success and mark the volumes as illegal.
>>
>>>
>>> ##################
>>> # END
>>> ##################
>>>
>>> If that's the case, then why (how) are the afflicted machines that have
>>> not been rebooted still running without thier backing disks ?
>>
>> It is possible to unlink a file while it is being used by another process.
>> The directory entry is removed so another process cannot access the file,
>> but processes that already opened the file are not affected.
>>
>> But this looks like the live merge issue, not like your backup script trying
>> too hard.
>>
>>>
>>> I can upload the logs and a copy of the backup script. Do you all have
>>> a repository you'd like meto upload to ? Let me know and i'll upload
>>> them right now.
>>
>> Please file a bug and attach the files there.
>>
>> Nir
>>
>>>
>>>
>>>
>>>> On Wed, 2016-04-20 at 13:33 -0400, users-request at ovirt.org wrote:
>>>> Send Users mailing list submissions to
>>>> users at ovirt.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>> or, via email, send a message with subject or body 'help' to
>>>> users-request at ovirt.org
>>>>
>>>> You can reach the person managing the list at
>>>> users-owner at ovirt.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of Users digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>> 1. Re: vhostmd vdsm-hook (Ars?ne Gschwind)
>>>> 2. Re: Disks Illegal State (Nir Soffer)
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> ---
>>>>
>>>> Message: 1
>>>> Date: Wed, 20 Apr 2016 19:09:39 +0200
>>>> From: Ars?ne Gschwind <arsene.gschwind at unibas.ch>
>>>> To: Simon Barrett <Simon.Barrett at tradingscreen.com>, "users at ov
>>>> irt.org"
>>>> <users at ovirt.org>
>>>> Subject: Re: [ovirt-users] vhostmd vdsm-hook
>>>> Message-ID: <5717B7D3.2070502 at unibas.ch>
>>>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>>>
>>>> I've never tried with 2 disks but I will assume that the next free
>>>> available disk will be used by the vdsm hook and the vm-dump-metrics
>>>> cmd
>>>> will check the kind of disk.
>>>> Let me know if you give a try....
>>>>
>>>> thanks,
>>>> Ars?ne
>>>>
>>>>> On 04/19/2016 02:43 PM, Simon Barrett wrote:
>>>>>
>>>>>
>>>>> Thanks again but how does that work when a VM is configured to
>>>>> have
>>>>> more than one disk?
>>>>>
>>>>> If I have a VM with a /dev/vda disk and a /dev/vdb disk, when I
>>>>> turn
>>>>> the vhostmd hook on the vm metric device gets created as /dev/vdb
>>>>> and
>>>>> the original /dev/vdb disk gets bumped to /dev/vdc.
>>>>>
>>>>> Is that expected behavior? Will that not cause problems?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Simon
>>>>>
>>>>> *From:*Ars?ne Gschwind [mailto:arsene.gschwind at unibas.ch]
>>>>> *Sent:* Tuesday, 19 April, 2016 13:06
>>>>> *To:* Simon Barrett <Simon.Barrett at tradingscreen.com>; users at ovirt.
>>>>> org
>>>>> *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
>>>>>
>>>>> The metric information are available on this additional disk
>>>>> /dev/vdb.
>>>>> You may install the package vm-dump-metrics and use the command
>>>>> vm-dump-metrics which will display all metrics in an xml format.
>>>>>
>>>>> Ars?ne
>>>>>
>>>>> On 04/19/2016 10:48 AM, Simon Barrett wrote:
>>>>>
>>>>> Thanks Ars?ne,
>>>>>
>>>>> I have vhostmd running on the ovirt node and have set the
>>>>> sap_agent to true on the VM configuration. I also stopped and
>>>>> started the VM to ensure that the config change took effect.
>>>>>
>>>>> On the oVirt node I see the vhostmd running and see the
>>>>> following
>>>>> entry in the qemu-kvm output:
>>>>>
>>>>> drive
>>>>> file=/dev/shm/vhostmd0,if=none,id=drive-virtio-
>>>>> disk701,readonly=on,format=raw
>>>>> -device
>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-
>>>>> disk701,id=virtio-disk701
>>>>>
>>>>> The part I wasn?t quite understanding was how this presented
>>>>> itself on the VM but I now see a new disk device ?/dev/vdb?. If
>>>>> I
>>>>> cat the contents of /dev/vdb I now see the information that is
>>>>> provided from the ovirt node, which is great news and very
>>>>> useful.
>>>>>
>>>>> Thanks for your help.
>>>>>
>>>>> Simon
>>>>>
>>>>> *From:*users-bounces at ovirt.org <mailto:users-bounces at ovirt.org>
>>>>> [mailto:users-bounces at ovirt.org] *On Behalf Of *Ars?ne Gschwind
>>>>> *Sent:* Monday, 18 April, 2016 16:03
>>>>> *To:* users at ovirt.org <mailto:users at ovirt.org>
>>>>> *Subject:* Re: [ovirt-users] vhostmd vdsm-hook
>>>>>
>>>>> Hi Simon,
>>>>>
>>>>> You will need to have vhostmd running on the oVirt node and set
>>>>> the "sap_agent" custom property for the vm as you may see on
>>>>> the
>>>>> screenshot.
>>>>>
>>>>> sap_agent
>>>>>
>>>>> Ars?ne
>>>>>
>>>>> On 04/15/2016 12:15 PM, Simon Barrett wrote:
>>>>>
>>>>> I?m trying to use the vhostmd vdsm host to access ovirt
>>>>> node
>>>>> metrics from within a VM. Vhostmd is running and updating
>>>>> the
>>>>> /dev/shm/vhostmd0 on the ovirt node.
>>>>>
>>>>> The part I?m stuck on is: ?This disk image is exported
>>>>> read-only to guests. Guests can read the disk image to see
>>>>> metrics? from
>>>>> http://www.ovirt.org/develop/developer-guide/vdsm/hook/vhos
>>>>> tmd/
>>>>>
>>>>> Does the hook do this by default? I don?t see any new
>>>>> read-only device mounted in the guest. Is there additional
>>>>> work I need to do to mount this and access the data from
>>>>> within the guest?
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>>
>>>>> Users mailing list
>>>>>
>>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL: <http://lists.ovirt.org/pipermail/users/attachments/20160420/d5d
>>>> 7d06b/attachment-0001.html>
>>>> -------------- next part --------------
>>>> A non-text attachment was scrubbed...
>>>> Name: not available
>>>> Type: image/png
>>>> Size: 6941 bytes
>>>> Desc: not available
>>>> URL: <http://lists.ovirt.org/pipermail/users/attachments/20160420/d5d
>>>> 7d06b/attachment-0001.png>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Wed, 20 Apr 2016 20:33:06 +0300
>>>> From: Nir Soffer <nsoffer at redhat.com>
>>>> To: Clint Boggio <clint at theboggios.com>, Ala Hino <ahino at redhat.com>,
>>>> Adam Litke <alitke at redhat.com>
>>>> Cc: users <users at ovirt.org>
>>>> Subject: Re: [ovirt-users] Disks Illegal State
>>>> Message-ID:
>>>> <CAMRbyyvfxT4h_T_qgyXNNm+OwQF7xPuwk_qOeiLP+xQ-JkkWdg at mail.gmail
>>>> .com>
>>>> Content-Type: text/plain; charset=UTF-8
>>>>
>>>> On Wed, Apr 20, 2016 at 5:34 PM, Clint Boggio <clint at theboggios.com>
>>>> wrote:
>>>>>
>>>>> The "vdsm-tool dump-volume-chains" command on the iSCSI storage
>>>>> domain
>>>>> shows one disk in "ILLEGAL" state while the gui shows 8 disk images
>>>>> in
>>>>> the same state.
>>>> Interesting - it would be useful to find the missing volume ids in
>>>> engine log and
>>>> understand wahy they are marked as illegal.
>>>>
>>>>>
>>>>>
>>>>> ###########################################
>>>>> # BEGIN COMMAND OUTPUT
>>>>> ###########################################
>>>>>
>>>>>
>>>>>
>>>>> [root at KVM01 ~]# vdsm-tool dump-volume-chains 045c7fda-ab98-4905-
>>>>> 876c-
>>>>> 00b5413a619f
>>>>>
>>>>> Images volume chains (base volume first)
>>>>>
>>>>> image: 477e73af-e7db-4914-81ed-89b3fbc876f7
>>>>>
>>>>> - c8320522-f839-472e-9707-a75f6fbe5cb6
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 882c73fc-a833-4e2e-8e6a-f714d80c0f0d
>>>>>
>>>>> - 689220c0-70f8-475f-98b2-6059e735cd1f
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 0ca8c49f-452e-4f61-a3fc-c4bf2711e200
>>>>>
>>>>> - dac06a5c-c5a8-4f82-aa8d-5c7a382da0b3
>>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>>> LEGAL,
>>>>> type: PREALLOCATED
>>>>>
>>>>>
>>>>> image: 0ca0b8f8-8802-46ae-a9f8-45d5647feeb7
>>>>>
>>>>> - 51a6de7b-b505-4c46-ae2a-25fb9faad810
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: ae6d2c62-cfbb-4765-930f-c0a0e3bc07d0
>>>>>
>>>>> - b2d39c7d-5b9b-498d-a955-0e99c9bd5f3c
>>>>> status: OK, voltype: INTERNAL, format: COW,
>>>>> legality:
>>>>> LEGAL, type: SPARSE
>>>>>
>>>>> - bf962809-3de7-4264-8c68-6ac12d65c151
>>>>> status: ILLEGAL, voltype: LEAF, format: COW,
>>>>> legality:
>>>>> ILLEGAL, type: SPARSE
>>>> Lets check vdsm and engine log, and find when and why this disk
>>>> became
>>>> illegal.
>>>>
>>>> If this was a result of a live merge that failed while finalizing the
>>>> merge
>>>> on the engine side, we can safely delete the illegal volume.
>>>>
>>>> If this is the case, we should find a live merge for volume
>>>> bf962809-3de7-4264-8c68-6ac12d65c151, and the live merge
>>>> should be successful on vdsm side. At this point, vdsm set the old
>>>> volume state to illegal. Engine should ask to delete this volume
>>>> later in this flow.
>>>>
>>>> Adding Ala and Adam to look at this case.
>>>>
>>>>>
>>>>>
>>>>>
>>>>> image: ff8c64c4-d52b-4812-b541-7f291f98d961
>>>>>
>>>>> - 85f77cd5-2f86-49a9-a411-8539114d3035
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 70fc19a2-75da-41bd-a1f6-eb857ed2f18f
>>>>>
>>>>> - a8f27397-395f-4b62-93c4-52699f59ea4b
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 2b315278-65f5-45e8-a51e-02b9bc84dcee
>>>>>
>>>>> - a6e2150b-57fa-46eb-b205-017fe01b0e4b
>>>>> status: OK, voltype: INTERNAL, format: COW,
>>>>> legality:
>>>>> LEGAL, type: SPARSE
>>>>>
>>>>> - 2d8e5c14-c923-49ac-8660-8e57b801e329
>>>>> status: OK, voltype: INTERNAL, format: COW,
>>>>> legality:
>>>>> LEGAL, type: SPARSE
>>>>>
>>>>> - 43100548-b849-4762-bfc5-18a0f281df2e
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: bf4594b0-242e-4823-abfd-9398ce5e31b7
>>>>>
>>>>> - 4608ce2e-f288-40da-b4e5-2a5e7f3bf837
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 00efca9d-932a-45b3-92c3-80065c1a40ce
>>>>>
>>>>> - a0bb00bc-cefa-4031-9b59-3cddc3a53a0a
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 5ce704eb-3508-4c36-b0ce-444ebdd27e66
>>>>>
>>>>> - e41f2c2d-0a79-49f1-8911-1535a82bd735
>>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>>> LEGAL,
>>>>> type: PREALLOCATED
>>>>>
>>>>>
>>>>> image: 11288fa5-0019-4ac0-8a7d-1d455e5e1549
>>>>>
>>>>> - 5df31efc-14dd-427c-b575-c0d81f47c6d8
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: a091f7df-5c64-4b6b-a806-f4bf3aad53bc
>>>>>
>>>>> - 38138111-2724-44a4-bde1-1fd9d60a1f63
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: c0b302c4-4b9d-4759-bb80-de1e865ecd58
>>>>>
>>>>> - d4db9ba7-1b39-4b48-b319-013ebc1d71ce
>>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>>> LEGAL,
>>>>> type: PREALLOCATED
>>>>>
>>>>>
>>>>> image: 21123edb-f74f-440b-9c42-4c16ba06a2b7
>>>>>
>>>>> - f3cc17aa-4336-4542-9ab0-9df27032be0b
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: ad486d26-4594-4d16-a402-68b45d82078a
>>>>>
>>>>> - e87e0c7c-4f6f-45e9-90ca-cf34617da3f6
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: c30c7f11-7818-4592-97ca-9d5be46e2d8e
>>>>>
>>>>> - cb53ad06-65e8-474d-94c3-9acf044d5a09
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 998ac54a-0d91-431f-8929-fe62f5d7290a
>>>>>
>>>>> - d11aa0ee-d793-4830-9120-3b118ca44b6c
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: a1e69838-0bdf-42f3-95a4-56e4084510a9
>>>>>
>>>>> - f687c727-ec06-49f1-9762-b0195e0b549a
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: a29598fe-f94e-4215-8508-19ac24b082c8
>>>>>
>>>>> - 29b9ff26-2386-4fb5-832e-b7129307ceb4
>>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>>> LEGAL,
>>>>> type: PREALLOCATED
>>>>>
>>>>>
>>>>> image: b151d4d7-d7fc-43ff-8bb2-75cf947ed626
>>>>>
>>>>> - 34676d55-695a-4d2a-a7fa-546971067829
>>>>> status: OK, voltype: LEAF, format: COW, legality:
>>>>> LEGAL,
>>>>> type: SPARSE
>>>>>
>>>>>
>>>>> image: 352a3a9a-4e1a-41bf-af86-717e374a7562
>>>>>
>>>>> - adcc7655-9586-48c1-90d2-1dc9a851bbe1
>>>>> status: OK, voltype: LEAF, format: RAW, legality:
>>>>> LEGAL,
>>>>> type: PREALLOCATED
>>>>>
>>>>>
>>>>> ###########################################
>>>>> # END COMMAND OUTPUT
>>>>> ########
>>>>> ###################################
>>>>>
>>>>>
>>>>> ###########################################
>>>>> # BEGIN LOG OUTPUT FROM ENGINE
>>>>> ###########################################
>>>>>
>>>>> The below output is an excerpt from the engine.log while attemtping
>>>>> to start one of the afflicted VM's. Following the storage chain out
>>>>> i have discovered that "919d6991-43e4-4f26-868e-031a01011191" does
>>>>> not exist. This is likely due to a failure of the backup python
>>>>> script that the client is using. I can provide that script if you
>>>>> all would like.
>>>> How is this related to backup? did you restore the files using your
>>>> backup script? or maybe
>>>> the backup script deleted files by mistake?
>>>>
>>>>>
>>>>>
>>>>> 2016-04-20 08:56:58,285
>>>>> INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLog
>>>>> Director] (org.ovirt.thread.pool-8-thread-8) [] Correlation ID:
>>>>> abd1342, Job ID: 1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack:
>>>>> null, Custom Event ID: -1, Message: VM Bill-V was started by admin@
>>>>> internal (Host: KVM03).
>>>>> 2016-04-20 08:57:00,392 ERROR
>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirect
>>>>> or] (ForkJoinPool-1-worker-1) [] Correlation ID: null, Call Stack:
>>>>> null, Custom Event ID: -1, Message: VM Bill-V is down with error.
>>>>> Exit message: Unable to get volume size for domain 045c7fda-ab98-
>>>>> 4905-876c- 00b5413a619f volume 919d6991-43e4-4f26-868e-
>>>>> 031a01011191.
>>>>> 2016-04-20 08:57:00,393
>>>>> INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-
>>>>> worker-1) [] VM '6ef30172-b010-46fa-9482-accd30682232(Bill-V) is
>>>>> running in db and not running in VDS 'KVM03'
>>>>> 2016-04-20 08:57:00,498
>>>>> WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLog
>>>>> Director] (org.ovirt.thread.pool-8-thread-5) [] Correlation ID:
>>>>> abd1342, Job ID: 1ad2ee48-2c2c-437e-997b-469e09498e41, Call Stack:
>>>>> null, Custom Event ID: -1, Message: Failed to run VM Bill-V on Host
>>>>> KVM03.
>>>> We need the entire engine to investigate.
>>>>
>>>>>
>>>>>
>>>>> ###########################################
>>>>> # END LOG OUTPUT FROM ENGINE
>>>>> ###########################################
>>>>>
>>>>> I have followed the storage chain out to where the UUID'ed
>>>>> snapshots live, and discovered that all of the "ILLEGAL" snapshots
>>>>> show to be broken symbolic links.
>>>>>
>>>>> Attached is a screenshot of the snapshots as they appear in the
>>>>> GUI. ALL of the UUID's illustrated show as broken symbolic links in
>>>>> the storage domains.
>>>> Unless you think that the issue is your backup script deleting files,
>>>> I think the best way to proceeed would be to file a bug:
>>>> https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-engine
>>>>
>>>> Use:
>>>> oVirt Team: storage
>>>> Severity: high
>>>>
>>>> Please include the information in this mail, and complete vdsm
>>>> and engine logs.
>>>>
>>>> Nir
>>>>
>>>>>
>>>>>> On Tue, 2016-04-19 at 21:28 +0300, Nir Soffer wrote:
>>>>>>
>>>>>> On Mon, Apr 18, 2016 at 3:16 PM, Clint Boggio <clint at theboggios.c
>>>>>> om>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> OVirt 3.6, 4 node cluster with dedicated engine. Main storage
>>>>>>> domain is iscsi, ISO and Export domains are NFS.
>>>>>>>
>>>>>>> Several of my VM snapshot disks show to be in an "illegal
>>>>>>> state".
>>>>>>> The system will not allow me to manipulate the snapshots in any
>>>>>>> way, nor clone the active system, or create a new snapshot.
>>>>>>>
>>>>>>> In the logs I see that the system complains about not being
>>>>>>> able to
>>>>>>> "get volume size for xxx", and also that the system appears to
>>>>>>> believe that the image is "locked" and is currently in the
>>>>>>> snapshot
>>>>>>> process.
>>>>>>>
>>>>>>> Of the VM's with this status, one rebooted and was lost due to
>>>>>>> "cannot get volume size for domain xxx".
>>>>>> Can you share the vdsm log showing these errors?
>>>>>>
>>>>>> Also it may helpful to get the output of this command:
>>>>>>
>>>>>> vdsm-tool dump-volume-chains SDUUID
>>>>>>
>>>>>> Nir
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I fear that in this current condition, should any of the other
>>>>>>> machine reboot, they too will be lost.
>>>>>>>
>>>>>>> How can I troubleshoot this problem further, and hopefully
>>>>>>> alleviate the condition ?
>>>>>>>
>>>>>>> Thank you for your help.
>>>>>>>
>>>>>>> Clint
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>> End of Users Digest, Vol 55, Issue 155
>>>> **************************************
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>
More information about the Users
mailing list