[Users] oVirt 3.3.3 RC EL6 Live Snapshot
Elad Ben Aharon
ebenahar at redhat.com
Tue Feb 4 08:20:56 UTC 2014
>From what I saw in the thread, libvirt pauses the VM, which effects the continuity of its operation.
I checked it also in one of the latest builds of 3.3 and I observed the same behaviour:
2014-02-04 10:18:29,441 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [5cbc7ab5] VM nfs2-1 86728f5c-3583-420f-9536-bfabaf11b235 moved from Up --> Paused
Dafna, I saw you've already opened a bug on that:
https://bugzilla.redhat.com/show_bug.cgi?id=1057587
----- Original Message -----
From: "Dafna Ron" <dron at redhat.com>
To: "Maor Lipchuk" <mlipchuk at redhat.com>
Cc: "Steve Dainard" <sdainard at miovision.com>, "Elad Ben Aharon" <ebenahar at redhat.com>, "Karli Sjöberg" <Karli.Sjoberg at slu.se>, users at ovirt.org
Sent: Tuesday, February 4, 2014 12:15:42 AM
Subject: Re: [Users] oVirt 3.3.3 RC EL6 Live Snapshot
Maor I am not saying that we are not doing a live snapshot :) I am
saying that we need a print in the log that states live snapshot command
was called i.e: Print in the log: LiveSnapshotCommand -> this can call
to the rest of snapshotVDSCreateCommand.
On 02/03/2014 07:38 PM, Maor Lipchuk wrote:
> On 02/03/2014 07:46 PM, Dafna Ron wrote:
>> On 02/03/2014 05:34 PM, Maor Lipchuk wrote:
>>> On 02/03/2014 07:18 PM, Dafna Ron wrote:
>>>> Maor,
>>>>
>>>> If snapshotVDSCommand is for live snapshot, what is the offline create
>>>> snapshot command?
>>> It is the CreateSnapshotVdsCommand which calls createVolume in VDSM
>> but we need to be able to know that a live snapshot was sent and not an
>> offline snapshot.
> Yes, at the logs we can see the all process :
>
> First a request to create a snapshot (new volume) sent to VDSM:
> 2014-02-02 09:41:09,557 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand]
> (pool-6-thread-49) [67ea047a] START, CreateSnapshotVDSCommand(
> storagePoolId = fcb89071-6cdb-4972-94d1-c9324cebf814,
> ignoreFailoverLimit = false, storageDomainId =
> a52938f7-2cf4-4771-acb2-0c78d14999e5, imageGroupId =
> c1cb6b66-655e-48c3-8568-4975295eb037, imageSizeInBytes = 21474836480,
> volumeFormat = COW, newImageId = 6d8c80a4-328f-4a53-86a2-a4080a2662ce,
> newImageDescription = , imageId = 5085422e-6592-415a-9da3-9e43dac9374b,
> sourceImageGroupId = c1cb6b66-655e-48c3-8568-4975295eb037), log id: 7875f3f5
>
> after the snapshot gets created :
> 2014-02-02 09:41:20,553 INFO
> [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
> (pool-6-thread-49) Ending command successfully:
> org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand
>
> then the engine calls the live snapshot (see also [1])
> 2014-02-02 09:41:30,234 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
> (pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872
>
>> Elad, somewhere in this flow we need to know that the snapshot was taken
>> on a running vm :) this seems like a bug to me.
>>>> we did not say that live snapshot did not succeed :) we said that the
>>>> vm is paused and restarted - which is something that should not happen
>>>> for live snapshot (or at least never did before).
>>> It's not sure that the restart is related to the live snapshot. but that
>>> should be observed in the libvirt/vdsm logs.
>> yes, I am sure because the user is reporting it and the logs show it...
>>>> as I wrote before, we know that vdsm is reporting the vm as paused, that
>>>> is because libvirt is reporting the vm as paused and I think that its
>>>> happening because libvirt is not doing a live snapshot and so pauses the
>>>> vm while taking the snapshot.
>>> That sounds logic to me, it's need to be checked with libvirt, if that
>>> kind of behaviour could happen.
>> Elad, can you please try to reproduce and open a bug to libvirt?
>>
>>>> Dafna
>>>>
>>>>
>>>> On 02/03/2014 05:08 PM, Maor Lipchuk wrote:
>>>>> From the engine logs it seems that indeed live snapshot is called
>>>>> (The
>>>>> command is snapshotVDSCommand see [1]).
>>>>> This is done right after the snapshot has been created in the VM and it
>>>>> signals the qemu process to start using the new volume created.
>>>>>
>>>>> When live snapshot does not succeed we should see in the log something
>>>>> like "Wasn't able to live snapshot due to error:...", but it does not
>>>>> appear so it seems that this worked out fine.
>>>>>
>>>>> At some point I can see in the logs that VDSM reports to the engine
>>>>> that
>>>>> the VM is paused.
>>>>>
>>>>>
>>>>> [1]
>>>>> 2014-02-02 09:41:20,564 INFO
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>>>>> (pool-6-thread-49) START, SnapshotVDSCommand(HostName = ovirt002,
>>>>> HostId
>>>>> = 3080fb61-2d03-4008-b47f-9b66276a4257,
>>>>> vmId=e261e707-a21f-4ae8-9cff-f535f4430446), log id: 7e0d7872
>>>>> 2014-02-02 09:41:21,119 INFO
>>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>>> (DefaultQuartzScheduler_Worker-93) VM snapshot-test
>>>>> e261e707-a21f-4ae8-9cff-f535f4430446 moved from Up --> Paused
>>>>> 2014-02-02 09:41:30,234 INFO
>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
>>>>> (pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872
>>>>> 2014-02-02 09:41:30,238 INFO
>>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] (pool-6-thread-49)
>>>>> [67ea047a] Ending command successfully:
>>>>> org.ovirt.engine.core.bll.CreateSnapshotCommand
>>>>> ...
>>>>>
>>>>> Regards,
>>>>> Maor
>>>>>
>>>>> On 02/03/2014 06:24 PM, Dafna Ron wrote:
>>>>>> Thanks Steve.
>>>>>>
>>>>>> from the logs I can see that the create snapshot succeeds and that the
>>>>>> vm is resumed.
>>>>>> the vm moves to pause as part of libvirt flows:
>>>>>>
>>>>>> 2014-02-02 14:41:20.872+0000: 5843: debug :
>>>>>> qemuProcessHandleStop:728 :
>>>>>> Transitioned guest snapshot-test to paused state
>>>>>> 2014-02-02 14:41:30.031+0000: 5843: debug :
>>>>>> qemuProcessHandleResume:776
>>>>>> : Transitioned guest snapshot-test out of paused into resumed state
>>>>>>
>>>>>> There are bugs here but I am not sure yet if this is libvirt
>>>>>> regression
>>>>>> or engine.
>>>>>>
>>>>>> I'm adding Elad and Maor since in engine logs I can't see anything
>>>>>> calling for live snapshot (only for snapshot) - Maor, shouldn't live
>>>>>> snapshot command be logged somewhere in the logs?
>>>>>> Is it possible that engine is calling to create snapshot and not
>>>>>> create
>>>>>> live snapshot which is why the vm pauses?
>>>>>>
>>>>>> Elad, if engine is not logging live snapshot anywhere I would open
>>>>>> a bug
>>>>>> for engine (to print that in the logs).
>>>>>> Also, there is a bug in vdsm log for sdc where the below is logged as
>>>>>> ERROR and not INFO:
>>>>>>
>>>>>> Thread-23::ERROR::2014-02-02
>>>>>> 09:51:19,497::sdc::137::Storage.StorageDomainCache::(_findDomain)
>>>>>> looking for unfetched domain a52938f7-2cf4-4771-acb2-0c78d14999e5
>>>>>> Thread-23::ERROR::2014-02-02
>>>>>> 09:51:19,497::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
>>>>>>
>>>>>>
>>>>>> looking for domain a52938f7-2cf4-4771-acb2-0c78d14999e5
>>>>>>
>>>>>> If the engine was sending live snapshot or if there is no
>>>>>> difference in
>>>>>> the two commands in engine side than I would open a bug for libvirt
>>>>>> for
>>>>>> pausing the vm during live snapshot.
>>>>>>
>>>>>> Dafna
>>>>>>
>>>>>> On 02/03/2014 02:41 PM, Steve Dainard wrote:
>>>>>>> [root at ovirt002 ~]# vdsClient -s 0 getStorageDomainInfo
>>>>>>> a52938f7-2cf4-4771-acb2-0c78d14999e5
>>>>>>> uuid = a52938f7-2cf4-4771-acb2-0c78d14999e5
>>>>>>> pool = ['fcb89071-6cdb-4972-94d1-c9324cebf814']
>>>>>>> lver = 5
>>>>>>> version = 3
>>>>>>> role = Master
>>>>>>> remotePath = gluster-store-vip:/rep1
>>>>>>> spm_id = 2
>>>>>>> type = NFS
>>>>>>> class = Data
>>>>>>> master_ver = 1
>>>>>>> name = gluster-store-rep1
>>>>>>>
>>>>>>>
>>>>>>> *Steve Dainard *
>>>>>>> IT Infrastructure Manager
>>>>>>> Miovision <http://miovision.com/> | /Rethink Traffic/
>>>>>>> 519-513-2407 ex.250
>>>>>>> 877-646-8476 (toll-free)
>>>>>>>
>>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn
>>>>>>> <https://www.linkedin.com/company/miovision-technologies> | Twitter
>>>>>>> <https://twitter.com/miovision> | Facebook
>>>>>>> <https://www.facebook.com/miovision>*
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101,
>>>>>>> Kitchener,
>>>>>>> ON, Canada | N2C 1L3
>>>>>>> This e-mail may contain information that is privileged or
>>>>>>> confidential. If you are not the intended recipient, please delete
>>>>>>> the
>>>>>>> e-mail and any attachments and notify us immediately.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Feb 2, 2014 at 2:55 PM, Dafna Ron <dron at redhat.com
>>>>>>> <mailto:dron at redhat.com>> wrote:
>>>>>>>
>>>>>>> please run vdsClient -s 0 getStorageDomainInfo
>>>>>>> a52938f7-2cf4-4771-acb2-0c78d14999e5
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Dafna
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 02/02/2014 03:02 PM, Steve Dainard wrote:
>>>>>>>
>>>>>>> Logs attached with VM running on qemu-kvm-rhev packages
>>>>>>> installed.
>>>>>>>
>>>>>>> *Steve Dainard *
>>>>>>> IT Infrastructure Manager
>>>>>>> Miovision <http://miovision.com/> | /Rethink Traffic/
>>>>>>> 519-513-2407 <tel:519-513-2407> ex.250
>>>>>>>
>>>>>>> 877-646-8476 <tel:877-646-8476> (toll-free)
>>>>>>>
>>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn
>>>>>>>
>>>>>>> <https://www.linkedin.com/company/miovision-technologies> |
>>>>>>> Twitter <https://twitter.com/miovision> | Facebook
>>>>>>> <https://www.facebook.com/miovision>*
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101,
>>>>>>> Kitchener, ON, Canada | N2C 1L3
>>>>>>> This e-mail may contain information that is privileged or
>>>>>>> confidential. If you are not the intended recipient, please
>>>>>>> delete the e-mail and any attachments and notify us
>>>>>>> immediately.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Feb 2, 2014 at 5:05 AM, Dafna Ron <dron at redhat.com
>>>>>>> <mailto:dron at redhat.com> <mailto:dron at redhat.com
>>>>>>> <mailto:dron at redhat.com>>> wrote:
>>>>>>>
>>>>>>> can you please upload full engine, vdsm, libvirt and
>>>>>>> vm's
>>>>>>> qemu logs?
>>>>>>>
>>>>>>>
>>>>>>> On 02/02/2014 02:08 AM, Steve Dainard wrote:
>>>>>>>
>>>>>>> I have two CentOS 6.5 Ovirt hosts (ovirt001,
>>>>>>> ovirt002)
>>>>>>>
>>>>>>> I've installed the applicable qemu-kvm-rhev
>>>>>>> packages
>>>>>>> from this
>>>>>>> site:
>>>>>>> http://www.dreyou.org/ovirt/vdsm32/Packages/ on
>>>>>>> ovirt002.
>>>>>>>
>>>>>>> On ovirt001 if I take a live snapshot:
>>>>>>>
>>>>>>> Snapshot 'test qemu-kvm' creation for VM
>>>>>>> 'snapshot-test' was
>>>>>>> initiated by admin at internal.
>>>>>>> The VM is paused
>>>>>>> Failed to create live snapshot 'test qemu-kvm'
>>>>>>> for VM
>>>>>>> 'snapshot-test'. VM restart is recommended.
>>>>>>> Failed to complete snapshot 'test qemu-kvm'
>>>>>>> creation
>>>>>>> for VM
>>>>>>> 'snapshot-test'.
>>>>>>> The VM is then started, and the status for the
>>>>>>> snapshot
>>>>>>> changes to OK.
>>>>>>>
>>>>>>> On ovirt002 (with the packages from dreyou) I don't
>>>>>>> get any
>>>>>>> messages about a snapshot failing, but my VM is
>>>>>>> still
>>>>>>> paused
>>>>>>> to complete the snapshot. Is there something else
>>>>>>> other than
>>>>>>> the qemu-kvm-rhev packages that would enable this
>>>>>>> functionality?
>>>>>>>
>>>>>>> I've looked for some information on when the
>>>>>>> packages
>>>>>>> would be
>>>>>>> built as required in the CentOS repos, but I
>>>>>>> don't see
>>>>>>> anything definitive.
>>>>>>>
>>>>>>>
>>>>>>> http://lists.ovirt.org/pipermail/users/2013-December/019126.html
>>>>>>> Looks like one of the maintainers is waiting for
>>>>>>> someone to
>>>>>>> tell him what flags need to be set.
>>>>>>>
>>>>>>> Also, another thread here:
>>>>>>>
>>>>>>> http://comments.gmane.org/gmane.comp.emulators.ovirt.arch/1618
>>>>>>> same maintainer, mentioning that he hasn't seen
>>>>>>> anything in
>>>>>>> the bug tracker.
>>>>>>>
>>>>>>> There is a bug here:
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 that
>>>>>>> seems
>>>>>>> to have ended in finding a way for qemu to expose
>>>>>>> whether it
>>>>>>> supports live snapshots, rather than figuring
>>>>>>> out how
>>>>>>> to get
>>>>>>> the CentOS team the info they need to build the
>>>>>>> packages with
>>>>>>> the proper flags set.
>>>>>>>
>>>>>>> I have bcc'd both dreyou (packaged the
>>>>>>> qemu-kvm-rhev
>>>>>>> packages
>>>>>>> listed above) and Russ (CentOS maintainer
>>>>>>> mentioned in
>>>>>>> the
>>>>>>> other threads) if they wish to chime in and perhaps
>>>>>>> collaborate on which flags, if any, should be set
>>>>>>> for the
>>>>>>> qemu-kvm builds so we can get a CentOS bug report
>>>>>>> going and
>>>>>>> hammer this out.
>>>>>>>
>>>>>>> Thanks everyone.
>>>>>>>
>>>>>>> **crosses fingers and hopes for live snapshots
>>>>>>> soon**
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Steve Dainard *
>>>>>>> IT Infrastructure Manager
>>>>>>> Miovision <http://miovision.com/> | /Rethink
>>>>>>> Traffic/
>>>>>>> 519-513-2407 <tel:519-513-2407> <tel:519-513-2407
>>>>>>> <tel:519-513-2407>> <tel:519-513-2407 <tel:519-513-2407>
>>>>>>> <tel:519-513-2407 <tel:519-513-2407>>> ex.250
>>>>>>> 877-646-8476 <tel:877-646-8476> <tel:877-646-8476
>>>>>>> <tel:877-646-8476>> <tel:877-646-8476 <tel:877-646-8476>
>>>>>>>
>>>>>>> <tel:877-646-8476 <tel:877-646-8476>>> (toll-free)
>>>>>>>
>>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn
>>>>>>>
>>>>>>> <https://www.linkedin.com/company/miovision-technologies> |
>>>>>>> Twitter <https://twitter.com/miovision> | Facebook
>>>>>>> <https://www.facebook.com/miovision>*
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Miovision Technologies Inc. | 148 Manitou Drive,
>>>>>>> Suite
>>>>>>> 101,
>>>>>>> Kitchener, ON, Canada | N2C 1L3
>>>>>>> This e-mail may contain information that is
>>>>>>> privileged or
>>>>>>> confidential. If you are not the intended
>>>>>>> recipient,
>>>>>>> please
>>>>>>> delete the e-mail and any attachments and notify us
>>>>>>> immediately.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 31, 2014 at 1:26 PM, Steve Dainard
>>>>>>> <sdainard at miovision.com
>>>>>>> <mailto:sdainard at miovision.com>
>>>>>>> <mailto:sdainard at miovision.com
>>>>>>> <mailto:sdainard at miovision.com>>
>>>>>>> <mailto:sdainard at miovision.com
>>>>>>> <mailto:sdainard at miovision.com>
>>>>>>>
>>>>>>> <mailto:sdainard at miovision.com
>>>>>>> <mailto:sdainard at miovision.com>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> How would you developers, speaking for the
>>>>>>> oVirt-community,
>>>>>>> propose to
>>>>>>> solve this for CentOS _now_ ?
>>>>>>>
>>>>>>> I would imagine that the easiest way is
>>>>>>> that
>>>>>>> you build and
>>>>>>> host this one
>>>>>>> package(qemu-kvm-rhev), since you´ve
>>>>>>> basically
>>>>>>> already
>>>>>>> have
>>>>>>> the source
>>>>>>> and recipe (since you´re already
>>>>>>> providing it
>>>>>>> for RHEV
>>>>>>> anyway). Then,
>>>>>>> once that´s in place, it´s more a
>>>>>>> question of
>>>>>>> where to
>>>>>>> host the
>>>>>>> packages, in what repository. Be it your
>>>>>>> own,
>>>>>>> or some
>>>>>>> other
>>>>>>> repo set up
>>>>>>> for the SIG.
>>>>>>>
>>>>>>> This is my view, how I as a user view this
>>>>>>> issue.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think this is a pretty valid view.
>>>>>>>
>>>>>>> What would it take to get the correct qemu
>>>>>>> package
>>>>>>> hosted
>>>>>>> in the
>>>>>>> ovirt repo?
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Med Vänliga Hälsningar
>>>>>>>
>>>>>>>
>>>>>>> -------------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Karli Sjöberg
>>>>>>> Swedish University of Agricultural Sciences
>>>>>>> Box 7079
>>>>>>> (Visiting
>>>>>>> Address
>>>>>>> Kronåsvägen 8)
>>>>>>> S-750 07 Uppsala, Sweden
>>>>>>> Phone: +46-(0)18-67 15 66
>>>>>>> <tel:%2B46-%280%2918-67%2015%2066>
>>>>>>> <tel:%2B46-%280%2918-67%2015%2066>
>>>>>>> <tel:%2B46-%280%2918-67%2015%2066>
>>>>>>> karli.sjoberg at slu.se <mailto:karli.sjoberg at slu.se>
>>>>>>> <mailto:karli.sjoberg at slu.se <mailto:karli.sjoberg at slu.se>>
>>>>>>> <mailto:karli.sjoberg at slu.se
>>>>>>> <mailto:karli.sjoberg at slu.se> <mailto:karli.sjoberg at slu.se
>>>>>>> <mailto:karli.sjoberg at slu.se>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>>> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>>>>>>> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>>> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>>
>>>>>>>
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>>> <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>>>>>>>
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Dafna Ron
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Dafna Ron
>>>>>>>
>>>>>>>
>>
--
Dafna Ron
More information about the Users
mailing list