Re: [Users] oVirt 3.3.3 RC EL6 Live Snapshot

From what I saw in the thread, libvirt pauses the VM, which effects the continuity of its operation. I checked it also in one of the latest builds of 3.3 and I observed the same behaviour: 2014-02-04 10:18:29,441 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [5cbc7ab5] VM nfs2-1 86728f5c-3583-420f-9536-bfabaf11b235 moved from Up --> Paused Dafna, I saw you've already opened a bug on that: https://bugzilla.redhat.com/show_bug.cgi?id=1057587

----- Original Message ----- From: "Dafna Ron" <dron(a)redhat.com&gt; To: "Maor Lipchuk" <mlipchuk(a)redhat.com&gt; Cc: "Steve Dainard" <sdainard(a)miovision.com&gt;, "Elad Ben Aharon" <ebenahar(a)redhat.com&gt;, "Karli Sjöberg" <Karli.Sjoberg(a)slu.se&gt;, users(a)ovirt.org Sent: Tuesday, February 4, 2014 12:15:42 AM Subject: Re: [Users] oVirt 3.3.3 RC EL6 Live Snapshot Maor I am not saying that we are not doing a live snapshot :) I am saying that we need a print in the log that states live snapshot command was called i.e: Print in the log: LiveSnapshotCommand -> this can call to the rest of snapshotVDSCreateCommand. On 02/03/2014 07:38 PM, Maor Lipchuk wrote: > On 02/03/2014 07:46 PM, Dafna Ron wrote: >> On 02/03/2014 05:34 PM, Maor Lipchuk wrote: >>> On 02/03/2014 07:18 PM, Dafna Ron wrote: >>>> Maor, >>>> >>>> If snapshotVDSCommand is for live snapshot, what is the offline create >>>> snapshot command? >>> It is the CreateSnapshotVdsCommand which calls createVolume in VDSM >> but we need to be able to know that a live snapshot was sent and not an >> offline snapshot. > Yes, at the logs we can see the all process : > > First a request to create a snapshot (new volume) sent to VDSM: > 2014-02-02 09:41:09,557 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] > (pool-6-thread-49) [67ea047a] START, CreateSnapshotVDSCommand( > storagePoolId = fcb89071-6cdb-4972-94d1-c9324cebf814, > ignoreFailoverLimit = false, storageDomainId = > a52938f7-2cf4-4771-acb2-0c78d14999e5, imageGroupId = > c1cb6b66-655e-48c3-8568-4975295eb037, imageSizeInBytes = 21474836480, > volumeFormat = COW, newImageId = 6d8c80a4-328f-4a53-86a2-a4080a2662ce, > newImageDescription = , imageId = 5085422e-6592-415a-9da3-9e43dac9374b, > sourceImageGroupId = c1cb6b66-655e-48c3-8568-4975295eb037), log id: 7875f3f5 > > after the snapshot gets created : > 2014-02-02 09:41:20,553 INFO > [org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand] > (pool-6-thread-49) Ending command successfully: > org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand > > then the engine calls the live snapshot (see also [1]) > 2014-02-02 09:41:30,234 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] > (pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872 > >> Elad, somewhere in this flow we need to know that the snapshot was taken >> on a running vm :) this seems like a bug to me. >>>> we did not say that live snapshot did not succeed :) we said that the >>>> vm is paused and restarted - which is something that should not happen >>>> for live snapshot (or at least never did before). >>> It's not sure that the restart is related to the live snapshot. but that >>> should be observed in the libvirt/vdsm logs. >> yes, I am sure because the user is reporting it and the logs show it... >>>> as I wrote before, we know that vdsm is reporting the vm as paused, that >>>> is because libvirt is reporting the vm as paused and I think that its >>>> happening because libvirt is not doing a live snapshot and so pauses the >>>> vm while taking the snapshot. >>> That sounds logic to me, it's need to be checked with libvirt, if that >>> kind of behaviour could happen. >> Elad, can you please try to reproduce and open a bug to libvirt? >> >>>> Dafna >>>> >>>> >>>> On 02/03/2014 05:08 PM, Maor Lipchuk wrote: >>>>> From the engine logs it seems that indeed live snapshot is called >>>>> (The >>>>> command is snapshotVDSCommand see [1]). >>>>> This is done right after the snapshot has been created in the VM and it >>>>> signals the qemu process to start using the new volume created. >>>>> >>>>> When live snapshot does not succeed we should see in the log something >>>>> like "Wasn't able to live snapshot due to error:...", but it does not >>>>> appear so it seems that this worked out fine. >>>>> >>>>> At some point I can see in the logs that VDSM reports to the engine >>>>> that >>>>> the VM is paused. >>>>> >>>>> >>>>> [1] >>>>> 2014-02-02 09:41:20,564 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>> (pool-6-thread-49) START, SnapshotVDSCommand(HostName = ovirt002, >>>>> HostId >>>>> = 3080fb61-2d03-4008-b47f-9b66276a4257, >>>>> vmId=e261e707-a21f-4ae8-9cff-f535f4430446), log id: 7e0d7872 >>>>> 2014-02-02 09:41:21,119 INFO >>>>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] >>>>> (DefaultQuartzScheduler_Worker-93) VM snapshot-test >>>>> e261e707-a21f-4ae8-9cff-f535f4430446 moved from Up --> Paused >>>>> 2014-02-02 09:41:30,234 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] >>>>> (pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872 >>>>> 2014-02-02 09:41:30,238 INFO >>>>> [org.ovirt.engine.core.bll.CreateSnapshotCommand] (pool-6-thread-49) >>>>> [67ea047a] Ending command successfully: >>>>> org.ovirt.engine.core.bll.CreateSnapshotCommand >>>>> ... >>>>> >>>>> Regards, >>>>> Maor >>>>> >>>>> On 02/03/2014 06:24 PM, Dafna Ron wrote: >>>>>> Thanks Steve. >>>>>> >>>>>> from the logs I can see that the create snapshot succeeds and that the >>>>>> vm is resumed. >>>>>> the vm moves to pause as part of libvirt flows: >>>>>> >>>>>> 2014-02-02 14:41:20.872+0000: 5843: debug : >>>>>> qemuProcessHandleStop:728 : >>>>>> Transitioned guest snapshot-test to paused state >>>>>> 2014-02-02 14:41:30.031+0000: 5843: debug : >>>>>> qemuProcessHandleResume:776 >>>>>> : Transitioned guest snapshot-test out of paused into resumed state >>>>>> >>>>>> There are bugs here but I am not sure yet if this is libvirt >>>>>> regression >>>>>> or engine. >>>>>> >>>>>> I'm adding Elad and Maor since in engine logs I can't see anything >>>>>> calling for live snapshot (only for snapshot) - Maor, shouldn't live >>>>>> snapshot command be logged somewhere in the logs? >>>>>> Is it possible that engine is calling to create snapshot and not >>>>>> create >>>>>> live snapshot which is why the vm pauses? >>>>>> >>>>>> Elad, if engine is not logging live snapshot anywhere I would open >>>>>> a bug >>>>>> for engine (to print that in the logs). >>>>>> Also, there is a bug in vdsm log for sdc where the below is logged as >>>>>> ERROR and not INFO: >>>>>> >>>>>> Thread-23::ERROR::2014-02-02 >>>>>> 09:51:19,497::sdc::137::Storage.StorageDomainCache::(_findDomain) >>>>>> looking for unfetched domain a52938f7-2cf4-4771-acb2-0c78d14999e5 >>>>>> Thread-23::ERROR::2014-02-02 >>>>>> 09:51:19,497::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) >>>>>> >>>>>> >>>>>> looking for domain a52938f7-2cf4-4771-acb2-0c78d14999e5 >>>>>> >>>>>> If the engine was sending live snapshot or if there is no >>>>>> difference in >>>>>> the two commands in engine side than I would open a bug for libvirt >>>>>> for >>>>>> pausing the vm during live snapshot. >>>>>> >>>>>> Dafna >>>>>> >>>>>> On 02/03/2014 02:41 PM, Steve Dainard wrote: >>>>>>> [root@ovirt002 ~]# vdsClient -s 0 getStorageDomainInfo >>>>>>> a52938f7-2cf4-4771-acb2-0c78d14999e5 >>>>>>> uuid = a52938f7-2cf4-4771-acb2-0c78d14999e5 >>>>>>> pool = ['fcb89071-6cdb-4972-94d1-c9324cebf814'] >>>>>>> lver = 5 >>>>>>> version = 3 >>>>>>> role = Master >>>>>>> remotePath = gluster-store-vip:/rep1 >>>>>>> spm_id = 2 >>>>>>> type = NFS >>>>>>> class = Data >>>>>>> master_ver = 1 >>>>>>> name = gluster-store-rep1 >>>>>>> >>>>>>> >>>>>>> *Steve Dainard * >>>>>>> IT Infrastructure Manager >>>>>>> Miovision <http://miovision.com/> | /Rethink Traffic/ >>>>>>> 519-513-2407 ex.250 >>>>>>> 877-646-8476 (toll-free) >>>>>>> >>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn >>>>>>> <https://www.linkedin.com/company/miovision-technologies> | Twitter >>>>>>> <https://twitter.com/miovision> | Facebook >>>>>>> <https://www.facebook.com/miovision>* >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> >>>>>>> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, >>>>>>> Kitchener, >>>>>>> ON, Canada | N2C 1L3 >>>>>>> This e-mail may contain information that is privileged or >>>>>>> confidential. If you are not the intended recipient, please delete >>>>>>> the >>>>>>> e-mail and any attachments and notify us immediately. >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 2, 2014 at 2:55 PM, Dafna Ron <dron(a)redhat.com >>>>>>> <mailto:dron@redhat.com>> wrote: >>>>>>> >>>>>>> please run vdsClient -s 0 getStorageDomainInfo >>>>>>> a52938f7-2cf4-4771-acb2-0c78d14999e5 >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 02/02/2014 03:02 PM, Steve Dainard wrote: >>>>>>> >>>>>>> Logs attached with VM running on qemu-kvm-rhev packages >>>>>>> installed. >>>>>>> >>>>>>> *Steve Dainard * >>>>>>> IT Infrastructure Manager >>>>>>> Miovision <http://miovision.com/> | /Rethink Traffic/ >>>>>>> 519-513-2407 <tel:519-513-2407> ex.250 >>>>>>> >>>>>>> 877-646-8476 <tel:877-646-8476> (toll-free) >>>>>>> >>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn >>>>>>> >>>>>>> <https://www.linkedin.com/company/miovision-technologies> | >>>>>>> Twitter <https://twitter.com/miovision> | Facebook >>>>>>> <https://www.facebook.com/miovision>* >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> >>>>>>> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, >>>>>>> Kitchener, ON, Canada | N2C 1L3 >>>>>>> This e-mail may contain information that is privileged or >>>>>>> confidential. If you are not the intended recipient, please >>>>>>> delete the e-mail and any attachments and notify us >>>>>>> immediately. >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 2, 2014 at 5:05 AM, Dafna Ron <dron(a)redhat.com >>>>>>> <mailto:dron@redhat.com> <mailto:dron@redhat.com >>>>>>> <mailto:dron@redhat.com>>> wrote: >>>>>>> >>>>>>> can you please upload full engine, vdsm, libvirt and >>>>>>> vm's >>>>>>> qemu logs? >>>>>>> >>>>>>> >>>>>>> On 02/02/2014 02:08 AM, Steve Dainard wrote: >>>>>>> >>>>>>> I have two CentOS 6.5 Ovirt hosts (ovirt001, >>>>>>> ovirt002) >>>>>>> >>>>>>> I've installed the applicable qemu-kvm-rhev >>>>>>> packages >>>>>>> from this >>>>>>> site: >>>>>>> http://www.dreyou.org/ovirt/vdsm32/Packages/ on >>>>>>> ovirt002. >>>>>>> >>>>>>> On ovirt001 if I take a live snapshot: >>>>>>> >>>>>>> Snapshot 'test qemu-kvm' creation for VM >>>>>>> 'snapshot-test' was >>>>>>> initiated by admin@internal. >>>>>>> The VM is paused >>>>>>> Failed to create live snapshot 'test qemu-kvm' >>>>>>> for VM >>>>>>> 'snapshot-test'. VM restart is recommended. >>>>>>> Failed to complete snapshot 'test qemu-kvm' >>>>>>> creation >>>>>>> for VM >>>>>>> 'snapshot-test'. >>>>>>> The VM is then started, and the status for the >>>>>>> snapshot >>>>>>> changes to OK. >>>>>>> >>>>>>> On ovirt002 (with the packages from dreyou) I don't >>>>>>> get any >>>>>>> messages about a snapshot failing, but my VM is >>>>>>> still >>>>>>> paused >>>>>>> to complete the snapshot. Is there something else >>>>>>> other than >>>>>>> the qemu-kvm-rhev packages that would enable this >>>>>>> functionality? >>>>>>> >>>>>>> I've looked for some information on when the >>>>>>> packages >>>>>>> would be >>>>>>> built as required in the CentOS repos, but I >>>>>>> don't see >>>>>>> anything definitive. >>>>>>> >>>>>>> >>>>>>> http://lists.ovirt.org/pipermail/users/2013-December/019126.html >>>>>>> Looks like one of the maintainers is waiting for >>>>>>> someone to >>>>>>> tell him what flags need to be set. >>>>>>> >>>>>>> Also, another thread here: >>>>>>> >>>>>>> http://comments.gmane.org/gmane.comp.emulators.ovirt.arch/1618 >>>>>>> same maintainer, mentioning that he hasn't seen >>>>>>> anything in >>>>>>> the bug tracker. >>>>>>> >>>>>>> There is a bug here: >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1009100 that >>>>>>> seems >>>>>>> to have ended in finding a way for qemu to expose >>>>>>> whether it >>>>>>> supports live snapshots, rather than figuring >>>>>>> out how >>>>>>> to get >>>>>>> the CentOS team the info they need to build the >>>>>>> packages with >>>>>>> the proper flags set. >>>>>>> >>>>>>> I have bcc'd both dreyou (packaged the >>>>>>> qemu-kvm-rhev >>>>>>> packages >>>>>>> listed above) and Russ (CentOS maintainer >>>>>>> mentioned in >>>>>>> the >>>>>>> other threads) if they wish to chime in and perhaps >>>>>>> collaborate on which flags, if any, should be set >>>>>>> for the >>>>>>> qemu-kvm builds so we can get a CentOS bug report >>>>>>> going and >>>>>>> hammer this out. >>>>>>> >>>>>>> Thanks everyone. >>>>>>> >>>>>>> **crosses fingers and hopes for live snapshots >>>>>>> soon** >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Steve Dainard * >>>>>>> IT Infrastructure Manager >>>>>>> Miovision <http://miovision.com/> | /Rethink >>>>>>> Traffic/ >>>>>>> 519-513-2407 <tel:519-513-2407> <tel:519-513-2407 >>>>>>> <tel:519-513-2407>> <tel:519-513-2407 <tel:519-513-2407> >>>>>>> <tel:519-513-2407 <tel:519-513-2407>>> ex.250 >>>>>>> 877-646-8476 <tel:877-646-8476> <tel:877-646-8476 >>>>>>> <tel:877-646-8476>> <tel:877-646-8476 <tel:877-646-8476> >>>>>>> >>>>>>> <tel:877-646-8476 <tel:877-646-8476>>> (toll-free) >>>>>>> >>>>>>> *Blog <http://miovision.com/blog> | **LinkedIn >>>>>>> >>>>>>> <https://www.linkedin.com/company/miovision-technologies> | >>>>>>> Twitter <https://twitter.com/miovision> | Facebook >>>>>>> <https://www.facebook.com/miovision>* >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Miovision Technologies Inc. | 148 Manitou Drive, >>>>>>> Suite >>>>>>> 101, >>>>>>> Kitchener, ON, Canada | N2C 1L3 >>>>>>> This e-mail may contain information that is >>>>>>> privileged or >>>>>>> confidential. If you are not the intended >>>>>>> recipient, >>>>>>> please >>>>>>> delete the e-mail and any attachments and notify us >>>>>>> immediately. >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 31, 2014 at 1:26 PM, Steve Dainard >>>>>>> <sdainard(a)miovision.com >>>>>>> <mailto:sdainard@miovision.com> >>>>>>> <mailto:sdainard@miovision.com >>>>>>> <mailto:sdainard@miovision.com>> >>>>>>> <mailto:sdainard@miovision.com >>>>>>> <mailto:sdainard@miovision.com> >>>>>>> >>>>>>> <mailto:sdainard@miovision.com >>>>>>> <mailto:sdainard@miovision.com>>>> wrote: >>>>>>> >>>>>>> >>>>>>> How would you developers, speaking for the >>>>>>> oVirt-community, >>>>>>> propose to >>>>>>> solve this for CentOS _now_ ? >>>>>>> >>>>>>> I would imagine that the easiest way is >>>>>>> that >>>>>>> you build and >>>>>>> host this one >>>>>>> package(qemu-kvm-rhev), since you´ve >>>>>>> basically >>>>>>> already >>>>>>> have >>>>>>> the source >>>>>>> and recipe (since you´re already >>>>>>> providing it >>>>>>> for RHEV >>>>>>> anyway). Then, >>>>>>> once that´s in place, it´s more a >>>>>>> question of >>>>>>> where to >>>>>>> host the >>>>>>> packages, in what repository. Be it your >>>>>>> own, >>>>>>> or some >>>>>>> other >>>>>>> repo set up >>>>>>> for the SIG. >>>>>>> >>>>>>> This is my view, how I as a user view this >>>>>>> issue. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I think this is a pretty valid view. >>>>>>> >>>>>>> What would it take to get the correct qemu >>>>>>> package >>>>>>> hosted >>>>>>> in the >>>>>>> ovirt repo? >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Med Vänliga Hälsningar >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> Karli Sjöberg >>>>>>> Swedish University of Agricultural Sciences >>>>>>> Box 7079 >>>>>>> (Visiting >>>>>>> Address >>>>>>> Kronåsvägen 8) >>>>>>> S-750 07 Uppsala, Sweden >>>>>>> Phone: +46-(0)18-67 15 66 >>>>>>> <tel:%2B46-%280%2918-67%2015%2066> >>>>>>> <tel:%2B46-%280%2918-67%2015%2066> >>>>>>> <tel:%2B46-%280%2918-67%2015%2066> >>>>>>> karli.sjoberg(a)slu.se <mailto:karli.sjoberg@slu.se> >>>>>>> <mailto:karli.sjoberg@slu.se <mailto:karli.sjoberg@slu.se>> >>>>>>> <mailto:karli.sjoberg@slu.se >>>>>>> <mailto:karli.sjoberg@slu.se> <mailto:karli.sjoberg@slu.se >>>>>>> <mailto:karli.sjoberg@slu.se>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org> >>>>>>> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >>>>>>> <mailto:Users@ovirt.org <mailto:Users@ovirt.org> >>>>>>> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>> >>>>>>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org> >>>>>>> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >>>>>>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- Dafna Ron >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- Dafna Ron >>>>>>> >>>>>>>