Re: [Users] oVirt 3.3.3 RC EL6 Live Snapshot

4 Feb 2014

      ...
From what I saw in the thread, libvirt pauses the VM, which effects the continuity of its operation.
I checked it also in one of the latest builds of 3.3 and I observed the same behaviour:
2014-02-04 10:18:29,441 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [5cbc7ab5] VM nfs2-1 86728f5c-3583-420f-9536-bfabaf11b235 moved from Up --> Paused
Dafna, I saw you've already opened a bug on that:
https://bugzilla.redhat.com/show_bug.cgi?id=1057587 

----- Original Message -----
From: "Dafna Ron" <dron@redhat.com>
To: "Maor Lipchuk" <mlipchuk@redhat.com>
Cc: "Steve Dainard" <sdainard@miovision.com>, "Elad Ben Aharon" <ebenahar@redhat.com>, "Karli Sjöberg" <Karli.Sjoberg@slu.se>, users@ovirt.org
Sent: Tuesday, February 4, 2014 12:15:42 AM
Subject: Re: [Users] oVirt 3.3.3 RC EL6 Live Snapshot

Maor I am not saying that we are not doing a live snapshot :) I am 
saying that we need a print in the log that states live snapshot command 
was called i.e: Print in the log: LiveSnapshotCommand -> this can call 
to the rest of snapshotVDSCreateCommand.

On 02/03/2014 07:38 PM, Maor Lipchuk wrote:
...
On 02/03/2014 07:46 PM, Dafna Ron wrote:
...
On 02/03/2014 05:34 PM, Maor Lipchuk wrote:
...
On 02/03/2014 07:18 PM, Dafna Ron wrote:
...
Maor,
If snapshotVDSCommand is for live snapshot, what is the offline create
snapshot command?
It is the CreateSnapshotVdsCommand which calls createVolume in VDSM
but we need to be able to know that a live snapshot was sent and not an
offline snapshot.
Yes, at the logs we can see the all process :
First a request to create a snapshot (new volume) sent to VDSM:
2014-02-02 09:41:09,557 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand]
(pool-6-thread-49) [67ea047a] START, CreateSnapshotVDSCommand(
storagePoolId = fcb89071-6cdb-4972-94d1-c9324cebf814,
ignoreFailoverLimit = false, storageDomainId =
a52938f7-2cf4-4771-acb2-0c78d14999e5, imageGroupId =
c1cb6b66-655e-48c3-8568-4975295eb037, imageSizeInBytes = 21474836480,
volumeFormat = COW, newImageId = 6d8c80a4-328f-4a53-86a2-a4080a2662ce,
newImageDescription = , imageId = 5085422e-6592-415a-9da3-9e43dac9374b,
sourceImageGroupId = c1cb6b66-655e-48c3-8568-4975295eb037), log id: 7875f3f5
after the snapshot gets created :
2014-02-02 09:41:20,553 INFO
[org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand]
(pool-6-thread-49) Ending command successfully:
org.ovirt.engine.core.bll.CreateAllSnapshotsFromVmCommand
then the engine calls the live snapshot (see also [1])
2014-02-02 09:41:30,234 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
(pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872
...
Elad, somewhere in this flow we need to know that the snapshot was taken
on a running vm :) this seems like a bug to me.
...
...
we did not say that live snapshot did not succeed :)  we said that the
vm is paused and restarted - which is something that should not happen
for live snapshot (or at least never did before).
It's not sure that the restart is related to the live snapshot. but that
should be observed in the libvirt/vdsm logs.
yes, I am sure because the user is reporting it and the logs show it...
as I wrote before, we know that vdsm is reporting the vm as paused, that
is because libvirt is reporting the vm as paused and I think that its
happening because libvirt is not doing a live snapshot and so pauses the
vm while taking the snapshot.
That sounds logic to me, it's need to be checked with libvirt, if that
kind of behaviour could happen.
Elad, can you please try to reproduce and open a bug to libvirt?
...
...
Dafna
On 02/03/2014 05:08 PM, Maor Lipchuk wrote:
...
From the engine logs it seems that indeed live snapshot is called
(The
command is snapshotVDSCommand see [1]).
This is done right after the snapshot has been created in the VM and it
signals the qemu process to start using the new volume created.
When live snapshot does not succeed we should see in the log something
like "Wasn't able to live snapshot due to error:...", but it does not
appear so it seems that this worked out fine.
At some point I can see in the logs that VDSM reports to the engine
that
the VM is paused.
[1]
2014-02-02 09:41:20,564 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
(pool-6-thread-49) START, SnapshotVDSCommand(HostName = ovirt002,
HostId
= 3080fb61-2d03-4008-b47f-9b66276a4257,
vmId=e261e707-a21f-4ae8-9cff-f535f4430446), log id: 7e0d7872
2014-02-02 09:41:21,119 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-93) VM snapshot-test
e261e707-a21f-4ae8-9cff-f535f4430446 moved from Up --> Paused
2014-02-02 09:41:30,234 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand]
(pool-6-thread-49) FINISH, SnapshotVDSCommand, log id: 7e0d7872
2014-02-02 09:41:30,238 INFO
[org.ovirt.engine.core.bll.CreateSnapshotCommand] (pool-6-thread-49)
[67ea047a] Ending command successfully:
org.ovirt.engine.core.bll.CreateSnapshotCommand
...
Regards,
Maor
On 02/03/2014 06:24 PM, Dafna Ron wrote:
...
Thanks Steve.
from the logs I can see that the create snapshot succeeds and that the
vm is resumed.
the vm moves to pause as part of libvirt flows:
2014-02-02 14:41:20.872+0000: 5843: debug :
qemuProcessHandleStop:728 :
Transitioned guest snapshot-test to paused state
2014-02-02 14:41:30.031+0000: 5843: debug :
qemuProcessHandleResume:776
: Transitioned guest snapshot-test out of paused into resumed state
There are bugs here but I am not sure yet if this is libvirt
regression
or engine.
I'm adding Elad and Maor since in engine logs I can't see anything
calling for live snapshot (only for snapshot) - Maor, shouldn't live
snapshot command be logged somewhere in the logs?
Is it possible that engine is calling to create snapshot and not
create
live snapshot which is why the vm pauses?
Elad, if engine is not logging live snapshot anywhere I would open
a bug
for engine (to print that in the logs).
Also, there is a bug in vdsm log for sdc where the below is logged as
ERROR and not INFO:
Thread-23::ERROR::2014-02-02
09:51:19,497::sdc::137::Storage.StorageDomainCache::(_findDomain)
looking for unfetched domain a52938f7-2cf4-4771-acb2-0c78d14999e5
Thread-23::ERROR::2014-02-02
09:51:19,497::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain)
looking for domain a52938f7-2cf4-4771-acb2-0c78d14999e5
If the engine was sending live snapshot or if there is no
difference in
the two commands in engine side than I would open a bug for libvirt
for
pausing the vm during live snapshot.
Dafna
On 02/03/2014 02:41 PM, Steve Dainard wrote:
> [root@ovirt002 ~]# vdsClient -s 0 getStorageDomainInfo
> a52938f7-2cf4-4771-acb2-0c78d14999e5
> uuid = a52938f7-2cf4-4771-acb2-0c78d14999e5
> pool = ['fcb89071-6cdb-4972-94d1-c9324cebf814']
> lver = 5
> version = 3
> role = Master
> remotePath = gluster-store-vip:/rep1
> spm_id = 2
> type = NFS
> class = Data
> master_ver = 1
> name = gluster-store-rep1
>
>
> *Steve Dainard *
> IT Infrastructure Manager
> Miovision <http://miovision.com/> | /Rethink Traffic/
> 519-513-2407 ex.250
> 877-646-8476 (toll-free)
>
> *Blog <http://miovision.com/blog> | **LinkedIn
> <https://www.linkedin.com/company/miovision-technologies>  | Twitter
> <https://twitter.com/miovision>  | Facebook
> <https://www.facebook.com/miovision>*
> ------------------------------------------------------------------------
>
>
> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101,
> Kitchener,
> ON, Canada | N2C 1L3
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete
> the
> e-mail and any attachments and notify us immediately.
>
>
> On Sun, Feb 2, 2014 at 2:55 PM, Dafna Ron <dron@redhat.com
> <mailto:dron@redhat.com>> wrote:
>
>        please run vdsClient -s 0 getStorageDomainInfo
>        a52938f7-2cf4-4771-acb2-0c78d14999e5
>
>        Thanks,
>
>        Dafna
>
>
>
>        On 02/02/2014 03:02 PM, Steve Dainard wrote:
>
>            Logs attached with VM running on qemu-kvm-rhev packages
> installed.
>
>            *Steve Dainard *
>            IT Infrastructure Manager
>            Miovision <http://miovision.com/> | /Rethink Traffic/
>            519-513-2407 <tel:519-513-2407> ex.250
>
>            877-646-8476 <tel:877-646-8476> (toll-free)
>
>            *Blog <http://miovision.com/blog> | **LinkedIn
>           
> <https://www.linkedin.com/company/miovision-technologies>  |
>            Twitter <https://twitter.com/miovision>  | Facebook
>            <https://www.facebook.com/miovision>*
>         
> ------------------------------------------------------------------------
>
>
>            Miovision Technologies Inc. | 148 Manitou Drive, Suite 101,
>            Kitchener, ON, Canada | N2C 1L3
>            This e-mail may contain information that is privileged or
>            confidential. If you are not the intended recipient, please
>            delete the e-mail and any attachments and notify us
> immediately.
>
>
>            On Sun, Feb 2, 2014 at 5:05 AM, Dafna Ron <dron@redhat.com
>            <mailto:dron@redhat.com> <mailto:dron@redhat.com
>            <mailto:dron@redhat.com>>> wrote:
>
>                can you please upload full engine, vdsm, libvirt and
> vm's
>            qemu logs?
>
>
>                On 02/02/2014 02:08 AM, Steve Dainard wrote:
>
>                    I have two CentOS 6.5 Ovirt hosts (ovirt001,
> ovirt002)
>
>                    I've installed the applicable qemu-kvm-rhev
> packages
>            from this
>                    site:
> http://www.dreyou.org/ovirt/vdsm32/Packages/ on
>            ovirt002.
>
>                    On ovirt001 if I take a live snapshot:
>
>                    Snapshot 'test qemu-kvm' creation for VM
>            'snapshot-test' was
>                    initiated by admin@internal.
>                    The VM is paused
>                    Failed to create live snapshot 'test qemu-kvm'
> for VM
>                    'snapshot-test'. VM restart is recommended.
>                    Failed to complete snapshot 'test qemu-kvm'
> creation
>            for VM
>                    'snapshot-test'.
>                    The VM is then started, and the status for the
> snapshot
>                    changes to OK.
>
>                    On ovirt002 (with the packages from dreyou) I don't
>            get any
>                    messages about a snapshot failing, but my VM is
> still
>            paused
>                    to complete the snapshot. Is there something else
>            other than
>                    the qemu-kvm-rhev packages that would enable this
>            functionality?
>
>                    I've looked for some information on when the
> packages
>            would be
>                    built as required in the CentOS repos, but I
> don't see
>                    anything definitive.
>
>          
> http://lists.ovirt.org/pipermail/users/2013-December/019126.html
>                    Looks like one of the maintainers is waiting for
>            someone to
>                    tell him what flags need to be set.
>
>                    Also, another thread here:
>           
> http://comments.gmane.org/gmane.comp.emulators.ovirt.arch/1618
>                    same maintainer, mentioning that he hasn't seen
>            anything in
>                    the bug tracker.
>
>                    There is a bug here:
>            https://bugzilla.redhat.com/show_bug.cgi?id=1009100 that
> seems
>                    to have ended in finding a way for qemu to expose
>            whether it
>                    supports live snapshots, rather than figuring
> out how
>            to get
>                    the CentOS team the info they need to build the
>            packages with
>                    the proper flags set.
>
>                    I have bcc'd both dreyou (packaged the
> qemu-kvm-rhev
>            packages
>                    listed above) and Russ (CentOS maintainer
> mentioned in
> the
>                    other threads) if they wish to chime in and perhaps
>                    collaborate on which flags, if any, should be set
> for the
>                    qemu-kvm builds so we can get a CentOS bug report
>            going and
>                    hammer this out.
>
>                    Thanks everyone.
>
>                    **crosses fingers and hopes for live snapshots
> soon**
>
>
>
>                    *Steve Dainard *
>                    IT Infrastructure Manager
>                    Miovision <http://miovision.com/> | /Rethink
> Traffic/
>            519-513-2407 <tel:519-513-2407> <tel:519-513-2407
>            <tel:519-513-2407>> <tel:519-513-2407 <tel:519-513-2407>
>                    <tel:519-513-2407 <tel:519-513-2407>>> ex.250
>            877-646-8476 <tel:877-646-8476> <tel:877-646-8476
>            <tel:877-646-8476>> <tel:877-646-8476 <tel:877-646-8476>
>
>                    <tel:877-646-8476 <tel:877-646-8476>>> (toll-free)
>
>                    *Blog <http://miovision.com/blog> | **LinkedIn
>                        
> <https://www.linkedin.com/company/miovision-technologies>  |
>                    Twitter <https://twitter.com/miovision>  | Facebook
>                    <https://www.facebook.com/miovision>*
>                        
> ------------------------------------------------------------------------
>
>
>
>
>                    Miovision Technologies Inc. | 148 Manitou Drive,
> Suite
>            101,
>                    Kitchener, ON, Canada | N2C 1L3
>                    This e-mail may contain information that is
> privileged or
>                    confidential. If you are not the intended
> recipient,
>            please
>                    delete the e-mail and any attachments and notify us
>            immediately.
>
>
>                    On Fri, Jan 31, 2014 at 1:26 PM, Steve Dainard
>                    <sdainard@miovision.com
>            <mailto:sdainard@miovision.com>
> <mailto:sdainard@miovision.com
>            <mailto:sdainard@miovision.com>>
>                    <mailto:sdainard@miovision.com
>            <mailto:sdainard@miovision.com>
>
>                    <mailto:sdainard@miovision.com
>            <mailto:sdainard@miovision.com>>>> wrote:
>
>
>                            How would you developers, speaking for the
>                    oVirt-community,
>                            propose to
>                            solve this for CentOS _now_ ?
>
>                            I would imagine that the easiest way is
> that
>            you build and
>                            host this one
>                            package(qemu-kvm-rhev), since you´ve
> basically
>            already
>                    have
>                            the source
>                            and recipe (since you´re already
> providing it
>            for RHEV
>                            anyway). Then,
>                            once that´s in place, it´s more a
> question of
>            where to
>                    host the
>                            packages, in what repository. Be it your
> own,
>            or some
>                    other
>                            repo set up
>                            for the SIG.
>
>                            This is my view, how I as a user view this
> issue.
>
>
>
>                        I think this is a pretty valid view.
>
>                        What would it take to get the correct qemu
> package
>            hosted
>                    in the
>                        ovirt repo?
>
>                            --
>
>                            Med Vänliga Hälsningar
>
>                                       
> -------------------------------------------------------------------------------
>
>
>
>                            Karli Sjöberg
>                            Swedish University of Agricultural Sciences
>            Box 7079
>                    (Visiting
>                            Address
>                            Kronåsvägen 8)
>                            S-750 07 Uppsala, Sweden
>                            Phone: +46-(0)18-67 15 66
>            <tel:%2B46-%280%2918-67%2015%2066>
>                    <tel:%2B46-%280%2918-67%2015%2066>
>                    <tel:%2B46-%280%2918-67%2015%2066>
>            karli.sjoberg@slu.se <mailto:karli.sjoberg@slu.se>
>            <mailto:karli.sjoberg@slu.se <mailto:karli.sjoberg@slu.se>>
>                    <mailto:karli.sjoberg@slu.se
>            <mailto:karli.sjoberg@slu.se> <mailto:karli.sjoberg@slu.se
>            <mailto:karli.sjoberg@slu.se>>>
>
>                          
> _______________________________________________
>                            Users mailing list
>            Users@ovirt.org <mailto:Users@ovirt.org>
>            <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
>                    <mailto:Users@ovirt.org <mailto:Users@ovirt.org>
>            <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>>
>
>            http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
>
>
>                    _______________________________________________
>                    Users mailing list
>            Users@ovirt.org <mailto:Users@ovirt.org>
>            <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
>
>            http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>                --     Dafna Ron
>
>
>
>
>        --     Dafna Ron
>
>
-- 
Dafna Ron