[ovirt-users] VM incremental backup

Nathanaël Blanchet blanchet at abes.fr
Wed Oct 12 05:08:25 EDT 2016


Hi,

After testing many ways to transfer sparse file between servers (in 
local cp is enough), here is the only way wich is time efficient:

Source side

# tar cSf - /var/lib/libvirt/images/acore_thin.img | nc -l 5000

Destination side

# nc 10.34.100.128 5000 | pv -btr | tar xSf -
3,54GB 0:02:19 [  26MB/s]

# ls -lsh /var/lib/libvirt/images/acore_thin.img

3,8G -rw-r--r--. 1 root root 40G  6 oct.  12:43 
/var/lib/libvirt/images/acore_thin.img

Le 07/10/2016 à 08:50, Arman Khalatyan a écrit :
>
> The sparse files are special files(please checkout the wiki pages) to 
> conserve some diskspace and copy time.
> The rsync --sparce does not take an advantage of it. It is trying to 
> checksum whole virtual diskspace which is obviously zero filled.
> If you "cp" file then "cp" will try to autodetect it otherwise your 
> file willbe converted to regular file. You can also copy files with 
> "tar -S ".
> To check the size of actual vs virtual(sparse) you can use:
> df -bsh filename vs  df -sh
>
>
> Am 06.10.2016 11:34 vorm. schrieb "Nathanaël Blanchet" 
> <blanchet at abes.fr <mailto:blanchet at abes.fr>>:
>
>     Hi all,
>
>     I didn't know this tool, after testing the original 40GB image
>     size is reduced as expected to 2,9GB with du -sh, but transfer
>     between servers continue to act as a 40GB image.
>
>     So if I understand virt-sparsify reduces the effective space on
>     the local disk, but the disk is seen with its original size by
>     other systems, and there is no advantage in transferring the image
>     from one server to an other.
>
>     Please tell me if I'm wrong.
>
>     Le 29/09/2016 à 15:10, Kai Wagner a écrit :
>>
>>     Hey,
>>
>>     I don't wont to break into your discussion, but I also never
>>     heard about virt-sparsfiy and the sparse option for dd.
>>
>>     I tested it and it works like a charm. I converted a block device
>>     65GB with dd sparse to 40GB raw image and afterwards I used
>>     virsh-sparsify to reduce the size down to 6.8GB into a qcow2
>>     image file.
>>
>>     Thanks a lot for that hint.
>>
>>     Kai
>>
>>     Am 27.09.2016 um 15:17 schrieb Sven Achtelik:
>>>
>>>     No, I never came across this approach. I didn’t know about
>>>     virt-sparsify.
>>>
>>>     I’ll look into that and give it a try.
>>>
>>>     Thank you
>>>
>>>     it-novum GmbH
>>>     i. A. Kai Wagner ● Team Lead Support & Presales openATTIC
>>>      _____________________________________________________________
>>>     Unser Newsletter Service: Jetzt Business Open Source News
>>>     abonnieren! <http://www.it-novum.com/newsletter?wm=mail>
>>>     <http://www.it-novum.com/newsletter?wm=mail>
>>>     Folgen Sie uns:
>>>     <http://www.it-novum.com/blog/?wm=mail><http://www.it-novum.com/blog/feed?wm=mail><https://twitter.com/itnovum><http://www.xing.com/companies/itnovumgmbh/about><http://www.linkedin.com/company/it-novum-gmbh><https://plus.google.com/105835582713486693501><http://www.youtube.com/user/itnovumGmbH?sub_confirmation=1%20><https://www.facebook.com/itnovum>
>>>
>>>
>>>
>>>     <http://www.it-novum.com/?wm=mail> 	Tel: +49 661 103-762
>>>     <tel:%2B49%20661%20103-762>
>>>     Fax: +49 661 10317762 <tel:%2B49%20661%2010317762>
>>>     Mail: Kai.Wagner at it-novum.com <mailto:Kai.Wagner at it-novum.com>
>>>
>>>
>>>     it-novum GmbH • Edelzeller Straße 44 • 36043 Fulda •
>>>     http://www.it-novum.com/
>>>     Handelsregister Amtsgericht Fulda, HRB 1934 • Geschäftsführer:
>>>     Michael Kienle • Sitz der Gesellschaft: Fulda
>>>
>>>     Der Inhalt dieser E-Mail ist vertraulich. Wenn Sie nicht der
>>>     eigentliche Empfänger sein sollten, informieren Sie bitte sofort
>>>     den Absender oder vernichten umgehend diese Mail. Jegliche
>>>     unerlaubte Vervielfältigung oder Weiterleitung dieser Mail ist
>>>     strengstens verboten.
>>>     This e-mail may contain confidential and/or priviledged
>>>     information. If you are not the intended recepient (or have
>>>     received this e-mail in error) please notify the sender
>>>     immediately and destroy this e-mail. Any unauthorised copying,
>>>     disclosure or distribution of material in this e-mail is
>>>     strictly forbidden.
>>>
>>>     *Von:*Yaniv Dary [mailto:ydary at redhat.com]
>>>     *Gesendet:* Dienstag, 27. September 2016 15:10
>>>     *An:* Sven Achtelik <Sven.Achtelik at mailpool.us>
>>>     <mailto:Sven.Achtelik at mailpool.us>
>>>     *Cc:* Maton, Brett <matonb at ltresources.co.uk>
>>>     <mailto:matonb at ltresources.co.uk>; vasily.lamykin at MegaFon.ru
>>>     <mailto:vasily.lamykin at MegaFon.ru>; Ovirt Users
>>>     <users at ovirt.org> <mailto:users at ovirt.org>
>>>     *Betreff:* Re: [ovirt-users] VM incremental backup
>>>
>>>     As I see it you have two options:
>>>     - In backup use 'dd' with 'conv=sparse' (or similar tool that
>>>     allows sparse).
>>>
>>>     - After backup use virt-sparsify [1] to reduce the size to the
>>>     real used size prior to restore.
>>>
>>>     To make this extra efficient you can use virt-sparsify anyways
>>>     after backup to make the file even smaller.
>>>
>>>     Have you considered this approach?
>>>
>>>     [1] http://libguestfs.org/virt-sparsify.1.html
>>>     <http://libguestfs.org/virt-sparsify.1.html>
>>>
>>>
>>>     Yaniv Dary
>>>     Technical Product Manager
>>>     Red Hat Israel Ltd.
>>>     34 Jerusalem Road
>>>     Building A, 4th floor
>>>     Ra'anana, Israel 4350109
>>>     Tel : +972 (9) 7692306 <tel:%2B972%20%289%29%207692306>
>>>             8272306
>>>     Email: ydary at redhat.com <mailto:ydary at redhat.com>
>>>     IRC : ydary
>>>
>>>     On Tue, Sep 27, 2016 at 4:01 PM, Sven Achtelik
>>>     <Sven.Achtelik at mailpool.us <mailto:Sven.Achtelik at mailpool.us>>
>>>     wrote:
>>>
>>>         Hi Yaniv,
>>>
>>>         how can this be done with DD ? Since it doesn’t know if the
>>>         block is free space ? I’ve been looking for such a solution
>>>         for a long time now. Everything I could find out was that I
>>>         have to use a utility that understands the FS and therefore
>>>         knows where the free space is.
>>>
>>>         Thank you,
>>>
>>>         Sven
>>>
>>>         *Von:*users-bounces at ovirt.org
>>>         <mailto:users-bounces at ovirt.org>
>>>         [mailto:users-bounces at ovirt.org
>>>         <mailto:users-bounces at ovirt.org>] *Im Auftrag von *Yaniv Dary
>>>         *Gesendet:* Dienstag, 27. September 2016 14:38
>>>         *An:* Maton, Brett <matonb at ltresources.co.uk
>>>         <mailto:matonb at ltresources.co.uk>>
>>>         *Cc:* vasily.lamykin at MegaFon.ru
>>>         <mailto:vasily.lamykin at MegaFon.ru>; Ovirt Users
>>>         <users at ovirt.org <mailto:users at ovirt.org>>
>>>         *Betreff:* Re: [ovirt-users] VM incremental backup
>>>
>>>         Full VM disk backup. If you use dd you can drop the 0 parts
>>>         of the disk.
>>>
>>>
>>>         Yaniv Dary
>>>
>>>         Technical Product Manager
>>>
>>>         Red Hat Israel Ltd.
>>>
>>>         34 Jerusalem Road
>>>
>>>         Building A, 4th floor
>>>
>>>         Ra'anana, Israel 4350109
>>>
>>>         Tel : +972 (9) 7692306 <tel:%2B972%20%289%29%207692306>
>>>
>>>                 8272306
>>>
>>>         Email: ydary at redhat.com <mailto:ydary at redhat.com>
>>>
>>>         IRC : ydary
>>>
>>>         2016-09-27 15:27 GMT+03:00 Maton, Brett
>>>         <matonb at ltresources.co.uk <mailto:matonb at ltresources.co.uk>>:
>>>
>>>             Is there a preferred or recommended backup method in the
>>>             meantime ?
>>>
>>>             2016-09-22 10:30 GMT+01:00 vasily.lamykin at MegaFon.ru
>>>             <mailto:vasily.lamykin at MegaFon.ru>
>>>             <vasily.lamykin at megafon.ru
>>>             <mailto:vasily.lamykin at megafon.ru>>:
>>>
>>>                 Hello!
>>>
>>>                 I would like to understand the existence of the
>>>                 possibility of incremental VM backups.
>>>
>>>                 And if it is, the means by which it is achieved.
>>>
>>>                 While I was only able to achieve a complete backup
>>>                 of the virtual machine (but it is a very large
>>>                 amount of information and time) by cloning a
>>>                 snapshot of the VM and export to data domain.
>>>
>>>                 *Ламыкин Василий*
>>>
>>>                 Старший инженер по эксплуатации сервисных платформ
>>>
>>>                 Столичный ф-ал ПАО "МегаФон
>>>
>>>                 +7 (926) 500-3308 <tel:%2B7%20%28926%29%20500-3308>
>>>
>>>                 МегаФон лого+знак РУС B2C
>>>
>>>                 ------------------------------------------------------------------------
>>>
>>>
>>>                 Информация в этом сообщении предназначена
>>>                 исключительно для конкретных лиц, которым она
>>>                 адресована. В сообщении может содержаться
>>>                 конфиденциальная информация, которая не может быть
>>>                 раскрыта или использована кем-либо, кроме адресатов.
>>>                 Если вы не адресат этого сообщения, то
>>>                 использование, переадресация, копирование или
>>>                 распространение содержания сообщения или его части
>>>                 незаконно и запрещено. Если Вы получили это
>>>                 сообщение ошибочно, пожалуйста, незамедлительно
>>>                 сообщите отправителю об этом и удалите со всем
>>>                 содержимым само сообщение и любые возможные его
>>>                 копии и приложения.
>>>
>>>                 The information contained in this communication is
>>>                 intended solely for the use of the individual or
>>>                 entity to whom it is addressed and others authorized
>>>                 to receive it. It may contain confidential or
>>>                 legally privileged information. The contents may not
>>>                 be disclosed or used by anyone other than the
>>>                 addressee. If you are not the intended recipient(s),
>>>                 any use, disclosure, copying, distribution or any
>>>                 action taken or omitted to be taken in reliance on
>>>                 it is prohibited and may be unlawful. If you have
>>>                 received this communication in error please notify
>>>                 us immediately by responding to this email and then
>>>                 delete the e-mail and all attachments and any copies
>>>                 thereof.
>>>
>>>                 (c)20mf50
>>>
>>>                 _______________________________________________
>>>                 Users mailing list
>>>                 Users at ovirt.org <mailto:Users at ovirt.org>
>>>                 http://lists.ovirt.org/mailman/listinfo/users
>>>                 <http://lists.ovirt.org/mailman/listinfo/users>
>>>
>>>
>>>             _______________________________________________
>>>             Users mailing list
>>>             Users at ovirt.org <mailto:Users at ovirt.org>
>>>             http://lists.ovirt.org/mailman/listinfo/users
>>>             <http://lists.ovirt.org/mailman/listinfo/users>
>>>
>>>
>>>
>>>     _______________________________________________
>>>     Users mailing list
>>>     Users at ovirt.org <mailto:Users at ovirt.org>
>>>     http://lists.ovirt.org/mailman/listinfo/users
>>>     <http://lists.ovirt.org/mailman/listinfo/users>
>>
>>     _______________________________________________
>>     Users mailing list
>>     Users at ovirt.org <mailto:Users at ovirt.org>
>>     http://lists.ovirt.org/mailman/listinfo/users
>>     <http://lists.ovirt.org/mailman/listinfo/users>
>
>     -- 
>     Nathanaël Blanchet
>
>     Supervision réseau
>     Pôle Infrastrutures Informatiques
>     227 avenue Professeur-Jean-Louis-Viala
>     34193 MONTPELLIER CEDEX 5 	
>     Tél. 33 (0)4 67 54 84 55
>     Fax  33 (0)4 67 54 84 14
>     blanchet at abes.fr <mailto:blanchet at abes.fr>  
>
>     _______________________________________________ Users mailing list
>     Users at ovirt.org <mailto:Users at ovirt.org>
>     http://lists.ovirt.org/mailman/listinfo/users
>     <http://lists.ovirt.org/mailman/listinfo/users> 
>
-- 
Nathanaël Blanchet

Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 	
Tél. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
blanchet at abes.fr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1317 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0010.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 24630 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0001.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 4050 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1422 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0012.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1318 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0013.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1359 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0014.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1276 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0015.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1378 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0016.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1399 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0017.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 1206 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0018.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/x-png
Size: 6808 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0019.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 18112 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161012/29b67d3c/attachment-0001.png>


More information about the Users mailing list