Backups take sooo long, how can I improve the situation, where is the bottleneck?
Hallo everyone, I would like to update my Ovirt installation but, before that I would like to create proper VM backups. Currently I try to automate the backups and I use the implementation of the following project. https://github.com/wefixit-AT/oVirtBackup The main idea is to use the Ovirt-API to trigger a snapshot, create a new VM from the snapshot, delete the snapshot, export the vm to the export domain, delete the vm. This process takes even with only a 20 GB vm disk 55 Minutes. I have now 21 VMs with average 30 GB HDD space, that would mean that the backup job runs 22 hours. Is there something wrong with my setup? Config? Is there a better way to automate backups? Is anyone else doing VM Exports as backup or are you just using snapshots? Specs: HA Cluster 3 Nodes each has: 2x 6x 2.96 GHz, 96 GB RAM, 120 GB HDD for CentOS7, 1 TB SSD VM-Storage (Gluster FS), 4x 1 GBit LAN Bond, 4x 1GBit SAN Bond (Gluster Network, connection to Backup NAS/ Export Domain) Ovirt 4.0.3 The script output: Jan 05 14:12:16: Start backup for: wiki (20 GB) Jan 05 14:12:17: Snapshot creation started ... Jan 05 14:12:30: Snapshot created Jan 05 14:12:40: Clone into VM started ... Jan 05 14:36:37: Cloning finished Jan 05 14:36:38: Snapshot deletion started ... Jan 05 14:59:27: Snapshots deleted Jan 05 14:59:28: Export started ... Jan 05 15:07:09: Exporting finished Jan 05 15:07:09: Delete cloned VM started ... Jan 05 15:07:15: Cloned VM deleted Jan 05 15:07:15: Duration: 55:0 minutes Jan 05 15:07:15: VM exported as wiki_BACKUP_010516 Jan 05 15:07:15: Backup done for: wiki Jan 05 15:07:15: All backups done Thanks, Florain
Hi Florian, Did you check the IO load on the SPM host during the backup? If you have 1Gbit network then the maximum theoretical IO is restricted to 100MB/s . So if you get 20GB raw preallocated disk then a single disk backup will be about: 20/0.08/3600 hours x 3times~10mins: .08 is 20% overhead,3time for the clone-export-deletesnap process. Vdsm has also restriction on the migration process to avoid the network saturation. Check the Cluster properties: this is available for the 4.0 capable clusters. or set the migration_max_bandwidth in vdsm.conf on the hosts. Am 05.01.2017 6:27 nachm. schrieb "Florian Nolden" <f.nolden@xilloc.com>:
Hallo everyone,
I would like to update my Ovirt installation but, before that I would like to create proper VM backups. Currently I try to automate the backups and I use the implementation of the following project. https://github.com/wefixit-AT/oVirtBackup
The main idea is to use the Ovirt-API to trigger a snapshot, create a new VM from the snapshot, delete the snapshot, export the vm to the export domain, delete the vm.
This process takes even with only a 20 GB vm disk 55 Minutes. I have now 21 VMs with average 30 GB HDD space, that would mean that the backup job runs 22 hours.
Is there something wrong with my setup? Config? Is there a better way to automate backups? Is anyone else doing VM Exports as backup or are you just using snapshots?
Specs: HA Cluster 3 Nodes each has: 2x 6x 2.96 GHz, 96 GB RAM, 120 GB HDD for CentOS7, 1 TB SSD VM-Storage (Gluster FS), 4x 1 GBit LAN Bond, 4x 1GBit SAN Bond (Gluster Network, connection to Backup NAS/ Export Domain) Ovirt 4.0.3
The script output: Jan 05 14:12:16: Start backup for: wiki (20 GB) Jan 05 14:12:17: Snapshot creation started ... Jan 05 14:12:30: Snapshot created Jan 05 14:12:40: Clone into VM started ... Jan 05 14:36:37: Cloning finished Jan 05 14:36:38: Snapshot deletion started ... Jan 05 14:59:27: Snapshots deleted Jan 05 14:59:28: Export started ... Jan 05 15:07:09: Exporting finished Jan 05 15:07:09: Delete cloned VM started ... Jan 05 15:07:15: Cloned VM deleted Jan 05 15:07:15: Duration: 55:0 minutes Jan 05 15:07:15: VM exported as wiki_BACKUP_010516 Jan 05 15:07:15: Backup done for: wiki Jan 05 15:07:15: All backups done
Thanks, Florain
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------E8DE5FE656CDFEAC6F09CEBF Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 05.01.2017 18:26, Florian Nolden wrote:
Hallo everyone,
I would like to update my Ovirt installation but, before that I would like to create proper VM backups. Currently I try to automate the backups and I use the implementation of the following project. https://github.com/wefixit-AT/oVirtBackup
The problem is that ovirt does not support exporting from a snapshot. Because of that, the data is copied twice, first for the VM and then for export. you can wait for ovirt to support direct export. Or hack it like I have, and export the data yourself directly from the SPM via lvm. -- Ernest Beinrohr, AXON PRO Ing <http://www.beinrohr.sk/ing.php>, RHCE <http://www.beinrohr.sk/rhce.php>, RHCVA <http://www.beinrohr.sk/rhce.php>, LPIC <http://www.beinrohr.sk/lpic.php>, VCA <http://www.beinrohr.sk/vca.php>, +421-2-62410360 +421-903-482603 --------------E8DE5FE656CDFEAC6F09CEBF Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 05.01.2017 18:26, Florian Nolden wrote:<br> </div> <blockquote type="cite" cite="mid:CAD7w93F2c_GW-2RKCJpXaY_n1vqA-Hr92_+U7eMt=gO8GYcNZQ@mail.gmail.com"> <div dir="ltr"> <div>Hallo everyone, </div> <div><br> </div> <div>I would like to update my Ovirt installation but, before that I would like to create proper VM backups. Currently I try to automate the backups and I use the implementation of the following project.</div> <div><a href="https://github.com/wefixit-AT/oVirtBackup" moz-do-not-send="true">https://github.com/wefixit-AT/oVirtBackup</a><br> </div> <div><br> </div> </div> </blockquote> The problem is that ovirt does not support exporting from a snapshot. Because of that, the data is copied twice, first for the VM and then for export. you can wait for ovirt to support direct export. Or hack it like I have, and export the data yourself directly from the SPM via lvm.<br> <div class="moz-signature">-- <br> <div id="oernii_footer" style="color: gray;"> <span style="font-family: Lucida Console, Luxi Mono, Courier, monospace; font-size: 90%;"> Ernest Beinrohr, AXON PRO<br> <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/ing.php">Ing</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCE</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCVA</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/lpic.php">LPIC</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/vca.php">VCA</a>, <br> +421-2-62410360 +421-903-482603 <br> </span> </div> <img src="http://nojsstats.appspot.com/UA-44497096-1/email.beinrohr.sk" moz-do-not-send="true" width="1" height="1" border="0"> </div> </body> </html> --------------E8DE5FE656CDFEAC6F09CEBF--
On Jan 5, 2017 7:27 PM, "Florian Nolden" <f.nolden@xilloc.com> wrote: Hallo everyone, I would like to update my Ovirt installation but, before that I would like to create proper VM backups. Currently I try to automate the backups and I use the implementation of the following project. https://github.com/wefixit-AT/oVirtBackup The main idea is to use the Ovirt-API to trigger a snapshot, create a new VM from the snapshot, delete the snapshot, export the vm to the export domain, delete the vm. This process takes even with only a 20 GB vm disk 55 Minutes. I have now 21 VMs with average 30 GB HDD space, that would mean that the backup job runs 22 hours. Is there something wrong with my setup? Config? Is there a better way to automate backups? Is anyone else doing VM Exports as backup or are you just using snapshots? Specs: HA Cluster 3 Nodes each has: 2x 6x 2.96 GHz, 96 GB RAM, 120 GB HDD for CentOS7, 1 TB SSD VM-Storage (Gluster FS), 4x 1 GBit LAN Bond, 4x 1GBit SAN Bond (Gluster Network, connection to Backup NAS/ Export Domain) If you are using Gluster I'd utilize geo-replication. See https://github.com/sabose/ovirt-georep-backup Ovirt 4.0.3 I'd upgrade. Y. The script output: Jan 05 14:12:16: Start backup for: wiki (20 GB) Jan 05 14:12:17: Snapshot creation started ... Jan 05 14:12:30: Snapshot created Jan 05 14:12:40: Clone into VM started ... Jan 05 14:36:37: Cloning finished Jan 05 14:36:38: Snapshot deletion started ... Jan 05 14:59:27: Snapshots deleted Jan 05 14:59:28: Export started ... Jan 05 15:07:09: Exporting finished Jan 05 15:07:09: Delete cloned VM started ... Jan 05 15:07:15: Cloned VM deleted Jan 05 15:07:15: Duration: 55:0 minutes Jan 05 15:07:15: VM exported as wiki_BACKUP_010516 Jan 05 15:07:15: Backup done for: wiki Jan 05 15:07:15: All backups done Thanks, Florain _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Arman Khalatyan -
Ernest Beinrohr -
Florian Nolden -
Yaniv Kaul