Thank you Nir for the below.

I am putting some comments inline in blue.


On Tue, Feb 13, 2018 at 7:33 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Jan 24, 2018 at 3:19 PM Alex K <rightkicktech@gmail.com> wrote:
Hi all,

I have a cluster with 3 nodes, using ovirt 4.1 in a self hosted setup on top glusterfs. 
On some VMs (especially one Windows server 2016 64bit with 500 GB of disk). Guest agents are installed at VMs. i almost always observe that during the backup of the VM the VM is rendered unresponsive (dashboard shows a question mark at the VM status and VM does not respond to ping or to anything).

For scheduled backups I use:

https://github.com/wefixit-AT/oVirtBackup

The script does the following:

1. snapshot VM (this is done ok without any failure)


This is a very cheap operation
 

2. Clone snapshot (this steps renders the VM unresponsive)


This copy 500g of data. In gluster case, it copies 1500g of data, since in glusterfs, the client
is doing the replication.

Maybe your network or gluster server is too slow? Can you describe the network topology?

Please attach also the volume info for the gluster volume, maybe it is not configured in the 
best way?

The network is 1Gbit. The hosts (3 hosts) are decent ones and new hardware with each host having: 32GB RAM, 16 CPU cores and 2 TB of storage in RAID10.
The VMS hosted (7 VMs) exhibit high performance. The VMs are Windows 2016 and Windows10.
The network topology is: two networks defined at ovirt: ovirtmgmt is for the managment and access network and "storage" is a separate network, where each server is connected with two network cables at a managed switch with mode 6 load balancing. this storage network is used for gluster traffic. Attached the volume configuration.

3. Export Clone


This copy 500g to the export domain. If the export domain is on glusterfs as well, you
copy now another 1500g of data.
 
Export domain a Synology NAS with NFS share.  If the cloning succeeds then export is completed ok.

4. Delete clone

5. Delete snapshot


Not clear why do you need to clone the vm before you export it, you can save half of 
the data copies.
Because I cannot export the VM while it is running. It does not provide such option. 

If you 4.2, you can backup the vm *while the vm is running* by:
- Take a snapshot
- Get the vm ovf from the engine api
- Download the vm disks using ovirt-imageio and store the snaphosts in your backup
  storage
- Delete a snapshot

In this flow, you would copy 500g.

I am not aware about this option. checking quickly at site this seems that it is still half implemented? Is there any script that I may use and test this? I am interested to have these backups scheduled.
 
Daniel, please correct me if I'm wrong regarding doing this online.

Regardless, a vm should not become non-responsive while cloning. Please file a bug
for this and attach engine, vdsm, and glusterfs logs.
 
Nir

Do you have any similar experience? Any suggestions to address this?

I have never seen such issue with hosted Linux VMs.

The cluster has enough storage to accommodate the clone.


Thanx,

Alex



 _______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users