On Fri, Jul 19, 2019 at 4:14 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

In engine.log the first error I see is 30 minutes after start

2019-07-19 12:25:31,563+02 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook.

In the mean time, as the playbook seems this one ( I run the job from engine) : /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml

Based on what described in bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1697301 I created at the moment the file /etc/ovirt-engine/engine.conf.d/99-ansible-playbook-timeout.conf with ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT=80 and restarted the engine and the python script to verify Just to see if it completes, even if in my case with a 30Gb preallocated disk, the source problem is qemu-img convert command very slow in I/O. It reads from iscsi multipath (2 paths) with 2x3MB/s and it writes on nfs If I run a dd command from iscsi device mapper device to an nfs file I have 140MB/s rate that is what expected based on my storage array performances and my network. Not understood why the qemu-img command is so slow The question still applies in case I have to do an appliance from a VM with a very big disk, where the copy could potentially have an elapsed of more that 30 minutes... Gianluca

Hi Gianluca. We have a problem wit "export as ova" too. In our env we back up all the vm with a python script that run an export. If we run multiple export at the same time, also on different datacenter[but same engine], the wait for each other and do not run in async mode. If only one of those export takes 10h, all of them taskes 10+h too. seems that all the export have to end a step of the playbook to go on, but we see only one "nasible-playbook" process at a time. Have you got any hint? Regards, Tommaso. Il 23/07/2019 11:21, Gianluca Cecchi ha scritto:

On Fri, Jul 19, 2019 at 5:59 PM Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:

On Fri, Jul 19, 2019 at 4:14 PM Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:

On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:

In engine.log the first error I see is 30 minutes after start

2019-07-19 12:25:31,563+02 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook.

In the mean time, as the playbook seems this one ( I run the job from engine) : /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml

Based on what described in bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1697301 I created at the moment the file /etc/ovirt-engine/engine.conf.d/99-ansible-playbook-timeout.conf with ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT=80 and restarted the engine and the python script to verify

Just to see if it completes, even if in my case with a 30Gb preallocated disk, the source problem is qemu-img convert command very slow in I/O. It reads from iscsi multipath (2 paths) with 2x3MB/s and it writes on nfs If I run a dd command from iscsi device mapper device to an nfs file I have 140MB/s rate that is what expected based on my storage array performances and my network.

Not understood why the qemu-img command is so slow The question still applies in case I have to do an appliance from a VM with a very big disk, where the copy could potentially have an elapsed of more that 30 minutes... Gianluca

I confirm that setting  ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT was the solution. I got the ova completed:

Starting to export Vm enginecopy1 as a Virtual Appliance 7/19/19 5:53:05 PM Vm enginecopy1 was exported successfully as a Virtual Appliance to path /save_ova/base/dump/myvm2.ova on Host ov301 7/19/19 6:58:07 PM

I have to understand why the conversion of the pre-allocated disk is so slow, because simulating I/O from iSCSI lun where VM disks live to the NFS share gives me about 110MB/s I'm going to update to 4.3.4, just to see if there is any bug fixed. The same operation on vSphere have an elapsed of 5 minutes. What is the eta for 4.3.5?

One notice: if I manually create a snapshot of the same VM and then clone the snapshot, the process is this one vdsm      5713 20116  6 10:50 ?        00:00:04 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b -O raw -W /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/d13a5c43-0138-4cbb-b663-f3ad5f9f5983/fd4e1b08-15fe-45ee-ab12-87dea2d29bc4

and its speed is quite better (up to 100MB/s read and 100MB/s write) with a total elapsed of 6 minutes and 30 seconds.

during the ova generation the process was instead:  vdsm     13505 13504  3 14:24 ?        00:01:26 qemu-img convert -T none -O qcow2 /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b /dev/loop0

could it be the "-O qcow2" the reason? Why qcow2 if origin is preallocated (raw)?

Gianluca

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UFB6ZACCJIN3DS... -- -- Shellrent - Il primo hosting italiano Security First

*Tommaso De Marchi* /COO - Chief Operating Officer/ Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155 <tel:+390444321155> | Fax 04441492177

I am testing the migration from CentOS7/oVirt 4.3 to CentOS8/oVirt 4.4. Exporting all VMs to OVAs, and re-importing them on a new cluster built from scratch seems the safest and best method, because in the step-by-step migration, there is simply far too many things that can go wrong and no easy way to fail-back after each step. But of course it requires that one of the most essential operations in a hypervisor actualy works. For me a hypervisor turns a machine into a file and a file into a machine: That's the most appreciated and fundamental quality of it. oVirt fails right there, repeatedly, and for different reasons and without even reporting an error. So I have manually put the single-line fix in, which settles udev to ensure that disks are not exported as zeros. That's the bug which renders the final release oVirt 4.3 forever unfit, 4 years before the end of maintenance of CentOS7, because it won't be fixed there. Exporting VMs and re-importing them on another farm generally seemed to work after that. But just as I was exporting not one of the trivial machines, that I have been using for testing, but one of the bigger ones, that actually contain a significant amout of data, I find myself hitting this timeout bug. The disks for both the trival and less-trivial are defined at 500GB, thinly allocated. The trivial is the naked OS at something like 7GB actually allocated, the 'real' has 113GB allocated. In both cases the OVA export file to a local SSD xfs partition is 500GB, with lots of zeros and sparse allocation in the case of the first one. The second came to 72GB of 500GB actually allocated, which didn't seem like a good sign already, but perhaps there was some compression involved? Still the export finished without error or incident and the import on the other side went just as well. The machine even boots and runs, it was only once I started using it, that I suddenly had all types of file system errors... it turns out 113-73GB were actually really cut off and missing from the OVA export, and there is nobody and nothing checking for that. I now know that qemu-img is used in the process, which actually runs in a pipe. There is no checksumming or any other logic involved to ensure that the format conversion of the disk image has retained the integrity of the image. There is no obvious simple solution that I can think of, but the combination of processing the image through a pipe and an impatient ansible timeout results in a hypervisor which fails on the most important elementary task: Turn a machine into a file and back into a machine. IMHO it makes oVirt a toy, not a tool. And worst of all, I am pretty sure that RHV has the same quality, even if the sticker price is probably quite different. I have the export domain backup running right now, but I'm not sure it's not using the same mechanism under the cover with potentially similar results. Yes, I know there is a Python script that will do the backup, and probably with full fidelity. And perhaps with this new infrastructure as code approach, that is how it should be done anyway. But if you have a GUI, that should just work: No excuses. P.S. The allocation size of the big VM in the export domain is again 72GB, with the file size at 500GB. I'll run the import on the other end, but by now I am pretty sure, the result will be no different. Unless you resort to Python or some external tool, there is no reliable way to back up and restore a VM of less than 80 seconds worth of data transfer and no warning, when corruption occurs. I am not sure you can compete with Nutanix and VMware at this level of reliability. P.P.S. So just where (and on which machine) do I need to change the timeout?

On Fri, Aug 28, 2020 at 2:31 AM <thomas(a)hoberg.net> wrote:

You should really try the attach/detach storage domain, this is the recommended way to move vms from one ovirt system to another.

You could detach the entire domain with all vms from the old system, and connect it to the new system, without copying even one bit.

I guess you cannot do this because you don't use shared storage?

These are all HCI setups with GlusterFS, so storage is shared in a way... I am also experimenting with a backup (not export) domain on NFS and/or removable media (just temp local storage, exported via NFS), but the handling is very odd, to say the least (see my other post for the full story). Basically the documentation says you move all VM disks to the backup domain after cloning the VM. And then it says nothing more... (how does the VM definition get carried over? Can I then destroy the remaing clone VM? Do I need to re-create a similar VM at the target? etc.) The in-place upgrade producedure in the docs for the HCI case has far too many tersly described steps that can go wrong with someone like me doing it: I even manage to fail a green-field setup many times somehow And even if I were to do the upgrade as described, I do need to know that the export/clean-migration/import is still a viable option, should something go wrong.

...

Using ovirt 4.3 when 4.4 was released is going to be painful, don't do this.

That's why I am migrating, but for that I need to prove a working plan B

...

You are hitting https://bugzilla.redhat.com/1854888

Unfortunately the description doesn't tell if the failure of the silent qemu-img was on the export side, resulting a corrupted image: I am assuming that qemu-img is used in both export and import. The failure on the import is not silent, just doesn't seem to make a lot of sense, because qemu-img is reporting a write error at the local single node HCI gluster target, which has plenty of space and is essentially a loopback in 1nHCI.

...

No, export domain is using qemu-img, which is the best tool for copying images. This is how all disks are copied in oVirt in all flows. There are no issues like ignored errors or silent failures in storage code. My problem is getting errors and now understanding what's causing them.

Qemu-img on the target HCI single node Gluster is reporting a write error at varying block numbers, often after dozens of gigabytes have already been transferred. There is plenty of space on the Gluster, an SSD with VDO underneath so the higher risk is actually the source, which is the NFS mount from the export domain. I've tried uploading the image using imageio and your Python sample from the SDK, but just as I had done that (with 50MB/s at 1/6 of the performance of the qemu-img transfer), I managed to kill the 4.4 cluster by downgrading the machine type of the hosted-engine, when I was really trying to make a successfully restored VM work with renamed Ethernet devices... The upload via imageio completed fully, I hadn't tested the disk image with a machine yet to see if it would boot.

...

There are no timeouts in storage code, e.g. attach/detach domain, or export to export domain.

Nir

Well, that was almost my last hope, because I don't know what could make the qemu-img import transfer fail on a write, when the very same image works with imageio... Actually, the big difference there is that the resulting disk, which is logically configured at 500GB is actually logically consuming 500GB in the domain, while sparse images that make it successfully through qemu-img, retain their much smaller actual size. VDO is still underneath so it may not matter, and I didn't have a chance to try sparsify before I killed the target cluster. I have also prepared a USB3 disk to act as export domain, which I'll physically move, just to ensure the NFS pipe in the qemu-img job isn't the real culprit. And I guess I'll try the export again, to see if I overlooked some error there.

Nir

On Sun, Aug 30, 2020 at 11:46 PM <thomas@hoberg.net> wrote:

BTW: This is the message I get on the import: VDSM nucvirt command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'qcow2', '/rhev/data-center/mnt/petitcent.mtk.hoberg.net:_flash_export/fe9fb0db-2743-457a-80f0-9a4edc509e9d/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3', '-O', 'raw', '/rhev/data-center/mnt/glusterSD/nucvirt.mtk.hoberg.net:_vmstore/ba410e27-458d-4b32-969c-ad0c37edaceb/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing sector 9566208: No such file or directory\\n')",) abortedcode=261

This is a gluster error that was already reported here several times. ENOENT is not a valid error for write. Please report a gluster bug for this. Nir