8:15 a.m.
Hello,
I'm playing with export_vm_as_ova.py downloaded from the examples github:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/export...
My environment is oVirt 4.3.3.7 with iSCSI storage domain.
It fails leaving an ova.tmp file
In webadmin gui:
Starting to export Vm enginecopy1 as a Virtual Appliance
7/19/1911:55:12 AM
VDSM ov301 command TeardownImageVDS failed: Cannot deactivate Logical
Volume: ('General Storage Exception: ("5 [] [\' Logical volume
fa33df49-b09d-4f86-9719-ede649542c21/0420ef47-0ad0-4cf9-babd-d89383f7536b
in
use.\']\\nfa33df49-b09d-4f86-9719-ede649542c21/[\'a7480dc5-b5ca-4cb3-986d-77bc12165be4\',
\'0420ef47-0ad0-4cf9-babd-d89383f7536b\']",)',)
7/19/1912:25:36 PM
Failed to export Vm enginecopy1 as a Virtual Appliance to path
/save_ova/base/dump/myvm2.ova on Host ov301
7/19/1912:25:37 PM
During export I have this qemu-img process creating the disk over the loop
device:
root 30878 30871 0 11:55 pts/2 00:00:00 su -p -c qemu-img convert
-T none -O qcow2
'/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b'
'/dev/loop1' vdsm
vdsm 30882 30878 10 11:55 ? 00:00:00 qemu-img convert -T none -O
qcow2
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
/dev/loop1
The ova.tmp file is getting filled while command runs
eg:
[root@ov301 ]# du -sh /save_ova/base/dump/myvm2.ova.tmp
416M /save_ova/base/dump/myvm2.ova.tmp
[root@ov301 sysctl.d]#
[root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp
911M /save_ova/base/dump/myvm2.ova.tmp
[root@ov301 ]#
and the final generated / not completed file is in this state:
[root@ov301 ]# qemu-img info /save_ova/base/dump/myvm2.ova.tmp
image: /save_ova/base/dump/myvm2.ova.tmp
file format: raw
virtual size: 30G (32217446400 bytes)
disk size: 30G
[root@ov301 sysctl.d]#
But I notice that the timestamp of the file is about 67 minutes after start
of job and well after the notice of its failure....
[root@ov301 sysctl.d]# ll /save_ova/base/dump/
total 30963632
-rw-------. 1 root root 32217446400 Jul 19 13:02 myvm2.ova.tmp
[root@ov301 sysctl.d]#
[root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp
30G /save_ova/base/dump/myvm2.ova.tmp
[root@ov301 sysctl.d]#
In engine.log the first error I see is 30 minutes after start
2019-07-19 12:25:31,563+02 ERROR
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible
playbook execution failed: Timeout occurred while executing Ansible
playbook.
2019-07-19 12:25:31,563+02 INFO
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible
playbook command has exited with value: 1
2019-07-19 12:25:31,564+02 ERROR
[org.ovirt.engine.core.bll.CreateOvaCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Failed to
create OVA. Please check logs for more details:
/var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20190719115531-ov301-2001ddf4.log
2019-07-19 12:25:31,565+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] START,
TeardownImageVDSCommand(HostName = ov301,
ImageActionsVDSCommandParameters:{hostId='8ef1ce6f-4e38-486c-b3a4-58235f1f1d06'}),
log id: 3d2246f7
2019-07-19 12:25:36,569+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ov301 command TeardownImageVDS
failed: Cannot deactivate Logical Volume: ('General Storage Exception: ("5
[] [\' Logical volume
fa33df49-b09d-4f86-9719-ede649542c21/0420ef47-0ad0-4cf9-babd-d89383f7536b
in
use.\']\\nfa33df49-b09d-4f86-9719-ede649542c21/[\'a7480dc5-b5ca-4cb3-986d-77bc12165be4\',
\'0420ef47-0ad0-4cf9-babd-d89383f7536b\']",)',)
In ansible playbook suggested log file I don't see anything useful.
It ends with timestamps when the script has been launched.
Last lines are:
2019-07-19 11:55:33,877 p=5699 u=ovirt | TASK [ovirt-ova-export-pre-pack :
Retrieving the temporary path for the OVA file] ***
2019-07-19 11:55:34,198 p=5699 u=ovirt | changed: [ov301] => {
"changed": true,
"dest": "/save_ova/base/dump/myvm2.ova.tmp",
"gid": 0,
"group": "root",
"mode": "0600",
"owner": "root",
"secontext": "system_u:object_r:nfs_t:s0",
"size": 32217446912,
"state": "file",
"uid": 0
}
2019-07-19 11:55:34,204 p=5699 u=ovirt | TASK [ovirt-ova-pack : Run
packing script] *************************************
It seems 30 minutes... for timeout? About what, ansible job?
Or possibly implicit user session created when running the python script?
The snapshot has been correctly deleted (as I see also in engine.log), I
don't see it in webadmin gui.
Any known problem?
Just for test I executed again at 14:24 and I see same Ansible error at
14:54
The snapshot gets deleted, while the qemu-img command still continues....
[root@ov301 sysctl.d]# ps -ef | grep qemu-img
root 13504 13501 0 14:24 pts/1 00:00:00 su -p -c qemu-img convert
-T none -O qcow2
'/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b'
'/dev/loop0' vdsm
vdsm 13505 13504 3 14:24 ? 00:01:26 qemu-img convert -T none -O
qcow2
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
/dev/loop0
root 17587 24530 0 15:05 pts/0 00:00:00 grep --color=auto qemu-img
[root@ov301 sysctl.d]#
[root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp
24G /save_ova/base/dump/myvm2.ova.tmp
[root@ov301 sysctl.d]# ll /save_ova/base/dump/myvm2.ova.tmp
-rw-------. 1 root root 32217446400 Jul 19 15:14
/save_ova/base/dump/myvm2.ova.tmp
[root@ov301 sysctl.d]#
and then continues until image copy completes, but at this time the job has
already aborted and so the completion of the ova composition doesn't go
ahead... and I remain with the ova.tmp file...
How to extend timeout?
Thanks in advance,
Gianluca
9:14 a.m.
On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
In engine.log the first error I see is 30 minutes after start
2019-07-19 12:25:31,563+02 ERROR
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible
playbook execution failed: Timeout occurred while executing Ansible
playbook.
In the mean time, as the playbook seems this one ( I run the job from
engine) : /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml
- hosts: all
remote_user: root
gather_facts: no
roles:
- ovirt-ova-export-pre-pack
- ovirt-ova-pack
- ovirt-ova-export-post-pack
and it seems async is not supported, I temporary changed the main.yml
inside ovirt-ova-pack role from
--
- name: Run packing script
script: >
pack_ova.py
...
to
---
- name: Copy packing script
copy:
src: pack_ova.py
dest: /tmp/pack_ova.py
mode: '0755'
- name: Run packing script
command: timeout 4800 /tmp/pack_ova.py
...
just to check... see how it goes now
10:59 a.m.
On Fri, Jul 19, 2019 at 4:14 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi
<gianluca.cecchi(a)gmail.com>
wrote:
>
>
> In engine.log the first error I see is 30 minutes after start
>
> 2019-07-19 12:25:31,563+02 ERROR
> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
> (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible
> playbook execution failed: Timeout occurred while executing Ansible
> playbook.
>
In the mean time, as the playbook seems this one ( I run the job from
engine) : /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml
Based on what described in bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1697301
I created at the moment the file
/etc/ovirt-engine/engine.conf.d/99-ansible-playbook-timeout.conf
with
ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT=80
and restarted the engine and the python script to verify
Just to see if it completes, even if in my case with a 30Gb preallocated
disk, the source problem is qemu-img convert command very slow in I/O.
It reads from iscsi multipath (2 paths) with 2x3MB/s and it writes on nfs
If I run a dd command from iscsi device mapper device to an nfs file I have
140MB/s rate that is what expected based on my storage array performances
and my network.
Not understood why the qemu-img command is so slow
The question still applies in case I have to do an appliance from a VM with
a very big disk, where the copy could potentially have an elapsed of more
that 30 minutes...
Gianluca
4:21 a.m.
On Fri, Jul 19, 2019 at 5:59 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Fri, Jul 19, 2019 at 4:14 PM Gianluca Cecchi
<gianluca.cecchi(a)gmail.com>
wrote:
> On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi <
> gianluca.cecchi(a)gmail.com> wrote:
>
>>
>>
>> In engine.log the first error I see is 30 minutes after start
>>
>> 2019-07-19 12:25:31,563+02 ERROR
>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible
>> playbook execution failed: Timeout occurred while executing Ansible
>> playbook.
>>
>
> In the mean time, as the playbook seems this one ( I run the job from
> engine) : /usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml
>
>
Based on what described in bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1697301
I created at the moment the file
/etc/ovirt-engine/engine.conf.d/99-ansible-playbook-timeout.conf
with
ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT=80
and restarted the engine and the python script to verify
Just to see if it completes, even if in my case with a 30Gb preallocated
disk, the source problem is qemu-img convert command very slow in I/O.
It reads from iscsi multipath (2 paths) with 2x3MB/s and it writes on nfs
If I run a dd command from iscsi device mapper device to an nfs file I
have 140MB/s rate that is what expected based on my storage array
performances and my network.
Not understood why the qemu-img command is so slow
The question still applies in case I have to do an appliance from a VM
with a very big disk, where the copy could potentially have an elapsed of
more that 30 minutes...
Gianluca
I confirm that setting ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT was the
solution.
I got the ova completed:
Starting to export Vm enginecopy1 as a Virtual Appliance 7/19/19 5:53:05 PM
Vm enginecopy1 was exported successfully as a Virtual Appliance to path
/save_ova/base/dump/myvm2.ova on Host ov301 7/19/19 6:58:07 PM
I have to understand why the conversion of the pre-allocated disk is so
slow, because simulating I/O from iSCSI lun where VM disks live to the NFS
share gives me about 110MB/s
I'm going to update to 4.3.4, just to see if there is any bug fixed. The
same operation on vSphere have an elapsed of 5 minutes.
What is the eta for 4.3.5?
One notice:
if I manually create a snapshot of the same VM and then clone the snapshot,
the process is this one
vdsm 5713 20116 6 10:50 ? 00:00:04 /usr/bin/qemu-img convert
-p -t none -T none -f raw
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
-O raw -W
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/d13a5c43-0138-4cbb-b663-f3ad5f9f5983/fd4e1b08-15fe-45ee-ab12-87dea2d29bc4
and its speed is quite better (up to 100MB/s read and 100MB/s write) with a
total elapsed of 6 minutes and 30 seconds.
during the ova generation the process was instead:
vdsm 13505 13504 3 14:24 ? 00:01:26 qemu-img convert -T none
-O qcow2
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
/dev/loop0
could it be the "-O qcow2" the reason? Why qcow2 if origin is preallocated
(raw)?
Gianluca
8:28 a.m.
Hi Gianluca.
We have a problem wit "export as ova" too.
In our env we back up all the vm with a python script that run an export.
If we run multiple export at the same time, also on different
datacenter[but same engine], the wait for each other and do not run in
async mode.
If only one of those export takes 10h, all of them taskes 10+h too.
seems that all the export have to end a step of the playbook to go on,
but we see only one "nasible-playbook" process at a time.
Have you got any hint?
Regards,
Tommaso.
Il 23/07/2019 11:21, Gianluca Cecchi ha scritto:
On Fri, Jul 19, 2019 at 5:59 PM Gianluca Cecchi
<gianluca.cecchi(a)gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:
On Fri, Jul 19, 2019 at 4:14 PM Gianluca Cecchi
<gianluca.cecchi(a)gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:
On Fri, Jul 19, 2019 at 3:15 PM Gianluca Cecchi
<gianluca.cecchi(a)gmail.com <mailto:gianluca.cecchi@gmail.com>>
wrote:
In engine.log the first error I see is 30 minutes after start
2019-07-19 12:25:31,563+02 ERROR
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engineScheduled-Thread-64)
[2001ddf4] Ansible playbook execution failed: Timeout
occurred while executing Ansible playbook.
In the mean time, as the playbook seems this one ( I run the
job from engine) :
/usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml
Based on what described in bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1697301
I created at the moment the file
/etc/ovirt-engine/engine.conf.d/99-ansible-playbook-timeout.conf
with
ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT=80
and restarted the engine and the python script to verify
Just to see if it completes, even if in my case with a 30Gb
preallocated disk, the source problem is qemu-img convert command
very slow in I/O.
It reads from iscsi multipath (2 paths) with 2x3MB/s and it writes
on nfs
If I run a dd command from iscsi device mapper device to an nfs
file I have 140MB/s rate that is what expected based on my storage
array performances and my network.
Not understood why the qemu-img command is so slow
The question still applies in case I have to do an appliance from
a VM with a very big disk, where the copy could potentially have
an elapsed of more that 30 minutes...
Gianluca
I confirm that setting ANSIBLE_PLAYBOOK_EXEC_DEFAULT_TIMEOUT was the
solution.
I got the ova completed:
Starting to export Vm enginecopy1 as a Virtual Appliance 7/19/19
5:53:05 PM
Vm enginecopy1 was exported successfully as a Virtual Appliance to
path /save_ova/base/dump/myvm2.ova on Host ov301 7/19/19 6:58:07 PM
I have to understand why the conversion of the pre-allocated disk is
so slow, because simulating I/O from iSCSI lun where VM disks live to
the NFS share gives me about 110MB/s
I'm going to update to 4.3.4, just to see if there is any bug fixed.
The same operation on vSphere have an elapsed of 5 minutes.
What is the eta for 4.3.5?
One notice:
if I manually create a snapshot of the same VM and then clone the
snapshot, the process is this one
vdsm 5713 20116 6 10:50 ? 00:00:04 /usr/bin/qemu-img
convert -p -t none -T none -f raw
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
-O raw -W
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/d13a5c43-0138-4cbb-b663-f3ad5f9f5983/fd4e1b08-15fe-45ee-ab12-87dea2d29bc4
and its speed is quite better (up to 100MB/s read and 100MB/s write)
with a total elapsed of 6 minutes and 30 seconds.
during the ova generation the process was instead:
vdsm 13505 13504 3 14:24 ? 00:01:26 qemu-img convert -T
none -O qcow2
/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b
/dev/loop0
could it be the "-O qcow2" the reason? Why qcow2 if origin is
preallocated (raw)?
Gianluca
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UFB6ZACCJIN...
--
--
Shellrent - Il primo hosting italiano Security First
*Tommaso De Marchi*
/COO - Chief Operating Officer/
Shellrent Srl
Via dell'Edilizia, 19 - 36100 Vicenza
Tel. 0444321155 <tel:+390444321155> | Fax 04441492177
4:15 a.m.
On Tue, Aug 25, 2020 at 3:29 PM Tommaso - Shellrent <tommaso(a)shellrent.com>
wrote:
Hi Gianluca.
We have a problem wit "export as ova" too.
In our env we back up all the vm with a python script that run an export.
If we run multiple export at the same time, also on different
datacenter[but same engine], the wait for each other and do not run in
async mode.
If only one of those export takes 10h, all of them taskes 10+h too.
seems that all the export have to end a step of the playbook to go on, but
we see only one "nasible-playbook" process at a time.
Have you got any hint?
Regards,
Tommaso.
My post was one year old.
In the meantime you can check these two posts in archives by Jayme (I cc
him):
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/U65CV5A6WC6S...
and
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JNSY6GYNS6L...
In the first post it appears that by default the exports were of type fire
and forget (that is what you need) and so he included a wait_for task to
instead have them sequential.
I think you can borrow from his github playbooks and adapt to your needs.
Or Jayme, you could apply a general change where you set a variable (eg
sequential_flow) that by default is true and then you modify the
export_vm.yml playbook in the task
- name: "Wait for export"
with the wait condition becoming of kind
when: (export_result is not failed) or (sequential_flow|bool == False)
This way by default the play workflow remains unchanged, but a user can set
sequential_flow to False
What do you think about it?
Gianluca
6 p.m.
I am testing the migration from CentOS7/oVirt 4.3 to CentOS8/oVirt 4.4.
Exporting all VMs to OVAs, and re-importing them on a new cluster built from scratch seems
the safest and best method, because in the step-by-step migration, there is simply far too
many things that can go wrong and no easy way to fail-back after each step.
But of course it requires that one of the most essential operations in a hypervisor
actualy works.
For me a hypervisor turns a machine into a file and a file into a machine: That's the
most appreciated and fundamental quality of it.
oVirt fails right there, repeatedly, and for different reasons and without even reporting
an error.
So I have manually put the single-line fix in, which settles udev to ensure that disks are
not exported as zeros. That's the bug which renders the final release oVirt 4.3
forever unfit, 4 years before the end of maintenance of CentOS7, because it won't be
fixed there.
Exporting VMs and re-importing them on another farm generally seemed to work after that.
But just as I was exporting not one of the trivial machines, that I have been using for
testing, but one of the bigger ones, that actually contain a significant amout of data, I
find myself hitting this timeout bug.
The disks for both the trival and less-trivial are defined at 500GB, thinly allocated. The
trivial is the naked OS at something like 7GB actually allocated, the 'real' has
113GB allocated. In both cases the OVA export file to a local SSD xfs partition is 500GB,
with lots of zeros and sparse allocation in the case of the first one.
The second came to 72GB of 500GB actually allocated, which didn't seem like a good
sign already, but perhaps there was some compression involved?
Still the export finished without error or incident and the import on the other side went
just as well. The machine even boots and runs, it was only once I started using it, that I
suddenly had all types of file system errors... it turns out 113-73GB were actually really
cut off and missing from the OVA export, and there is nobody and nothing checking for
that.
I now know that qemu-img is used in the process, which actually runs in a pipe. There is
no checksumming or any other logic involved to ensure that the format conversion of the
disk image has retained the integrity of the image. There is no obvious simple solution
that I can think of, but the combination of processing the image through a pipe and an
impatient ansible timeout results in a hypervisor which fails on the most important
elementary task: Turn a machine into a file and back into a machine.
IMHO it makes oVirt a toy, not a tool. And worst of all, I am pretty sure that RHV has the
same quality, even if the sticker price is probably quite different.
I have the export domain backup running right now, but I'm not sure it's not using
the same mechanism under the cover with potentially similar results.
Yes, I know there is a Python script that will do the backup, and probably with full
fidelity. And perhaps with this new infrastructure as code approach, that is how it should
be done anyway.
But if you have a GUI, that should just work: No excuses.
P.S. The allocation size of the big VM in the export domain is again 72GB, with the file
size at 500GB. I'll run the import on the other end, but by now I am pretty sure, the
result will be no different.
Unless you resort to Python or some external tool, there is no reliable way to back up and
restore a VM of less than 80 seconds worth of data transfer and no warning, when
corruption occurs.
I am not sure you can compete with Nutanix and VMware at this level of reliability.
P.P.S. So just where (and on which machine) do I need to change the timeout?
4:47 a.m.
On Fri, Aug 28, 2020 at 2:31 AM <thomas(a)hoberg.net> wrote:
I am testing the migration from CentOS7/oVirt 4.3 to CentOS8/oVirt 4.4.
Exporting all VMs to OVAs, and re-importing them on a new cluster built from scratch
seems the safest and best method, because in the step-by-step migration, there is simply
far too many things that can go wrong and no easy way to fail-back after each step.
You should really try the attach/detach storage domain, this is the
recommended way to move
vms from one ovirt system to another.
You could detach the entire domain with all vms from the old system,
and connect it to the new
system, without copying even one bit.
I guess you cannot do this because you don't use shared storage?
...
So I have manually put the single-line fix in, which settles udev to
ensure that disks are not exported as zeros. That's the bug which renders the final
release oVirt 4.3 forever unfit, 4 years before the end of maintenance of CentOS7, because
it won't be fixed there.
Using ovirt 4.3 when 4.4 was released is going to be painful, don't do this.
...
But just as I was exporting not one of the trivial machines, that I
have been using for testing, but one of the bigger ones, that actually contain a
significant amout of data, I find myself hitting this timeout bug.
The disks for both the trival and less-trivial are defined at 500GB, thinly allocated.
The trivial is the naked OS at something like 7GB actually allocated, the 'real'
has 113GB allocated. In both cases the OVA export file to a local SSD xfs partition is
500GB, with lots of zeros and sparse allocation in the case of the first one.
The second came to 72GB of 500GB actually allocated, which didn't seem like a good
sign already, but perhaps there was some compression involved?
Still the export finished without error or incident and the import on the other side went
just as well. The machine even boots and runs, it was only once I started using it, that I
suddenly had all types of file system errors... it turns out 113-73GB were actually really
cut off and missing from the OVA export, and there is nobody and nothing checking for
that.
You are hitting https://bugzilla.redhat.com/1854888
...
I have the export domain backup running right now, but I'm not
sure it's not using the same mechanism under the cover with potentially similar
results.
No, export domain is using qemu-img, which is the best tool for copying images.
This is how all disks are copied in oVirt in all flows. There are no
issues like ignored
errors or silent failures in storage code.
...
P.P.S. So just where (and on which machine) do I need to change the
timeout?
There are no timeouts in storage code, e.g. attach/detach domain, or
export to export
domain.
Nir
Nir
12:37 p.m.
On Fri, Aug 28, 2020 at 2:31 AM <thomas(a)hoberg.net>
wrote:
You should really try the attach/detach storage domain, this is the
recommended way to move
vms from one ovirt system to another.
You could detach the entire domain with all vms from the old system,
and connect it to the new
system, without copying even one bit.
I guess you cannot do this because you don't use shared storage?
These are all HCI setups with GlusterFS, so storage is shared in a way...
I am also experimenting with a backup (not export) domain on NFS and/or removable media
(just temp local storage, exported via NFS), but the handling is very odd, to say the
least (see my other post for the full story).
Basically the documentation says you move all VM disks to the backup domain after cloning
the VM. And then it says nothing more... (how does the VM definition get carried over? Can
I then destroy the remaing clone VM? Do I need to re-create a similar VM at the target?
etc.)
The in-place upgrade producedure in the docs for the HCI case has far too many tersly
described steps that can go wrong with someone like me doing it: I even manage to fail a
green-field setup many times somehow
And even if I were to do the upgrade as described, I do need to know that the
export/clean-migration/import is still a viable option, should something go wrong.
...
Using ovirt 4.3 when 4.4 was released is going to be painful, don't do this.
That's why I am migrating, but for that I need to prove a working plan B
...
You are hitting https://bugzilla.redhat.com/1854888
Unfortunately the description doesn't tell if the failure of the silent
qemu-img was on the export side, resulting a corrupted image: I am assuming that qemu-img
is used in both export and import.
The failure on the import is not silent, just doesn't seem to make a lot of sense,
because qemu-img is reporting a write error at the local single node HCI gluster target,
which has plenty of space and is essentially a loopback in 1nHCI.
...
No, export domain is using qemu-img, which is the best tool for copying images.
This is how all disks are copied in oVirt in all flows. There are no
issues like ignored
errors or silent failures in storage code.
My problem is getting errors and now
understanding what's causing them.
Qemu-img on the target HCI single node Gluster is reporting a write error at varying block
numbers, often after dozens of gigabytes have already been transferred. There is plenty of
space on the Gluster, an SSD with VDO underneath so the higher risk is actually the
source, which is the NFS mount from the export domain.
I've tried uploading the image using imageio and your Python sample from the SDK, but
just as I had done that (with 50MB/s at 1/6 of the performance of the qemu-img transfer),
I managed to kill the 4.4 cluster by downgrading the machine type of the hosted-engine,
when I was really trying to make a successfully restored VM work with renamed Ethernet
devices...
The upload via imageio completed fully, I hadn't tested the disk image with a machine
yet to see if it would boot.
...
There are no timeouts in storage code, e.g. attach/detach domain, or
export to export
domain.
Nir
Well, that was almost my last hope, because I don't know what could make
the qemu-img import transfer fail on a write, when the very same image works with
imageio... Actually, the big difference there is that the resulting disk, which is
logically configured at 500GB is actually logically consuming 500GB in the domain, while
sparse images that make it successfully through qemu-img, retain their much smaller actual
size. VDO is still underneath so it may not matter, and I didn't have a chance to try
sparsify before I killed the target cluster.
I have also prepared a USB3 disk to act as export domain, which I'll physically move,
just to ensure the NFS pipe in the qemu-img job isn't the real culprit.
And I guess I'll try the export again, to see if I overlooked some error there.
>
>
> Nir
3:44 p.m.
BTW: This is the message I get on the import:
VDSM nucvirt command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed:
("Command ['/usr/bin/qemu-img', 'convert', '-p',
'-t', 'none', '-T', 'none', '-f', 'qcow2',
'/rhev/data-center/mnt/petitcent.mtk.hoberg.net:_flash_export/fe9fb0db-2743-457a-80f0-9a4edc509e9d/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3',
'-O', 'raw',
'/rhev/data-center/mnt/glusterSD/nucvirt.mtk.hoberg.net:_vmstore/ba410e27-458d-4b32-969c-ad0c37edaceb/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3']
failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing sector
9566208: No such file or directory\\n')",) abortedcode=261
8/29/2011:41:21 AM
2:03 a.m.
On Sun, Aug 30, 2020 at 11:46 PM <thomas(a)hoberg.net> wrote:
BTW: This is the message I get on the import:
VDSM nucvirt command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed:
("Command ['/usr/bin/qemu-img', 'convert', '-p',
'-t', 'none', '-T', 'none', '-f', 'qcow2',
'/rhev/data-center/mnt/petitcent.mtk.hoberg.net:_flash_export/fe9fb0db-2743-457a-80f0-9a4edc509e9d/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3',
'-O', 'raw',
'/rhev/data-center/mnt/glusterSD/nucvirt.mtk.hoberg.net:_vmstore/ba410e27-458d-4b32-969c-ad0c37edaceb/images/3be7c1bb-377c-4d5e-b4f6-1a6574b8a52b/845cdd93-def8-4d84-9a08-f8c991f89fe3']
failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing sector
9566208: No such file or directory\\n')",) abortedcode=261
This is a gluster error that was already reported here several times.
ENOENT is not a valid error for write. Please report a gluster bug for this.
Nir
1:47 a.m.
Hi Gianluca, we have read the massage from Jayme, but we have problem
with the INTERNAL ansible playbook.
Wen we run an export, ovirt internally run the playbook
/usr/share/ovirt-engine/playbooks/ovirt-ova-export.yml
but only export at time....
Il 27/08/2020 11:15, Gianluca Cecchi ha scritto:
On Tue, Aug 25, 2020 at 3:29 PM Tommaso - Shellrent
<tommaso(a)shellrent.com <mailto:tommaso@shellrent.com>> wrote:
Hi Gianluca.
We have a problem wit "export as ova" too.
In our env we back up all the vm with a python script that run an
export.
If we run multiple export at the same time, also on different
datacenter[but same engine], the wait for each other and do not
run in async mode.
If only one of those export takes 10h, all of them taskes 10+h too.
seems that all the export have to end a step of the playbook to go
on, but we see only one "nasible-playbook" process at a time.
Have you got any hint?
Regards,
Tommaso.
My post was one year old.
In the meantime you can check these two posts in archives by Jayme (I
cc him):
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/U65CV5A6WC6S...
and
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JNSY6GYNS6L...
In the first post it appears that by default the exports were of type
fire and forget (that is what you need) and so he included a wait_for
task to instead have them sequential.
I think you can borrow from his github playbooks and adapt to your needs.
Or Jayme, you could apply a general change where you set a variable
(eg sequential_flow) that by default is true and then you modify the
export_vm.yml playbook in the task
- name: "Wait for export"
with the wait condition becoming of kind
when: (export_result is not failed) or (sequential_flow|bool == False)
This way by default the play workflow remains unchanged, but a user
can set sequential_flow to False
What do you think about it?
Gianluca
--
--
Shellrent - Il primo hosting italiano Security First
*Tommaso De Marchi*
/COO - Chief Operating Officer/
Shellrent Srl
Via dell'Edilizia, 19 - 36100 Vicenza
Tel. 0444321155 <tel:+390444321155> | Fax 04441492177
1573
days inactive
1982
days old
11 comments
4 participants
participants (4)
-
Gianluca Cecchi -
Nir Soffer -
thomas＠hoberg.net -
Tommaso - Shellrent