9:42 a.m.
Hello,
when trying to deploy the engine using "hosted-engine --deploy
--restore-from-file=myenginebackup" the ansible playbook errors out at
[ INFO ] TASK [ovirt.hosted_engine_setup : Trigger hosted engine OVF update and enable
the serial console]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait until OVF update finishes]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Parse OVF_STORE disk list]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Check OVF_STORE volume status]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE disk content]
[ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records
in', u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000141645 s,
72.3 MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in
archive', u'tar: Exiting with failure status due to previous errors'],
u'changed': True, u'end': u'2019-05-08 15:21:47.595195',
u'_ansible_item_label': {u'image_id':
u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE',
u'id': u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'stdout':
u'', u'failed': True, u'_ansible_item_result': True,
u'msg': u'non-zero return code', u'rc': 2, u'start':
u'2019-05-08 15:21:46.906877', u'attempts': 12, u'cmd':
u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=9ff8b389-5e24-4166-9842-f1d6104b662b volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98
| grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40
b18612a54.ovf", u'item': {u'image_id':
u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE',
u'id': u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'delta':
u'0:00:00.688318', u'invocation': {u'module_args':
{u'warn': False, u'executable': None, u'_uses_shell': True,
u'_raw_params': u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=9ff8b389-5e24-4166-9842-f1d6104b662b volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98
| grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None, u'argv':
None, u'creates': None, u'chdir': None, u'stdin': None}},
u'stdout_lines': [], u'stderr': u'20+0 records in\n20+0 records
out\n10240 bytes (10 kB) copied, 0.000141645 s, 72.3 MB/s\ntar:
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive\ntar: Exiting with failure
status due to previous errors', u'_ansible_no_log': False}
[ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records
in', u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000140541 s,
72.9 MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in
archive', u'tar: Exiting with failure status due to previous errors'],
u'changed': True, u'end': u'2019-05-08 15:24:01.387469',
u'_ansible_item_label': {u'image_id':
u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE',
u'id': u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'stdout':
u'', u'failed': True, u'_ansible_item_result': True,
u'msg': u'non-zero return code', u'rc': 2, u'start':
u'2019-05-08 15:24:00.660309', u'attempts': 12, u'cmd':
u"vdsm-client Image prepare storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=8691076a-8e45-4429-a18a-5faebef866cc volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a
| grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40
b18612a54.ovf", u'item': {u'image_id':
u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE',
u'id': u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'delta':
u'0:00:00.727160', u'invocation': {u'module_args':
{u'warn': False, u'executable': None, u'_uses_shell': True,
u'_raw_params': u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=8691076a-8e45-4429-a18a-5faebef866cc volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a
| grep path | awk '{ print $2 }' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None, u'argv':
None, u'creates': None, u'chdir': None, u'stdin': None}},
u'stdout_lines': [], u'stderr': u'20+0 records in\n20+0 records
out\n10240 bytes (10 kB) copied, 0.000140541 s, 72.9 MB/s\ntar:
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in archive\ntar: Exiting with failure
status due to previous errors', u'_ansible_no_log': False}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
I tried twice. Same result. Should I retry?
Is it safe to use the local hosted engine for starting stopping vms? I'm kind of
headless for some days :-)
Best regards.
10:08 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
On Wed, May 8, 2019 at 3:44 PM Andreas Elvers <
andreas.elvers+ovirtforum(a)solutions.work> wrote:
Hello,
when trying to deploy the engine using "hosted-engine --deploy
--restore-from-file=myenginebackup" the ansible playbook errors out at
[ INFO ] TASK [ovirt.hosted_engine_setup : Trigger hosted engine OVF
update and enable the serial console]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait until OVF update finishes]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Parse OVF_STORE disk list]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Check OVF_STORE volume status]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE disk
content]
[ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records
in',
u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000141645 s, 72.3
MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in
archive', u'tar: Exiting with failure status due to previous errors'],
u'changed': True, u'end': u'2019-05-08 15:21:47.595195',
u'_ansible_item_label': {u'image_id':
u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE',
u'id':
u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'stdout': u'',
u'failed': True,
u'_ansible_item_result': True, u'msg': u'non-zero return code',
u'rc': 2,
u'start': u'2019-05-08 15:21:46.906877', u'attempts': 12,
u'cmd':
u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=9ff8b389-5e24-4166-9842-f1d6104b662b
volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98 | grep path | awk '{ print $2
}' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40
b18612a54.ovf", u'item': {u'image_id':
u'65fd6c57-033c-4c95-87c1-b16c26e4bc98', u'name': u'OVF_STORE',
u'id':
u'9ff8b389-5e24-4166-9842-f1d6104b662b'}, u'delta':
u'0:00:00.688318',
u'invocation': {u'module_args': {u'warn': False,
u'executable': None,
u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=9ff8b389-5e24-4166-9842-f1d6104b662b
volumeID=65fd6c57-033c-4c95-87c1-b16c26e4bc98 | grep path | awk '{ print $2
}' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None,
u'argv': None,
u'creates': None, u'chdir': None, u'stdin': None}},
u'stdout_lines': [],
u'stderr': u'20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied,
0.000141645 s, 72.3 MB/s\ntar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf:
Not found in archive\ntar: Exiting with failure status due to previous
errors', u'_ansible_no_log': False}
[ ERROR ] {u'_ansible_parsed': True, u'stderr_lines': [u'20+0 records
in',
u'20+0 records out', u'10240 bytes (10 kB) copied, 0.000140541 s, 72.9
MB/s', u'tar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf: Not found in
archive', u'tar: Exiting with failure status due to previous errors'],
u'changed': True, u'end': u'2019-05-08 15:24:01.387469',
u'_ansible_item_label': {u'image_id':
u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE',
u'id':
u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'stdout': u'',
u'failed': True,
u'_ansible_item_result': True, u'msg': u'non-zero return code',
u'rc': 2,
u'start': u'2019-05-08 15:24:00.660309', u'attempts': 12,
u'cmd':
u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=8691076a-8e45-4429-a18a-5faebef866cc
volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a | grep path | awk '{ print $2
}' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40
b18612a54.ovf", u'item': {u'image_id':
u'dacf9ad8-77b9-4205-8ca2-d6877627ad4a', u'name': u'OVF_STORE',
u'id':
u'8691076a-8e45-4429-a18a-5faebef866cc'}, u'delta':
u'0:00:00.727160',
u'invocation': {u'module_args': {u'warn': False,
u'executable': None,
u'_uses_shell': True, u'_raw_params': u"vdsm-client Image prepare
storagepoolID=597f329c-0296-03af-0369-000000000139
storagedomainID=f708ced4-e339-4d02-a07f-78f1a30fc2a8
imageID=8691076a-8e45-4429-a18a-5faebef866cc
volumeID=dacf9ad8-77b9-4205-8ca2-d6877627ad4a | grep path | awk '{ print $2
}' | xargs -I{} sudo -u vdsm dd if={} | tar -tvf -
ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf", u'removes': None,
u'argv': None,
u'creates': None, u'chdir': None, u'stdin': None}},
u'stdout_lines': [],
u'stderr': u'20+0 records in\n20+0 records out\n10240 bytes (10 kB) copied,
0.000140541 s, 72.9 MB/s\ntar: ebb09b0e-2d03-40f0-8fa4-c40b18612a54.ovf:
Not found in archive\ntar: Exiting with failure status due to previous
errors', u'_ansible_no_log': False}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing
ansible-playbook
I tried twice. Same result. Should I retry?
We had a bug about that in the past:
https://bugzilla.redhat.com/1644748
<https://bugzilla.redhat.com/show_bug.cgi?id=1644748>
but it's reported as CLOSED CURRENTRELEASE.
Can I ask which versions of ovirt-hosted-engine-setup,
ovirt-ansible-hosted-engine-setup
and ovirt-engine-appliance are you using?
I see that you sent more than one email in the last days and in general all
the issue you are reporting are due to timeouts/race conditions.
Can you please provide some more info about your storage configuration?
Is it safe to use the local hosted engine for starting stopping vms? I'm
kind of headless for some days :-)
Best regards.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UNBCYRKXM24...
--
Simone Tiraboschi
He / Him / His
Principal Software Engineer
Red Hat <https://www.redhat.com/>
stirabos(a)redhat.com
@redhatjobs <https://twitter.com/redhatjobs> redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://red.ht/sig>
<https://redhat.com/summit>
4:02 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
Yeah. I digged that bug report up as well.
Our oVirt setup comprises of
6 oVirt hosts which form 2 datacenters
Datacenter: Luise (has a ceph storage domain)
node01 to node03 are a hyper converged oVirt glusterfs. This is cluster Luise01. My
original is to move the engine onto this cluster.
Datacenter: Default Cluster Default
node04 and node05. It uses an nfs storage: see below. The engine used to live here.
Datacenter Luise Cluster: Luise02
node06 is currently the only node. This cluster will gain all nodes from the Default
Datacenter as soon as it is decommissioned.
All nodes are by Supermicro 4 cores 48GB Intel Xeon E5506 @ 2.13 GHz
They use two bonded 1Gbit Links for connectivity. I was sceptical with performance, but we
are running multiple around 10-20 TYPO3 CMSes plus lesser IO hungry things in around 3 to
6 VMs per Node. They all use Ceph as storage domain.
All nodes I tried to deploy to are without virtual load.
Storage:
========
The glusterfs is provided by the hyper converged setup provided by oVirt Node NG. It is
quite small because is just our gateway to boot the cinder VM that will guide our ceph
nodes to their rbd destiny. Every Node has a 250GB SSD. This gives us 90GB for the
:/engine gluster volume. There is also a 100GB volume for the supporting VMs that provide
the bootstrap to cinder. There is little IO going to these volumes.
The ceph is a 10Gbit connected cluster of 3 Nodes with SSD OSDs. The oVirt connect via a
Cisco
The NFS server is serving a Raid-10 btrfs formatted filesystem added with a bcache.
My migration history so far:
Tried to Migrate to Luise Datacenter I did a typo with entering Luise1 instead the correct
cluster Luise01. Ironically that migration succeeded.
Tried to Migrate back to Default. First one erred out with "Disk Image is not
available in the cluster". Second one succeeded after following your advice to try it
a second time. I removed the accidentally created Luise1 cluster.
Tried to Migrate to Luise again. This time I forgot to completely shut down the hosted
engine on Default Datacenter. I shut it down and tried again.
All subsequent retires to deploy to the hyper converged Luise01 cluster failed. I used the
Local Engine to re-install node05 which was hosting the engine I forget to shut down. I
was not able to access the web interface for some reason, so I supposed it was damaged in
a way.
After 4 tries of deployment to our hyper converged cluster that all erred with the
"Disk Image is not available" I'm not trying to go back to the Default
Datacenter. Twice it erred out at "Wait for OVF_STORE disk content"
5:15 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
Short correction: All nodes oVirt and ceph are connected with 3 stacked Cisco 3750e
providing 10 and 1Gbit ports.
5:32 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
I'm current on 4.3.3.1
The software versions I am using:
ovirt-hosted-engine-setup: 2.3.7
ovirt-ansible-hosted-engine-setup: 1.0.17
ovirt-engine-appliance: 4.2
I think I found the bug. :-/
6:55 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
I updated all nodes to 4.3.3.1, but I did not remove the 4.2 repo from /etc/yum.repos.d.
I did now remove the old repos, ovirt-engine-appliance is now shown as 4.3.
I tried deployment to the hyper converged cluster. First try failed again with "The
target Data Center does not contain the Virtual Disk.". I'm retrying. Should this
fail again, I will try a deployment to the old NFS backed Default Datacenter.
8:43 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
I re-checked the versions with
[root@node05 ~]# rpm -qa | grep engine
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-engine-appliance-4.3-20190416.1.el7.x86_64
ovirt-hosted-engine-ha-2.3.1-1.el7.noarch
ovirt-hosted-engine-setup-2.3.7-1.el7.noarch
ovirt-ansible-hosted-engine-setup-1.0.17-1.el7.noarch
python-ovirt-engine-sdk4-4.3.1-2.el7.x86_64
so probably I was already on the latest versions. I did not do any updated. First I was
doing a "rpm info" to get the version.
9:06 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
Deployment on NFS again failed with TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE
disk content].
I'm lost.
3:49 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
Any update about this? If still stuck, please provide more
information, such as the relevant parts of relevant log files (setup
logs, engine/vdsm, etc.). Thanks and best regards,
On Thu, May 9, 2019 at 4:07 PM Andreas Elvers
<andreas.elvers+ovirtforum(a)solutions.work> wrote:
Deployment on NFS again failed with TASK [ovirt.hosted_engine_setup : Wait for OVF_STORE
disk content].
I'm lost.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/DFG5ECI2DAD...
--
Didi
8:02 a.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
Sorry. The thread became separated. Yeah. I finished the migration. Simone proposed I had
a corrupted backup. And yes. I was backtracking my backups to that point where I switched
all datacenters to 4.3 compatibility and before doing any engine migration. Lucky me, I
had that backup available.
I think my backups became corrupted, because I did a backup from the engine after I moved
it to a cluster that was originally not defined. I had a typo in the cluster name. And
used that backup and tried to get the engine back to my nfs storage. That worked. I did
another backup at that place again and from that point on the backup would always break
the deployment for whatever reason.
The logs are available at
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HXJZWN7L6WXS...
3:26 p.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
On Wed, May 22, 2019 at 3:04 PM Andreas Elvers
<andreas.elvers+ovirtforum(a)solutions.work> wrote:
Sorry. The thread became separated. Yeah. I finished the migration. Simone proposed I had
a corrupted backup. And yes. I was backtracking my backups to that point where I switched
all datacenters to 4.3 compatibility and before doing any engine migration. Lucky me, I
had that backup available.
I think my backups became corrupted, because I did a backup from the engine after I moved
it to a cluster that was originally not defined. I had a typo in the cluster name. And
used that backup and tried to get the engine back to my nfs storage. That worked. I did
another backup at that place again and from that point on the backup would always break
the deployment for whatever reason.
The logs are available at
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HXJZWN7L6WXS...
Sorry that I didn't follow all the threads. It sounds like you went
through a lot of hard-work with your oVirt setups lately :-(
If you can now take the time, and think about changes/improvements
that would have helped you, or (more importantly) prevent you from
doing some mistakes you did, please open RFEs/bugs.
I'll also discuss restore with Simone in private, we might come up
with our own RFE.
Thanks and best regards,
--
Didi
4:04 p.m.
New subject: Engine restore errors out on "Wait for OVF_STORE disk content"
No need to worry. I'm so much impressed with the quality of oVirt. It has so many
different setup scenarios, mine even a tech preview style one ;-)
And good idea to file a bug for usability. To wrap things up:
Me typing Luise1 instead of Luise01 was my error to begin with. It triggered everything.
Luise01 is our hyper converged oVirt node cluster. It is our ceph bootstrap. Yeah... How
could you forget the zero..
node01, node02 and node03 are forming Luise01.
I ordered deployment of the engine to node01 on Luise1 and hosted-engine happily did so.
Creating a new cluster Luise1 on node01 masking the glusterfs bricks on node01 for the
remaining Luise01 cluster nodes.
This most probably also triggered this issue:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/L3YCRPRAGPUM...
It has already been addressed by Nir Soffer. See
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OSZGHT65OUU...
I will create a bug report for that. A node that has already joined a cluster should not
be a viable destination for an engine deployment to a new to create cluster.
2042
days inactive
2056
days old
11 comments
3 participants
participants (3)
-
Andreas Elvers -
Simone Tiraboschi -
Yedidyah Bar David