Hi Simone,

“bootstrap VMs for some reasons fails to get an address from DHCP over libvirt default natted network”. 

I respectfully disagree.  Line 1784 of the most recent log I sent you is:

2018-10-24 14:12:26,524+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 local_vm_ip: {'stderr_lines': [], u'changed': True, u'end': u'2018-10-24 14:12:25.437958', u'stdout': u'192.168.124.175', u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:3d:13:d8 | awk '{ print $5 }' | cut -f1 -d'/'", 'failed': False, 'attempts': 2, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.070076', 'stdout_lines': [u'192.168.124.175'], u'start': u'2018-10-24 14:12:25.367882'}

I am able to ping the natted IP.

Also I wrote after an earlier attempt: “ovirt-engine.example.com resolves to a natted IP: 192.168.124.51”.  (https://www.mail-archive.com/users@ovirt.org/msg51515.html). 

The process fails as the bootstrap VM is moved to the final engine VM on the shared storage.  I still think if I solve the DHCP problem on my network, the engine VM will be successfully created.  Are you able to confirm the Node VM (https://www.ovirt.org/node/) has DHCP enabled?  If so, then since this has the same DHCP problem as the engine VM (I have to type “dhclient” each boot to get an IP), I can use Node to troubleshoot.

Many thanks,

Brendan

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 26 October 2018 08:39
To: B Holmes <me@brendanh.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] Re: Diary of hosted engine install woes

On Thu, Oct 25, 2018 at 4:29 PM Brendan Holmes <me@brendanh.com> wrote:

“The issue here is that it seams that your VM never got an address from libvirt default network DHCP” – but aren’t I obviating DHCP by using parameter:

OVEHOSTED_VM/cloudinitVMStaticCIDR=str:10.0.0.109/24

?  VM should receive a static IP instead.

Perhaps there is a bug in hosted-engine setup that forces it to require DHCP. 

No, this is expected.

The deploy process will launch a bootstrap local VM that will get a temporary address via DHCP over libvirt default natted network.

The engine running on that VM will be used to configure the environment and create a VM on the shared storage.

Only at the end, the local VM will be shutdown and its disk moved over the disk of the VM created by the engine on the shared storage.

In your case it seams that the bootstrap VMs for some reasons fails to get an address from DHCP over libvirt default natted network.

I'd suggest to try connecting to that VM via VNC to see what's going on there.

If you have no more suggestions, I will buy a new router with different DHCP server.

Many thanks,

Brendan

From: Brendan Holmes <me@brendanh.com>
Sent: 24 October 2018 16:48
To: 'Simone Tiraboschi' <stirabos@redhat.com>
Cc: 'users' <users@ovirt.org>
Subject: RE: [ovirt-users] Re: Diary of hosted engine install woes

Hi Simone,

[root@host ~]# virsh -r net-dhcp-leases default

Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID

-------------------------------------------------------------------------------------------------------------------

[root@host ~]#

No, node is running on bare-metal.  Node was installed by plugging in a bootable USB.

Many thanks,

Brendan

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 24 October 2018 16:11
To: B Holmes <me@brendanh.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] Re: Diary of hosted engine install woes

On Wed, Oct 24, 2018 at 4:56 PM Brendan Holmes <me@brendanh.com> wrote:

Hi Simone,

My attempt to install the RPM output hundreds of lines similar to:

/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/__init__.pyc from install of ovirt-hosted-engine-setup-2.2.29-1.el7.noarch conflicts with file from package ovirt-hosted-engine-setup-2.2.26-1.el7.noarch

Do I need to install the matching version of ovirt node?

“using it interactively without manually filing an answerfile” – if you mean using command: hosted-engine –deploy

and then manually typing each value, I tried this (choosing the static IP option) and the usual error occurs.  I attach the log from this attempt.

The issue here is that it seams that your VM never got an address from libvirt default network DHCP:

2018-10-24 15:36:14,370+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 local_vm_ip: {'stderr_lines': [], u'changed': True, u'end': u'2018-10-24 15:36:13.196641', u'stdout': u'', u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:3d:13:d8 | awk '{ print $5 }' | cut -f1 -d'/'", 'failed': False, u'delta': u'0:00:00.054641', u'stderr': u'', u'rc': 0, 'stdout_lines': [], u'start': u'2018-10-24 15:36:13.142000'}

Can you please share the output of 

  virsh -r net-dhcp-leases default 

Are you running nested on a different hypervisor?

Many thanks,

Brendan

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 24 October 2018 08:44
To: B Holmes <me@brendanh.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] Re: Diary of hosted engine install woes

Can you please retry installing https://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_4.2_build-artifacts-el7-x86_64/157/artifact/exported-artifacts/ovirt-hosted-engine-setup-2.2.29-1.el7.noarch.rpm and using it interactively without manually filing an answerfile?

On Tue, Oct 23, 2018 at 11:19 PM Brendan Holmes <me@brendanh.com> wrote:

Hi Simone,

I attempted to use a fixed IP address instead of DHCP.  I attach an attempt using answer-file parameter:

OVEHOSTED_VM/cloudinitVMStaticCIDR=str:10.0.0.109/24

which is in accordance with this example on your website:

https://ovirt.org/develop/release-management/features/integration/heapplianceflow/

However, instead of this, you will notice that the log has:

OVEHOSTED_VM/cloudinitVMStaticCIDR=bool:'False'

The same error as usual occurred.  Is this:

1. The wrong syntax\format for specifying a “cloudinitVMStaticCIDR” IP?
2. ovirt setup is ignoring it: either failing to assign the IP to the engine VM or to variable “local_vm_ip.stdout_lines”?
3. Something else?

“You can also connect to that VM with VNC to check what's going on.” – Doesn’t work.  Port querying 10.0.0.109 on 5800\5900 fails, so I don’t think the VM is networked.  Which port does engine VNC use?

Many thanks,

Brendan

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 23 October 2018 09:09
To: B Holmes <me@brendanh.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] Re: Diary of hosted engine install woes

OK,

according to this log file, the target engine VM didn't got an IPv4 address:

Are you sure that your DHCP server is properly working?

You can also connect to that VM with VNC to check what's going on.

On Mon, Oct 22, 2018 at 10:37 PM Brendan Holmes <me@brendanh.com> wrote:

Hi Simone,

Pls see attached the log without any of my variable-enumeration changes.

Many thanks,

Brendan

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 22 October 2018 13:33
To: B Holmes <me@brendanh.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] Re: Diary of hosted engine install woes

On Sun, Oct 21, 2018 at 11:50 PM Brendan Holmes <me@brendanh.com> wrote:

Hi Simone,

Sorry for late response, I’ve been unwell.  Attached is the whole log you requested.  I hope it reveals why variable “local_vm_ip.stdout_lines” is not being populated.

Hi,

according to your log file the deployment was successful: 

at this point I can only suggest again to simply run the vanilla code without any custom patch.

From: Simone Tiraboschi <stirabos@redhat.com>
Sent: 15 October 2018 04:49

On Sun, Oct 14, 2018 at 6:08 AM Brendan Holmes <me@brendanh.com> wrote:

Hi Simone,

[ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: list object has no element 0\n\nThe error appears to have been in '/usr/share/ovirt-hosted-engine-setup/ansible/create_target_vm.yml': line 396, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    changed_when: True\n  - name: Clean /etc/hosts on the host\n    ^ here\n"}

I think that this is just an error in the clean up on failures phase due to something bad before.

Can you please attach the whole deployment log from /var/log/ovirt-hosted-engine/hosted-engine.log ?