On Fri, Jun 29, 2018 at 12:32 PM fsoyer <fsoyer@systea.fr> wrote:

Hi,

hi Frank,

thanks for the report.

I must say it : I'm -totally- lost.
To try to find a reason to this error, I've re-installed the first host from scratch - CentOS 7.5-1804, ovirt 4.2.3-1, gluster 3.12.9.
The first attemp was made with only em1 declared. Result = SUCCESS, the install pass "Get local VM IP", then through "Wait for the host to be up" without difficulty and wait at "Please specify the storage...".
At this time I even notice that I've forgot to stop/disable NetworkManager, that had no impact !
So : I re-install the host from scratch (yes, sometimes I'm a fool) to be absolutly sure that there is no problem coming from the preceding install. Now I declare em1 (10.0.0.230) and em2 (10.0.0.229, without gateway nor DNS, for futur vmnetwork). NetworkManager off and disabled. Result = SUCCESS... Oo
OK : Re-install host !! Now I declare, as I did some days ago, em1, em2 and bond0(em3+em4 with IP 192.168.0.30). Result : SUCCESS !!! Oo

So I'm unable to say what append tuesday. Actually I see only two differences :
- gluster is not active (I don't configure it to go faster)

did you tried with gdeploy from the cockpit web ui?

- the version of ovirt (ovirt-release, ovirt-host, appliance...) has sligthly changed.

AFAIK we didn't have any specific fix for that kind of issue in the recent weeks.

I've no more time for another attempt re-installing the host(s) with gluster activated, I must now go on as I need an operational system for other tasks with VMs this afternoon. So I leave the first host waiting for the end of install in a screen, I re-install the 2 other hosts and activate gluster and volumes on the 3 nodes. Then I'll end the install on the gluster volume.
I'll tell you if this works finally, but I hope so !
however, I'm in doubt with this problem. I have no explanation of what append tuesday, this is really annoying...

Yes, the same for me.

Maybe have you the ability to test on a same configuration (3 hosts with 2 nics on the same network for ovirtmgmt and a futur vmnetwork, and gluster on a separate network) to try to understand ?

In the past moths we had a lot of successful tests also on complex network environments; I'll try to reproduce on something really close to your env.

Thank you for the time spent.
Frank

PS : to answer to your question : yes, tuesday I ran ovirt-hosted-engine-cleanup between each attempt.

Le Jeudi, Juin 28, 2018 16:26 CEST, Simone Tiraboschi <stirabos@redhat.com> a écrit:

On Wed, Jun 27, 2018 at 5:48 PM fsoyer@systea.fr <fsoyer@systea.fr> wrote:
Hi again,
In fact, the hour in file is exactly 2hours before, I guess a timezone problem (in the process of install ?), as the file itself is correctly timed at 11:17am (correct hour here in France). So the messages are synchrone.

Yes, sorry, fault of mine.
From the logs I don't see anything strange.

Can you please try again on your environment and connect to the bootstrap VM via virsh console or VNC to check what's happening there?

Did you also run ovirt-hosted-engine-cleanup between one attempt and the next?

-------- Message original --------
Objet : Re: [ovirt-users] Re: Install hosted-engine - Task Get local VM IP failed
De : Simone Tiraboschi
À : fsoyer@systea.fr
Cc : users

Hi,
HostedEngineLocal was started at 2018-06-26 09:17:26 but /var/log/messages starts only at Jun 26 11:02:32.
Can you please reattach it fro the relevant time frame?

On Wed, Jun 27, 2018 at 10:54 AM fsoyer <fsoyer@systea.fr> wrote:
Hi Simone,
here are the revelant part of messages and the engine install log (there were only this file in /var/log/libvirt/qemu) .

Thanks for your time.

Frank

Le Mardi, Juin 26, 2018 11:43 CEST, Simone Tiraboschi <stirabos@redhat.com> a écrit:

On Tue, Jun 26, 2018 at 11:39 AM fsoyer <fsoyer@systea.fr> wrote:
Well,
unfortunatly, it was a "false-positive". This morning I tried again, with the idea that at one moment the deploy will ask for the final destination for the engine, I will restart bond0+gluster+volume engine at thos moment.
Re-launching the deploy on the second "fresh" host (the first one with all errors yesterday let it in a doutful state) with em2 and gluster+bond0 off :
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.227/8 brd 10.255.255.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff
4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff
5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state DOWN group default qlen 1000
link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff

# ip r
default via 10.0.1.254 dev em1
10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227
169.254.0.0/16 dev em1 scope link metric 1002
... does NOT work this morning
[ INFO ] TASK [Get local VM IP]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:01:c6:32 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.083587", "end": "2018-06-26 11:26:07.581706", "rc": 0, "start": "2018-06-26 11:26:07.498119", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
I'm sure that the network was the same yesterday when my attempt finally pass the "get local vm ip". Why not today ?
After the error, the network was :
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.227/8 brd 10.255.255.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff
4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff
5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state DOWN group default qlen 1000
link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:01:c6:32 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe01:c632/64 scope link
valid_lft forever preferred_lft forever

# ip r
default via 10.0.1.254 dev em1
10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227
169.254.0.0/16 dev em1 scope link metric 1002
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

So, finally, I have no idea why this appends :(((

Can you please attach /var/log/messages and /var/log/libvirt/qemu/* ?

Le Mardi, Juin 26, 2018 09:21 CEST, Simone Tiraboschi <stirabos@redhat.com> a écrit:

On Mon, Jun 25, 2018 at 6:32 PM fsoyer <fsoyer@systea.fr> wrote:
Well, answering to myself for more informations.
Thinking that the network was part of the problem, I tried to stop gluster volumes, stop gluster on host, and stop bond0.
So, the host now had just em1 with one IP.
And... The winner is... Yes : the install passed the "[Get local VM IP]" and continued !!

I hit ctrl-c, restart the bond0, restart deploy : it crashed. So it seems that more than one network is the problem. But ! How do I install engine on gluster on a separate - bonding - jumbo network in this case ???

Can you reproduce this on your side ?

Can you please attach the output of 'ip a' in both the case?

Frank

Le Lundi, Juin 25, 2018 16:50 CEST, "fsoyer" <fsoyer@systea.fr> a écrit:

Hi staff,
Installing a fresh ovirt - CentOS 7.5.1804 up to date, ovirt version :
# rpm -qa | grep ovirt
ovirt-hosted-engine-ha-2.2.11-1.el7.centos.noarch
ovirt-imageio-common-1.3.1.2-0.el7.centos.noarch
ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64
ovirt-vmconsole-1.0.5-4.el7.centos.noarch
ovirt-provider-ovn-driver-1.2.10-1.el7.centos.noarch
ovirt-hosted-engine-setup-2.2.20-1.el7.centos.noarch
ovirt-engine-appliance-4.2-20180504.1.el7.centos.noarch
python-ovirt-engine-sdk4-4.2.6-2.el7.centos.x86_64
ovirt-host-deploy-1.7.3-1.el7.centos.noarch
ovirt-release42-4.2.3.1-1.el7.noarch
ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch
cockpit-ovirt-dashboard-0.11.24-1.el7.centos.noarch
ovirt-setup-lib-1.1.4-1.el7.centos.noarch
ovirt-imageio-daemon-1.3.1.2-0.el7.centos.noarch
ovirt-host-4.2.2-2.el7.centos.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch

ON PHYSICAL SERVERS (not on VMware, why should I be ?? ;) I got exactly the same error :
[ INFO ] TASK [Get local VM IP]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:69:3a:c6 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.073313", "end": "2018-06-25 16:11:36.025277", "rc": 0, "start": "2018-06-25 16:11:35.951964", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
[ INFO ] TASK [include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [Remove local vm dir]
[ INFO ] changed: [localhost]
[ INFO ] TASK [Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO ] Stage: Clean up

I have 4 NIC :
em1 10.0.0.230/8 is for ovirmgmt, it have the gateway
em2 10.0.0.229/8 is for a vmnetwork
em3+em4 in bond0 192.168.0.30 are for gluster with jumbo frames, volumes (ENGINE, ISO,EXPORT,DATA) are up and operationals.

I tried to stop em2 (ONBOOT=No and restart network), so the network is actually :
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether e0:db:55:15:eb:70 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.230/8 brd 10.255.255.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::e2db:55ff:fe15:eb70/64 scope link
valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether e0:db:55:15:eb:71 brd ff:ff:ff:ff:ff:ff
4: em3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff
5: em4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.30/24 brd 192.168.0.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::e2db:55ff:fe15:eb72/64 scope link
valid_lft forever preferred_lft forever

# ip r
default via 10.0.1.254 dev em1
10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.230
169.254.0.0/16 dev em1 scope link metric 1002
169.254.0.0/16 dev bond0 scope link metric 1006
192.168.0.0/24 dev bond0 proto kernel scope link src 192.168.0.30

but same issue, after "/usr/sbin/ovirt-hosted-engine-cleanup" and restarting the deployment.
NetworkManager was stopped and disabled at the node install, and it is still stopped.
After the error, the network shows this after device 6 (bond0) :
7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff
11: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:69:3a:c6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe69:3ac6/64 scope link
valid_lft forever preferred_lft forever

I do not see ovirmgmt... And I don't know if I can access the engine vm as I have not its IP :(
I tried to ping addresses after 192.168.122.1, but no one are accessible so I stopped at 122.10. The VM seems up (kvm process), qemu-kvm process taking 150% of cpu in "top"...

I pasted the log here : https://pastebin.com/Ebzh1uEh

PLEASE ! This issue seems to be reccurent since the beginning of 2018 (see messages here on list !
Jamie Lawrence in February, suporte@logicworks.pt in april, shamilkpm@gmail.com and Yaniv Kaul in May,
florentl on june 01...). Can anyone give us a way to solve this ?
--
Cordialement,

Frank Soyer

Le Lundi, Juin 04, 2018 16:07 CEST, Simone Tiraboschi <stirabos@redhat.com> a écrit:

On Mon, Jun 4, 2018 at 2:20 PM, Phillip Bailey <phbailey@redhat.com> wrote:
Hi Florent,

Could you please provide the log for the stage in which the wizard is failing? Logs can be found in /var/log/ovirt-hosted-engine-setup.

Thanks!

-Phillip Bailey

On Fri, Jun 1, 2018 at 7:57 AM, florentl <florentl.info@laposte.net> wrote:
Hi all,
I try to install hosted-engine on node : ovirt-node-ng-4.2.3-0.20180518.
Every times I get stuck on :
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:5a:91 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.108872", "end": "2018-06-01 11:17:34.421769", "rc": 0, "start": "2018-06-01 11:17:34.312897", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
I tried with static IP Address and with DHCP but both failed.

To be more specific, I installed three nodes, deployed glusterfs with the wizard. I'm in a nested virtualization environment for this lab (Vmware Esxi Hypervisor).

Unfortunately I think that the issue is trying to run a nested env over ESXi.
AFAIk nesting KVM VMs over ESX is still problematic.

I'd suggest to repeat the experiment nesting over KVM on L0.

My node IP is : 192.168.176.40 / and I want the hosted-engine vm has 192.168.176.43.

Thanks,
Florent
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/F3BNUQ2T434EASIX56F7KQQJVF7OCDUM/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RU34XDM2W6GPDCRRGWORBTPH2BUN3CJR/