[ovirt-users] Re: Install hosted-engine - Task Get local VM IP failed

Hi,

I must say it : I'm -totally- lost. To try to find a reason to this error, I've re-installed the first host from scratch - CentOS 7.5-1804, ovirt 4.2.3-1, gluster 3.12.9. The first attemp was made with only em1 declared. Result = SUCCESS, the install pass "Get local VM IP", then through "Wait for the host to be up" without difficulty and wait at "Please specify the storage...". At this time I even notice that I've forgot to stop/disable NetworkManager, that had no impact ! So : I re-install the host from scratch (yes, sometimes I'm a fool) to be absolutly sure that there is no problem coming from the preceding install. Now I declare em1 (10.0.0.230) and em2 (10.0.0.229, without gateway nor DNS, for futur vmnetwork). NetworkManager off and disabled. Result = SUCCESS... Oo OK : Re-install host !! Now I declare, as I did some days ago, em1, em2 and bond0(em3+em4 with IP 192.168.0.30). Result : SUCCESS !!! Oo So I'm unable to say what append tuesday. Actually I see only two differences : - gluster is not active (I don't configure it to go faster)

- the version of ovirt (ovirt-release, ovirt-host, appliance...) has sligthly changed.

I've no more time for another attempt re-installing the host(s) with gluster activated, I must now go on as I need an operational system for other tasks with VMs this afternoon. So I leave the first host waiting for the end of install in a screen, I re-install the 2 other hosts and activate gluster and volumes on the 3 nodes. Then I'll end the install on the gluster volume. I'll tell you if this works finally, but I hope so ! however, I'm in doubt with this problem. I have no explanation of what append tuesday, this is really annoying...

Maybe have you the ability to test on a same configuration (3 hosts with 2 nics on the same network for ovirtmgmt and a futur vmnetwork, and gluster on a separate network) to try to understand ?

Thank you for the time spent. Frank PS : to answer to your question : yes, tuesday I ran ovirt-hosted-engine-cleanup between each attempt. Le Jeudi, Juin 28, 2018 16:26 CEST, Simone Tiraboschi <stirabos(a)redhat.com&gt; a écrit: On Wed, Jun 27, 2018 at 5:48 PM fsoyer(a)systea.fr <fsoyer(a)systea.fr&gt; wrote: > Hi again, > In fact, the hour in file is exactly 2hours before, I guess a timezone > problem (in the process of install ?), as the file itself is correctly > timed at 11:17am (correct hour here in France). So the messages are > synchrone. > > Yes, sorry, fault of mine. From the logs I don't see anything strange. Can you please try again on your environment and connect to the bootstrap VM via virsh console or VNC to check what's happening there? Did you also run ovirt-hosted-engine-cleanup between one attempt and the next? > > -------- Message original -------- > Objet : Re: [ovirt-users] Re: Install hosted-engine - Task Get local VM > IP failed > De : Simone Tiraboschi > À : fsoyer(a)systea.fr > Cc : users > > > > Hi, > HostedEngineLocal was started at 2018-06-26 09:17:26 but > /var/log/messages starts only at Jun 26 11:02:32. > Can you please reattach it fro the relevant time frame? > > On Wed, Jun 27, 2018 at 10:54 AM fsoyer <fsoyer(a)systea.fr&gt; wrote: > >> Hi Simone, >> here are the revelant part of messages and the engine install log (there >> were only this file in /var/log/libvirt/qemu) . >> >> Thanks for your time. >> >> Frank >> >> >> Le Mardi, Juin 26, 2018 11:43 CEST, Simone Tiraboschi < >> stirabos(a)redhat.com&gt; a écrit: >> >> >> >> On Tue, Jun 26, 2018 at 11:39 AM fsoyer <fsoyer(a)systea.fr&gt; wrote: >> >>> Well, >>> unfortunatly, it was a "false-positive". This morning I tried again, >>> with the idea that at one moment the deploy will ask for the final >>> destination for the engine, I will restart bond0+gluster+volume engine at >>> thos moment. >>> Re-launching the deploy on the second "fresh" host (the first one with >>> all errors yesterday let it in a doutful state) with em2 and gluster+bond0 >>> off : >>> >>> # ip a >>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >>> group default qlen 1000 >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>> inet 127.0.0.1/8 scope host lo >>> valid_lft forever preferred_lft forever >>> inet6 ::1/128 scope host >>> valid_lft forever preferred_lft forever >>> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >>> group default qlen 1000 >>> link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff >>> inet 10.0.0.227/8 brd 10.255.255.255 scope global em1 >>> valid_lft forever preferred_lft forever >>> inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link >>> valid_lft forever preferred_lft forever >>> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff >>> 4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff >>> 5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff >>> 6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state >>> DOWN group default qlen 1000 >>> link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff >>> >>> # ip r >>> default via 10.0.1.254 dev em1 >>> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227 >>> 169.254.0.0/16 dev em1 scope link metric 1002 >>> >>> ... does NOT work this morning >>> >>> [ INFO ] TASK [Get local VM IP] >>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:01:c6:32 >>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.083587", "end": >>> "2018-06-26 11:26:07.581706", "rc": 0, "start": "2018-06-26 >>> 11:26:07.498119", "stderr": "", "stderr_lines": [], "stdout": "", >>> "stdout_lines": []} >>> >>> I'm sure that the network was the same yesterday when my attempt >>> finally pass the "get local vm ip". Why not today ? >>> After the error, the network was : >>> >>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >>> group default qlen 1000 >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>> inet 127.0.0.1/8 scope host lo >>> valid_lft forever preferred_lft forever >>> inet6 ::1/128 scope host >>> valid_lft forever preferred_lft forever >>> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >>> group default qlen 1000 >>> link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff >>> inet 10.0.0.227/8 brd 10.255.255.255 scope global em1 >>> valid_lft forever preferred_lft forever >>> inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link >>> valid_lft forever preferred_lft forever >>> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff >>> 4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff >>> 5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff >>> 6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state >>> DOWN group default qlen 1000 >>> link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff >>> 7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >>> state UP group default qlen 1000 >>> link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff >>> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 >>> valid_lft forever preferred_lft forever >>> 8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master >>> virbr0 state DOWN group default qlen 1000 >>> link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff >>> 9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >>> master virbr0 state UNKNOWN group default qlen 1000 >>> link/ether fe:16:3e:01:c6:32 brd ff:ff:ff:ff:ff:ff >>> inet6 fe80::fc16:3eff:fe01:c632/64 scope link >>> valid_lft forever preferred_lft forever >>> >>> # ip r >>> default via 10.0.1.254 dev em1 >>> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227 >>> 169.254.0.0/16 dev em1 scope link metric 1002 >>> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 >>> >>> >>> >>> So, finally, I have no idea why this appends :((( >> >> >> Can you please attach /var/log/messages and /var/log/libvirt/qemu/* ? >> >> >> >>> >>> >>> Le Mardi, Juin 26, 2018 09:21 CEST, Simone Tiraboschi < >>> stirabos(a)redhat.com&gt; a écrit: >>> >>> >>> >>> On Mon, Jun 25, 2018 at 6:32 PM fsoyer <fsoyer(a)systea.fr&gt; wrote: >>> >>>> Well, answering to myself for more informations. >>>> Thinking that the network was part of the problem, I tried to stop >>>> gluster volumes, stop gluster on host, and stop bond0. >>>> So, the host now had just em1 with one IP. >>>> And... The winner is... Yes : the install passed the "[Get local VM >>>> IP]" and continued !! >>>> >>>> I hit ctrl-c, restart the bond0, restart deploy : it crashed. So it >>>> seems that more than one network is the problem. But ! How do I install >>>> engine on gluster on a separate - bonding - jumbo network in this case ??? >>>> >>>> Can you reproduce this on your side ? >>> >>> >>> Can you please attach the output of 'ip a' in both the case? >>> >>> >>>> >>>> Frank >>>> >>>> >>>> >>>> >>>> Le Lundi, Juin 25, 2018 16:50 CEST, "fsoyer" <fsoyer(a)systea.fr&gt; a >>>> écrit: >>>> >>>> >>>> >>>> >>>> Hi staff, >>>> Installing a fresh ovirt - CentOS 7.5.1804 up to date, ovirt version : >>>> # rpm -qa | grep ovirt >>>> ovirt-hosted-engine-ha-2.2.11-1.el7.centos.noarch >>>> ovirt-imageio-common-1.3.1.2-0.el7.centos.noarch >>>> ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64 >>>> ovirt-vmconsole-1.0.5-4.el7.centos.noarch >>>> ovirt-provider-ovn-driver-1.2.10-1.el7.centos.noarch >>>> ovirt-hosted-engine-setup-2.2.20-1.el7.centos.noarch >>>> ovirt-engine-appliance-4.2-20180504.1.el7.centos.noarch >>>> python-ovirt-engine-sdk4-4.2.6-2.el7.centos.x86_64 >>>> ovirt-host-deploy-1.7.3-1.el7.centos.noarch >>>> ovirt-release42-4.2.3.1-1.el7.noarch >>>> ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch >>>> cockpit-ovirt-dashboard-0.11.24-1.el7.centos.noarch >>>> ovirt-setup-lib-1.1.4-1.el7.centos.noarch >>>> ovirt-imageio-daemon-1.3.1.2-0.el7.centos.noarch >>>> ovirt-host-4.2.2-2.el7.centos.x86_64 >>>> ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch >>>> >>>> ON PHYSICAL SERVERS (not on VMware, why should I be ?? ;) I got >>>> exactly the same error : >>>> [ INFO ] TASK [Get local VM IP] >>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:69:3a:c6 >>>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.073313", "end": >>>> "2018-06-25 16:11:36.025277", "rc": 0, "start": "2018-06-25 >>>> 16:11:35.951964", "stderr": "", "stderr_lines": [], "stdout": "", >>>> "stdout_lines": []} >>>> [ INFO ] TASK [include_tasks] >>>> [ INFO ] ok: [localhost] >>>> [ INFO ] TASK [Remove local vm dir] >>>> [ INFO ] changed: [localhost] >>>> [ INFO ] TASK [Notify the user about a failure] >>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>> "The system may not be provisioned according to the playbook results: >>>> please check the logs for the issue, fix accordingly or re-deploy from >>>> scratch.\n"} >>>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >>>> ansible-playbook >>>> [ INFO ] Stage: Clean up >>>> >>>> >>>> I have 4 NIC : >>>> em1 10.0.0.230/8 is for ovirmgmt, it have the gateway >>>> em2 10.0.0.229/8 is for a vmnetwork >>>> em3+em4 in bond0 192.168.0.30 are for gluster with jumbo frames, >>>> volumes (ENGINE, ISO,EXPORT,DATA) are up and operationals. >>>> >>>> I tried to stop em2 (ONBOOT=No and restart network), so the network is >>>> actually : >>>> # ip a >>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >>>> group default qlen 1000 >>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>>> inet 127.0.0.1/8 scope host lo >>>> valid_lft forever preferred_lft forever >>>> inet6 ::1/128 scope host >>>> valid_lft forever preferred_lft forever >>>> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >>>> group default qlen 1000 >>>> link/ether e0:db:55:15:eb:70 brd ff:ff:ff:ff:ff:ff >>>> inet 10.0.0.230/8 brd 10.255.255.255 scope global em1 >>>> valid_lft forever preferred_lft forever >>>> inet6 fe80::e2db:55ff:fe15:eb70/64 scope link >>>> valid_lft forever preferred_lft forever >>>> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>>> default qlen 1000 >>>> link/ether e0:db:55:15:eb:71 brd ff:ff:ff:ff:ff:ff >>>> 4: em3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq >>>> master bond0 state UP group default qlen 1000 >>>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>>> 5: em4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq >>>> master bond0 state UP group default qlen 1000 >>>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>>> 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc >>>> noqueue state UP group default qlen 1000 >>>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>>> inet 192.168.0.30/24 brd 192.168.0.255 scope global bond0 >>>> valid_lft forever preferred_lft forever >>>> inet6 fe80::e2db:55ff:fe15:eb72/64 scope link >>>> valid_lft forever preferred_lft forever >>>> >>>> # ip r >>>> default via 10.0.1.254 dev em1 >>>> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.230 >>>> 169.254.0.0/16 dev em1 scope link metric 1002 >>>> 169.254.0.0/16 dev bond0 scope link metric 1006 >>>> 192.168.0.0/24 dev bond0 proto kernel scope link src 192.168.0.30 >>>> >>>> but same issue, after "/usr/sbin/ovirt-hosted-engine-cleanup" and >>>> restarting the deployment. >>>> >>>> NetworkManager was stopped and disabled at the node install, and it is >>>> still stopped. >>>> After the error, the network shows this after device 6 (bond0) : >>>> 7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >>>> state UP group default qlen 1000 >>>> link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff >>>> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 >>>> valid_lft forever preferred_lft forever >>>> 8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master >>>> virbr0 state DOWN group default qlen 1000 >>>> link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff >>>> 11: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >>>> master virbr0 state UNKNOWN group default qlen 1000 >>>> link/ether fe:16:3e:69:3a:c6 brd ff:ff:ff:ff:ff:ff >>>> inet6 fe80::fc16:3eff:fe69:3ac6/64 scope link >>>> valid_lft forever preferred_lft forever >>>> >>>> I do not see ovirmgmt... And I don't know if I can access the engine >>>> vm as I have not its IP :( >>>> I tried to ping addresses after 192.168.122.1, but no one are >>>> accessible so I stopped at 122.10. The VM seems up (kvm process), qemu-kvm >>>> process taking 150% of cpu in "top"... >>>> >>>> I pasted the log here : https://pastebin.com/Ebzh1uEh >>>> >>>> PLEASE ! This issue seems to be reccurent since the beginning of 2018 >>>> (see messages here on list ! >>>> Jamie Lawrence in February, suporte(a)logicworks.pt in april, >>>> shamilkpm(a)gmail.com and Yaniv Kaul in May, >>>> florentl on june 01...). Can anyone give us a way to solve this ? >>>> -- >>>> >>>> Cordialement, >>>> >>>> *Frank Soyer * >>>> >>>> >>>> Le Lundi, Juin 04, 2018 16:07 CEST, Simone Tiraboschi < >>>> stirabos(a)redhat.com&gt; a écrit: >>>> >>>> >>>> >>>> >>>> On Mon, Jun 4, 2018 at 2:20 PM, Phillip Bailey <phbailey(a)redhat.com&gt; >>>> wrote: >>>>> >>>>> Hi Florent, >>>>> >>>>> Could you please provide the log for the stage in which the wizard is >>>>> failing? Logs can be found in /var/log/ovirt-hosted-engine-setup. >>>>> >>>>> Thanks! >>>>> >>>>> -Phillip Bailey >>>>> >>>>> On Fri, Jun 1, 2018 at 7:57 AM, florentl <florentl.info(a)laposte.net&gt; >>>>> wrote: >>>>>> >>>>>> Hi all, >>>>>> I try to install hosted-engine on node : >>>>>> ovirt-node-ng-4.2.3-0.20180518. >>>>>> Every times I get stuck on : >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>>>>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:5a:91 >>>>>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.108872", "end": >>>>>> "2018-06-01 11:17:34.421769", "rc": 0, "start": "2018-06-01 >>>>>> 11:17:34.312897", "stderr": "", "stderr_lines": [], "stdout": "", >>>>>> "stdout_lines": []} >>>>>> I tried with static IP Address and with DHCP but both failed. >>>>>> >>>>>> To be more specific, I installed three nodes, deployed glusterfs >>>>>> with the wizard. I'm in a nested virtualization environment for this lab >>>>>> (Vmware Esxi Hypervisor). >>>>> >>>>> >>>> Unfortunately I think that the issue is trying to run a nested env >>>> over ESXi. >>>> AFAIk nesting KVM VMs over ESX is still problematic. >>>> >>>> I'd suggest to repeat the experiment nesting over KVM on L0. >>>> >>>> >>>> >>>>> My node IP is : 192.168.176.40 / and I want the hosted-engine vm has >>>>>> 192.168.176.43. >>>>>> >>>>>> Thanks, >>>>>> Florent >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/F3BNUQ2T434... >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/RU34XDM2W6G... >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > >