[ovirt-users] Re: Install hosted-engine - Task Get local VM IP failed

Hi again, In fact, the hour in file is exactly 2hours before, I guess a timezone problem (in the process of install ?), as the file itself is correctly timed at 11:17am (correct hour here in France). So the messages are synchrone.

-------- Message original -------- Objet : Re: [ovirt-users] Re: Install hosted-engine - Task Get local VM IP failed De : Simone Tiraboschi À : fsoyer(a)systea.fr Cc : users Hi, HostedEngineLocal was started at 2018-06-26 09:17:26 but /var/log/messages starts only at Jun 26 11:02:32. Can you please reattach it fro the relevant time frame? On Wed, Jun 27, 2018 at 10:54 AM fsoyer <fsoyer(a)systea.fr&gt; wrote: > Hi Simone, > here are the revelant part of messages and the engine install log (there > were only this file in /var/log/libvirt/qemu) . > > Thanks for your time. > > Frank > > > Le Mardi, Juin 26, 2018 11:43 CEST, Simone Tiraboschi < > stirabos(a)redhat.com&gt; a écrit: > > > > On Tue, Jun 26, 2018 at 11:39 AM fsoyer <fsoyer(a)systea.fr&gt; wrote: > >> Well, >> unfortunatly, it was a "false-positive". This morning I tried again, >> with the idea that at one moment the deploy will ask for the final >> destination for the engine, I will restart bond0+gluster+volume engine at >> thos moment. >> Re-launching the deploy on the second "fresh" host (the first one with >> all errors yesterday let it in a doutful state) with em2 and gluster+bond0 >> off : >> >> # ip a >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> group default qlen 1000 >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >> inet 127.0.0.1/8 scope host lo >> valid_lft forever preferred_lft forever >> inet6 ::1/128 scope host >> valid_lft forever preferred_lft forever >> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >> group default qlen 1000 >> link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff >> inet 10.0.0.227/8 brd 10.255.255.255 scope global em1 >> valid_lft forever preferred_lft forever >> inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link >> valid_lft forever preferred_lft forever >> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff >> 4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff >> 5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff >> 6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state DOWN >> group default qlen 1000 >> link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff >> >> # ip r >> default via 10.0.1.254 dev em1 >> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227 >> 169.254.0.0/16 dev em1 scope link metric 1002 >> >> ... does NOT work this morning >> >> [ INFO ] TASK [Get local VM IP] >> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:01:c6:32 >> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.083587", "end": >> "2018-06-26 11:26:07.581706", "rc": 0, "start": "2018-06-26 >> 11:26:07.498119", "stderr": "", "stderr_lines": [], "stdout": "", >> "stdout_lines": []} >> >> I'm sure that the network was the same yesterday when my attempt finally >> pass the "get local vm ip". Why not today ? >> After the error, the network was : >> >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> group default qlen 1000 >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >> inet 127.0.0.1/8 scope host lo >> valid_lft forever preferred_lft forever >> inet6 ::1/128 scope host >> valid_lft forever preferred_lft forever >> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >> group default qlen 1000 >> link/ether e0:db:55:15:f0:f0 brd ff:ff:ff:ff:ff:ff >> inet 10.0.0.227/8 brd 10.255.255.255 scope global em1 >> valid_lft forever preferred_lft forever >> inet6 fe80::e2db:55ff:fe15:f0f0/64 scope link >> valid_lft forever preferred_lft forever >> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f1 brd ff:ff:ff:ff:ff:ff >> 4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f2 brd ff:ff:ff:ff:ff:ff >> 5: em4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default >> qlen 1000 >> link/ether e0:db:55:15:f0:f3 brd ff:ff:ff:ff:ff:ff >> 6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 9000 qdisc noqueue state DOWN >> group default qlen 1000 >> link/ether 3a:ab:a2:f2:38:5c brd ff:ff:ff:ff:ff:ff >> 7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >> state UP group default qlen 1000 >> link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff >> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 >> valid_lft forever preferred_lft forever >> 8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master >> virbr0 state DOWN group default qlen 1000 >> link/ether 52:54:00:ae:8d:93 brd ff:ff:ff:ff:ff:ff >> 9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >> master virbr0 state UNKNOWN group default qlen 1000 >> link/ether fe:16:3e:01:c6:32 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc16:3eff:fe01:c632/64 scope link >> valid_lft forever preferred_lft forever >> >> # ip r >> default via 10.0.1.254 dev em1 >> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.227 >> 169.254.0.0/16 dev em1 scope link metric 1002 >> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 >> >> >> >> So, finally, I have no idea why this appends :((( > > > Can you please attach /var/log/messages and /var/log/libvirt/qemu/* ? > > > >> >> >> Le Mardi, Juin 26, 2018 09:21 CEST, Simone Tiraboschi < >> stirabos(a)redhat.com&gt; a écrit: >> >> >> >> On Mon, Jun 25, 2018 at 6:32 PM fsoyer <fsoyer(a)systea.fr&gt; wrote: >> >>> Well, answering to myself for more informations. >>> Thinking that the network was part of the problem, I tried to stop >>> gluster volumes, stop gluster on host, and stop bond0. >>> So, the host now had just em1 with one IP. >>> And... The winner is... Yes : the install passed the "[Get local VM >>> IP]" and continued !! >>> >>> I hit ctrl-c, restart the bond0, restart deploy : it crashed. So it >>> seems that more than one network is the problem. But ! How do I install >>> engine on gluster on a separate - bonding - jumbo network in this case ??? >>> >>> Can you reproduce this on your side ? >> >> >> Can you please attach the output of 'ip a' in both the case? >> >> >>> >>> Frank >>> >>> >>> >>> >>> Le Lundi, Juin 25, 2018 16:50 CEST, "fsoyer" <fsoyer(a)systea.fr&gt; a >>> écrit: >>> >>> >>> >>> >>> Hi staff, >>> Installing a fresh ovirt - CentOS 7.5.1804 up to date, ovirt version : >>> # rpm -qa | grep ovirt >>> ovirt-hosted-engine-ha-2.2.11-1.el7.centos.noarch >>> ovirt-imageio-common-1.3.1.2-0.el7.centos.noarch >>> ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64 >>> ovirt-vmconsole-1.0.5-4.el7.centos.noarch >>> ovirt-provider-ovn-driver-1.2.10-1.el7.centos.noarch >>> ovirt-hosted-engine-setup-2.2.20-1.el7.centos.noarch >>> ovirt-engine-appliance-4.2-20180504.1.el7.centos.noarch >>> python-ovirt-engine-sdk4-4.2.6-2.el7.centos.x86_64 >>> ovirt-host-deploy-1.7.3-1.el7.centos.noarch >>> ovirt-release42-4.2.3.1-1.el7.noarch >>> ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch >>> cockpit-ovirt-dashboard-0.11.24-1.el7.centos.noarch >>> ovirt-setup-lib-1.1.4-1.el7.centos.noarch >>> ovirt-imageio-daemon-1.3.1.2-0.el7.centos.noarch >>> ovirt-host-4.2.2-2.el7.centos.x86_64 >>> ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch >>> >>> ON PHYSICAL SERVERS (not on VMware, why should I be ?? ;) I got exactly >>> the same error : >>> [ INFO ] TASK [Get local VM IP] >>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:69:3a:c6 >>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.073313", "end": >>> "2018-06-25 16:11:36.025277", "rc": 0, "start": "2018-06-25 >>> 16:11:35.951964", "stderr": "", "stderr_lines": [], "stdout": "", >>> "stdout_lines": []} >>> [ INFO ] TASK [include_tasks] >>> [ INFO ] ok: [localhost] >>> [ INFO ] TASK [Remove local vm dir] >>> [ INFO ] changed: [localhost] >>> [ INFO ] TASK [Notify the user about a failure] >>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The >>> system may not be provisioned according to the playbook results: please >>> check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} >>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >>> ansible-playbook >>> [ INFO ] Stage: Clean up >>> >>> >>> I have 4 NIC : >>> em1 10.0.0.230/8 is for ovirmgmt, it have the gateway >>> em2 10.0.0.229/8 is for a vmnetwork >>> em3+em4 in bond0 192.168.0.30 are for gluster with jumbo frames, >>> volumes (ENGINE, ISO,EXPORT,DATA) are up and operationals. >>> >>> I tried to stop em2 (ONBOOT=No and restart network), so the network is >>> actually : >>> # ip a >>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >>> group default qlen 1000 >>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >>> inet 127.0.0.1/8 scope host lo >>> valid_lft forever preferred_lft forever >>> inet6 ::1/128 scope host >>> valid_lft forever preferred_lft forever >>> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP >>> group default qlen 1000 >>> link/ether e0:db:55:15:eb:70 brd ff:ff:ff:ff:ff:ff >>> inet 10.0.0.230/8 brd 10.255.255.255 scope global em1 >>> valid_lft forever preferred_lft forever >>> inet6 fe80::e2db:55ff:fe15:eb70/64 scope link >>> valid_lft forever preferred_lft forever >>> 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group >>> default qlen 1000 >>> link/ether e0:db:55:15:eb:71 brd ff:ff:ff:ff:ff:ff >>> 4: em3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq >>> master bond0 state UP group default qlen 1000 >>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>> 5: em4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq >>> master bond0 state UP group default qlen 1000 >>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>> 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc >>> noqueue state UP group default qlen 1000 >>> link/ether e0:db:55:15:eb:72 brd ff:ff:ff:ff:ff:ff >>> inet 192.168.0.30/24 brd 192.168.0.255 scope global bond0 >>> valid_lft forever preferred_lft forever >>> inet6 fe80::e2db:55ff:fe15:eb72/64 scope link >>> valid_lft forever preferred_lft forever >>> >>> # ip r >>> default via 10.0.1.254 dev em1 >>> 10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.230 >>> 169.254.0.0/16 dev em1 scope link metric 1002 >>> 169.254.0.0/16 dev bond0 scope link metric 1006 >>> 192.168.0.0/24 dev bond0 proto kernel scope link src 192.168.0.30 >>> >>> but same issue, after "/usr/sbin/ovirt-hosted-engine-cleanup" and >>> restarting the deployment. >>> >>> NetworkManager was stopped and disabled at the node install, and it is >>> still stopped. >>> After the error, the network shows this after device 6 (bond0) : >>> 7: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >>> state UP group default qlen 1000 >>> link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff >>> inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 >>> valid_lft forever preferred_lft forever >>> 8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master >>> virbr0 state DOWN group default qlen 1000 >>> link/ether 52:54:00:38:e0:5a brd ff:ff:ff:ff:ff:ff >>> 11: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast >>> master virbr0 state UNKNOWN group default qlen 1000 >>> link/ether fe:16:3e:69:3a:c6 brd ff:ff:ff:ff:ff:ff >>> inet6 fe80::fc16:3eff:fe69:3ac6/64 scope link >>> valid_lft forever preferred_lft forever >>> >>> I do not see ovirmgmt... And I don't know if I can access the engine vm >>> as I have not its IP :( >>> I tried to ping addresses after 192.168.122.1, but no one are >>> accessible so I stopped at 122.10. The VM seems up (kvm process), qemu-kvm >>> process taking 150% of cpu in "top"... >>> >>> I pasted the log here : https://pastebin.com/Ebzh1uEh >>> >>> PLEASE ! This issue seems to be reccurent since the beginning of 2018 >>> (see messages here on list ! >>> Jamie Lawrence in February, suporte(a)logicworks.pt in april, >>> shamilkpm(a)gmail.com and Yaniv Kaul in May, >>> florentl on june 01...). Can anyone give us a way to solve this ? >>> -- >>> >>> Cordialement, >>> >>> *Frank Soyer * >>> >>> >>> Le Lundi, Juin 04, 2018 16:07 CEST, Simone Tiraboschi < >>> stirabos(a)redhat.com&gt; a écrit: >>> >>> >>> >>> >>> On Mon, Jun 4, 2018 at 2:20 PM, Phillip Bailey <phbailey(a)redhat.com&gt; >>> wrote: >>>> >>>> Hi Florent, >>>> >>>> Could you please provide the log for the stage in which the wizard is >>>> failing? Logs can be found in /var/log/ovirt-hosted-engine-setup. >>>> >>>> Thanks! >>>> >>>> -Phillip Bailey >>>> >>>> On Fri, Jun 1, 2018 at 7:57 AM, florentl <florentl.info(a)laposte.net&gt; >>>> wrote: >>>>> >>>>> Hi all, >>>>> I try to install hosted-engine on node : >>>>> ovirt-node-ng-4.2.3-0.20180518. >>>>> Every times I get stuck on : >>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>>>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:5a:91 >>>>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.108872", "end": >>>>> "2018-06-01 11:17:34.421769", "rc": 0, "start": "2018-06-01 >>>>> 11:17:34.312897", "stderr": "", "stderr_lines": [], "stdout": "", >>>>> "stdout_lines": []} >>>>> I tried with static IP Address and with DHCP but both failed. >>>>> >>>>> To be more specific, I installed three nodes, deployed glusterfs with >>>>> the wizard. I'm in a nested virtualization environment for this lab (Vmware >>>>> Esxi Hypervisor). >>>> >>>> >>> Unfortunately I think that the issue is trying to run a nested env over >>> ESXi. >>> AFAIk nesting KVM VMs over ESX is still problematic. >>> >>> I'd suggest to repeat the experiment nesting over KVM on L0. >>> >>> >>> >>>> My node IP is : 192.168.176.40 / and I want the hosted-engine vm has >>>>> 192.168.176.43. >>>>> >>>>> Thanks, >>>>> Florent >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/F3BNUQ2T434... >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/RU34XDM2W6G... >>>> >>> >>> >>> >>> >> >> >> > > > >