Failed to deploy ovirt engine with "hosted-engine --deploy"
Hi: I'm new to ovirt. I followed the guide at the page https://www.ovirt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_E..., started to run "hosted-engine --deploy". It got stuck at the stage TASK [Get local VM IP] for about 5 minutes, and failed. Did a search but came out with nothing. From the log files these 2 lines are shown: 2018-07-25 19:59:11,269+0800 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2018-07-25 19:59:11.081420', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'start': u'2018-07-25 19:59:10.949464', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.131956', u'stdout_lines': []} 2018-07-25 19:59:11,370+0800 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.131956", "end": "2018-07-25 19:59:11.081420", "rc": 0, "start": "2018-07-25 19:59:10.949464", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Can someone please enlighten me on what has happened? Many thanks in advance. regards, Bong SF
On Wed, 2018-07-25 at 12:05 +0000, Bong Shau Fui wrote:
Hi: I'm new to ovirt. I followed the guide at the page https://www.ovi rt.org/documentation/self-hosted/chap-Deploying_Self-Hosted_Engine/, started to run "hosted-engine --deploy". It got stuck at the stage TASK [Get local VM IP] for about 5 minutes, and failed. Did a search but came out with nothing. From the log files these 2 lines are shown:
2018-07-25 19:59:11,269+0800 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:94 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2018-07-25 19:59:11.081420', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'start': u'2018-07-25 19:59:10.949464', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.131956', u'stdout_lines': []} 2018-07-25 19:59:11,370+0800 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:98 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:5e:43:32 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.131956", "end": "2018-07-25 19:59:11.081420", "rc": 0, "start": "2018-07-25 19:59:10.949464", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
Can someone please enlighten me on what has happened? Many thanks in advance.
That is exactly the same thing that happened to me today! Could you try deploying again and in another terminal run this[*] after it's printed "Get local VM IP"? What happened to me was that the VM never started because it had kernel panicked on boot, rather fast too, so you'll need to be quick in order to catch it! [*]: virsh console HostedEngineLocal PS. The console will then hang if we're experiencing the same issue. /K
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/communit y-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/WRYIJNH2C6FJSEYBWUYKBZGQVTVNPNAJ/
I tried redeploying the ovirt-engine and started the console for the HostedEngineLocal vm right after the task [ INFO ] TASK [Create local VM] [ INFO ] changed: [localhost] But my console is blank. How did you manage to catch the kernel panic message? But as you said, the domain is listed as 'running', with a hung console. regards, Bong SF
On Thu, Jul 26, 2018 at 3:21 AM Bong Shau Fui <bongsf@gmail.com> wrote:
I tried redeploying the ovirt-engine and started the console for the HostedEngineLocal vm right after the task
[ INFO ] TASK [Create local VM] [ INFO ] changed: [localhost]
But my console is blank. How did you manage to catch the kernel panic message? But as you said, the domain is listed as 'running', with a hung console.
Are you trying to deploy on a nested environment as well? on which hypervisor on L0?
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/M76WBC3TF7MB24...
Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5. regards, Bong SF
On Mon, Jul 30, 2018 at 6:04 AM Bong Shau Fui <bongsf@gmail.com> wrote:
Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5.
I know for sue that a nested kvm env over vmware esxi is still problematic; kvm over kvm works fine instead.
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQZHWNK2CPSWGW...
Hi Simone: Ok, noted on the config part. Is there any bug report that points to KVM on ESXi problem? I'm just curious. Thanks. regards, Bong SF
Hi Simone, sorry for the late comment (I just found this thread while researching on nested oVirt-on-VMware issues). It seems that the problem is generally known and has been solved upstream in kernel 4.16: https://bugs.launchpad.net/qemu/+bug/1636217 Any ETA on backports in RHEL7? ;-) (Note: should I open a Bugzilla on kernel package for this?) I seems that Hyper-V would experience the same troubles and I think that making nested-inside-Windows-host work should be deemed fairly important for demo/trial/lab uses... :-) In the meanwhile, workarounds have been published for the new Ansible-based setup flow: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/UDVRS5I64WLFJJF... on the other side, I can confirm that with the legacy setup flow (with option --noansible) adding the following to the answer file avoids the problem (did not test whether newer types like pc-i440fx-rhel7.2.0 solve it too): OVEHOSTED_VM/emulatedMschine=str:rhel6.0.0 Many thanks. Best regards, Giuseppe ________________________________________ Da: Simone Tiraboschi <stirabos@redhat.com> Inviato: lunedì 30 luglio 2018 09:20 A: bongsf@gmail.com Cc: users Oggetto: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy" On Mon, Jul 30, 2018 at 6:04 AM Bong Shau Fui <bongsf@gmail.com<mailto:bongsf@gmail.com>> wrote: Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5. I know for sue that a nested kvm env over vmware esxi is still problematic; kvm over kvm works fine instead. regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQZHWNK2CPSWGW... </mailto:users-leave@ovirt.org></mailto:users@ovirt.org></bongsf@gmail.com<mailto:bongsf@gmail.com></stirabos@redhat.com>
Hi all, I confirm that: 1) the machine type pc-i440fx-rhel7.2.0 is enough to solve the problem 2) the OVEHOSTED_VM/emulatedMachine variable in answers file is NOT enough (the engine vm hangs again as soon as it gets rebooted, after the successful initial setup), but the VDSM hook in https://gist.github.com/RabidCicada/40655db1582ca5d07c9bbf2c429cdd01 solves the problem (for further vms too, arguably) Many thanks. Best regards, Giuseppe ________________________________________ Da: Giuseppe Ragusa <giuseppe.ragusa@hotmail.com> Inviato: venerdì 7 settembre 2018 16:04 A: Simone Tiraboschi; bongsf@gmail.com Cc: users Oggetto: R: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy" Hi Simone, sorry for the late comment (I just found this thread while researching on nested oVirt-on-VMware issues). It seems that the problem is generally known and has been solved upstream in kernel 4.16: https://bugs.launchpad.net/qemu/+bug/1636217 Any ETA on backports in RHEL7? ;-) (Note: should I open a Bugzilla on kernel package for this?) I seems that Hyper-V would experience the same troubles and I think that making nested-inside-Windows-host work should be deemed fairly important for demo/trial/lab uses... :-) In the meanwhile, workarounds have been published for the new Ansible-based setup flow: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/UDVRS5I64WLFJJF... on the other side, I can confirm that with the legacy setup flow (with option --noansible) adding the following to the answer file avoids the problem (did not test whether newer types like pc-i440fx-rhel7.2.0 solve it too): OVEHOSTED_VM/emulatedMschine=str:rhel6.0.0 Many thanks. Best regards, Giuseppe ________________________________________ Da: Simone Tiraboschi <stirabos@redhat.com> Inviato: lunedì 30 luglio 2018 09:20 A: bongsf@gmail.com Cc: users Oggetto: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy" On Mon, Jul 30, 2018 at 6:04 AM Bong Shau Fui <bongsf@gmail.com<mailto:bongsf@gmail.com>> wrote: Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5. I know for sue that a nested kvm env over vmware esxi is still problematic; kvm over kvm works fine instead. regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQZHWNK2CPSWGW... </mailto:users-leave@ovirt.org></mailto:users@ovirt.org></bongsf@gmail.com<mailto:bongsf@gmail.com></stirabos@redhat.com> </giuseppe.ragusa@hotmail.com>
Hey there, I posted solutions here earlier: https://mail.google.com/mail/u/0/?tab=wm#search/in%3Asent+ovirt+users/LXphbR... Thanks for the gist reference giuseppe. I haven't updated the approach in a while...but should absolutely explain the reasoning and I think should still just work:). On Sat, Sep 8, 2018 at 11:31 AM Giuseppe Ragusa <giuseppe.ragusa@hotmail.com> wrote:
Hi all, I confirm that:
1) the machine type pc-i440fx-rhel7.2.0 is enough to solve the problem
2) the OVEHOSTED_VM/emulatedMachine variable in answers file is NOT enough (the engine vm hangs again as soon as it gets rebooted, after the successful initial setup), but the VDSM hook in https://gist.github.com/RabidCicada/40655db1582ca5d07c9bbf2c429cdd01 solves the problem (for further vms too, arguably)
Many thanks.
Best regards, Giuseppe ________________________________________ Da: Giuseppe Ragusa <giuseppe.ragusa@hotmail.com> Inviato: venerdì 7 settembre 2018 16:04 A: Simone Tiraboschi; bongsf@gmail.com Cc: users Oggetto: R: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy"
Hi Simone, sorry for the late comment (I just found this thread while researching on nested oVirt-on-VMware issues).
It seems that the problem is generally known and has been solved upstream in kernel 4.16:
https://bugs.launchpad.net/qemu/+bug/1636217
Any ETA on backports in RHEL7? ;-) (Note: should I open a Bugzilla on kernel package for this?) I seems that Hyper-V would experience the same troubles and I think that making nested-inside-Windows-host work should be deemed fairly important for demo/trial/lab uses... :-)
In the meanwhile, workarounds have been published for the new Ansible-based setup flow:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/UDVRS5I64WLFJJF...
on the other side, I can confirm that with the legacy setup flow (with option --noansible) adding the following to the answer file avoids the problem (did not test whether newer types like pc-i440fx-rhel7.2.0 solve it too):
OVEHOSTED_VM/emulatedMschine=str:rhel6.0.0
Many thanks.
Best regards, Giuseppe
________________________________________ Da: Simone Tiraboschi <stirabos@redhat.com> Inviato: lunedì 30 luglio 2018 09:20 A: bongsf@gmail.com Cc: users Oggetto: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy"
On Mon, Jul 30, 2018 at 6:04 AM Bong Shau Fui <bongsf@gmail.com<mailto: bongsf@gmail.com>> wrote: Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5.
I know for sue that a nested kvm env over vmware esxi is still problematic; kvm over kvm works fine instead.
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto: users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQZHWNK2CPSWGW... </mailto:users-leave@ovirt.org></mailto:users@ovirt.org></bongsf@gmail.com <mailto:bongsf@gmail.com></stirabos@redhat.com> </giuseppe.ragusa@hotmail.com> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/W4GOU6N43OGPPD...
On Fri, Sep 7, 2018 at 4:04 PM Giuseppe Ragusa <giuseppe.ragusa@hotmail.com> wrote:
Hi Simone, sorry for the late comment (I just found this thread while researching on nested oVirt-on-VMware issues).
It seems that the problem is generally known and has been solved upstream in kernel 4.16:
https://bugs.launchpad.net/qemu/+bug/1636217
Any ETA on backports in RHEL7? ;-) (Note: should I open a Bugzilla on kernel package for this?)
Ciao Giuseppe, Thanks for the accurate report. Honestly I'm not aware of backport ETA but I'd suggest to file a bug on product: Red Hat Enterprise Linux 7, component kernel, subcomponent KVM pointing to the upstream patch. Not sure if we are still on time for RHEL 7.6.
I seems that Hyper-V would experience the same troubles and I think that making nested-inside-Windows-host work should be deemed fairly important for demo/trial/lab uses... :-)
In the meanwhile, workarounds have been published for the new Ansible-based setup flow:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/UDVRS5I64WLFJJF...
on the other side, I can confirm that with the legacy setup flow (with option --noansible) adding the following to the answer file avoids the problem (did not test whether newer types like pc-i440fx-rhel7.2.0 solve it too):
OVEHOSTED_VM/emulatedMschine=str:rhel6.0.0
Many thanks.
Best regards, Giuseppe
________________________________________ Da: Simone Tiraboschi <stirabos@redhat.com> Inviato: lunedì 30 luglio 2018 09:20 A: bongsf@gmail.com Cc: users Oggetto: [ovirt-users] Re: Failed to deploy ovirt engine with "hosted-engine --deploy"
On Mon, Jul 30, 2018 at 6:04 AM Bong Shau Fui <bongsf@gmail.com<mailto: bongsf@gmail.com>> wrote: Hi Simone: Yes, it's in a nested environment. L0 is vmware esxi 5.5.
I know for sue that a nested kvm env over vmware esxi is still problematic; kvm over kvm works fine instead.
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto: users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQZHWNK2CPSWGW... </mailto:users-leave@ovirt.org></mailto:users@ovirt.org></bongsf@gmail.com <mailto:bongsf@gmail.com></stirabos@redhat.com>
Hi guys: For some reasons I stopped with this experimentation and only to pick it up again these few days. I took your advice up there to try it with KVM on KVM. Guess what? It failed at the same stage! I'm using CentOS 7.5 + yum update, so I suppose everything is the latest release However, if I were to use cockpit to do the deployment, it'll fail me at the last stage, that is to bring the VM into the NFS (I'm using an NFS, again presented from the KVM host). I don't know whether this is due to the nested virtualisation issue. My colleague who's never done this before managed to do it with ease on physical machines + FC SAN. regards, Bong SF
Hi: Some more updates. While running the hosted-engine --deploy, it seems like the VM is created, and still get stuck at the line [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] I opened up another terminal and run brctl show bridge name bridge id STP enabled interfaces ;vdsmdummy; 8000.000000000000 no ovirtmgmt 8000.525400da358f no eth0 virbr0 8000.5254002d053d yes virbr0-nic vnet0 Looks like the engine vm's network interface is connected to the default NAT network bridge of virbr0! When selecting the interface to bridge ovirtmgmt I specifically choose 'eth0'. I'm very sure I didn't choose anything but eth0 as I've been trying for many times already. But why vnet0 is bridged to virbr0? regards, Bong SF
On Thu, Mar 7, 2019 at 2:10 PM Bong Shau Fui <bongsf@gmail.com> wrote:
Hi: Some more updates. While running the hosted-engine --deploy, it seems like the VM is created, and still get stuck at the line
[ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP]
I opened up another terminal and run brctl show
bridge name bridge id STP enabled interfaces ;vdsmdummy; 8000.000000000000 no ovirtmgmt 8000.525400da358f no eth0 virbr0 8000.5254002d053d yes virbr0-nic vnet0
Looks like the engine vm's network interface is connected to the default NAT network bridge of virbr0! When selecting the interface to bridge ovirtmgmt I specifically choose 'eth0'. I'm very sure I didn't choose anything but eth0 as I've been trying for many times already. But why vnet0 is bridged to virbr0?
This is fine: the bootstrap engine VM is started over libvirt default network then we will use that engine to configure your system.
regards, Bong SF _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XRL6V6XHHKW3G6...
Hi Guys: Good news. Finally I was able to deploy my ovirt engine under nested kvm virtualisation. What I did was I spin up a fresh vm, added the ovirt repo and use cockpit to deploy instead of command-line. I know it sounded silly, but I believe my previous installation failed due to I took out the default eth0 from Network Manager's control. Now I'm trying to add in another ovirt host through the ovirt engine GUI. Thanks. regards, Bong SF
participants (5)
-
Bong Shau Fui -
Giuseppe Ragusa -
Karli Sjöberg -
RabidCicada -
Simone Tiraboschi