I did that but no success yet.

I see that "Get local VM IP" task tries the following :
virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | awk '{ print $5 }' | cut -f1 -d'/'

However while the task is running, and vm running in qemu, "virsh -r net-dhcp-leases default" never returns anything :
[root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default
 Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID
-------------------------------------------------------------------------------------------------------------------

[root@vs-inf-int-kvm-fr-301-210 ~]# 



Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
OK, try this:
temporary edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml around line 120
and edit tasks "Get local VM IP"
changing from "retries: 50" to  "retries: 500" so that you have more time to debug it



On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
I retried after killing the remaining qemu process and doing ovirt-hosted-engine-cleanup
The new attempt failed again at the same step. Then after it fails, it cleans the temporary files (and vm disk) but qemu still runs! :

[ INFO  ] TASK [ovirt.hosted_engine_setup : Get local VM IP]

[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
[ INFO  ] Stage: Clean up
[ INFO  ] Cleaning temporary resources
...
[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO  ] ok: [localhost]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.


[root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu
root      4021  0.0  0.0  24844  1788 ?        Ss   févr.22   0:00 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status -F/etc/qemu-ga/fsfreeze-hook
qemu     26463 22.9  4.8 17684512 1088844 ?    Sl   16:01   3:09 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
root     28416  0.0  0.0 112712   980 pts/3    S+   16:14   0:00 grep --color=auto qemu

Before the first Error, while the vm was running for sure and the disk was there, I also unsuccessfuly tried to connect to it with VNC  and got the same error I got before : 

[root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested.
debug1: channel 3: new [direct-tcpip]
channel 3: open failed: connect failed: Connection refused
debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4

Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
Something was definitely wrong ; as indicated, qemu process for guest=HostedEngineLocal was running but the disk file did not exist anymore...
No surprise I could not connect

I am retrying


Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
It fails too :
I made sure PermitTunnel=yes in sshd config but when I try to connect to the forwarded port I get the following error on the openened host ssh session :
...
[root@vs-inf-int-kvm-fr-301-210 ~]#
debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 port 42144 to ::1 port 5900, nchannels 4
debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested.
debug1: channel 3: new [direct-tcpip]
channel 3: open failed: connect failed: Connection refused
debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4

and in journalctl : 
févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: failed.


Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi <stirabos@redhat.com> wrote:



On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
I made sure of everything and even stopped firewalld but still can't connect :

[root@vs-inf-int-kvm-fr-301-210 ~]# cat /var/run/libvirt/qemu/HostedEngineLocal.xml
     <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
        <listen type='address' address='127.0.0.1' fromConfig='1' autoGenerated='no'/>

[root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN      13376/qemu-kvm


I suggest to try ssh tunneling, run
ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 root@vs-inf-int-kvm-fr-301-210.hostics.fr

and then
remote-viewer vnc://localhost:5900
 
 

[root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Stopped firewalld - dynamic firewall daemon.

From my laptop :
[gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr 5900
Trying 10.199.210.11...
[nothing gets through...]
^C

For making sure :
[gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr 9090
Trying 10.199.210.11...
Escape character is '^]'.





Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth@redhat.com> wrote:
Hey!

You can check under /var/run/libvirt/qemu/HostedEngine.xml
Search for 'vnc'
From there you can look up the port on which the HE VM is available and connect to the same.


On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
1) I am running in a Nested env, but under libvirt/kvm on remote Centos 7.4 Hosts

Please advise how to connect with VNC to the local HE vm. I see it's running, but this is on a remote host, not my local machine :
qemu     13376  100  3.7 17679424 845216 ?     Sl   12:46  85:08 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on


2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat /etc/libvirt/qemu/networks/default.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit default
or other application using the libvirt API.
-->

<network>
  <name>default</name>
  <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:e5:fe:3b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>
You have new mail in /var/spool/mail/root
[root@vs-inf-int-kvm-fr-301-210 ~]



Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group


On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi <stirabos@redhat.com> wrote:


On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese <guillaume.pavese@interactiv-group.com> wrote:
He deployment with "hosted-engine --deploy" fails at TASK [ovirt.hosted_engine_setup : Get local VM IP]

See following Error :

2019-02-25 12:46:50,154+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get local VM IP]
2019-02-25 12:55:26,823+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {u'_ansible_parsed': True, u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00
:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': True, u'executable':
None, u'_uses_shell': True, u'_raw_params': u"virsh -r net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'std
in': None}}, u'start': u'2019-02-25 12:55:26.584686', u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', u'stdout_lines': []}
2019-02-25 12:55:26,924+0100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:107 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Here we are just waiting for the bootstrap engine VM to fetch an IP address from default libvirt network over DHCP but it your case it never happened.
Possible issues: something went wrong in the bootstrap process for the engine VM or the default libvirt network is not correctly configured.

1. can you try to reach the engine VM via VNC and check what's happening there? (another question, are you running it nested? AFAIK it will not work if nested over ESXi)
2. can you please share the output of 
cat /etc/libvirt/qemu/networks/default.xml
 

Guillaume Pavese
Ingénieur Système et Réseau
Interactiv-Group
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTMB2YYNMOMD7I5NX7RZQ2IW/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV7YVE6GLJWEV2AV22TIGXK/