It fails too : I made sure PermitTunnel=yes in sshd config but when I try to connect to the forwarded port I get the following error on the openened host ssh session : [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: vs-inf-int-kvm-fr-301-210.hostics.fr:5900 root(a)vs-inf-int-kvm-fr-301-210.hostics.fr ... [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 port 42144 to ::1 port 5900, nchannels 4 debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. debug1: channel 3: new [direct-tcpip] channel 3: open failed: connect failed: Connection refused debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4 and in journalctl : févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: failed. Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > > On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> I made sure of everything and even stopped firewalld but still can't >> connect : >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >> /var/run/libvirt/qemu/HostedEngineLocal.xml >> <graphics type='vnc' port='*5900*' autoport='yes' >> *listen='127.0.0.1*'> >> <listen type='address' address='*127.0.0.1*' fromConfig='1' >> autoGenerated='no'/> >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >> LISTEN 13376/qemu-kvm >> > > > I suggest to try ssh tunneling, run > ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 > root(a)vs-inf-int-kvm-fr-301-210.hostics.fr > > and then > remote-viewer vnc://localhost:5900 > > > >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status firewalld.service >> ● firewalld.service - firewalld - dynamic firewall daemon >> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; >> vendor preset: enabled) >> *Active: inactive (dead)* >> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >> - dynamic firewall daemon.* >> >> From my laptop : >> [gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr >> *5900* >> Trying 10.199.210.11... >> [*nothing gets through...*] >> ^C >> >> For making sure : >> [gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr >> *9090* >> Trying 10.199.210.11... >> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >> Escape character is '^]'. >> >> >> >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth(a)redhat.com&gt; >> wrote: >> >>> Hey! >>> >>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>> Search for 'vnc' >>> From there you can look up the port on which the HE VM is available and >>> connect to the same. >>> >>> >>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> 1) I am running in a Nested env, but under libvirt/kvm on remote >>>> Centos 7.4 Hosts >>>> >>>> Please advise how to connect with VNC to the local HE vm. I see it's >>>> running, but this is on a remote host, not my local machine : >>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 85:08 >>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>> -object >>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>> -no-user-config -nodefaults -chardev >>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>> menu=off,strict=on -device >>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>> -device >>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> -drive >>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>> -chardev pty,id=charserial0 -device >>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>> socket,id=charchannel0,fd=31,server,nowait -device >>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>> -msg timestamp=on >>>> >>>> >>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>> /etc/libvirt/qemu/networks/default.xml >>>> <!-- >>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE >>>> OVERWRITTEN AND LOST. Changes to this xml configuration should be made >>>> using: >>>> virsh net-edit default >>>> or other application using the libvirt API. >>>> --> >>>> >>>> <network> >>>> <name>default</name> >>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>> <forward mode='nat'/> >>>> <bridge name='virbr0' stp='on' delay='0'/> >>>> <mac address='52:54:00:e5:fe:3b'/> >>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>> <dhcp> >>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>> </dhcp> >>>> </ip> >>>> </network> >>>> You have new mail in /var/spool/mail/root >>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>> >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>> >>>>>> See following Error : >>>>>> >>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>> local VM IP] >>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': >>>>>> u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', >>>>>> u'changed': True, u'invocation': {u'module_args': {u'warn': True, >>>>>> u'executable': >>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>> u'chdir': None, u'std >>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', u'attempts': >>>>>> 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', u'stdout_lines': >>>>>> []} >>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>> "", "stdout_lines": []} >>>>>> >>>>> >>>>> Here we are just waiting for the bootstrap engine VM to fetch an IP >>>>> address from default libvirt network over DHCP but it your case it never >>>>> happened. >>>>> Possible issues: something went wrong in the bootstrap process for >>>>> the engine VM or the default libvirt network is not correctly configured. >>>>> >>>>> 1. can you try to reach the engine VM via VNC and check what's >>>>> happening there? (another question, are you running it nested? AFAIK it >>>>> will not work if nested over ESXi) >>>>> 2. can you please share the output of >>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>> >>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>> >>>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>> >>>

I retried after killing the remaining qemu process and doing ovirt-hosted-engine-cleanup The new attempt failed again at the same step. Then after it fails, it cleans the temporary files (and vm disk) but *qemu still runs!* : [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources ... [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] ok: [localhost] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. [root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu root 4021 0.0 0.0 24844 1788 ? Ss févr.22 0:00 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status -F/etc/qemu-ga/fsfreeze-hook qemu 26463 22.9 4.8 17684512 1088844 ? Sl 16:01 3:09 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on root 28416 0.0 0.0 112712 980 pts/3 S+ 16:14 0:00 grep --color=auto qemu Before the first Error, while the vm was running for sure and the disk was there, I also unsuccessfuly tried to connect to it with VNC and got the same error I got before : [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. debug1: channel 3: new [direct-tcpip] channel 3: open failed: connect failed: Connection refused debug1: channel 3: free: direct-tcpip: listening port 5900 for vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4 Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese < guillaume.pavese(a)interactiv-group.com&gt; wrote: > Something was definitely wrong ; as indicated, qemu process > for guest=HostedEngineLocal was running but the disk file did not exist > anymore... > No surprise I could not connect > > I am retrying > > > Guillaume Pavese > Ingénieur Système et Réseau > Interactiv-Group > > > On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> It fails too : >> I made sure PermitTunnel=yes in sshd config but when I try to connect to >> the forwarded port I get the following error on the openened host ssh >> session : >> >> [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: >> vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >> ... >> [root@vs-inf-int-kvm-fr-301-210 ~]# >> debug1: channel 3: free: direct-tcpip: listening port 5900 for >> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 port >> 42144 to ::1 port 5900, nchannels 4 >> debug1: Connection to port 5900 forwarding to >> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >> debug1: channel 3: new [direct-tcpip] >> channel 3: open failed: connect failed: Connection refused >> debug1: channel 3: free: direct-tcpip: listening port 5900 for >> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 >> port 32778 to 127.0.0.1 port 5900, nchannels 4 >> >> >> and in journalctl : >> >> févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: >> error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: >> failed. >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> >>> On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> I made sure of everything and even stopped firewalld but still can't >>>> connect : >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>> /var/run/libvirt/qemu/HostedEngineLocal.xml >>>> <graphics type='vnc' port='*5900*' autoport='yes' >>>> *listen='127.0.0.1*'> >>>> <listen type='address' address='*127.0.0.1*' fromConfig='1' >>>> autoGenerated='no'/> >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >>>> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >>>> LISTEN 13376/qemu-kvm >>>> >>> >>> >>> I suggest to try ssh tunneling, run >>> ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>> >>> and then >>> remote-viewer vnc://localhost:5900 >>> >>> >>> >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status >>>> firewalld.service >>>> ● firewalld.service - firewalld - dynamic firewall daemon >>>> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; >>>> vendor preset: enabled) >>>> *Active: inactive (dead)* >>>> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >>>> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >>>> - dynamic firewall daemon.* >>>> >>>> From my laptop : >>>> [gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr >>>> *5900* >>>> Trying 10.199.210.11... >>>> [*nothing gets through...*] >>>> ^C >>>> >>>> For making sure : >>>> [gpavese@sheepora-X230 ~]$ telnet vs-inf-int-kvm-fr-301-210.hostics.fr >>>> *9090* >>>> Trying 10.199.210.11... >>>> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >>>> Escape character is '^]'. >>>> >>>> >>>> >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Hey! >>>>> >>>>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>>>> Search for 'vnc' >>>>> From there you can look up the port on which the HE VM is available >>>>> and connect to the same. >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> 1) I am running in a Nested env, but under libvirt/kvm on remote >>>>>> Centos 7.4 Hosts >>>>>> >>>>>> Please advise how to connect with VNC to the local HE vm. I see it's >>>>>> running, but this is on a remote host, not my local machine : >>>>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 85:08 >>>>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>>>> -object >>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>>>> -no-user-config -nodefaults -chardev >>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>> menu=off,strict=on -device >>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>> -device >>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>> -drive >>>>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>>>> -chardev pty,id=charserial0 -device >>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>> -msg timestamp=on >>>>>> >>>>>> >>>>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>> /etc/libvirt/qemu/networks/default.xml >>>>>> <!-- >>>>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO >>>>>> BE >>>>>> OVERWRITTEN AND LOST. Changes to this xml configuration should be >>>>>> made using: >>>>>> virsh net-edit default >>>>>> or other application using the libvirt API. >>>>>> --> >>>>>> >>>>>> <network> >>>>>> <name>default</name> >>>>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>>>> <forward mode='nat'/> >>>>>> <bridge name='virbr0' stp='on' delay='0'/> >>>>>> <mac address='52:54:00:e5:fe:3b'/> >>>>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>>>> <dhcp> >>>>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>>>> </dhcp> >>>>>> </ip> >>>>>> </network> >>>>>> You have new mail in /var/spool/mail/root >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>>>> >>>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi < >>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>> >>>>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>> >>>>>>>> See following Error : >>>>>>>> >>>>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>>>> local VM IP] >>>>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': >>>>>>>> u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', >>>>>>>> u'changed': True, u'invocation': {u'module_args': {u'warn': True, >>>>>>>> u'executable': >>>>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>>>> u'chdir': None, u'std >>>>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', u'attempts': >>>>>>>> 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', u'stdout_lines': >>>>>>>> []} >>>>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>> "", "stdout_lines": []} >>>>>>>> >>>>>>> >>>>>>> Here we are just waiting for the bootstrap engine VM to fetch an IP >>>>>>> address from default libvirt network over DHCP but it your case it never >>>>>>> happened. >>>>>>> Possible issues: something went wrong in the bootstrap process for >>>>>>> the engine VM or the default libvirt network is not correctly configured. >>>>>>> >>>>>>> 1. can you try to reach the engine VM via VNC and check what's >>>>>>> happening there? (another question, are you running it nested? AFAIK it >>>>>>> will not work if nested over ESXi) >>>>>>> 2. can you please share the output of >>>>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Guillaume Pavese >>>>>>>> Ingénieur Système et Réseau >>>>>>>> Interactiv-Group >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>>>> >>>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>>>> >>>>>

I did that but no success yet. I see that "Get local VM IP" task tries the following : virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | awk '{ print $5 }' | cut -f1 -d'/' However while the task is running, and vm running in qemu, "virsh -r net-dhcp-leases default" never returns anything :

[root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------------- [root@vs-inf-int-kvm-fr-301-210 ~]# Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > OK, try this: > temporary > edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml > around line 120 > and edit tasks "Get local VM IP" > changing from "retries: 50" to "retries: 500" so that you have more time > to debug it > > > > On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> I retried after killing the remaining qemu process and >> doing ovirt-hosted-engine-cleanup >> The new attempt failed again at the same step. Then after it fails, it >> cleans the temporary files (and vm disk) but *qemu still runs!* : >> >> [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 >> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": >> "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 >> 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", >> "stdout_lines": []} >> [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >> [ INFO ] changed: [localhost] >> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in >> /etc/hosts for the local VM] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a >> failure] >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The >> system may not be provisioned according to the playbook results: please >> check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} >> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >> ansible-playbook >> [ INFO ] Stage: Clean up >> [ INFO ] Cleaning temporary resources >> ... >> >> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in >> /etc/hosts for the local VM] >> [ INFO ] ok: [localhost] >> [ INFO ] Generating answer file >> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf' >> [ INFO ] Stage: Pre-termination >> [ INFO ] Stage: Termination >> [ ERROR ] Hosted Engine deployment failed: please check the logs for the >> issue, fix accordingly or re-deploy from scratch. >> >> >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu >> root 4021 0.0 0.0 24844 1788 ? Ss févr.22 0:00 >> /usr/bin/qemu-ga --method=virtio-serial >> --path=/dev/virtio-ports/org.qemu.guest_agent.0 >> --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status >> -F/etc/qemu-ga/fsfreeze-hook >> qemu 26463 22.9 4.8 17684512 1088844 ? Sl 16:01 3:09 >> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >> -object >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >> 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e >> -no-user-config -nodefaults -chardev >> socket,id=charmonitor,fd=27,server,nowait -mon >> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >> menu=off,strict=on -device >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >> file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >> -device >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >> -drive >> file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 >> -chardev pty,id=charserial0 -device >> isa-serial,chardev=charserial0,id=serial0 -chardev >> socket,id=charchannel0,fd=31,server,nowait -device >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >> -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >> -object rng-random,id=objrng0,filename=/dev/random -device >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >> -msg timestamp=on >> root 28416 0.0 0.0 112712 980 pts/3 S+ 16:14 0:00 grep >> --color=auto qemu >> >> >> Before the first Error, while the vm was running for sure and the disk >> was there, I also unsuccessfuly tried to connect to it with VNC and got >> the same error I got before : >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 >> forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >> debug1: channel 3: new [direct-tcpip] >> channel 3: open failed: connect failed: Connection refused >> debug1: channel 3: free: direct-tcpip: listening port 5900 for >> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from 127.0.0.1 >> port 37002 to 127.0.0.1 port 5900, nchannels 4 >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese < >> guillaume.pavese(a)interactiv-group.com&gt; wrote: >> >>> Something was definitely wrong ; as indicated, qemu process >>> for guest=HostedEngineLocal was running but the disk file did not exist >>> anymore... >>> No surprise I could not connect >>> >>> I am retrying >>> >>> >>> Guillaume Pavese >>> Ingénieur Système et Réseau >>> Interactiv-Group >>> >>> >>> On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> It fails too : >>>> I made sure PermitTunnel=yes in sshd config but when I try to connect >>>> to the forwarded port I get the following error on the openened host ssh >>>> session : >>>> >>>> [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: >>>> vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>> ... >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 port >>>> 42144 to ::1 port 5900, nchannels 4 >>>> debug1: Connection to port 5900 forwarding to >>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >>>> debug1: channel 3: new [direct-tcpip] >>>> channel 3: open failed: connect failed: Connection refused >>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>> 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4 >>>> >>>> >>>> and in journalctl : >>>> >>>> févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: >>>> error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: >>>> failed. >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi < >>>> stirabos(a)redhat.com&gt; wrote: >>>> >>>>> >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> I made sure of everything and even stopped firewalld but still can't >>>>>> connect : >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>> /var/run/libvirt/qemu/HostedEngineLocal.xml >>>>>> <graphics type='vnc' port='*5900*' autoport='yes' >>>>>> *listen='127.0.0.1*'> >>>>>> <listen type='address' address='*127.0.0.1*' fromConfig='1' >>>>>> autoGenerated='no'/> >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >>>>>> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >>>>>> LISTEN 13376/qemu-kvm >>>>>> >>>>> >>>>> >>>>> I suggest to try ssh tunneling, run >>>>> ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>> >>>>> and then >>>>> remote-viewer vnc://localhost:5900 >>>>> >>>>> >>>>> >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status >>>>>> firewalld.service >>>>>> ● firewalld.service - firewalld - dynamic firewall daemon >>>>>> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; >>>>>> enabled; vendor preset: enabled) >>>>>> *Active: inactive (dead)* >>>>>> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >>>>>> - dynamic firewall daemon.* >>>>>> >>>>>> From my laptop : >>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *5900* >>>>>> Trying 10.199.210.11... >>>>>> [*nothing gets through...*] >>>>>> ^C >>>>>> >>>>>> For making sure : >>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *9090* >>>>>> Trying 10.199.210.11... >>>>>> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >>>>>> Escape character is '^]'. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hey! >>>>>>> >>>>>>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>>>>>> Search for 'vnc' >>>>>>> From there you can look up the port on which the HE VM is available >>>>>>> and connect to the same. >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>> >>>>>>>> 1) I am running in a Nested env, but under libvirt/kvm on remote >>>>>>>> Centos 7.4 Hosts >>>>>>>> >>>>>>>> Please advise how to connect with VNC to the local HE vm. I see >>>>>>>> it's running, but this is on a remote host, not my local machine : >>>>>>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 85:08 >>>>>>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>>>>>> -object >>>>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>>>>>> -no-user-config -nodefaults -chardev >>>>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>>>> menu=off,strict=on -device >>>>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>>>> -device >>>>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>>>> -drive >>>>>>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>>>>>> -chardev pty,id=charserial0 -device >>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>>>> -msg timestamp=on >>>>>>>> >>>>>>>> >>>>>>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>> /etc/libvirt/qemu/networks/default.xml >>>>>>>> <!-- >>>>>>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY >>>>>>>> TO BE >>>>>>>> OVERWRITTEN AND LOST. Changes to this xml configuration should be >>>>>>>> made using: >>>>>>>> virsh net-edit default >>>>>>>> or other application using the libvirt API. >>>>>>>> --> >>>>>>>> >>>>>>>> <network> >>>>>>>> <name>default</name> >>>>>>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>>>>>> <forward mode='nat'/> >>>>>>>> <bridge name='virbr0' stp='on' delay='0'/> >>>>>>>> <mac address='52:54:00:e5:fe:3b'/> >>>>>>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>>>>>> <dhcp> >>>>>>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>>>>>> </dhcp> >>>>>>>> </ip> >>>>>>>> </network> >>>>>>>> You have new mail in /var/spool/mail/root >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Guillaume Pavese >>>>>>>> Ingénieur Système et Réseau >>>>>>>> Interactiv-Group >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi < >>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>> >>>>>>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>>>> >>>>>>>>>> See following Error : >>>>>>>>>> >>>>>>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>>>>>> local VM IP] >>>>>>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': >>>>>>>>>> u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', >>>>>>>>>> u'changed': True, u'invocation': {u'module_args': {u'warn': True, >>>>>>>>>> u'executable': >>>>>>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>>>>>> u'chdir': None, u'std >>>>>>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', >>>>>>>>>> u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', >>>>>>>>>> u'stdout_lines': []} >>>>>>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>>>> "", "stdout_lines": []} >>>>>>>>>> >>>>>>>>> >>>>>>>>> Here we are just waiting for the bootstrap engine VM to fetch an >>>>>>>>> IP address from default libvirt network over DHCP but it your case it never >>>>>>>>> happened. >>>>>>>>> Possible issues: something went wrong in the bootstrap process >>>>>>>>> for the engine VM or the default libvirt network is not correctly >>>>>>>>> configured. >>>>>>>>> >>>>>>>>> 1. can you try to reach the engine VM via VNC and check what's >>>>>>>>> happening there? (another question, are you running it nested? AFAIK it >>>>>>>>> will not work if nested over ESXi) >>>>>>>>> 2. can you please share the output of >>>>>>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Guillaume Pavese >>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>> Interactiv-Group >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>>>>>> >>>>>>>

I still can't connect with VNC remotely but locally with X forwarding it works. However my connection has too high latency for that to be usable (I'm in Japan, my hosts in France, ~250 ms ping) But I could see that the VM is booted! and in Hosts logs there is : févr. 25 18:51:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14719]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPDISCOVER(virbr0) 00:16:3e:1d:4b:b6 févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPOFFER(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPREQUEST(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: DHCPACK(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 vs-inf-int-ovt-fr-301-210 févr. 25 18:51:42 vs-inf-int-kvm-fr-301-210.hostics.fr python[14757]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None févr. 25 18:52:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14789]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None févr. 25 18:52:43 vs-inf-int-kvm-fr-301-210.hostics.fr python[14818]: ansible-command Invoked with warn=True executable=None _uses_shell=True _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None chdir=None stdin=None .... ssh to the vm works too : [root@vs-inf-int-kvm-fr-301-210 ~]# ssh root(a)192.168.122.14 The authenticity of host '192.168.122.14 (192.168.122.14)' can't be established. ECDSA key fingerprint is SHA256:+/pUzTGVA4kCyICb7XgqrxWYYkqzmDjVmdAahiBFgOQ. ECDSA key fingerprint is MD5:4b:ef:ff:4a:7c:1a:af:c2:af:4a:0f:14:a3:c5:31:fb. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.122.14' (ECDSA) to the list of known hosts. root(a)192.168.122.14&#39;s password: [root@vs-inf-int-ovt-fr-301-210 ~]# But the test that the playbook tries still fails with empty result : [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------------- [root@vs-inf-int-kvm-fr-301-210 ~]#

Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Tue, Feb 26, 2019 at 1:54 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Feb 25, 2019 at 5:50 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> I did that but no success yet. >> >> I see that "Get local VM IP" task tries the following : >> >> virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | awk '{ >> print $5 }' | cut -f1 -d'/' >> >> >> However while the task is running, and vm running in qemu, "virsh -r >> net-dhcp-leases default" never returns anything : >> > > Yes, I think that libvirt will never provide a DHCP lease since the > appliance OS never correctly complete the boot process. > I'd suggest to connect to the running VM via vnc DURING the boot process > and check what's wrong. > > >> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >> Expiry Time MAC address Protocol IP address >> Hostname Client ID or DUID >> >> ------------------------------------------------------------------------------------------------------------------- >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# >> >> >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> OK, try this: >>> temporary >>> edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml >>> around line 120 >>> and edit tasks "Get local VM IP" >>> changing from "retries: 50" to "retries: 500" so that you have more >>> time to debug it >>> >>> >>> >>> On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> I retried after killing the remaining qemu process and >>>> doing ovirt-hosted-engine-cleanup >>>> The new attempt failed again at the same step. Then after it fails, it >>>> cleans the temporary files (and vm disk) but *qemu still runs!* : >>>> >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] >>>> >>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 >>>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": >>>> "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 >>>> 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", >>>> "stdout_lines": []} >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] >>>> [ INFO ] ok: [localhost] >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>> [ INFO ] changed: [localhost] >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in >>>> /etc/hosts for the local VM] >>>> [ INFO ] ok: [localhost] >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a >>>> failure] >>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>> "The system may not be provisioned according to the playbook results: >>>> please check the logs for the issue, fix accordingly or re-deploy from >>>> scratch.\n"} >>>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >>>> ansible-playbook >>>> [ INFO ] Stage: Clean up >>>> [ INFO ] Cleaning temporary resources >>>> ... >>>> >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>> [ INFO ] ok: [localhost] >>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in >>>> /etc/hosts for the local VM] >>>> [ INFO ] ok: [localhost] >>>> [ INFO ] Generating answer file >>>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf' >>>> [ INFO ] Stage: Pre-termination >>>> [ INFO ] Stage: Termination >>>> [ ERROR ] Hosted Engine deployment failed: please check the logs for >>>> the issue, fix accordingly or re-deploy from scratch. >>>> >>>> >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu >>>> root 4021 0.0 0.0 24844 1788 ? Ss févr.22 0:00 >>>> /usr/bin/qemu-ga --method=virtio-serial >>>> --path=/dev/virtio-ports/org.qemu.guest_agent.0 >>>> --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status >>>> -F/etc/qemu-ga/fsfreeze-hook >>>> qemu 26463 22.9 4.8 17684512 1088844 ? Sl 16:01 3:09 >>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>> -object >>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>> 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e >>>> -no-user-config -nodefaults -chardev >>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>> menu=off,strict=on -device >>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>> file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>> -device >>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> -drive >>>> file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 >>>> -chardev pty,id=charserial0 -device >>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>> socket,id=charchannel0,fd=31,server,nowait -device >>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>> -vnc 127.0.0.1:0 -device >>>> VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object >>>> rng-random,id=objrng0,filename=/dev/random -device >>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>> -msg timestamp=on >>>> root 28416 0.0 0.0 112712 980 pts/3 S+ 16:14 0:00 grep >>>> --color=auto qemu >>>> >>>> >>>> Before the first Error, while the vm was running for sure and the disk >>>> was there, I also unsuccessfuly tried to connect to it with VNC and got >>>> the same error I got before : >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 >>>> forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 >>>> requested. >>>> debug1: channel 3: new [direct-tcpip] >>>> channel 3: open failed: connect failed: Connection refused >>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>> 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4 >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese < >>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>> >>>>> Something was definitely wrong ; as indicated, qemu process >>>>> for guest=HostedEngineLocal was running but the disk file did not exist >>>>> anymore... >>>>> No surprise I could not connect >>>>> >>>>> I am retrying >>>>> >>>>> >>>>> Guillaume Pavese >>>>> Ingénieur Système et Réseau >>>>> Interactiv-Group >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> It fails too : >>>>>> I made sure PermitTunnel=yes in sshd config but when I try to >>>>>> connect to the forwarded port I get the following error on the openened >>>>>> host ssh session : >>>>>> >>>>>> [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>> ... >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 >>>>>> port 42144 to ::1 port 5900, nchannels 4 >>>>>> debug1: Connection to port 5900 forwarding to >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >>>>>> debug1: channel 3: new [direct-tcpip] >>>>>> channel 3: open failed: connect failed: Connection refused >>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>> 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4 >>>>>> >>>>>> >>>>>> and in journalctl : >>>>>> >>>>>> févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr sshd[19595]: >>>>>> error: connect_to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: >>>>>> failed. >>>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi < >>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < >>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>> >>>>>>>> I made sure of everything and even stopped firewalld but still >>>>>>>> can't connect : >>>>>>>> >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>> /var/run/libvirt/qemu/HostedEngineLocal.xml >>>>>>>> <graphics type='vnc' port='*5900*' autoport='yes' >>>>>>>> *listen='127.0.0.1*'> >>>>>>>> <listen type='address' address='*127.0.0.1*' >>>>>>>> fromConfig='1' autoGenerated='no'/> >>>>>>>> >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >>>>>>>> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >>>>>>>> LISTEN 13376/qemu-kvm >>>>>>>> >>>>>>> >>>>>>> >>>>>>> I suggest to try ssh tunneling, run >>>>>>> ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>> >>>>>>> and then >>>>>>> remote-viewer vnc://localhost:5900 >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status >>>>>>>> firewalld.service >>>>>>>> ● firewalld.service - firewalld - dynamic firewall daemon >>>>>>>> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; >>>>>>>> enabled; vendor preset: enabled) >>>>>>>> *Active: inactive (dead)* >>>>>>>> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >>>>>>>> - dynamic firewall daemon.* >>>>>>>> >>>>>>>> From my laptop : >>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *5900* >>>>>>>> Trying 10.199.210.11... >>>>>>>> [*nothing gets through...*] >>>>>>>> ^C >>>>>>>> >>>>>>>> For making sure : >>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *9090* >>>>>>>> Trying 10.199.210.11... >>>>>>>> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >>>>>>>> Escape character is '^]'. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Guillaume Pavese >>>>>>>> Ingénieur Système et Réseau >>>>>>>> Interactiv-Group >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal <dparth(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hey! >>>>>>>>> >>>>>>>>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>>>>>>>> Search for 'vnc' >>>>>>>>> From there you can look up the port on which the HE VM is >>>>>>>>> available and connect to the same. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>> >>>>>>>>>> 1) I am running in a Nested env, but under libvirt/kvm on remote >>>>>>>>>> Centos 7.4 Hosts >>>>>>>>>> >>>>>>>>>> Please advise how to connect with VNC to the local HE vm. I see >>>>>>>>>> it's running, but this is on a remote host, not my local machine : >>>>>>>>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 85:08 >>>>>>>>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>>>>>>>> -object >>>>>>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>>>>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>>>>>>>> -no-user-config -nodefaults -chardev >>>>>>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>>>>>> menu=off,strict=on -device >>>>>>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>>>>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>>>>>> -device >>>>>>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>>>>>> -drive >>>>>>>>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>>>>>>>> -chardev pty,id=charserial0 -device >>>>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>>>>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>>>>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>>>>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>>>>>> -msg timestamp=on >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>>>> /etc/libvirt/qemu/networks/default.xml >>>>>>>>>> <!-- >>>>>>>>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE >>>>>>>>>> LIKELY TO BE >>>>>>>>>> OVERWRITTEN AND LOST. Changes to this xml configuration should >>>>>>>>>> be made using: >>>>>>>>>> virsh net-edit default >>>>>>>>>> or other application using the libvirt API. >>>>>>>>>> --> >>>>>>>>>> >>>>>>>>>> <network> >>>>>>>>>> <name>default</name> >>>>>>>>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>>>>>>>> <forward mode='nat'/> >>>>>>>>>> <bridge name='virbr0' stp='on' delay='0'/> >>>>>>>>>> <mac address='52:54:00:e5:fe:3b'/> >>>>>>>>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>>>>>>>> <dhcp> >>>>>>>>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>>>>>>>> </dhcp> >>>>>>>>>> </ip> >>>>>>>>>> </network> >>>>>>>>>> You have new mail in /var/spool/mail/root >>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Guillaume Pavese >>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>> Interactiv-Group >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi < >>>>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>>>>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>>>>>> >>>>>>>>>>>> See following Error : >>>>>>>>>>>> >>>>>>>>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>>>>>>>> local VM IP] >>>>>>>>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>>>>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>>>>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", u'end': >>>>>>>>>>>> u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, u'stdout': u'', >>>>>>>>>>>> u'changed': True, u'invocation': {u'module_args': {u'warn': True, >>>>>>>>>>>> u'executable': >>>>>>>>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>>>>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>>>>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>>>>>>>> u'chdir': None, u'std >>>>>>>>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', >>>>>>>>>>>> u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', >>>>>>>>>>>> u'stdout_lines': []} >>>>>>>>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>>>>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>>>>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>>>>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>>>>>> "", "stdout_lines": []} >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Here we are just waiting for the bootstrap engine VM to fetch >>>>>>>>>>> an IP address from default libvirt network over DHCP but it your case it >>>>>>>>>>> never happened. >>>>>>>>>>> Possible issues: something went wrong in the bootstrap process >>>>>>>>>>> for the engine VM or the default libvirt network is not correctly >>>>>>>>>>> configured. >>>>>>>>>>> >>>>>>>>>>> 1. can you try to reach the engine VM via VNC and check what's >>>>>>>>>>> happening there? (another question, are you running it nested? AFAIK it >>>>>>>>>>> will not work if nested over ESXi) >>>>>>>>>>> 2. can you please share the output of >>>>>>>>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>> List Archives: >>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>>>>>>>> >>>>>>>>>

No, as indicated previously, still : [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default Expiry Time MAC address Protocol IP address Hostname Client ID or DUID ------------------------------------------------------------------------------------------------------------------- [root@vs-inf-int-kvm-fr-301-210 ~]# I did not see any relevant log on the HE vm. Is there something I should look for there?

Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Tue, Feb 26, 2019 at 3:12 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Feb 25, 2019 at 7:04 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> I still can't connect with VNC remotely but locally with X forwarding it >> works. >> However my connection has too high latency for that to be usable (I'm in >> Japan, my hosts in France, ~250 ms ping) >> >> But I could see that the VM is booted! >> >> and in Hosts logs there is : >> >> févr. 25 18:51:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14719]: >> ansible-command Invoked with warn=True executable=None _uses_shell=True >> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >> chdir=None stdin=None >> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >> dnsmasq-dhcp[6310]: DHCPDISCOVER(virbr0) 00:16:3e:1d:4b:b6 >> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >> dnsmasq-dhcp[6310]: DHCPOFFER(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >> dnsmasq-dhcp[6310]: DHCPREQUEST(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >> dnsmasq-dhcp[6310]: DHCPACK(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >> vs-inf-int-ovt-fr-301-210 >> févr. 25 18:51:42 vs-inf-int-kvm-fr-301-210.hostics.fr python[14757]: >> ansible-command Invoked with warn=True executable=None _uses_shell=True >> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >> chdir=None stdin=None >> févr. 25 18:52:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14789]: >> ansible-command Invoked with warn=True executable=None _uses_shell=True >> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >> chdir=None stdin=None >> févr. 25 18:52:43 vs-inf-int-kvm-fr-301-210.hostics.fr python[14818]: >> ansible-command Invoked with warn=True executable=None _uses_shell=True >> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >> chdir=None stdin=None >> .... >> >> ssh to the vm works too : >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# ssh root(a)192.168.122.14 >> The authenticity of host '192.168.122.14 (192.168.122.14)' can't be >> established. >> ECDSA key fingerprint is >> SHA256:+/pUzTGVA4kCyICb7XgqrxWYYkqzmDjVmdAahiBFgOQ. >> ECDSA key fingerprint is >> MD5:4b:ef:ff:4a:7c:1a:af:c2:af:4a:0f:14:a3:c5:31:fb. >> Are you sure you want to continue connecting (yes/no)? yes >> Warning: Permanently added '192.168.122.14' (ECDSA) to the list of known >> hosts. >> root(a)192.168.122.14&#39;s password: >> [root@vs-inf-int-ovt-fr-301-210 ~]# >> >> >> But the test that the playbook tries still fails with empty result : >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >> Expiry Time MAC address Protocol IP address >> Hostname Client ID or DUID >> >> ------------------------------------------------------------------------------------------------------------------- >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# >> >> > This smells like a bug to me: > and nothing at all in the output of > virsh -r net-dhcp-leases default > > ? > > >> >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Tue, Feb 26, 2019 at 1:54 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Mon, Feb 25, 2019 at 5:50 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> I did that but no success yet. >>>> >>>> I see that "Get local VM IP" task tries the following : >>>> >>>> virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | awk >>>> '{ print $5 }' | cut -f1 -d'/' >>>> >>>> >>>> However while the task is running, and vm running in qemu, "virsh -r >>>> net-dhcp-leases default" never returns anything : >>>> >>> >>> Yes, I think that libvirt will never provide a DHCP lease since the >>> appliance OS never correctly complete the boot process. >>> I'd suggest to connect to the running VM via vnc DURING the boot >>> process and check what's wrong. >>> >>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >>>> Expiry Time MAC address Protocol IP address >>>> Hostname Client ID or DUID >>>> >>>> ------------------------------------------------------------------------------------------------------------------- >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>> >>>> >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi < >>>> stirabos(a)redhat.com&gt; wrote: >>>> >>>>> OK, try this: >>>>> temporary >>>>> edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml >>>>> around line 120 >>>>> and edit tasks "Get local VM IP" >>>>> changing from "retries: 50" to "retries: 500" so that you have more >>>>> time to debug it >>>>> >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> I retried after killing the remaining qemu process and >>>>>> doing ovirt-hosted-engine-cleanup >>>>>> The new attempt failed again at the same step. Then after it fails, >>>>>> it cleans the temporary files (and vm disk) but *qemu still runs!* : >>>>>> >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] >>>>>> >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": >>>>>> true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:e8:f9 >>>>>> | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.092436", "end": >>>>>> "2019-02-25 16:09:38.863263", "rc": 0, "start": "2019-02-25 >>>>>> 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": "", >>>>>> "stdout_lines": []} >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] >>>>>> [ INFO ] ok: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>>>> [ INFO ] changed: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry >>>>>> in /etc/hosts for the local VM] >>>>>> [ INFO ] ok: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a >>>>>> failure] >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>>>> "The system may not be provisioned according to the playbook results: >>>>>> please check the logs for the issue, fix accordingly or re-deploy from >>>>>> scratch.\n"} >>>>>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >>>>>> ansible-playbook >>>>>> [ INFO ] Stage: Clean up >>>>>> [ INFO ] Cleaning temporary resources >>>>>> ... >>>>>> >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>>>> [ INFO ] ok: [localhost] >>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry >>>>>> in /etc/hosts for the local VM] >>>>>> [ INFO ] ok: [localhost] >>>>>> [ INFO ] Generating answer file >>>>>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf' >>>>>> [ INFO ] Stage: Pre-termination >>>>>> [ INFO ] Stage: Termination >>>>>> [ ERROR ] Hosted Engine deployment failed: please check the logs for >>>>>> the issue, fix accordingly or re-deploy from scratch. >>>>>> >>>>>> >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu >>>>>> root 4021 0.0 0.0 24844 1788 ? Ss févr.22 0:00 >>>>>> /usr/bin/qemu-ga --method=virtio-serial >>>>>> --path=/dev/virtio-ports/org.qemu.guest_agent.0 >>>>>> --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status >>>>>> -F/etc/qemu-ga/fsfreeze-hook >>>>>> qemu 26463 22.9 4.8 17684512 1088844 ? Sl 16:01 3:09 >>>>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>>>> -object >>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>> 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e >>>>>> -no-user-config -nodefaults -chardev >>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>> menu=off,strict=on -device >>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>> file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>> -device >>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>> -drive >>>>>> file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 >>>>>> -chardev pty,id=charserial0 -device >>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>> -vnc 127.0.0.1:0 -device >>>>>> VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object >>>>>> rng-random,id=objrng0,filename=/dev/random -device >>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>> -msg timestamp=on >>>>>> root 28416 0.0 0.0 112712 980 pts/3 S+ 16:14 0:00 >>>>>> grep --color=auto qemu >>>>>> >>>>>> >>>>>> Before the first Error, while the vm was running for sure and the >>>>>> disk was there, I also unsuccessfuly tried to connect to it with VNC and >>>>>> got the same error I got before : >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port 5900 >>>>>> forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 >>>>>> requested. >>>>>> debug1: channel 3: new [direct-tcpip] >>>>>> channel 3: open failed: connect failed: Connection refused >>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>> 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4 >>>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese < >>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>> >>>>>>> Something was definitely wrong ; as indicated, qemu process >>>>>>> for guest=HostedEngineLocal was running but the disk file did not exist >>>>>>> anymore... >>>>>>> No surprise I could not connect >>>>>>> >>>>>>> I am retrying >>>>>>> >>>>>>> >>>>>>> Guillaume Pavese >>>>>>> Ingénieur Système et Réseau >>>>>>> Interactiv-Group >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese < >>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>> >>>>>>>> It fails too : >>>>>>>> I made sure PermitTunnel=yes in sshd config but when I try to >>>>>>>> connect to the forwarded port I get the following error on the openened >>>>>>>> host ssh session : >>>>>>>> >>>>>>>> [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>> ... >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from ::1 >>>>>>>> port 42144 to ::1 port 5900, nchannels 4 >>>>>>>> debug1: Connection to port 5900 forwarding to >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >>>>>>>> debug1: channel 3: new [direct-tcpip] >>>>>>>> channel 3: open failed: connect failed: Connection refused >>>>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>>>> 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4 >>>>>>>> >>>>>>>> >>>>>>>> and in journalctl : >>>>>>>> >>>>>>>> févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>> sshd[19595]: error: connect_to >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: failed. >>>>>>>> >>>>>>>> >>>>>>>> Guillaume Pavese >>>>>>>> Ingénieur Système et Réseau >>>>>>>> Interactiv-Group >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi < >>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < >>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>> >>>>>>>>>> I made sure of everything and even stopped firewalld but still >>>>>>>>>> can't connect : >>>>>>>>>> >>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>>>> /var/run/libvirt/qemu/HostedEngineLocal.xml >>>>>>>>>> <graphics type='vnc' port='*5900*' autoport='yes' >>>>>>>>>> *listen='127.0.0.1*'> >>>>>>>>>> <listen type='address' address='*127.0.0.1*' >>>>>>>>>> fromConfig='1' autoGenerated='no'/> >>>>>>>>>> >>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >>>>>>>>>> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >>>>>>>>>> LISTEN 13376/qemu-kvm >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I suggest to try ssh tunneling, run >>>>>>>>> ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>> >>>>>>>>> and then >>>>>>>>> remote-viewer vnc://localhost:5900 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status >>>>>>>>>> firewalld.service >>>>>>>>>> ● firewalld.service - firewalld - dynamic firewall daemon >>>>>>>>>> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; >>>>>>>>>> enabled; vendor preset: enabled) >>>>>>>>>> *Active: inactive (dead)* >>>>>>>>>> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>>> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >>>>>>>>>> - dynamic firewall daemon.* >>>>>>>>>> >>>>>>>>>> From my laptop : >>>>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *5900* >>>>>>>>>> Trying 10.199.210.11... >>>>>>>>>> [*nothing gets through...*] >>>>>>>>>> ^C >>>>>>>>>> >>>>>>>>>> For making sure : >>>>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *9090* >>>>>>>>>> Trying 10.199.210.11... >>>>>>>>>> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >>>>>>>>>> Escape character is '^]'. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Guillaume Pavese >>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>> Interactiv-Group >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal < >>>>>>>>>> dparth(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hey! >>>>>>>>>>> >>>>>>>>>>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>>>>>>>>>> Search for 'vnc' >>>>>>>>>>> From there you can look up the port on which the HE VM is >>>>>>>>>>> available and connect to the same. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> 1) I am running in a Nested env, but under libvirt/kvm on >>>>>>>>>>>> remote Centos 7.4 Hosts >>>>>>>>>>>> >>>>>>>>>>>> Please advise how to connect with VNC to the local HE vm. I >>>>>>>>>>>> see it's running, but this is on a remote host, not my local machine : >>>>>>>>>>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 >>>>>>>>>>>> 85:08 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on >>>>>>>>>>>> -S -object >>>>>>>>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>>>>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>>>>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>>>>>>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>>>>>>>>>> -no-user-config -nodefaults -chardev >>>>>>>>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>>>>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>>>>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>>>>>>>> menu=off,strict=on -device >>>>>>>>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>>>>>>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>>>>>>>> -device >>>>>>>>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>>>>>>>> -drive >>>>>>>>>>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>>>>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>>>>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>>>>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>>>>>>>>>> -chardev pty,id=charserial0 -device >>>>>>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>>>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>>>>>>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>>>>>>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>>>>>>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>>>>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>>>>>>>> -msg timestamp=on >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>>>>>> /etc/libvirt/qemu/networks/default.xml >>>>>>>>>>>> <!-- >>>>>>>>>>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE >>>>>>>>>>>> LIKELY TO BE >>>>>>>>>>>> OVERWRITTEN AND LOST. Changes to this xml configuration should >>>>>>>>>>>> be made using: >>>>>>>>>>>> virsh net-edit default >>>>>>>>>>>> or other application using the libvirt API. >>>>>>>>>>>> --> >>>>>>>>>>>> >>>>>>>>>>>> <network> >>>>>>>>>>>> <name>default</name> >>>>>>>>>>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>>>>>>>>>> <forward mode='nat'/> >>>>>>>>>>>> <bridge name='virbr0' stp='on' delay='0'/> >>>>>>>>>>>> <mac address='52:54:00:e5:fe:3b'/> >>>>>>>>>>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>>>>>>>>>> <dhcp> >>>>>>>>>>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>>>>>>>>>> </dhcp> >>>>>>>>>>>> </ip> >>>>>>>>>>>> </network> >>>>>>>>>>>> You have new mail in /var/spool/mail/root >>>>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi < >>>>>>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>>>>>>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>>>>>>>> >>>>>>>>>>>>>> See following Error : >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>>>>>>>>>> local VM IP] >>>>>>>>>>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>>>>>>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>>>>>>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", >>>>>>>>>>>>>> u'end': u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, >>>>>>>>>>>>>> u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': >>>>>>>>>>>>>> True, u'executable': >>>>>>>>>>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>>>>>>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>>>>>>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>>>>>>>>>> u'chdir': None, u'std >>>>>>>>>>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', >>>>>>>>>>>>>> u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', >>>>>>>>>>>>>> u'stdout_lines': []} >>>>>>>>>>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>>>>>>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>>>>>>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>>>>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>>>>>>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>>>>>>>> "", "stdout_lines": []} >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Here we are just waiting for the bootstrap engine VM to fetch >>>>>>>>>>>>> an IP address from default libvirt network over DHCP but it your case it >>>>>>>>>>>>> never happened. >>>>>>>>>>>>> Possible issues: something went wrong in the bootstrap >>>>>>>>>>>>> process for the engine VM or the default libvirt network is not correctly >>>>>>>>>>>>> configured. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. can you try to reach the engine VM via VNC and check >>>>>>>>>>>>> what's happening there? (another question, are you running it nested? AFAIK >>>>>>>>>>>>> it will not work if nested over ESXi) >>>>>>>>>>>>> 2. can you please share the output of >>>>>>>>>>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>>>> Privacy Statement: >>>>>>>>>>>>>> https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>>> List Archives: >>>>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>> List Archives: >>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>>>>>>>>>> >>>>>>>>>>>

journalctl -u libvirtd.service : févr. 25 18:47:24 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Stopping Virtualization daemon... févr. 25 18:47:24 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Stopped Virtualization daemon. févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Starting Virtualization daemon... févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr systemd[1]: Started Virtualization daemon. févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq[6310]: read /etc/hosts - 4 addresses févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq[6310]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr dnsmasq-dhcp[6310]: read /var/lib/libvirt/dnsmasq/default.hostsfile févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.739+0000: 13551: info : libvirt version: 4.5.0, package: 10.el7_6.4 (CentOS BuildSystem <http://bugs.centos.org>;, 2019-01-29-17:31:22, x86-01.bsys.centos.org) févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.739+0000: 13551: info : hostname: vs-inf-int-kvm-fr-301-210.hostics.fr févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.739+0000: 13551: error : virDirOpenInternal:2936 : cannot open directory '/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : storageDriverAutostartCallback:209 : internal error: Failed to autostart storage pool '15023c8a-e3a7-4851-a97d-3b90996b423b': cannot open directory '/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : virDirOpenInternal:2936 : cannot open directory '/var/tmp/localvmdRIozH': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : storageDriverAutostartCallback:209 : internal error: Failed to autostart storage pool 'localvmdRIozH': cannot open directory '/var/tmp/localvmdRIozH': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : virDirOpenInternal:2936 : cannot open directory '/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : storageDriverAutostartCallback:209 : internal error: Failed to autostart storage pool '15023c8a-e3a7-4851-a97d-3b90996b423b-1': cannot open directory '/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : virDirOpenInternal:2936 : cannot open directory '/var/tmp/localvmgmyYik': No such file or directory févr. 25 18:47:34 vs-inf-int-kvm-fr-301-210.hostics.fr libvirtd[13535]: 2019-02-25 17:47:34.740+0000: 13551: error : storageDriverAutostartCallback:209 : internal error: Failed to autostart storage pool 'localvmgmyYik': cannot open directory '/var/tmp/localvmgmyYik': No such file or directory /var/log/libvirt/qemu/HostedEngineLocal.log : 2019-02-25 17:50:08.694+0000: starting up libvirt version: 4.5.0, package: 10.el7_6.4 (CentOS BuildSystem <http://bugs.centos.org>;, 2019-01-29-17:31:22, x86-01.bsys.centos.org), qemu version: 2.12.0qemu-kvm-ev-2.12.0-18.el7_6.3.1, kernel: 3.10.0-957.5.1.el7.x86_64, hostname: vs-inf-int-kvm-fr-301-210.hostics.fr LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 8ba608c8-b721-4b5b-b839-b62f5e919814 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=off,strict=on -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/tmp/localvmlF5yTM/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/localvmlF5yTM/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:1d:4b:b6,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=31,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -vnc 127.0.0.1:0 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on 2019-02-25T17:50:08.904663Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/4 (label charserial0) 2019-02-25T17:50:08.911239Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10] 2019-02-25T17:50:08.917723Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10] 2019-02-25T17:50:08.918494Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10] 2019-02-25T17:50:08.919217Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10] I guess there is something about those last warnings? It should be noted that I previously successfully deployed oVirt 4.2 in the same Nested environment Running libvirt in debug mode will need to wait tomorrow ; my night is already cut to nothing much anymore XD Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Tue, Feb 26, 2019 at 3:33 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Feb 25, 2019 at 7:15 PM Guillaume Pavese < > guillaume.pavese(a)interactiv-group.com&gt; wrote: > >> No, as indicated previously, still : >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >> Expiry Time MAC address Protocol IP address >> Hostname Client ID or DUID >> >> ------------------------------------------------------------------------------------------------------------------- >> >> [root@vs-inf-int-kvm-fr-301-210 ~]# >> >> >> I did not see any relevant log on the HE vm. Is there something I should >> look for there? >> > > This smells really bad: I'd suggest to check /var/log/messages > and /var/log/libvirt/qemu/HostedEngineLocal.log for libvirt errors; > if nothing is there can I ask you to try reexecuting with libvirt debug > logs (edit /etc/libvirt/libvirtd.conf). > > Honestly I'm not able to reproduce it on my side. > > >> >> >> Guillaume Pavese >> Ingénieur Système et Réseau >> Interactiv-Group >> >> >> On Tue, Feb 26, 2019 at 3:12 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Mon, Feb 25, 2019 at 7:04 PM Guillaume Pavese < >>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>> >>>> I still can't connect with VNC remotely but locally with X forwarding >>>> it works. >>>> However my connection has too high latency for that to be usable (I'm >>>> in Japan, my hosts in France, ~250 ms ping) >>>> >>>> But I could see that the VM is booted! >>>> >>>> and in Hosts logs there is : >>>> >>>> févr. 25 18:51:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14719]: >>>> ansible-command Invoked with warn=True executable=None _uses_shell=True >>>> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >>>> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >>>> chdir=None stdin=None >>>> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >>>> dnsmasq-dhcp[6310]: DHCPDISCOVER(virbr0) 00:16:3e:1d:4b:b6 >>>> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >>>> dnsmasq-dhcp[6310]: DHCPOFFER(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >>>> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >>>> dnsmasq-dhcp[6310]: DHCPREQUEST(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >>>> févr. 25 18:51:30 vs-inf-int-kvm-fr-301-210.hostics.fr >>>> dnsmasq-dhcp[6310]: DHCPACK(virbr0) 192.168.122.14 00:16:3e:1d:4b:b6 >>>> vs-inf-int-ovt-fr-301-210 >>>> févr. 25 18:51:42 vs-inf-int-kvm-fr-301-210.hostics.fr python[14757]: >>>> ansible-command Invoked with warn=True executable=None _uses_shell=True >>>> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >>>> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >>>> chdir=None stdin=None >>>> févr. 25 18:52:12 vs-inf-int-kvm-fr-301-210.hostics.fr python[14789]: >>>> ansible-command Invoked with warn=True executable=None _uses_shell=True >>>> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >>>> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >>>> chdir=None stdin=None >>>> févr. 25 18:52:43 vs-inf-int-kvm-fr-301-210.hostics.fr python[14818]: >>>> ansible-command Invoked with warn=True executable=None _uses_shell=True >>>> _raw_params=virsh -r net-dhcp-leases default | grep -i 00:16:3e:1d:4b:b6 | >>>> awk '{ print $5 }' | cut -f1 -d'/' removes=None argv=None creates=None >>>> chdir=None stdin=None >>>> .... >>>> >>>> ssh to the vm works too : >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# ssh root(a)192.168.122.14 >>>> The authenticity of host '192.168.122.14 (192.168.122.14)' can't be >>>> established. >>>> ECDSA key fingerprint is >>>> SHA256:+/pUzTGVA4kCyICb7XgqrxWYYkqzmDjVmdAahiBFgOQ. >>>> ECDSA key fingerprint is >>>> MD5:4b:ef:ff:4a:7c:1a:af:c2:af:4a:0f:14:a3:c5:31:fb. >>>> Are you sure you want to continue connecting (yes/no)? yes >>>> Warning: Permanently added '192.168.122.14' (ECDSA) to the list of >>>> known hosts. >>>> root(a)192.168.122.14&#39;s password: >>>> [root@vs-inf-int-ovt-fr-301-210 ~]# >>>> >>>> >>>> But the test that the playbook tries still fails with empty result : >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >>>> Expiry Time MAC address Protocol IP address >>>> Hostname Client ID or DUID >>>> >>>> ------------------------------------------------------------------------------------------------------------------- >>>> >>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>> >>>> >>> This smells like a bug to me: >>> and nothing at all in the output of >>> virsh -r net-dhcp-leases default >>> >>> ? >>> >>> >>>> >>>> >>>> >>>> Guillaume Pavese >>>> Ingénieur Système et Réseau >>>> Interactiv-Group >>>> >>>> >>>> On Tue, Feb 26, 2019 at 1:54 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Feb 25, 2019 at 5:50 PM Guillaume Pavese < >>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>> >>>>>> I did that but no success yet. >>>>>> >>>>>> I see that "Get local VM IP" task tries the following : >>>>>> >>>>>> virsh -r net-dhcp-leases default | grep -i {{ he_vm_mac_addr }} | >>>>>> awk '{ print $5 }' | cut -f1 -d'/' >>>>>> >>>>>> >>>>>> However while the task is running, and vm running in qemu, "virsh -r >>>>>> net-dhcp-leases default" never returns anything : >>>>>> >>>>> >>>>> Yes, I think that libvirt will never provide a DHCP lease since the >>>>> appliance OS never correctly complete the boot process. >>>>> I'd suggest to connect to the running VM via vnc DURING the boot >>>>> process and check what's wrong. >>>>> >>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# virsh -r net-dhcp-leases default >>>>>> Expiry Time MAC address Protocol IP address >>>>>> Hostname Client ID or DUID >>>>>> >>>>>> ------------------------------------------------------------------------------------------------------------------- >>>>>> >>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Guillaume Pavese >>>>>> Ingénieur Système et Réseau >>>>>> Interactiv-Group >>>>>> >>>>>> >>>>>> On Tue, Feb 26, 2019 at 12:33 AM Simone Tiraboschi < >>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>> >>>>>>> OK, try this: >>>>>>> temporary >>>>>>> edit /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/bootstrap_local_vm/02_create_local_vm.yml >>>>>>> around line 120 >>>>>>> and edit tasks "Get local VM IP" >>>>>>> changing from "retries: 50" to "retries: 500" so that you have >>>>>>> more time to debug it >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 25, 2019 at 4:20 PM Guillaume Pavese < >>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>> >>>>>>>> I retried after killing the remaining qemu process and >>>>>>>> doing ovirt-hosted-engine-cleanup >>>>>>>> The new attempt failed again at the same step. Then after it >>>>>>>> fails, it cleans the temporary files (and vm disk) but *qemu >>>>>>>> still runs!* : >>>>>>>> >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>> >>>>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, >>>>>>>> "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i >>>>>>>> 00:16:3e:6c:e8:f9 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>> "0:00:00.092436", "end": "2019-02-25 16:09:38.863263", "rc": 0, "start": >>>>>>>> "2019-02-25 16:09:38.770827", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>> "", "stdout_lines": []} >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] >>>>>>>> [ INFO ] ok: [localhost] >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>>>>>> [ INFO ] changed: [localhost] >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry >>>>>>>> in /etc/hosts for the local VM] >>>>>>>> [ INFO ] ok: [localhost] >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about >>>>>>>> a failure] >>>>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>>>>>> "The system may not be provisioned according to the playbook results: >>>>>>>> please check the logs for the issue, fix accordingly or re-deploy from >>>>>>>> scratch.\n"} >>>>>>>> [ ERROR ] Failed to execute stage 'Closing up': Failed executing >>>>>>>> ansible-playbook >>>>>>>> [ INFO ] Stage: Clean up >>>>>>>> [ INFO ] Cleaning temporary resources >>>>>>>> ... >>>>>>>> >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] >>>>>>>> [ INFO ] ok: [localhost] >>>>>>>> [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry >>>>>>>> in /etc/hosts for the local VM] >>>>>>>> [ INFO ] ok: [localhost] >>>>>>>> [ INFO ] Generating answer file >>>>>>>> '/var/lib/ovirt-hosted-engine-setup/answers/answers-20190225161011.conf' >>>>>>>> [ INFO ] Stage: Pre-termination >>>>>>>> [ INFO ] Stage: Termination >>>>>>>> [ ERROR ] Hosted Engine deployment failed: please check the logs >>>>>>>> for the issue, fix accordingly or re-deploy from scratch. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# ps aux | grep qemu >>>>>>>> root 4021 0.0 0.0 24844 1788 ? Ss févr.22 0:00 >>>>>>>> /usr/bin/qemu-ga --method=virtio-serial >>>>>>>> --path=/dev/virtio-ports/org.qemu.guest_agent.0 >>>>>>>> --blacklist=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status >>>>>>>> -F/etc/qemu-ga/fsfreeze-hook >>>>>>>> qemu 26463 22.9 4.8 17684512 1088844 ? Sl 16:01 3:09 >>>>>>>> /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on -S >>>>>>>> -object >>>>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>>>> 4,sockets=4,cores=1,threads=1 -uuid 316eca5f-81de-4a0b-af1f-58f910402a8e >>>>>>>> -no-user-config -nodefaults -chardev >>>>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>>>> menu=off,strict=on -device >>>>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>>>> file=/var/tmp/localvmdRIozH/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>>>> -device >>>>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>>>> -drive >>>>>>>> file=/var/tmp/localvmdRIozH/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:6c:e8:f9,bus=pci.0,addr=0x3 >>>>>>>> -chardev pty,id=charserial0 -device >>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>>>> -vnc 127.0.0.1:0 -device >>>>>>>> VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -object >>>>>>>> rng-random,id=objrng0,filename=/dev/random -device >>>>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>>>> -msg timestamp=on >>>>>>>> root 28416 0.0 0.0 112712 980 pts/3 S+ 16:14 0:00 >>>>>>>> grep --color=auto qemu >>>>>>>> >>>>>>>> >>>>>>>> Before the first Error, while the vm was running for sure and the >>>>>>>> disk was there, I also unsuccessfuly tried to connect to it with VNC and >>>>>>>> got the same error I got before : >>>>>>>> >>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# debug1: Connection to port >>>>>>>> 5900 forwarding to vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 >>>>>>>> requested. >>>>>>>> debug1: channel 3: new [direct-tcpip] >>>>>>>> channel 3: open failed: connect failed: Connection refused >>>>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>>>> 127.0.0.1 port 37002 to 127.0.0.1 port 5900, nchannels 4 >>>>>>>> >>>>>>>> >>>>>>>> Guillaume Pavese >>>>>>>> Ingénieur Système et Réseau >>>>>>>> Interactiv-Group >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Feb 25, 2019 at 11:57 PM Guillaume Pavese < >>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>> >>>>>>>>> Something was definitely wrong ; as indicated, qemu process >>>>>>>>> for guest=HostedEngineLocal was running but the disk file did not exist >>>>>>>>> anymore... >>>>>>>>> No surprise I could not connect >>>>>>>>> >>>>>>>>> I am retrying >>>>>>>>> >>>>>>>>> >>>>>>>>> Guillaume Pavese >>>>>>>>> Ingénieur Système et Réseau >>>>>>>>> Interactiv-Group >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 25, 2019 at 11:15 PM Guillaume Pavese < >>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>> >>>>>>>>>> It fails too : >>>>>>>>>> I made sure PermitTunnel=yes in sshd config but when I try to >>>>>>>>>> connect to the forwarded port I get the following error on the openened >>>>>>>>>> host ssh session : >>>>>>>>>> >>>>>>>>>> [gpavese@sheepora-X230 ~]$ ssh -v -L 5900: >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>>> ... >>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# >>>>>>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>>>>>> ::1 port 42144 to ::1 port 5900, nchannels 4 >>>>>>>>>> debug1: Connection to port 5900 forwarding to >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900 requested. >>>>>>>>>> debug1: channel 3: new [direct-tcpip] >>>>>>>>>> channel 3: open failed: connect failed: Connection refused >>>>>>>>>> debug1: channel 3: free: direct-tcpip: listening port 5900 for >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900, connect from >>>>>>>>>> 127.0.0.1 port 32778 to 127.0.0.1 port 5900, nchannels 4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and in journalctl : >>>>>>>>>> >>>>>>>>>> févr. 25 14:55:38 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>>> sshd[19595]: error: connect_to >>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr port 5900: failed. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Guillaume Pavese >>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>> Interactiv-Group >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 25, 2019 at 10:44 PM Simone Tiraboschi < >>>>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 25, 2019 at 2:35 PM Guillaume Pavese < >>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> I made sure of everything and even stopped firewalld but still >>>>>>>>>>>> can't connect : >>>>>>>>>>>> >>>>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>>>>>> /var/run/libvirt/qemu/HostedEngineLocal.xml >>>>>>>>>>>> <graphics type='vnc' port='*5900*' autoport='yes' >>>>>>>>>>>> *listen='127.0.0.1*'> >>>>>>>>>>>> <listen type='address' address='*127.0.0.1*' >>>>>>>>>>>> fromConfig='1' autoGenerated='no'/> >>>>>>>>>>>> >>>>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# netstat -pan | grep 59 >>>>>>>>>>>> tcp 0 0 127.0.0.1:5900 0.0.0.0:* >>>>>>>>>>>> LISTEN 13376/qemu-kvm >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I suggest to try ssh tunneling, run >>>>>>>>>>> ssh -L 5900:vs-inf-int-kvm-fr-301-210.hostics.fr:5900 >>>>>>>>>>> root(a)vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>>>> >>>>>>>>>>> and then >>>>>>>>>>> remote-viewer vnc://localhost:5900 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~]# systemctl status >>>>>>>>>>>> firewalld.service >>>>>>>>>>>> ● firewalld.service - firewalld - dynamic firewall daemon >>>>>>>>>>>> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; >>>>>>>>>>>> enabled; vendor preset: enabled) >>>>>>>>>>>> *Active: inactive (dead)* >>>>>>>>>>>> *févr. 25 14:24:03 vs-inf-int-kvm-fr-301-210.hostics.fr >>>>>>>>>>>> <http://vs-inf-int-kvm-fr-301-210.hostics.fr> systemd[1]: Stopped firewalld >>>>>>>>>>>> - dynamic firewall daemon.* >>>>>>>>>>>> >>>>>>>>>>>> From my laptop : >>>>>>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *5900* >>>>>>>>>>>> Trying 10.199.210.11... >>>>>>>>>>>> [*nothing gets through...*] >>>>>>>>>>>> ^C >>>>>>>>>>>> >>>>>>>>>>>> For making sure : >>>>>>>>>>>> [gpavese@sheepora-X230 ~]$ telnet >>>>>>>>>>>> vs-inf-int-kvm-fr-301-210.hostics.fr *9090* >>>>>>>>>>>> Trying 10.199.210.11... >>>>>>>>>>>> *Connected* to vs-inf-int-kvm-fr-301-210.hostics.fr. >>>>>>>>>>>> Escape character is '^]'. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 25, 2019 at 10:24 PM Parth Dhanjal < >>>>>>>>>>>> dparth(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey! >>>>>>>>>>>>> >>>>>>>>>>>>> You can check under /var/run/libvirt/qemu/HostedEngine.xml >>>>>>>>>>>>> Search for 'vnc' >>>>>>>>>>>>> From there you can look up the port on which the HE VM is >>>>>>>>>>>>> available and connect to the same. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 25, 2019 at 6:47 PM Guillaume Pavese < >>>>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> 1) I am running in a Nested env, but under libvirt/kvm on >>>>>>>>>>>>>> remote Centos 7.4 Hosts >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please advise how to connect with VNC to the local HE vm. I >>>>>>>>>>>>>> see it's running, but this is on a remote host, not my local machine : >>>>>>>>>>>>>> qemu 13376 100 3.7 17679424 845216 ? Sl 12:46 >>>>>>>>>>>>>> 85:08 /usr/libexec/qemu-kvm -name guest=HostedEngineLocal,debug-threads=on >>>>>>>>>>>>>> -S -object >>>>>>>>>>>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes >>>>>>>>>>>>>> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu >>>>>>>>>>>>>> Haswell-noTSX,+kvmclock -m 16384 -realtime mlock=off -smp >>>>>>>>>>>>>> 4,sockets=4,cores=1,threads=1 -uuid 6fe7c1c3-ea93-4343-a385-0d9e14bb563a >>>>>>>>>>>>>> -no-user-config -nodefaults -chardev >>>>>>>>>>>>>> socket,id=charmonitor,fd=27,server,nowait -mon >>>>>>>>>>>>>> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown >>>>>>>>>>>>>> -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>>>>>>>>>>> menu=off,strict=on -device >>>>>>>>>>>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>>>>>>>>>>>> file=/var/tmp/localvmgmyYik/images/15023c8a-e3a7-4851-a97d-3b90996b423b/07fdcff3-11ce-4f7c-af05-0a878593e78e,format=qcow2,if=none,id=drive-virtio-disk0 >>>>>>>>>>>>>> -device >>>>>>>>>>>>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>>>>>>>>>>>> -drive >>>>>>>>>>>>>> file=/var/tmp/localvmgmyYik/seed.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on >>>>>>>>>>>>>> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev >>>>>>>>>>>>>> tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device >>>>>>>>>>>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:3e:fe:28,bus=pci.0,addr=0x3 >>>>>>>>>>>>>> -chardev pty,id=charserial0 -device >>>>>>>>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>>>>>>>>> socket,id=charchannel0,fd=31,server,nowait -device >>>>>>>>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 >>>>>>>>>>>>>> *-vnc 127.0.0.1:0 <http://127.0.0.1:0> -device VGA*,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>>>>>>>>>> -object rng-random,id=objrng0,filename=/dev/random -device >>>>>>>>>>>>>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox >>>>>>>>>>>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>>>>>>>>>>> -msg timestamp=on >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) [root@vs-inf-int-kvm-fr-301-210 ~]# cat >>>>>>>>>>>>>> /etc/libvirt/qemu/networks/default.xml >>>>>>>>>>>>>> <!-- >>>>>>>>>>>>>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE >>>>>>>>>>>>>> LIKELY TO BE >>>>>>>>>>>>>> OVERWRITTEN AND LOST. Changes to this xml configuration >>>>>>>>>>>>>> should be made using: >>>>>>>>>>>>>> virsh net-edit default >>>>>>>>>>>>>> or other application using the libvirt API. >>>>>>>>>>>>>> --> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <network> >>>>>>>>>>>>>> <name>default</name> >>>>>>>>>>>>>> <uuid>ba7bbfc8-28b8-459e-a42d-c2d6218e2cb6</uuid> >>>>>>>>>>>>>> <forward mode='nat'/> >>>>>>>>>>>>>> <bridge name='virbr0' stp='on' delay='0'/> >>>>>>>>>>>>>> <mac address='52:54:00:e5:fe:3b'/> >>>>>>>>>>>>>> <ip address='192.168.122.1' netmask='255.255.255.0'> >>>>>>>>>>>>>> <dhcp> >>>>>>>>>>>>>> <range start='192.168.122.2' end='192.168.122.254'/> >>>>>>>>>>>>>> </dhcp> >>>>>>>>>>>>>> </ip> >>>>>>>>>>>>>> </network> >>>>>>>>>>>>>> You have new mail in /var/spool/mail/root >>>>>>>>>>>>>> [root@vs-inf-int-kvm-fr-301-210 ~] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Feb 25, 2019 at 9:57 PM Simone Tiraboschi < >>>>>>>>>>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Feb 25, 2019 at 1:14 PM Guillaume Pavese < >>>>>>>>>>>>>>> guillaume.pavese(a)interactiv-group.com&gt; wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> He deployment with "hosted-engine --deploy" fails at TASK >>>>>>>>>>>>>>>> [ovirt.hosted_engine_setup : Get local VM IP] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> See following Error : >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2019-02-25 12:46:50,154+0100 INFO >>>>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>>>> ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup : Get >>>>>>>>>>>>>>>> local VM IP] >>>>>>>>>>>>>>>> 2019-02-25 12:55:26,823+0100 DEBUG >>>>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>>>> ansible_utils._process_output:103 {u'_ansible_parsed': True, >>>>>>>>>>>>>>>> u'stderr_lines': [], u'cmd': u"virsh -r net-dhcp-leases default | grep -i 00 >>>>>>>>>>>>>>>> :16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", >>>>>>>>>>>>>>>> u'end': u'2019-02-25 12:55:26.666925', u'_ansible_no_log': False, >>>>>>>>>>>>>>>> u'stdout': u'', u'changed': True, u'invocation': {u'module_args': {u'warn': >>>>>>>>>>>>>>>> True, u'executable': >>>>>>>>>>>>>>>> None, u'_uses_shell': True, u'_raw_params': u"virsh -r >>>>>>>>>>>>>>>> net-dhcp-leases default | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | >>>>>>>>>>>>>>>> cut -f1 -d'/'", u'removes': None, u'argv': None, u'creates': None, >>>>>>>>>>>>>>>> u'chdir': None, u'std >>>>>>>>>>>>>>>> in': None}}, u'start': u'2019-02-25 12:55:26.584686', >>>>>>>>>>>>>>>> u'attempts': 50, u'stderr': u'', u'rc': 0, u'delta': u'0:00:00.082239', >>>>>>>>>>>>>>>> u'stdout_lines': []} >>>>>>>>>>>>>>>> 2019-02-25 12:55:26,924+0100 ERROR >>>>>>>>>>>>>>>> otopi.ovirt_hosted_engine_setup.ansible_utils >>>>>>>>>>>>>>>> ansible_utils._process_output:107 fatal: [localhost]: FAILED! => >>>>>>>>>>>>>>>> {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default >>>>>>>>>>>>>>>> | grep -i 00:16:3e:3e:fe:28 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": >>>>>>>>>>>>>>>> "0:00:00.082239", "end": "2019-02-25 12:55:26.666925", "rc": 0, "start": >>>>>>>>>>>>>>>> "2019-02-25 12:55:26.584686", "stderr": "", "stderr_lines": [], "stdout": >>>>>>>>>>>>>>>> "", "stdout_lines": []} >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here we are just waiting for the bootstrap engine VM to >>>>>>>>>>>>>>> fetch an IP address from default libvirt network over DHCP but it your case >>>>>>>>>>>>>>> it never happened. >>>>>>>>>>>>>>> Possible issues: something went wrong in the bootstrap >>>>>>>>>>>>>>> process for the engine VM or the default libvirt network is not correctly >>>>>>>>>>>>>>> configured. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. can you try to reach the engine VM via VNC and check >>>>>>>>>>>>>>> what's happening there? (another question, are you running it nested? AFAIK >>>>>>>>>>>>>>> it will not work if nested over ESXi) >>>>>>>>>>>>>>> 2. can you please share the output of >>>>>>>>>>>>>>> cat /etc/libvirt/qemu/networks/default.xml >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Guillaume Pavese >>>>>>>>>>>>>>>> Ingénieur Système et Réseau >>>>>>>>>>>>>>>> Interactiv-Group >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>>>>>> Privacy Statement: >>>>>>>>>>>>>>>> https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>>>>> List Archives: >>>>>>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/VXRMU3SQWTM... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>>>> Privacy Statement: >>>>>>>>>>>>>> https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>>> List Archives: >>>>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/45UR44ITQTV... >>>>>>>>>>>>>> >>>>>>>>>>>>>