On Fri, Oct 23, 2015 at 4:56 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Oct 23, 2015 at 4:42 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:

Are ovirt-ha-agent and ovirt-ha-broker up and running?
Can you please try to restart them via systemd?

In the mean time I found inside the logs they failed to start.

I found in broker log the message
Thread-1730::ERROR::2015-10-22 17:31:47,016::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=nfs3 sd_uuid=f53854cd-8767-4011-9564-36dc36e0a5d1'
Traceback (most recent call last):
...
BackendFailureException: path to storage domain f53854cd-8767-4011-9564-36dc36e0a5d1 not found in /rhev/data-center/mnt

so probably the NFS part was not in lace yet when the broker attempted to start?
I saw that actually I had now

[root@ovc71 ovirt-hosted-engine-ha]# ll /rhev/data-center/mnt/ovc71.localdomain.local:_NFS__DOMAIN
total 0
-rwxr-xr-x. 1 vdsm kvm 0 Oct 23 16:46 __DIRECT_IO_TEST__
drwxr-xr-x. 5 vdsm kvm 47 Oct 22 15:49 f53854cd-8767-4011-9564-36dc36e0a5d1

and I was able to run

systemctl start ovirt-ha-broker.service
and verify it correctly started.
and the same for
systemctl start ovirt-ha-agent

after a couple of minutes the sh engine VM was powered on and I was able to access web admin portal.

OK, can you please try again the whole reboot procedure just to ensure that it was just a temporary NFS glitch?

But if I try to connect to its console with

[root@ovc71 ovirt-hosted-engine-ha]# hosted-engine --add-console-password
Enter password:
code = 0
message = 'Done'

and then
# remote-viewer --spice-ca-file=/etc/pki/vdsm/libvirt-spice/ca-cert.pem spice://localhost?tls-port=5900 --spice-host-subject="C=EN, L=Test, O=Test, CN=Test"

** (remote-viewer:7173): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-Gb5xXSKiKK: Connection refused
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
(/usr/bin/remote-viewer:7173): Spice-Warning **: ssl_verify.c:492:openssl_verify: ssl: subject 'C=EN, L=Test, O=Test, CN=Test' verification failed
(/usr/bin/remote-viewer:7173): Spice-Warning **: ssl_verify.c:494:openssl_verify: ssl: verification failed

The issue was here: --spice-host-subject="C=EN, L=Test, O=Test, CN=Test"

This one was just the temporary subject used by hosted-engine-setup during the bootstrap sequence when your engine was still to come.

At the end that cert got replace by the engine CA signed ones and so you have to substitute that subject to match the one you used during your setup.

(remote-viewer:7173): GSpice-WARNING **: main-1:0: SSL_connect: error:00000001:lib(0):func(0):reason(1)

I get an error window with
Unable to connect to the graphic server spice://localhost?tls-port=5900

[root@ovc71 ovirt-hosted-engine-ha]# netstat -tan | grep 5900
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN

the qemu command line of the sh engine is:
qemu 4489 1 23 16:41 ? 00:02:35 /usr/libexec/qemu-kvm -name HostedEngine -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Nehalem -m 8192 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9e654c4a-925c-48ba-9818-6908b7714d3a -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-1.1503.el7.centos.2.8,serial=97F39B57-FA7D-2A47-9E0E-304705DE227D,uuid=9e654c4a-925c-48ba-9818-6908b7714d3a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/HostedEngine.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-10-23T14:41:23,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/run/vdsm/storage/f53854cd-8767-4011-9564-36dc36e0a5d1/45ae3a4a-2190-4494-9419-b7c2af8a7aef/52b97c5b-96ae-4efc-b2e0-f56cde243384,if=none,id=drive-virtio-disk0,format=raw,serial=45ae3a4a-2190-4494-9419-b7c2af8a7aef,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:16:6a:b6,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/9e654c4a-925c-48ba-9818-6908b7714d3a.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/9e654c4a-925c-48ba-9818-6908b7714d3a.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/9e654c4a-925c-48ba-9818-6908b7714d3a.org.ovirt.hosted-engine-setup.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -msg timestamp=on