<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 23, 2015 at 5:55 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Fri, Oct 23, 2015 at 5:05 PM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><br></div></div></div></blockquote><div><br></div></span><div>OK, can you please try again the whole reboot procedure just to ensure that it was just a temporary NFS glitch?</div></div></div></div></blockquote><div><br></div><div><br></div></span><div>It seems reproducible.</div><div><br></div><div>This time I was able to shutdown the hypervisor without manual power off.</div><div>Only strange thing is that I ran </div><div><br></div><div>shutdown -h now</div><div><br></div><div>and actually the VM at some point (I was able to see that the watchdog stopped...) booted.... ?</div><div><br></div><div>Related lines in messages:</div><div><div>Oct 23 17:33:32 ovc71 systemd: Unmounting RPC Pipe File System...</div><div>Oct 23 17:33:32 ovc71 systemd: Stopping Session 11 of user root.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopped Session 11 of user root.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping user-0.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Removed slice user-0.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping vdsm-dhclient.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Removed slice vdsm-dhclient.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping vdsm.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Removed slice vdsm.slice.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping Sound Card.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopped target Sound Card.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping LVM2 PV scan on device 8:2...</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping LVM2 PV scan on device 8:16...</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping Dump dmesg to /var/log/dmesg...</div><div>Oct 23 17:33:33 ovc71 systemd: Stopped Dump dmesg to /var/log/dmesg.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping Watchdog Multiplexing Daemon...</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping Multi-User System.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopped target Multi-User System.</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping ABRT kernel log watcher...</div><div>Oct 23 17:33:33 ovc71 systemd: Stopping Command Scheduler...</div><div>Oct 23 17:33:33 ovc71 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="690" x-info="<a href="http://www.rsyslog.com" target="_blank">http://www.rsyslog.com</a>"] exiting on signal 15.</div><div>Oct 23 17:36:24 ovc71 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="697" x-info="<a href="http://www.rsyslog.com" target="_blank">http://www.rsyslog.com</a>"] start</div><div>Oct 23 17:36:21 ovc71 journal: Runtime journal is using 8.0M (max 500.0M, leaving 750.0M of free 4.8G, current limit 500.0M).</div><div>Oct 23 17:36:21 ovc71 kernel: Initializing cgroup subsys cpuset</div></div><div><br></div><div><br></div><div>Coming back with the ovrt processes I see:</div><div><br></div><div><div>[root@ovc71 ~]# systemctl status ovirt-ha-broker</div><div>ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker</div><div> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled)</div><div> Active: inactive (dead) since Fri 2015-10-23 17:36:25 CEST; 31s ago</div><div> Process: 849 ExecStop=/usr/lib/systemd/systemd-ovirt-ha-broker stop (code=exited, status=0/SUCCESS)</div><div> Process: 723 ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start (code=exited, status=0/SUCCESS)</div><div> Main PID: 844 (code=exited, status=0/SUCCESS)</div><div> CGroup: /system.slice/ovirt-ha-broker.service</div><div><br></div><div>Oct 23 17:36:24 ovc71.localdomain.local systemd-ovirt-ha-broker[723]: Starting ovirt-ha-broker: [...</div><div>Oct 23 17:36:24 ovc71.localdomain.local systemd[1]: Started oVirt Hosted Engine High Availabili...r.</div><div>Oct 23 17:36:25 ovc71.localdomain.local systemd-ovirt-ha-broker[849]: Stopping ovirt-ha-broker: [...</div><div>Hint: Some lines were ellipsized, use -l to show in full.</div></div><div><br></div><div>ANd</div><div><div>[root@ovc71 ~]# systemctl status nfs-server</div><div>nfs-server.service - NFS server and services</div><div> Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled)</div><div> Active: active (exited) since Fri 2015-10-23 17:36:27 CEST; 1min 9s ago</div><div> Process: 1123 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)</div><div> Process: 1113 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)</div><div> Main PID: 1123 (code=exited, status=0/SUCCESS)</div><div> CGroup: /system.slice/nfs-server.service</div><div><br></div><div>Oct 23 17:36:27 ovc71.localdomain.local systemd[1]: Starting NFS server and services...</div><div>Oct 23 17:36:27 ovc71.localdomain.local systemd[1]: Started NFS server and services.</div></div><div><br></div><div>So it seems that the broker tries to start and fails (17:36:25) before NFS server start phase completes (17:36:27)...?</div><div><br></div><div>Again if I then manually start ha-broker and ha-agent, they start ok and I'm able to become operational again with the sh engine up</div><div><br></div><div>systemd file for broker is this</div><div><br></div><div><div>[Unit]</div><div>Description=oVirt Hosted Engine High Availability Communications Broker</div><div><br></div><div>[Service]</div><div>Type=forking</div><div>EnvironmentFile=-/etc/sysconfig/ovirt-ha-broker</div><div>ExecStart=/usr/lib/systemd/systemd-ovirt-ha-broker start</div><div>ExecStop=/usr/lib/systemd/systemd-ovirt-ha-broker stop</div><div><br></div><div>[Install]</div><div>WantedBy=multi-user.target</div></div><div><br></div><div>Probably inside the [unit] section I should add</div><div>After=nfs-server.service<br></div><div><br></div></div></div></div></blockquote><div><br></div><div>Ok, I understood.</div><div>You are right: the broker was failing cause the NFS storage was not ready cause it was served in loopback and there isn't any explicit service dependency on that. </div><div><br></div><div>We are not imposing it cause generally an NFS shared domain is generally thought to be served from and external system while a loopback NFS is just a degenerate case.</div><div>Simply fix it manually.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>but this should be true only for sh engine configured with NFS.... so to be done at install/setup time?</div><div><br></div><div>If you want I can set this change for my environment and verify...</div><span class=""><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> </div><div><br></div></span><div>The issue was here: --spice-host-subject="C=EN, L=Test, O=Test, CN=Test"</div><div>This one was just the temporary subject used by hosted-engine-setup during the bootstrap sequence when your engine was still to come.</div><div>At the end that cert got replace by the engine CA signed ones and so you have to substitute that subject to match the one you used during your setup.</div><span><div> </div></span></div></div></div></blockquote><div><br></div></span><div>Even using correct certificate I have problem</div><div>On hypervisor</div><div><br></div><div><div>[root@ovc71 ~]# openssl x509 -in /etc/pki/vdsm/libvirt-spice/ca-cert.pem -text | grep Subject</div><div> Subject: C=US, O=localdomain.local, CN=shengine.localdomain.local.75331</div><div> Subject Public Key Info:</div><div> X509v3 Subject Key Identifier: </div></div><div><br></div><div>On engine</div><div><div>[root@shengine ~]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -text | grep Subject</div><div> Subject: C=US, O=localdomain.local, CN=shengine.localdomain.local.75331</div><div> Subject Public Key Info:</div><div> X509v3 Subject Key Identifier: </div></div><div><br></div><div>but</div><div><br></div><div><div>[root@ovc71 ~]# hosted-engine --add-console-password</div><span class=""><div>Enter password: </div><div><span style="white-space:pre-wrap"> </span>code = 0</div><div><span style="white-space:pre-wrap"> </span>message = 'Done'</div></span></div><div><br></div><div><div>[root@ovc71 ~]# remote-viewer --spice-ca-file=/etc/pki/vdsm/libvirt-spice/ca-cert.pem spice://localhost?tls-port=5900 --spice-host-subject="C=US, O=localdomain.local, CN=shengine.localdomain.local.75331"</div></div></div></div></div></blockquote><div><br></div><div>it should be: </div><div>remote-viewer --spice-ca-file=/etc/pki/vdsm/libvirt-spice/ca-cert.pem spice://ovc71.localdomain.local?tls-port=5900 --spice-host-subject="C=US, O=localdomain.local, CN=ovc71.localdomain.local" </div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div><br></div><div>** (remote-viewer:4297): WARNING **: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-Gb5xXSKiKK: Connection refused</div><span class=""><div>GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.</div></span><div>(/usr/bin/remote-viewer:4297): Spice-Warning **: ssl_verify.c:492:openssl_verify: ssl: subject 'C=US, O=localdomain.local, CN=shengine.localdomain.local.75331' verification failed</div><div>(/usr/bin/remote-viewer:4297): Spice-Warning **: ssl_verify.c:494:openssl_verify: ssl: verification failed</div><div><br></div><div>(remote-viewer:4297): GSpice-WARNING **: main-1:0: SSL_connect: error:00000001:lib(0):func(0):reason(1)</div></div><div><br></div><div><br></div><div>and the remote-viewer window with</div><span class=""><div><br></div><div><br></div><div> Unable to connect to the graphic server spice://localhost?tls-port=5900</div></span></div></div></div>
</blockquote></div><br></div></div>