[ovirt-users] Hosts temporarily in "Non Operational" state after upgrade

[Will Dennis]

Hi all,

Had run updates tonight on my three oVirt hosts (3.6 hyperconverged) on on two of them, they went into “non Operational” state for a few minutes each before springing back to life… The synopsis was this:

- Ran updates throughout the web Admin UI
...then I got the following series of messages via the “Events” tab in the UI:
- Updates successfully ran
- VDSM “command failed: Heartbeat exceeded” message
- host is not responding message
- "Failed to connect to hosted_storage" message
- “The error message for connection localhost:/engine returned by VDSM was: Problem while trying to mount target”
- "Host <name> reports about one of the Active Storage Domains as Problematic”
- “Host <name> cannot access the Storage Domain(s) hosted_storage attached to the data center Default. Setting host state to Non-Operational.”
- "Detected change in status of brick {…} of volume {…} from DOWN to UP.” (once for every brick on the host for every Gluster volume.)
- "Host <name> was autorecovered.”
- "Status of host <name> was set to Up."

(BTW, it would be awesome if the UI’s Events log could be copied and pasted… Doesn’t work for me at least…)

Duration of outage was ~3 mins per each affected host. Didn’t happen on the first host I upgraded, but did on the last two.

I know I’m a little over the bleeding edge running hyperconverged on 3.6 :) but, should this behavior be expected?

Also, if I go onto the hosts directly and run a ‘yum update’ after this upgrade process (not that I went thru with it, just wanted to see what was available to be upgraded) I see a bunch of ovirt-* packages that can be upgraded, which didn’t get updated thru the web UI’s upgrade process —
ovirt-engine-sdk-python            noarch       3.6.5.0-1.el7.centos         ovirt-3.6                      480 k
ovirt-hosted-engine-ha             noarch       1.3.5.3-1.1.el7              centos-ovirt36                 295 k
ovirt-hosted-engine-setup          noarch       1.3.5.0-1.1.el7              centos-ovirt36                 270 k
ovirt-release36                    noarch       007-1                        ovirt-3.6                      9.5 k

Are these packages not related to the “Upgrade” process available thru the web UI?

FYI, here’s what did get updated thru the web UI “Upgrade” process —
Apr 27 21:36:28 Updated: libvirt-client-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: vdsm-infra-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: vdsm-python-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: vdsm-xmlrpc-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: mom-0.5.3-1.1.el7.noarch
Apr 27 21:36:29 Updated: libvirt-lock-sanlock-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-kvm-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: 1:libguestfs-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:29 Updated: 1:libguestfs-tools-c-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:29 Installed: libguestfs-winsupport-7.2-1.el7.x86_64
Apr 27 21:36:29 Updated: vdsm-yajsonrpc-4.17.26-1.el7.noarch
Apr 27 21:36:29 Updated: vdsm-jsonrpc-4.17.26-1.el7.noarch
Apr 27 21:36:29 Installed: unzip-6.0-15.el7.x86_64
Apr 27 21:36:30 Installed: gtk2-2.24.28-8.el7.x86_64
Apr 27 21:36:31 Installed: 1:virt-v2v-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:31 Updated: safelease-1.0-7.el7.x86_64
Apr 27 21:36:31 Updated: vdsm-hook-vmfex-dev-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-gluster-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-cli-4.17.26-1.el7.noarch

Thanks,
Will