6 a.m.
Hi all,
Had run updates tonight on my three oVirt hosts (3.6 hyperconverged) on on two of them,
they went into “non Operational” state for a few minutes each before springing back to
life… The synopsis was this:
- Ran updates throughout the web Admin UI
...then I got the following series of messages via the “Events” tab in the UI:
- Updates successfully ran
- VDSM “command failed: Heartbeat exceeded” message
- host is not responding message
- "Failed to connect to hosted_storage" message
- “The error message for connection localhost:/engine returned by VDSM was: Problem while
trying to mount target”
- "Host <name> reports about one of the Active Storage Domains as
Problematic”
- “Host <name> cannot access the Storage Domain(s) hosted_storage attached to the
data center Default. Setting host state to Non-Operational.”
- "Detected change in status of brick {…} of volume {…} from DOWN to UP.” (once for
every brick on the host for every Gluster volume.)
- "Host <name> was autorecovered.”
- "Status of host <name> was set to Up."
(BTW, it would be awesome if the UI’s Events log could be copied and pasted… Doesn’t work
for me at least…)
Duration of outage was ~3 mins per each affected host. Didn’t happen on the first host I
upgraded, but did on the last two.
I know I’m a little over the bleeding edge running hyperconverged on 3.6 :) but, should
this behavior be expected?
Also, if I go onto the hosts directly and run a ‘yum update’ after this upgrade process
(not that I went thru with it, just wanted to see what was available to be upgraded) I see
a bunch of ovirt-* packages that can be upgraded, which didn’t get updated thru the web
UI’s upgrade process —
ovirt-engine-sdk-python noarch 3.6.5.0-1.el7.centos ovirt-3.6
480 k
ovirt-hosted-engine-ha noarch 1.3.5.3-1.1.el7
centos-ovirt36 295 k
ovirt-hosted-engine-setup noarch 1.3.5.0-1.1.el7
centos-ovirt36 270 k
ovirt-release36 noarch 007-1 ovirt-3.6
9.5 k
Are these packages not related to the “Upgrade” process available thru the web UI?
FYI, here’s what did get updated thru the web UI “Upgrade” process —
Apr 27 21:36:28 Updated: libvirt-client-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:28 Updated: vdsm-infra-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: vdsm-python-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: vdsm-xmlrpc-4.17.26-1.el7.noarch
Apr 27 21:36:28 Updated: libvirt-daemon-config-nwfilter-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: mom-0.5.3-1.1.el7.noarch
Apr 27 21:36:29 Updated: libvirt-lock-sanlock-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: libvirt-daemon-kvm-1.2.17-13.el7_2.4.x86_64
Apr 27 21:36:29 Updated: 1:libguestfs-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:29 Updated: 1:libguestfs-tools-c-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:29 Installed: libguestfs-winsupport-7.2-1.el7.x86_64
Apr 27 21:36:29 Updated: vdsm-yajsonrpc-4.17.26-1.el7.noarch
Apr 27 21:36:29 Updated: vdsm-jsonrpc-4.17.26-1.el7.noarch
Apr 27 21:36:29 Installed: unzip-6.0-15.el7.x86_64
Apr 27 21:36:30 Installed: gtk2-2.24.28-8.el7.x86_64
Apr 27 21:36:31 Installed: 1:virt-v2v-1.28.1-1.55.el7.centos.2.x86_64
Apr 27 21:36:31 Updated: safelease-1.0-7.el7.x86_64
Apr 27 21:36:31 Updated: vdsm-hook-vmfex-dev-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-gluster-4.17.26-1.el7.noarch
Apr 27 21:36:32 Updated: vdsm-cli-4.17.26-1.el7.noarch
Thanks,
Will