[ovirt-users] unable to bring up hosted engine after botched 4.2 upgrade

Yaniv Kaul ykaul at redhat.com
Fri Jan 12 10:17:58 UTC 2018 

On Jan 12, 2018 11:43 AM, "Jayme" <jaymef at gmail.com> wrote:

Please help, I'm really not sure what else to try at this point.  Thank you
for reading!


I'm still working on trying to get my hosted engine running after a botched
upgrade to 4.2.  Storage is NFS mounted from within one of the hosts.  Right
now I have 3 centos7 hosts that are fully



That's not a good idea. What happens when the host fails?
Y.

updated with yum packages from ovirt 4.2, the engine was fully updated with
yum packages and failed to come up after reboot.  As of right now,
everything should have full yum updates and all having 4.2 rpms.  I have
global maintenance mode on right now and started hosted-engine on one of
the three host and the status is currently: Engine status : {"reason":
"failed liveliness check”; "health": "bad", "vm": "up", "detail": "Up"}


this is what I get when trying to enter hosted-vm --console


The engine VM is running on this host

error: failed to get domain 'HostedEngine'

error: Domain not found: no domain with matching name 'HostedEngine'


Here are logs from various sources when I start the VM on HOST3:


hosted-engine --vm-start

Command VM.getStats with args {'vmID': '4013c829-c9d7-4b72-90d5-6fe58137504c'}
failed:

(code=1, message=Virtual machine does not exist: {'vmId':
u'4013c829-c9d7-4b72-90d5-6fe58137504c'})


Jan 11 16:55:57 cultivar3 systemd-machined: New machine qemu-110-Cultivar.

Jan 11 16:55:57 cultivar3 systemd: Started Virtual Machine
qemu-110-Cultivar.

Jan 11 16:55:57 cultivar3 systemd: Starting Virtual Machine
qemu-110-Cultivar.

Jan 11 16:55:57 cultivar3 kvm: 3 guests now active


==> /var/log/vdsm/vdsm.log <==

  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
method

    ret = func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2718,
in getStorageDomainInfo

    dom = self.validateSdUUID(sdUUID)

  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 304, in
validateSdUUID

    sdDom.validate()

  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 515,
in validate

    raise se.StorageDomainAccessError(self.sdUUID)

StorageDomainAccessError: Domain is either partially accessible or entirely
inaccessible: (u'248f46f0-d793-4581-9810-c9d965e2f286',)

jsonrpc/2::ERROR::2018-01-11
16:55:16,144::dispatcher::82::storage.Dispatcher::(wrapper)
FINISH getStorageDomainInfo error=Domain is either partially accessible or
entirely inaccessible: (u'248f46f0-d793-4581-9810-c9d965e2f286',)


==> /var/log/libvirt/qemu/Cultivar.log <==

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
guest=Cultivar,debug-threads=on -S -object secret,id=masterKey0,format=
raw,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes -machine
pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m
8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1
-uuid 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
'type=1,manufacturer=oVirt,product=oVirt Node,version=7-4.1708.el7.
centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
4013c829-c9d7-4b72-90d5-6fe58137504c' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
108-Cultivar/monitor.sock,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control
-rtc base=2018-01-11T20:33:19,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=
pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive
if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev
spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-
serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-chardev socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=4,chardev=
charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -chardev
pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0
-spice tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
tls-channel=default,seamless-migration=on -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
rng-random,id=objrng0,filename=/dev/urandom
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg
timestamp=on

2018-01-11T20:33:19.699999Z qemu-kvm: -chardev pty,id=charconsole0: char
device redirected to /dev/pts/2 (label charconsole0)

2018-01-11 20:38:11.640+0000: shutting down, reason=shutdown

2018-01-11 20:39:02.122+0000: starting up libvirt version: 3.2.0, package:
14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: cultivar3

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
guest=Cultivar,debug-threads=on -S -object secret,id=masterKey0,format=
raw,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes -machine
pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m
8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1
-uuid 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
'type=1,manufacturer=oVirt,product=oVirt Node,version=7-4.1708.el7.
centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
4013c829-c9d7-4b72-90d5-6fe58137504c' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
109-Cultivar/monitor.sock,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control
-rtc base=2018-01-11T20:39:02,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=
pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive
if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev
spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-
serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-chardev socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=4,chardev=
charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -chardev
pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0
-spice tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
tls-channel=default,seamless-migration=on -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
rng-random,id=objrng0,filename=/dev/urandom
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg
timestamp=on

2018-01-11T20:39:02.380773Z qemu-kvm: -chardev pty,id=charconsole0: char
device redirected to /dev/pts/2 (label charconsole0)

2018-01-11 20:53:11.407+0000: shutting down, reason=shutdown

2018-01-11 20:55:57.210+0000: starting up libvirt version: 3.2.0, package:
14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
cultivar3.grove.silverorange.com

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
guest=Cultivar,debug-threads=on -S -object secret,id=masterKey0,format=
raw,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes -machine
pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m
8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1
-uuid 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
'type=1,manufacturer=oVirt,product=oVirt Node,version=7-4.1708.el7.
centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
4013c829-c9d7-4b72-90d5-6fe58137504c' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
110-Cultivar/monitor.sock,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control
-rtc base=2018-01-11T20:55:57,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=
pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive
if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev
spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-
serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-chardev socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=4,chardev=
charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -chardev
pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0
-spice tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
tls-channel=default,seamless-migration=on -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
rng-random,id=objrng0,filename=/dev/urandom
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg
timestamp=on

2018-01-11T20:55:57.468037Z qemu-kvm: -chardev pty,id=charconsole0: char
device redirected to /dev/pts/2 (label charconsole0)


==> /var/log/ovirt-hosted-engine-ha/broker.log <==

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 151, in get_raw_stats

    f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)

OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/
248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/
8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'

StatusStorageThread::ERROR::2018-01-11 16:55:15,761::status_broker::
92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed to read state.

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 88, in run

    self._storage_broker.get_raw_stats()

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 162, in get_raw_stats

    .format(str(e)))

RequestError: failed to read metadata: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-
4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'


==> /var/log/ovirt-hosted-engine-ha/agent.log <==

    result = refresh_method()

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
line 519, in refresh_vm_conf

    content = self._get_file_content_from_shared_storage(VM)

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
line 484, in _get_file_content_from_shared_storage

    config_volume_path = self._get_config_volume_path()

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
line 188, in _get_config_volume_path

    conf_vol_uuid

  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py",
line 358, in get_volume_path

    root=envconst.SD_RUN_DIR,

RuntimeError: Path to volume 4838749f-216d-406b-b245-98d0343fcf7f not found
in /run/vdsm/storag



==> /var/log/vdsm/vdsm.log <==

periodic/42::ERROR::2018-01-11
16:56:11,446::vmstats::260::virt.vmstats::(send_metrics)
VM metrics collection failed

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 197,
in send_metrics

    data[prefix + '.cpu.usage'] = stat['cpuUsage']

KeyError: 'cpuUsage'

_______________________________________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180112/f807e411/attachment.html>

More information about the Users mailing list