[ovirt-users] unable to bring up hosted engine after botched 4.2 upgrade
Jayme
jaymef at gmail.com
Fri Jan 12 16:47:43 UTC 2018
The lock space issue was an issue I needed to clear but I don't believe it
has resolved the problem. I shutdown agent and broker on all hosts and
disconnected hosted-storage then enabled broker/agent on just one host and
connected storage. I started the VM and actually didn't get any errors in
the logs barely at all which was good to see, however the VM is still not
running:
HOST3:
Engine status : {"reason": "failed liveliness check",
"health": "bad", "vm": "up", "detail": "Up"}
==> /var/log/messages <==
Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled
state
Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode
Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking
state
Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered
forwarding state
Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer
space available
Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar.
Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine
qemu-111-Cultivar.
Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine
qemu-111-Cultivar.
Jan 12 12:42:57 cultivar3 kvm: 3 guests now active
Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535:
error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU
guest agent is not connected
Interestingly though, now I'm seeing this in the logs which may be a new
clue:
==> /var/log/vdsm/vdsm.log <==
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 126,
in findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 116,
in findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'248f46f0-d793-4581-9810-c9d965e2f286',)
jsonrpc/4::ERROR::2018-01-12
12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH
getStorageDomainInfo error=Storage domain does not exist:
(u'248f46f0-d793-4581-9810-c9d965e2f286',)
periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo)
failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
directory'Is the Hosted Engine setup finished?
periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo)
failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
directory'Is the Hosted Engine setup finished?
periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo)
failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
directory'Is the Hosted Engine setup finished?
periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo)
failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
directory'Is the Hosted Engine setup finished?
==> /var/log/ovirt-hosted-engine-ha/broker.log <==
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 151, in get_raw_stats
f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)
OSError: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
StatusStorageThread::ERROR::2018-01-12
12:32:06,049::status_broker::92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed to read state.
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 88, in run
self._storage_broker.get_raw_stats()
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 162, in get_raw_stats
.format(str(e)))
RequestError: failed to read metadata: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak at redhat.com> wrote:
> The lock is the issue.
>
> - try running sanlock client status on all hosts
> - also make sure you do not have some forgotten host still connected
> to the lockspace, but without ha daemons running (and with the VM)
>
> I need to go to our president election now, I might check the email
> later tonight.
>
> Martin
>
> On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef at gmail.com> wrote:
> > Here are the newest logs from me trying to start hosted vm:
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> blocking
> > state
> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> disabled
> > state
> > Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered promiscuous mode
> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> blocking
> > state
> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> > forwarding state
> > Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No buffer
> space
> > available
> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8715]
> > manager: (vnet4): new Tun device
> > (/org/freedesktop/NetworkManager/Devices/140)
> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8795]
> > device (vnet4): state change: unmanaged -> unavailable (reason
> > 'connection-assumed') [10 20 41]
> >
> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> > 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0,
> package:
> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
> > cultivar0.grove.silverorange.com
> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> > guest=Cultivar,debug-threads=on -S -object
> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
> qemu/domain-119-Cultivar/master-key.aes
> > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> > Conroe -m 8192 -realtime mlock=off -smp
> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> > 'type=1,manufacturer=oVirt,product=oVirt
> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-
> 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
> > -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
> 119-Cultivar/monitor.sock,server,nowait
> > -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2018-01-12T15:58:14,driftfix=slew -global
> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
> -device
> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
> fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
> serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
> werror=stop,rerror=stop,aio=threads
> > -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-
> virtio-disk0,id=virtio-disk0,bootindex=1
> > -drive if=none,id=drive-ide0-1-0,readonly=on -device
> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:
> 7f:d6:83,bus=pci.0,addr=0x3
> > -chardev
> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> > -chardev
> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> > -chardev spicevmc,id=charchannel2,name=vdagent -device
> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> > -chardev
> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-
> engine-setup.0,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> > -chardev pty,id=charconsole0 -device
> > virtconsole,chardev=charconsole0,id=console0 -spice
> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
> tls-channel=default,seamless-migration=on
> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
> > rng-random,id=objrng0,filename=/dev/urandom -device
> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8807]
> > device (vnet4): state change: unavailable -> disconnected (reason 'none')
> > [20 30 0]
> > Jan 12 11:58:14 cultivar0 systemd-machined: New machine
> qemu-119-Cultivar.
> > Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine
> > qemu-119-Cultivar.
> > Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine
> > qemu-119-Cultivar.
> >
> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> > 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev pty,id=charconsole0: char
> > device redirected to /dev/pts/1 (label charconsole0)
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:15 cultivar0 kvm: 5 guests now active
> >
> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> > 2018-01-12 15:58:15.217+0000: shutting down, reason=failed
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: 1908:
> > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to
> acquire
> > lock: Lease is held by another host
> >
> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> > 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from pid
> 1773
> > (/usr/sbin/libvirtd)
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> disabled
> > state
> > Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode
> > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> disabled
> > state
> > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> [1515772695.2348]
> > device (vnet4): state change: disconnected -> unmanaged (reason
> 'unmanaged')
> > [30 10 3]
> > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> [1515772695.2349]
> > device (vnet4): released from master device ovirtmgmt
> > Jan 12 11:58:15 cultivar0 kvm: 4 guests now active
> > Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar
> > terminated.
> >
> > ==> /var/log/vdsm/vdsm.log <==
> > vm/4013c829::ERROR::2018-01-12
> > 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm)
> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process
> failed
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, in
> > _startUnderlyingVm
> > self._run()
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, in
> > _run
> > dom.createWithFlags(flags)
> > File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
> line
> > 126, in wrapper
> > ret = f(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in
> > wrapper
> > return func(inst, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in
> > createWithFlags
> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> failed',
> > dom=self)
> > libvirtError: resource busy: Failed to acquire lock: Lease is held by
> > another host
> > jsonrpc/6::ERROR::2018-01-12
> > 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_handle_request)
> > Internal server error
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> 606,
> > in _handle_request
> > res = method(**params)
> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201,
> in
> > _dynamicMethod
> > result = fn(*methodArgs)
> > File "<string>", line 2, in getAllVmIoTunePolicies
> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48,
> in
> > method
> > ret = func(*args, **kwargs)
> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
> > getAllVmIoTunePolicies
> > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
> > getAllVmIoTunePolicies
> > 'current_values': v.getIoTune()}
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
> > getIoTune
> > result = self.getIoTuneResponse()
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
> > getIoTuneResponse
> > res = self._dom.blockIoTune(
> > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
> 47,
> > in __getattr__
> > % self.vmid)
> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not
> defined
> > yet or was undefined
> >
> > ==> /var/log/messages <==
> > Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR
> Internal
> > server error#012Traceback (most recent call last):#012 File
> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in
> > _handle_request#012 res = method(**params)#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
> > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>",
> line 2,
> > in getAllVmIoTunePolicies#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
> > method#012 ret = func(*args, **kwargs)#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
> > getAllVmIoTunePolicies#012 io_tune_policies_dict =
> > self._cif.getAllVmIoTunePolicies()#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
> > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
> > getIoTune#012 result = self.getIoTuneResponse()#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File
> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in
> > __getattr__#012 % self.vmid)#012NotConnectedError: VM
> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was
> undefined
> >
> > On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef at gmail.com> wrote:
> >>
> >> One other tidbit I noticed is that it seems like there are less errors
> if
> >> I started in paused mode:
> >>
> >> but still shows: Engine status : {"reason": "bad vm
> >> status", "health": "bad", "vm": "up", "detail": "Paused"}
> >>
> >> ==> /var/log/messages <==
> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >> blocking state
> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >> disabled state
> >> Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered promiscuous mode
> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >> blocking state
> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >> forwarding state
> >> Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No buffer
> >> space available
> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
> [1515772505.3625]
> >> manager: (vnet4): new Tun device
> >> (/org/freedesktop/NetworkManager/Devices/139)
> >>
> >> ==> /var/log/libvirt/qemu/Cultivar.log <==
> >> 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0,
> package:
> >> 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
> >> 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
> >> 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
> >> cultivar0.grove.silverorange.com
> >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> >> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> >> guest=Cultivar,debug-threads=on -S -object
> >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
> qemu/domain-118-Cultivar/master-key.aes
> >> -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> >> Conroe -m 8192 -realtime mlock=off -smp
> >> 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> >> 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> >> 'type=1,manufacturer=oVirt,product=oVirt
> >> Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-
> 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
> >> -no-user-config -nodefaults -chardev
> >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
> 118-Cultivar/monitor.sock,server,nowait
> >> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> >> base=2018-01-12T15:55:05,driftfix=slew -global
> >> kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
> -device
> >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
> >> file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
> fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
> serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
> werror=stop,rerror=stop,aio=threads
> >> -device
> >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-
> virtio-disk0,id=virtio-disk0,bootindex=1
> >> -drive if=none,id=drive-ide0-1-0,readonly=on -device
> >> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> >> tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:
> 7f:d6:83,bus=pci.0,addr=0x3
> >> -chardev
> >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> >> -device
> >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> >> -chardev
> >> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
> >> -device
> >> virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> >> -chardev spicevmc,id=charchannel2,name=vdagent -device
> >> virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> >> -chardev
> >> socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-
> engine-setup.0,server,nowait
> >> -device
> >> virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> >> -chardev pty,id=charconsole0 -device
> >> virtconsole,chardev=charconsole0,id=console0 -spice
> >> tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
> tls-channel=default,seamless-migration=on
> >> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
> >> rng-random,id=objrng0,filename=/dev/urandom -device
> >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on
> >>
> >> ==> /var/log/messages <==
> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
> [1515772505.3689]
> >> device (vnet4): state change: unmanaged -> unavailable (reason
> >> 'connection-assumed') [10 20 41]
> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
> [1515772505.3702]
> >> device (vnet4): state change: unavailable -> disconnected (reason
> 'none')
> >> [20 30 0]
> >> Jan 12 11:55:05 cultivar0 systemd-machined: New machine
> qemu-118-Cultivar.
> >> Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine
> >> qemu-118-Cultivar.
> >> Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine
> >> qemu-118-Cultivar.
> >>
> >> ==> /var/log/libvirt/qemu/Cultivar.log <==
> >> 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev pty,id=charconsole0: char
> >> device redirected to /dev/pts/1 (label charconsole0)
> >>
> >> ==> /var/log/messages <==
> >> Jan 12 11:55:05 cultivar0 kvm: 5 guests now active
> >>
> >> On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef at gmail.com> wrote:
> >>>
> >>> Yeah I am in global maintenance:
> >>>
> >>> state=GlobalMaintenance
> >>>
> >>> host0: {"reason": "vm not running on this host", "health": "bad",
> "vm":
> >>> "down", "detail": "unknown"}
> >>> host2: {"reason": "vm not running on this host", "health": "bad", "vm":
> >>> "down", "detail": "unknown"}
> >>> host3: {"reason": "vm not running on this host", "health": "bad", "vm":
> >>> "down", "detail": "unknown"}
> >>>
> >>> I understand the lock is an issue, I'll try to make sure it is fully
> >>> stopped on all three before starting but I don't think that is the
> issue at
> >>> hand either. What concerns me is mostly that it seems to be unable
> to read
> >>> the meta data, I think that might be the heart of the problem but I'm
> not
> >>> sure what is causing it.
> >>>
> >>> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak <msivak at redhat.com>
> wrote:
> >>>>
> >>>> > On all three hosts I ran hosted-engine --vm-shutdown; hosted-engine
> >>>> > --vm-poweroff
> >>>>
> >>>> Are you in global maintenance? I think you were in one of the previous
> >>>> emails, but worth checking.
> >>>>
> >>>> > I started ovirt-ha-broker with systemctl as root user but it does
> >>>> > appear to be running under vdsm:
> >>>>
> >>>> That is the correct behavior.
> >>>>
> >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held
> by
> >>>> > another host
> >>>>
> >>>> sanlock seems to think the VM runs somewhere and it is possible that
> >>>> some other host tried to start the VM as well unless you are in global
> >>>> maintenance (that is why I asked the first question here).
> >>>>
> >>>> Martin
> >>>>
> >>>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef at gmail.com> wrote:
> >>>> > Martin,
> >>>> >
> >>>> > Thanks so much for keeping with me, this is driving me crazy! I
> >>>> > really do
> >>>> > appreciate it, thanks again
> >>>> >
> >>>> > Let's go through this:
> >>>> >
> >>>> > HE VM is down - YES
> >>>> >
> >>>> >
> >>>> > HE agent fails when opening metadata using the symlink - YES
> >>>> >
> >>>> >
> >>>> > the symlink is there and readable by vdsm:kvm - it appears to be:
> >>>> >
> >>>> >
> >>>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20
> >>>> > 14a20941-1b84-4b82-be8f-ace38d7c037a
> >>>> > ->
> >>>> >
> >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_hosted__engine/248f46f0-d793-4581-9810-
> c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
> >>>> >
> >>>> >
> >>>> > And the files in the linked directory exist and have vdsm:kvm perms
> as
> >>>> > well:
> >>>> >
> >>>> >
> >>>> > # cd
> >>>> >
> >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_hosted__engine/248f46f0-d793-4581-9810-
> c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
> >>>> >
> >>>> > [root at cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al
> >>>> >
> >>>> > total 2040
> >>>> >
> >>>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 .
> >>>> >
> >>>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 ..
> >>>> >
> >>>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19
> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
> >>>> >
> >>>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016
> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease
> >>>> >
> >>>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016
> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta
> >>>> >
> >>>> >
> >>>> > I started ovirt-ha-broker with systemctl as root user but it does
> >>>> > appear to
> >>>> > be running under vdsm:
> >>>> >
> >>>> >
> >>>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18
> >>>> > /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker
> >>>> >
> >>>> >
> >>>> >
> >>>> > Here is something I tried:
> >>>> >
> >>>> >
> >>>> > - On all three hosts I ran hosted-engine --vm-shutdown;
> hosted-engine
> >>>> > --vm-poweroff
> >>>> >
> >>>> > - On HOST0 (cultivar0) I disconnected and reconnected storage using
> >>>> > hosted-engine
> >>>> >
> >>>> > - Tried starting up the hosted VM on cultivar0 while tailing the
> logs:
> >>>> >
> >>>> >
> >>>> > # hosted-engine --vm-start
> >>>> >
> >>>> > VM exists and is down, cleaning up and restarting
> >>>> >
> >>>> >
> >>>> >
> >>>> > ==> /var/log/vdsm/vdsm.log <==
> >>>> >
> >>>> > jsonrpc/2::ERROR::2018-01-12
> >>>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats)
> >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm
> stats
> >>>> >
> >>>> > Traceback (most recent call last):
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> 1762,
> >>>> > in
> >>>> > _getRunningVmStats
> >>>> >
> >>>> > vm_sample.interval)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
> line
> >>>> > 45, in
> >>>> > produce
> >>>> >
> >>>> > networks(vm, stats, first_sample, last_sample, interval)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
> line
> >>>> > 322, in
> >>>> > networks
> >>>> >
> >>>> > if nic.name.startswith('hostdev'):
> >>>> >
> >>>> > AttributeError: name
> >>>> >
> >>>> > jsonrpc/3::ERROR::2018-01-12
> >>>> > 11:27:27,221::__init__::611::jsonrpc.JsonRpcServer::(_
> handle_request)
> >>>> > Internal server error
> >>>> >
> >>>> > Traceback (most recent call last):
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> line
> >>>> > 606,
> >>>> > in _handle_request
> >>>> >
> >>>> > res = method(**params)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line
> >>>> > 201, in
> >>>> > _dynamicMethod
> >>>> >
> >>>> > result = fn(*methodArgs)
> >>>> >
> >>>> > File "<string>", line 2, in getAllVmIoTunePolicies
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
> 48,
> >>>> > in
> >>>> > method
> >>>> >
> >>>> > ret = func(*args, **kwargs)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354,
> in
> >>>> > getAllVmIoTunePolicies
> >>>> >
> >>>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line
> 524,
> >>>> > in
> >>>> > getAllVmIoTunePolicies
> >>>> >
> >>>> > 'current_values': v.getIoTune()}
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> 3481,
> >>>> > in
> >>>> > getIoTune
> >>>> >
> >>>> > result = self.getIoTuneResponse()
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> 3500,
> >>>> > in
> >>>> > getIoTuneResponse
> >>>> >
> >>>> > res = self._dom.blockIoTune(
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
> line
> >>>> > 47,
> >>>> > in __getattr__
> >>>> >
> >>>> > % self.vmid)
> >>>> >
> >>>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was
> not
> >>>> > defined
> >>>> > yet or was undefined
> >>>> >
> >>>> >
> >>>> > ==> /var/log/messages <==
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR
> >>>> > Internal
> >>>> > server error#012Traceback (most recent call last):#012 File
> >>>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> 606, in
> >>>> > _handle_request#012 res = method(**params)#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
> >>>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>",
> >>>> > line 2,
> >>>> > in getAllVmIoTunePolicies#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
> >>>> > method#012 ret = func(*args, **kwargs)#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
> >>>> > getAllVmIoTunePolicies#012 io_tune_policies_dict =
> >>>> > self._cif.getAllVmIoTunePolicies()#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
> >>>> > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012
> >>>> > File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
> >>>> > getIoTune#012 result = self.getIoTuneResponse()#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
> >>>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File
> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
> 47, in
> >>>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM
> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was
> >>>> > undefined
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > blocking
> >>>> > state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > disabled
> >>>> > state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered promiscuous
> >>>> > mode
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > blocking
> >>>> > state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > forwarding state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No
> buffer
> >>>> > space
> >>>> > available
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
> >>>> > [1515770847.4264]
> >>>> > manager: (vnet4): new Tun device
> >>>> > (/org/freedesktop/NetworkManager/Devices/135)
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
> >>>> > [1515770847.4342]
> >>>> > device (vnet4): state change: unmanaged -> unavailable (reason
> >>>> > 'connection-assumed') [10 20 41]
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
> >>>> > [1515770847.4353]
> >>>> > device (vnet4): state change: unavailable -> disconnected (reason
> >>>> > 'none')
> >>>> > [20 30 0]
> >>>> >
> >>>> >
> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> >>>> >
> >>>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: 3.2.0,
> >>>> > package:
> >>>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
> >>>> > cultivar0.grove.silverorange.com
> >>>> >
> >>>> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> >>>> > guest=Cultivar,debug-threads=on -S -object
> >>>> >
> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
> qemu/domain-114-Cultivar/master-key.aes
> >>>> > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off
> >>>> > -cpu
> >>>> > Conroe -m 8192 -realtime mlock=off -smp
> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
> >>>> >
> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-
> 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
> >>>> > -no-user-config -nodefaults -chardev
> >>>> >
> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
> 114-Cultivar/monitor.sock,server,nowait
> >>>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc
> >>>> > base=2018-01-12T15:27:27,driftfix=slew -global
> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
> >>>> > -device
> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
> >>>> >
> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-
> fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,
> serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,
> werror=stop,rerror=stop,aio=threads
> >>>> > -device
> >>>> >
> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-
> virtio-disk0,id=virtio-disk0,bootindex=1
> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device
> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> >>>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
> >>>> >
> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:
> 7f:d6:83,bus=pci.0,addr=0x3
> >>>> > -chardev
> >>>> >
> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> >>>> > -device
> >>>> >
> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> >>>> > -chardev
> >>>> >
> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
> >>>> > -device
> >>>> >
> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent -device
> >>>> >
> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> >>>> > -chardev
> >>>> >
> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/
> 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-
> engine-setup.0,server,nowait
> >>>> > -device
> >>>> >
> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> >>>> > -chardev pty,id=charconsole0 -device
> >>>> > virtconsole,chardev=charconsole0,id=console0 -spice
> >>>> >
> >>>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,
> tls-channel=default,seamless-migration=on
> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
> >>>> > rng-random,id=objrng0,filename=/dev/urandom -device
> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg
> >>>> > timestamp=on
> >>>> >
> >>>> >
> >>>> > ==> /var/log/messages <==
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine
> >>>> > qemu-114-Cultivar.
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine
> >>>> > qemu-114-Cultivar.
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine
> >>>> > qemu-114-Cultivar.
> >>>> >
> >>>> >
> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> >>>> >
> >>>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev pty,id=charconsole0:
> >>>> > char
> >>>> > device redirected to /dev/pts/2 (label charconsole0)
> >>>> >
> >>>> >
> >>>> > ==> /var/log/messages <==
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active
> >>>> >
> >>>> >
> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> >>>> >
> >>>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed
> >>>> >
> >>>> >
> >>>> > ==> /var/log/messages <==
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 15:27:27.773+0000:
> >>>> > 1910:
> >>>> > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed
> to
> >>>> > acquire
> >>>> > lock: Lease is held by another host
> >>>> >
> >>>> >
> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> >>>> >
> >>>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 from
> >>>> > pid 1773
> >>>> > (/usr/sbin/libvirtd)
> >>>> >
> >>>> >
> >>>> > ==> /var/log/messages <==
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > disabled
> >>>> > state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left promiscuous mode
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
> >>>> > disabled
> >>>> > state
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
> >>>> > [1515770847.7989]
> >>>> > device (vnet4): state change: disconnected -> unmanaged (reason
> >>>> > 'unmanaged')
> >>>> > [30 10 3]
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
> >>>> > [1515770847.7989]
> >>>> > device (vnet4): released from master device ovirtmgmt
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active
> >>>> >
> >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine
> qemu-114-Cultivar
> >>>> > terminated.
> >>>> >
> >>>> >
> >>>> > ==> /var/log/vdsm/vdsm.log <==
> >>>> >
> >>>> > vm/4013c829::ERROR::2018-01-12
> >>>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm)
> >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process
> >>>> > failed
> >>>> >
> >>>> > Traceback (most recent call last):
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> 843,
> >>>> > in
> >>>> > _startUnderlyingVm
> >>>> >
> >>>> > self._run()
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
> 2721,
> >>>> > in
> >>>> > _run
> >>>> >
> >>>> > dom.createWithFlags(flags)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/
> libvirtconnection.py",
> >>>> > line
> >>>> > 126, in wrapper
> >>>> >
> >>>> > ret = f(*args, **kwargs)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512,
> in
> >>>> > wrapper
> >>>> >
> >>>> > return func(inst, *args, **kwargs)
> >>>> >
> >>>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069,
> in
> >>>> > createWithFlags
> >>>> >
> >>>> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> >>>> > failed',
> >>>> > dom=self)
> >>>> >
> >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held
> by
> >>>> > another host
> >>>> >
> >>>> > periodic/47::ERROR::2018-01-12
> >>>> > 11:27:32,858::periodic::215::virt.periodic.Operation::(__call__)
> >>>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590>
> operation
> >>>> > failed
> >>>> >
> >>>> > Traceback (most recent call last):
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py",
> line
> >>>> > 213,
> >>>> > in __call__
> >>>> >
> >>>> > self._func()
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py",
> line
> >>>> > 522,
> >>>> > in __call__
> >>>> >
> >>>> > self._send_metrics()
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py",
> line
> >>>> > 538,
> >>>> > in _send_metrics
> >>>> >
> >>>> > vm_sample.interval)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
> line
> >>>> > 45, in
> >>>> > produce
> >>>> >
> >>>> > networks(vm, stats, first_sample, last_sample, interval)
> >>>> >
> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
> line
> >>>> > 322, in
> >>>> > networks
> >>>> >
> >>>> > if nic.name.startswith('hostdev'):
> >>>> >
> >>>> > AttributeError: name
> >>>> >
> >>>> >
> >>>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak <msivak at redhat.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Hmm that rules out most of NFS related permission issues.
> >>>> >>
> >>>> >> So the current status is (I need to sum it up to get the full
> >>>> >> picture):
> >>>> >>
> >>>> >> - HE VM is down
> >>>> >> - HE agent fails when opening metadata using the symlink
> >>>> >> - the symlink is there
> >>>> >> - the symlink is readable by vdsm:kvm
> >>>> >>
> >>>> >> Hmm can you check under which user is ovirt-ha-broker started?
> >>>> >>
> >>>> >> Martin
> >>>> >>
> >>>> >>
> >>>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef at gmail.com> wrote:
> >>>> >> > Same thing happens with data images of other VMs as well though,
> >>>> >> > and
> >>>> >> > those
> >>>> >> > seem to be running ok so I'm not sure if it's the problem.
> >>>> >> >
> >>>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme <jaymef at gmail.com>
> wrote:
> >>>> >> >>
> >>>> >> >> Martin,
> >>>> >> >>
> >>>> >> >> I can as VDSM user but not as root . I get permission denied
> >>>> >> >> trying to
> >>>> >> >> touch one of the files as root, is that normal?
> >>>> >> >>
> >>>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak <
> msivak at redhat.com>
> >>>> >> >> wrote:
> >>>> >> >>>
> >>>> >> >>> Hmm, then it might be a permission issue indeed. Can you touch
> >>>> >> >>> the
> >>>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not
> >>>> >> >>> prevent
> >>>> >> >>> you from doing that.
> >>>> >> >>>
> >>>> >> >>> Martin
> >>>> >> >>>
> >>>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme <jaymef at gmail.com>
> wrote:
> >>>> >> >>> > Sorry, I think we got confused about the symlink, there are
> >>>> >> >>> > symlinks
> >>>> >> >>> > in
> >>>> >> >>> > /var/run that point the /rhev when I was doing an LS it was
> >>>> >> >>> > listing
> >>>> >> >>> > the
> >>>> >> >>> > files in /rhev
> >>>> >> >>> >
> >>>> >> >>> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286
> >>>> >> >>> >
> >>>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a ->
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_hosted__engine/248f46f0-d793-4581-9810-
> c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
> >>>> >> >>> >
> >>>> >> >>> > ls -al
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_hosted__engine/248f46f0-d793-4581-9810-
> c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
> >>>> >> >>> > total 2040
> >>>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 .
> >>>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 ..
> >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56
> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
> >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016
> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease
> >>>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016
> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta
> >>>> >> >>> >
> >>>> >> >>> > Is it possible that this is the wrong image for hosted
> engine?
> >>>> >> >>> >
> >>>> >> >>> > this is all I get in vdsm log when running hosted-engine
> >>>> >> >>> > --connect-storage
> >>>> >> >>> >
> >>>> >> >>> > jsonrpc/4::ERROR::2018-01-12
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>> > 10:52:53,019::__init__::611::jsonrpc.JsonRpcServer::(_
> handle_request)
> >>>> >> >>> > Internal server error
> >>>> >> >>> > Traceback (most recent call last):
> >>>> >> >>> > File
> >>>> >> >>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> >>>> >> >>> > line
> >>>> >> >>> > 606,
> >>>> >> >>> > in _handle_request
> >>>> >> >>> > res = method(**params)
> >>>> >> >>> > File "/usr/lib/python2.7/site-
> packages/vdsm/rpc/Bridge.py",
> >>>> >> >>> > line
> >>>> >> >>> > 201,
> >>>> >> >>> > in
> >>>> >> >>> > _dynamicMethod
> >>>> >> >>> > result = fn(*methodArgs)
> >>>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies
> >>>> >> >>> > File "/usr/lib/python2.7/site-
> packages/vdsm/common/api.py",
> >>>> >> >>> > line
> >>>> >> >>> > 48,
> >>>> >> >>> > in
> >>>> >> >>> > method
> >>>> >> >>> > ret = func(*args, **kwargs)
> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line
> >>>> >> >>> > 1354, in
> >>>> >> >>> > getAllVmIoTunePolicies
> >>>> >> >>> > io_tune_policies_dict = self._cif.
> getAllVmIoTunePolicies()
> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
> >>>> >> >>> > line
> >>>> >> >>> > 524,
> >>>> >> >>> > in
> >>>> >> >>> > getAllVmIoTunePolicies
> >>>> >> >>> > 'current_values': v.getIoTune()}
> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> line
> >>>> >> >>> > 3481,
> >>>> >> >>> > in
> >>>> >> >>> > getIoTune
> >>>> >> >>> > result = self.getIoTuneResponse()
> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> line
> >>>> >> >>> > 3500,
> >>>> >> >>> > in
> >>>> >> >>> > getIoTuneResponse
> >>>> >> >>> > res = self._dom.blockIoTune(
> >>>> >> >>> > File
> >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
> >>>> >> >>> > line
> >>>> >> >>> > 47,
> >>>> >> >>> > in __getattr__
> >>>> >> >>> > % self.vmid)
> >>>> >> >>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c'
> >>>> >> >>> > was not
> >>>> >> >>> > defined
> >>>> >> >>> > yet or was undefined
> >>>> >> >>> >
> >>>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak
> >>>> >> >>> > <msivak at redhat.com>
> >>>> >> >>> > wrote:
> >>>> >> >>> >>
> >>>> >> >>> >> Hi,
> >>>> >> >>> >>
> >>>> >> >>> >> what happens when you try hosted-engine --connect-storage?
> Do
> >>>> >> >>> >> you
> >>>> >> >>> >> see
> >>>> >> >>> >> any errors in the vdsm log?
> >>>> >> >>> >>
> >>>> >> >>> >> Best regards
> >>>> >> >>> >>
> >>>> >> >>> >> Martin Sivak
> >>>> >> >>> >>
> >>>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme <jaymef at gmail.com>
> >>>> >> >>> >> wrote:
> >>>> >> >>> >> > Ok this is what I've done:
> >>>> >> >>> >> >
> >>>> >> >>> >> > - All three hosts in global maintenance mode
> >>>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop
> >>>> >> >>> >> > ovirt-ha-broker --
> >>>> >> >>> >> > on
> >>>> >> >>> >> > all three hosts
> >>>> >> >>> >> > - Moved ALL files in
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/
> >>>> >> >>> >> > to
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup
> >>>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl start
> >>>> >> >>> >> > ovirt-ha-broker
> >>>> >> >>> >> > --
> >>>> >> >>> >> > on all three hosts
> >>>> >> >>> >> >
> >>>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0):
> >>>> >> >>> >> > hosted-engine
> >>>> >> >>> >> > --vm-start
> >>>> >> >>> >> >
> >>>> >> >>> >> > Lots of errors in the logs still, it appears to be having
> >>>> >> >>> >> > problems
> >>>> >> >>> >> > with
> >>>> >> >>> >> > that
> >>>> >> >>> >> > directory still:
> >>>> >> >>> >> >
> >>>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker
> >>>> >> >>> >> > ovirt_hosted_engine_ha.broker.
> storage_broker.StorageBroker
> >>>> >> >>> >> > ERROR
> >>>> >> >>> >> > Failed
> >>>> >> >>> >> > to
> >>>> >> >>> >> > write metadata for host 1 to
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8#012Traceback
> >>>> >> >>> >> > (most recent call last):#012 File
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_
> ha/broker/storage_broker.py",
> >>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path,
> direct_flag
> >>>> >> >>> >> > |
> >>>> >> >>> >> > os.O_WRONLY |
> >>>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or
> directory:
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8'
> >>>> >> >>> >> >
> >>>> >> >>> >> > There are no new files or symlinks in
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/
> >>>> >> >>> >> >
> >>>> >> >>> >> > - Jayme
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak
> >>>> >> >>> >> > <msivak at redhat.com>
> >>>> >> >>> >> > wrote:
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> > Can you please stop all hosted engine tooling (
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> On all hosts I should have added.
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> Martin
> >>>> >> >>> >> >>
> >>>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak
> >>>> >> >>> >> >> <msivak at redhat.com>
> >>>> >> >>> >> >> wrote:
> >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No
> such
> >>>> >> >>> >> >> >> file
> >>>> >> >>> >> >> >> or
> >>>> >> >>> >> >> >> directory:
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8'
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> ls -al
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8
> >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are
> >>>> >> >>> >> >> >> referring to
> >>>> >> >>> >> >> >> that
> >>>> >> >>> >> >> >> was
> >>>> >> >>> >> >> >> addressed in RC1 or something else?
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > No, this file is the symlink. It should point to
> >>>> >> >>> >> >> > somewhere
> >>>> >> >>> >> >> > inside
> >>>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is
> >>>> >> >>> >> >> > really
> >>>> >> >>> >> >> > interesting.
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Can you please stop all hosted engine tooling
> >>>> >> >>> >> >> > (ovirt-ha-agent,
> >>>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is not
> >>>> >> >>> >> >> > important
> >>>> >> >>> >> >> > when
> >>>> >> >>> >> >> > services are stopped, but better safe than sorry) and
> >>>> >> >>> >> >> > restart
> >>>> >> >>> >> >> > all
> >>>> >> >>> >> >> > services again?
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> >> Could there possibly be a permissions
> >>>> >> >>> >> >> >> problem somewhere?
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Maybe, but the file itself looks out of the ordinary. I
> >>>> >> >>> >> >> > wonder
> >>>> >> >>> >> >> > how it
> >>>> >> >>> >> >> > got there.
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Best regards
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > Martin Sivak
> >>>> >> >>> >> >> >
> >>>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme <
> jaymef at gmail.com>
> >>>> >> >>> >> >> > wrote:
> >>>> >> >>> >> >> >> Thanks for the help thus far. Storage could be
> related
> >>>> >> >>> >> >> >> but
> >>>> >> >>> >> >> >> all
> >>>> >> >>> >> >> >> other
> >>>> >> >>> >> >> >> VMs on
> >>>> >> >>> >> >> >> same storage are running ok. The storage is mounted
> via
> >>>> >> >>> >> >> >> NFS
> >>>> >> >>> >> >> >> from
> >>>> >> >>> >> >> >> within one
> >>>> >> >>> >> >> >> of the three hosts, I realize this is not ideal. This
> >>>> >> >>> >> >> >> was
> >>>> >> >>> >> >> >> setup
> >>>> >> >>> >> >> >> by
> >>>> >> >>> >> >> >> a
> >>>> >> >>> >> >> >> previous admin more as a proof of concept and VMs were
> >>>> >> >>> >> >> >> put on
> >>>> >> >>> >> >> >> there
> >>>> >> >>> >> >> >> that
> >>>> >> >>> >> >> >> should not have been placed in a proof of concept
> >>>> >> >>> >> >> >> environment..
> >>>> >> >>> >> >> >> it
> >>>> >> >>> >> >> >> was
> >>>> >> >>> >> >> >> intended to be rebuilt with proper storage down the
> >>>> >> >>> >> >> >> road.
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts mount
> NFS
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/data
> >>>> >> >>> >> >> >> 4861742080
> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_data
> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso
> >>>> >> >>> >> >> >> 4861742080
> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_iso
> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.
> com:/exports/import_export
> >>>> >> >>> >> >> >> 4861742080
> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_import__export
> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.
> com:/exports/hosted_engine
> >>>> >> >>> >> >> >> 4861742080
> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.
> com:_exports_hosted__engine
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> Like I said, the VM data storage itself seems to be
> >>>> >> >>> >> >> >> working
> >>>> >> >>> >> >> >> ok,
> >>>> >> >>> >> >> >> as
> >>>> >> >>> >> >> >> all
> >>>> >> >>> >> >> >> other
> >>>> >> >>> >> >> >> VMs appear to be running.
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> I'm curious why the broker log says this file is not
> >>>> >> >>> >> >> >> found
> >>>> >> >>> >> >> >> when
> >>>> >> >>> >> >> >> it
> >>>> >> >>> >> >> >> is
> >>>> >> >>> >> >> >> correct and I can see the file at that path:
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No
> such
> >>>> >> >>> >> >> >> file
> >>>> >> >>> >> >> >> or
> >>>> >> >>> >> >> >> directory:
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8'
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> ls -al
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8
> >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are
> >>>> >> >>> >> >> >> referring to
> >>>> >> >>> >> >> >> that
> >>>> >> >>> >> >> >> was
> >>>> >> >>> >> >> >> addressed in RC1 or something else? Could there
> >>>> >> >>> >> >> >> possibly be
> >>>> >> >>> >> >> >> a
> >>>> >> >>> >> >> >> permissions
> >>>> >> >>> >> >> >> problem somewhere?
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms installed
> >>>> >> >>> >> >> >> and the
> >>>> >> >>> >> >> >> host
> >>>> >> >>> >> >> >> engine
> >>>> >> >>> >> >> >> will not start is it safe for me to update hosts to
> 4.2
> >>>> >> >>> >> >> >> RC1
> >>>> >> >>> >> >> >> rpms?
> >>>> >> >>> >> >> >> Or
> >>>> >> >>> >> >> >> perhaps install that repo and *only* update the ovirt
> HA
> >>>> >> >>> >> >> >> packages?
> >>>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates to
> the
> >>>> >> >>> >> >> >> inaccessible
> >>>> >> >>> >> >> >> hosted
> >>>> >> >>> >> >> >> engine VM.
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> I should also mention one more thing. I originally
> >>>> >> >>> >> >> >> upgraded
> >>>> >> >>> >> >> >> the
> >>>> >> >>> >> >> >> engine
> >>>> >> >>> >> >> >> VM
> >>>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed due
> >>>> >> >>> >> >> >> to not
> >>>> >> >>> >> >> >> being
> >>>> >> >>> >> >> >> in
> >>>> >> >>> >> >> >> global maintenance, so I set global maintenance and
> ran
> >>>> >> >>> >> >> >> it
> >>>> >> >>> >> >> >> again,
> >>>> >> >>> >> >> >> which
> >>>> >> >>> >> >> >> appeared to complete as intended but never came back
> up
> >>>> >> >>> >> >> >> after.
> >>>> >> >>> >> >> >> Just
> >>>> >> >>> >> >> >> in
> >>>> >> >>> >> >> >> case
> >>>> >> >>> >> >> >> this might have anything at all to do with what could
> >>>> >> >>> >> >> >> have
> >>>> >> >>> >> >> >> happened.
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> Thanks very much again, I very much appreciate the
> help!
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> - Jayme
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi
> >>>> >> >>> >> >> >> <stirabos at redhat.com>
> >>>> >> >>> >> >> >> wrote:
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak
> >>>> >> >>> >> >> >>> <msivak at redhat.com>
> >>>> >> >>> >> >> >>> wrote:
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> Hi,
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by
> >>>> >> >>> >> >> >>>> restarting
> >>>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest
> >>>> >> >>> >> >> >>>> ovirt-hosted-engine-ha
> >>>> >> >>> >> >> >>>> and
> >>>> >> >>> >> >> >>>> -setup. We improved handling of the missing symlink.
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> All the other issues seem to point to some storage
> >>>> >> >>> >> >> >>>> problem
> >>>> >> >>> >> >> >>>> I
> >>>> >> >>> >> >> >>>> am
> >>>> >> >>> >> >> >>>> afraid.
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> You said you started the VM, do you see it in virsh
> -r
> >>>> >> >>> >> >> >>>> list?
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> Best regards
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> Martin Sivak
> >>>> >> >>> >> >> >>>>
> >>>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme
> >>>> >> >>> >> >> >>>> <jaymef at gmail.com>
> >>>> >> >>> >> >> >>>> wrote:
> >>>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to try
> at
> >>>> >> >>> >> >> >>>> > this
> >>>> >> >>> >> >> >>>> > point.
> >>>> >> >>> >> >> >>>> > Thank
> >>>> >> >>> >> >> >>>> > you
> >>>> >> >>> >> >> >>>> > for reading!
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted
> engine
> >>>> >> >>> >> >> >>>> > running
> >>>> >> >>> >> >> >>>> > after a
> >>>> >> >>> >> >> >>>> > botched
> >>>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from
> within
> >>>> >> >>> >> >> >>>> > one
> >>>> >> >>> >> >> >>>> > of
> >>>> >> >>> >> >> >>>> > the
> >>>> >> >>> >> >> >>>> > hosts.
> >>>> >> >>> >> >> >>>> > Right
> >>>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully updated
> >>>> >> >>> >> >> >>>> > with
> >>>> >> >>> >> >> >>>> > yum
> >>>> >> >>> >> >> >>>> > packages
> >>>> >> >>> >> >> >>>> > from
> >>>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with yum
> >>>> >> >>> >> >> >>>> > packages
> >>>> >> >>> >> >> >>>> > and
> >>>> >> >>> >> >> >>>> > failed to
> >>>> >> >>> >> >> >>>> > come
> >>>> >> >>> >> >> >>>> > up after reboot. As of right now, everything
> should
> >>>> >> >>> >> >> >>>> > have
> >>>> >> >>> >> >> >>>> > full
> >>>> >> >>> >> >> >>>> > yum
> >>>> >> >>> >> >> >>>> > updates
> >>>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global
> maintenance
> >>>> >> >>> >> >> >>>> > mode
> >>>> >> >>> >> >> >>>> > on
> >>>> >> >>> >> >> >>>> > right
> >>>> >> >>> >> >> >>>> > now
> >>>> >> >>> >> >> >>>> > and
> >>>> >> >>> >> >> >>>> > started hosted-engine on one of the three host and
> >>>> >> >>> >> >> >>>> > the
> >>>> >> >>> >> >> >>>> > status is
> >>>> >> >>> >> >> >>>> > currently:
> >>>> >> >>> >> >> >>>> > Engine status : {"reason": "failed liveliness
> >>>> >> >>> >> >> >>>> > check”;
> >>>> >> >>> >> >> >>>> > "health":
> >>>> >> >>> >> >> >>>> > "bad",
> >>>> >> >>> >> >> >>>> > "vm":
> >>>> >> >>> >> >> >>>> > "up", "detail": "Up"}
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > this is what I get when trying to enter hosted-vm
> >>>> >> >>> >> >> >>>> > --console
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > The engine VM is running on this host
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine'
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > error: Domain not found: no domain with matching
> >>>> >> >>> >> >> >>>> > name
> >>>> >> >>> >> >> >>>> > 'HostedEngine'
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Here are logs from various sources when I start
> the
> >>>> >> >>> >> >> >>>> > VM on
> >>>> >> >>> >> >> >>>> > HOST3:
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > hosted-engine --vm-start
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID':
> >>>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} failed:
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not exist:
> >>>> >> >>> >> >> >>>> > {'vmId':
> >>>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'})
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: New
> >>>> >> >>> >> >> >>>> > machine
> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started Virtual
> >>>> >> >>> >> >> >>>> > Machine
> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting
> Virtual
> >>>> >> >>> >> >> >>>> > Machine
> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now active
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <==
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/vdsm/common/api.py",
> >>>> >> >>> >> >> >>>> > line
> >>>> >> >>> >> >> >>>> > 48,
> >>>> >> >>> >> >> >>>> > in
> >>>> >> >>> >> >> >>>> > method
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ret = func(*args, **kwargs)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/vdsm/storage/hsm.py",
> >>>> >> >>> >> >> >>>> > line
> >>>> >> >>> >> >> >>>> > 2718, in
> >>>> >> >>> >> >> >>>> > getStorageDomainInfo
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/vdsm/storage/hsm.py",
> >>>> >> >>> >> >> >>>> > line
> >>>> >> >>> >> >> >>>> > 304, in
> >>>> >> >>> >> >> >>>> > validateSdUUID
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > sdDom.validate()
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/vdsm/storage/fileSD.py",
> >>>> >> >>> >> >> >>>> > line
> >>>> >> >>> >> >> >>>> > 515,
> >>>> >> >>> >> >> >>>> > in validate
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > raise se.StorageDomainAccessError(
> self.sdUUID)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either
> partially
> >>>> >> >>> >> >> >>>> > accessible
> >>>> >> >>> >> >> >>>> > or
> >>>> >> >>> >> >> >>>> > entirely
> >>>> >> >>> >> >> >>>> > inaccessible:
> >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82::
> storage.Dispatcher::(wrapper)
> >>>> >> >>> >> >> >>>> > FINISH
> >>>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either
> >>>> >> >>> >> >> >>>> > partially
> >>>> >> >>> >> >> >>>> > accessible
> >>>> >> >>> >> >> >>>> > or
> >>>> >> >>> >> >> >>>> > entirely
> >>>> >> >>> >> >> >>>> > inaccessible:
> >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > LC_ALL=C
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/
> local/bin:/usr/sbin:/usr/bin
> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=
> raw,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes
> >>>> >> >>> >> >> >>>> > -machine
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,
> usb=off,dump-guest-core=off
> >>>> >> >>> >> >> >>>> > -cpu
> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.
> centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
> 4013c829-c9d7-4b72-90d5-6fe58137504c'
> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/
> var/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait
> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
> >>>> >> >>> >> >> >>>> > -rtc
> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global
> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot
> >>>> >> >>> >> >> >>>> > -boot
> >>>> >> >>> >> >> >>>> > strict=on
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-
> serial0,bus=pci.0,addr=0x4
> >>>> >> >>> >> >> >>>> > -drive
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/
> 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/
> 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=
> none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-
> a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
> pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=
> drive-ide0-1-0,id=ide0-1-0
> >>>> >> >>> >> >> >>>> > -netdev
> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=
> hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.qemu.guest_agent.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
> -spice
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=
> /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
> >>>> >> >>> >> >> >>>> > -object
> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=
> rng0,bus=pci.0,addr=0x5
> >>>> >> >>> >> >> >>>> > -msg
> >>>> >> >>> >> >> >>>> > timestamp=on
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev
> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
> >>>> >> >>> >> >> >>>> > char
> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
> charconsole0)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down,
> >>>> >> >>> >> >> >>>> > reason=shutdown
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up libvirt
> >>>> >> >>> >> >> >>>> > version:
> >>>> >> >>> >> >> >>>> > 3.2.0,
> >>>> >> >>> >> >> >>>> > package:
> >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem
> >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>,
> >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu
> >>>> >> >>> >> >> >>>> > version:
> >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
> >>>> >> >>> >> >> >>>> > cultivar3
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > LC_ALL=C
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/
> local/bin:/usr/sbin:/usr/bin
> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=
> raw,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes
> >>>> >> >>> >> >> >>>> > -machine
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,
> usb=off,dump-guest-core=off
> >>>> >> >>> >> >> >>>> > -cpu
> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.
> centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
> 4013c829-c9d7-4b72-90d5-6fe58137504c'
> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/
> var/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait
> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
> >>>> >> >>> >> >> >>>> > -rtc
> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global
> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot
> >>>> >> >>> >> >> >>>> > -boot
> >>>> >> >>> >> >> >>>> > strict=on
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-
> serial0,bus=pci.0,addr=0x4
> >>>> >> >>> >> >> >>>> > -drive
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/
> 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/
> 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=
> none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-
> a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
> pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=
> drive-ide0-1-0,id=ide0-1-0
> >>>> >> >>> >> >> >>>> > -netdev
> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=
> hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.qemu.guest_agent.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
> -spice
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=
> /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
> >>>> >> >>> >> >> >>>> > -object
> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=
> rng0,bus=pci.0,addr=0x5
> >>>> >> >>> >> >> >>>> > -msg
> >>>> >> >>> >> >> >>>> > timestamp=on
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev
> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
> >>>> >> >>> >> >> >>>> > char
> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
> charconsole0)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down,
> >>>> >> >>> >> >> >>>> > reason=shutdown
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up libvirt
> >>>> >> >>> >> >> >>>> > version:
> >>>> >> >>> >> >> >>>> > 3.2.0,
> >>>> >> >>> >> >> >>>> > package:
> >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem
> >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>,
> >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu
> >>>> >> >>> >> >> >>>> > version:
> >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
> >>>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > LC_ALL=C
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/
> local/bin:/usr/sbin:/usr/bin
> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=
> raw,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes
> >>>> >> >>> >> >> >>>> > -machine
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,
> usb=off,dump-guest-core=off
> >>>> >> >>> >> >> >>>> > -cpu
> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.
> centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=
> 4013c829-c9d7-4b72-90d5-6fe58137504c'
> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/
> var/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait
> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
> >>>> >> >>> >> >> >>>> > -rtc
> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global
> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot
> >>>> >> >>> >> >> >>>> > -boot
> >>>> >> >>> >> >> >>>> > strict=on
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-
> serial0,bus=pci.0,addr=0x4
> >>>> >> >>> >> >> >>>> > -drive
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/
> 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/
> 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=
> none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-
> a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
> pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=
> drive-ide0-1-0,id=ide0-1-0
> >>>> >> >>> >> >> >>>> > -netdev
> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=
> hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.qemu.guest_agent.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0
> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=
> charchannel2,id=channel2,name=com.redhat.spice.0
> >>>> >> >>> >> >> >>>> > -chardev
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/
> var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-
> 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
> >>>> >> >>> >> >> >>>> > -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=
> charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
> -spice
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=
> /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
> >>>> >> >>> >> >> >>>> > -object
> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
> -device
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=
> rng0,bus=pci.0,addr=0x5
> >>>> >> >>> >> >> >>>> > -msg
> >>>> >> >>> >> >> >>>> > timestamp=on
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev
> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
> >>>> >> >>> >> >> >>>> > char
> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
> charconsole0)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/broker.log
> <==
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> >>>> >> >>> >> >> >>>> > line 151, in get_raw_stats
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | os.O_RDONLY |
> >>>> >> >>> >> >> >>>> > os.O_SYNC)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or directory:
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8'
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 16:55:15,761::status_broker::
> 92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
> >>>> >> >>> >> >> >>>> > Failed to read state.
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Traceback (most recent call last):
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> >>>> >> >>> >> >> >>>> > line 88, in run
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats()
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> >>>> >> >>> >> >> >>>> > line 162, in get_raw_stats
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > .format(str(e)))
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno 2]
> No
> >>>> >> >>> >> >> >>>> > such
> >>>> >> >>> >> >> >>>> > file or
> >>>> >> >>> >> >> >>>> > directory:
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-
> c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8'
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > result = refresh_method()
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/env/config.py",
> >>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > content =
> >>>> >> >>> >> >> >>>> > self._get_file_content_from_shared_storage(VM)
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/env/config.py",
> >>>> >> >>> >> >> >>>> > line 484, in _get_file_content_from_shared_
> storage
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > config_volume_path =
> >>>> >> >>> >> >> >>>> > self._get_config_volume_path()
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/env/config.py",
> >>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > conf_vol_uuid
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/ovirt_hosted_engine_ha/lib/heconflib.py",
> >>>> >> >>> >> >> >>>> > line 358, in get_volume_path
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR,
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > RuntimeError: Path to volume
> >>>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f
> >>>> >> >>> >> >> >>>> > not
> >>>> >> >>> >> >> >>>> > found
> >>>> >> >>> >> >> >>>> > in /run/vdsm/storag
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <==
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260::
> virt.vmstats::(send_metrics)
> >>>> >> >>> >> >> >>>> > VM
> >>>> >> >>> >> >> >>>> > metrics
> >>>> >> >>> >> >> >>>> > collection failed
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > Traceback (most recent call last):
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > File
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
> packages/vdsm/virt/vmstats.py",
> >>>> >> >>> >> >> >>>> > line
> >>>> >> >>> >> >> >>>> > 197, in
> >>>> >> >>> >> >> >>>> > send_metrics
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] = stat['cpuUsage']
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > KeyError: 'cpuUsage'
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> > _______________________________________________
> >>>> >> >>> >> >> >>>> > Users mailing list
> >>>> >> >>> >> >> >>>> > Users at ovirt.org
> >>>> >> >>> >> >> >>>> > http://lists.ovirt.org/mailman/listinfo/users
> >>>> >> >>> >> >> >>>> >
> >>>> >> >>> >> >> >>>> _______________________________________________
> >>>> >> >>> >> >> >>>> Users mailing list
> >>>> >> >>> >> >> >>>> Users at ovirt.org
> >>>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman/listinfo/users
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>>
> >>>> >> >>> >> >> >>
> >>>> >> >>> >> >
> >>>> >> >>> >> >
> >>>> >> >>> >
> >>>> >> >>> >
> >>>> >> >>
> >>>> >> >>
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180112/f5aeeb14/attachment.html>
More information about the Users
mailing list