[ovirt-users] unable to bring up hosted engine after botched 4.2 upgrade
Martin Sivak
msivak at redhat.com
Fri Jan 12 17:49:08 UTC 2018
Hi,
the VM is up according to the status (at least for a while). You
should be able to use console and diagnose anything that happened
inside (line the need for fsck and such) now.
Check the presence of those links again now, the metadata file content
is not important, but the file has to exist (agents will populate it
with status data). I have no new idea about what is wrong with that
though.
Best regards
Martin
On Fri, Jan 12, 2018 at 5:47 PM, Jayme <jaymef at gmail.com> wrote:
> The lock space issue was an issue I needed to clear but I don't believe it
> has resolved the problem. I shutdown agent and broker on all hosts and
> disconnected hosted-storage then enabled broker/agent on just one host and
> connected storage. I started the VM and actually didn't get any errors in
> the logs barely at all which was good to see, however the VM is still not
> running:
>
> HOST3:
>
> Engine status : {"reason": "failed liveliness check",
> "health": "bad", "vm": "up", "detail": "Up"}
>
> ==> /var/log/messages <==
> Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled
> state
> Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode
> Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking
> state
> Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered
> forwarding state
> Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer space
> available
> Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar.
> Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine
> qemu-111-Cultivar.
> Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine
> qemu-111-Cultivar.
> Jan 12 12:42:57 cultivar3 kvm: 3 guests now active
> Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535:
> error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU
> guest agent is not connected
>
> Interestingly though, now I'm seeing this in the logs which may be a new
> clue:
>
>
> ==> /var/log/vdsm/vdsm.log <==
> File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 126,
> in findDomain
> return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
> File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 116,
> in findDomainPath
> raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> (u'248f46f0-d793-4581-9810-c9d965e2f286',)
> jsonrpc/4::ERROR::2018-01-12
> 12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH
> getStorageDomainInfo error=Storage domain does not exist:
> (u'248f46f0-d793-4581-9810-c9d965e2f286',)
> periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo)
> failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
> directory'Is the Hosted Engine setup finished?
> periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo)
> failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
> directory'Is the Hosted Engine setup finished?
> periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo)
> failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
> directory'Is the Hosted Engine setup finished?
> periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo)
> failed to retrieve Hosted Engine HA score '[Errno 2] No such file or
> directory'Is the Hosted Engine setup finished?
>
> ==> /var/log/ovirt-hosted-engine-ha/broker.log <==
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> line 151, in get_raw_stats
> f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)
> OSError: [Errno 2] No such file or directory:
> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
> StatusStorageThread::ERROR::2018-01-12
> 12:32:06,049::status_broker::92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
> Failed to read state.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> line 88, in run
> self._storage_broker.get_raw_stats()
> File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> line 162, in get_raw_stats
> .format(str(e)))
> RequestError: failed to read metadata: [Errno 2] No such file or directory:
> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>
> On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak at redhat.com> wrote:
>>
>> The lock is the issue.
>>
>> - try running sanlock client status on all hosts
>> - also make sure you do not have some forgotten host still connected
>> to the lockspace, but without ha daemons running (and with the VM)
>>
>> I need to go to our president election now, I might check the email
>> later tonight.
>>
>> Martin
>>
>> On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef at gmail.com> wrote:
>> > Here are the newest logs from me trying to start hosted vm:
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > blocking
>> > state
>> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > disabled
>> > state
>> > Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered promiscuous mode
>> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > blocking
>> > state
>> > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > forwarding state
>> > Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No buffer
>> > space
>> > available
>> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info>
>> > [1515772694.8715]
>> > manager: (vnet4): new Tun device
>> > (/org/freedesktop/NetworkManager/Devices/140)
>> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info>
>> > [1515772694.8795]
>> > device (vnet4): state change: unmanaged -> unavailable (reason
>> > 'connection-assumed') [10 20 41]
>> >
>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> > 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0,
>> > package:
>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
>> > cultivar0.grove.silverorange.com
>> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> > guest=Cultivar,debug-threads=on -S -object
>> >
>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-119-Cultivar/master-key.aes
>> > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu
>> > Conroe -m 8192 -realtime mlock=off -smp
>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> > 'type=1,manufacturer=oVirt,product=oVirt
>> >
>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> > -no-user-config -nodefaults -chardev
>> >
>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-119-Cultivar/monitor.sock,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> > base=2018-01-12T15:58:14,driftfix=slew -global
>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
>> > -device
>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> >
>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> > -device
>> >
>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device
>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
>> >
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> > -chardev
>> >
>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> > -device
>> >
>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> > -chardev
>> >
>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> > -device
>> >
>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> > -chardev spicevmc,id=charchannel2,name=vdagent -device
>> >
>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> > -chardev
>> >
>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> > -device
>> >
>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> > -chardev pty,id=charconsole0 -device
>> > virtconsole,chardev=charconsole0,id=console0 -spice
>> >
>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
>> > rng-random,id=objrng0,filename=/dev/urandom -device
>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info>
>> > [1515772694.8807]
>> > device (vnet4): state change: unavailable -> disconnected (reason
>> > 'none')
>> > [20 30 0]
>> > Jan 12 11:58:14 cultivar0 systemd-machined: New machine
>> > qemu-119-Cultivar.
>> > Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine
>> > qemu-119-Cultivar.
>> > Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine
>> > qemu-119-Cultivar.
>> >
>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> > 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev pty,id=charconsole0: char
>> > device redirected to /dev/pts/1 (label charconsole0)
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:15 cultivar0 kvm: 5 guests now active
>> >
>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> > 2018-01-12 15:58:15.217+0000: shutting down, reason=failed
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: 1908:
>> > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to
>> > acquire
>> > lock: Lease is held by another host
>> >
>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> > 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from pid
>> > 1773
>> > (/usr/sbin/libvirtd)
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > disabled
>> > state
>> > Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode
>> > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> > disabled
>> > state
>> > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info>
>> > [1515772695.2348]
>> > device (vnet4): state change: disconnected -> unmanaged (reason
>> > 'unmanaged')
>> > [30 10 3]
>> > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info>
>> > [1515772695.2349]
>> > device (vnet4): released from master device ovirtmgmt
>> > Jan 12 11:58:15 cultivar0 kvm: 4 guests now active
>> > Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar
>> > terminated.
>> >
>> > ==> /var/log/vdsm/vdsm.log <==
>> > vm/4013c829::ERROR::2018-01-12
>> > 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm)
>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process
>> > failed
>> > Traceback (most recent call last):
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, in
>> > _startUnderlyingVm
>> > self._run()
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, in
>> > _run
>> > dom.createWithFlags(flags)
>> > File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
>> > line
>> > 126, in wrapper
>> > ret = f(*args, **kwargs)
>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in
>> > wrapper
>> > return func(inst, *args, **kwargs)
>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in
>> > createWithFlags
>> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
>> > failed',
>> > dom=self)
>> > libvirtError: resource busy: Failed to acquire lock: Lease is held by
>> > another host
>> > jsonrpc/6::ERROR::2018-01-12
>> > 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_handle_request)
>> > Internal server error
>> > Traceback (most recent call last):
>> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
>> > 606,
>> > in _handle_request
>> > res = method(**params)
>> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201,
>> > in
>> > _dynamicMethod
>> > result = fn(*methodArgs)
>> > File "<string>", line 2, in getAllVmIoTunePolicies
>> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48,
>> > in
>> > method
>> > ret = func(*args, **kwargs)
>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
>> > getAllVmIoTunePolicies
>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
>> > getAllVmIoTunePolicies
>> > 'current_values': v.getIoTune()}
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
>> > getIoTune
>> > result = self.getIoTuneResponse()
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
>> > getIoTuneResponse
>> > res = self._dom.blockIoTune(
>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
>> > 47,
>> > in __getattr__
>> > % self.vmid)
>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not
>> > defined
>> > yet or was undefined
>> >
>> > ==> /var/log/messages <==
>> > Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR
>> > Internal
>> > server error#012Traceback (most recent call last):#012 File
>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in
>> > _handle_request#012 res = method(**params)#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>",
>> > line 2,
>> > in getAllVmIoTunePolicies#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
>> > method#012 ret = func(*args, **kwargs)#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
>> > getAllVmIoTunePolicies#012 io_tune_policies_dict =
>> > self._cif.getAllVmIoTunePolicies()#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
>> > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
>> > getIoTune#012 result = self.getIoTuneResponse()#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File
>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in
>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM
>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was
>> > undefined
>> >
>> > On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef at gmail.com> wrote:
>> >>
>> >> One other tidbit I noticed is that it seems like there are less errors
>> >> if
>> >> I started in paused mode:
>> >>
>> >> but still shows: Engine status : {"reason": "bad
>> >> vm
>> >> status", "health": "bad", "vm": "up", "detail": "Paused"}
>> >>
>> >> ==> /var/log/messages <==
>> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >> blocking state
>> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >> disabled state
>> >> Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered promiscuous mode
>> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >> blocking state
>> >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >> forwarding state
>> >> Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No buffer
>> >> space available
>> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
>> >> [1515772505.3625]
>> >> manager: (vnet4): new Tun device
>> >> (/org/freedesktop/NetworkManager/Devices/139)
>> >>
>> >> ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >> 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0,
>> >> package:
>> >> 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
>> >> 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
>> >> 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
>> >> cultivar0.grove.silverorange.com
>> >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> >> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> >> guest=Cultivar,debug-threads=on -S -object
>> >>
>> >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-118-Cultivar/master-key.aes
>> >> -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu
>> >> Conroe -m 8192 -realtime mlock=off -smp
>> >> 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> >> 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> >> 'type=1,manufacturer=oVirt,product=oVirt
>> >>
>> >> Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >> -no-user-config -nodefaults -chardev
>> >>
>> >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-118-Cultivar/monitor.sock,server,nowait
>> >> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> >> base=2018-01-12T15:55:05,driftfix=slew -global
>> >> kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
>> >> -device
>> >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
>> >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> >>
>> >> file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> >> -device
>> >>
>> >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> >> -drive if=none,id=drive-ide0-1-0,readonly=on -device
>> >> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
>> >> tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
>> >>
>> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> >> -chardev
>> >>
>> >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> >> -device
>> >>
>> >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> >> -chardev
>> >>
>> >> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> >> -device
>> >>
>> >> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> >> -chardev spicevmc,id=charchannel2,name=vdagent -device
>> >>
>> >> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> >> -chardev
>> >>
>> >> socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> >> -device
>> >>
>> >> virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> >> -chardev pty,id=charconsole0 -device
>> >> virtconsole,chardev=charconsole0,id=console0 -spice
>> >>
>> >> tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> >> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
>> >> rng-random,id=objrng0,filename=/dev/urandom -device
>> >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on
>> >>
>> >> ==> /var/log/messages <==
>> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
>> >> [1515772505.3689]
>> >> device (vnet4): state change: unmanaged -> unavailable (reason
>> >> 'connection-assumed') [10 20 41]
>> >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info>
>> >> [1515772505.3702]
>> >> device (vnet4): state change: unavailable -> disconnected (reason
>> >> 'none')
>> >> [20 30 0]
>> >> Jan 12 11:55:05 cultivar0 systemd-machined: New machine
>> >> qemu-118-Cultivar.
>> >> Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine
>> >> qemu-118-Cultivar.
>> >> Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine
>> >> qemu-118-Cultivar.
>> >>
>> >> ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >> 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev pty,id=charconsole0:
>> >> char
>> >> device redirected to /dev/pts/1 (label charconsole0)
>> >>
>> >> ==> /var/log/messages <==
>> >> Jan 12 11:55:05 cultivar0 kvm: 5 guests now active
>> >>
>> >> On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef at gmail.com> wrote:
>> >>>
>> >>> Yeah I am in global maintenance:
>> >>>
>> >>> state=GlobalMaintenance
>> >>>
>> >>> host0: {"reason": "vm not running on this host", "health": "bad",
>> >>> "vm":
>> >>> "down", "detail": "unknown"}
>> >>> host2: {"reason": "vm not running on this host", "health": "bad",
>> >>> "vm":
>> >>> "down", "detail": "unknown"}
>> >>> host3: {"reason": "vm not running on this host", "health": "bad",
>> >>> "vm":
>> >>> "down", "detail": "unknown"}
>> >>>
>> >>> I understand the lock is an issue, I'll try to make sure it is fully
>> >>> stopped on all three before starting but I don't think that is the
>> >>> issue at
>> >>> hand either. What concerns me is mostly that it seems to be unable
>> >>> to read
>> >>> the meta data, I think that might be the heart of the problem but I'm
>> >>> not
>> >>> sure what is causing it.
>> >>>
>> >>> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak <msivak at redhat.com>
>> >>> wrote:
>> >>>>
>> >>>> > On all three hosts I ran hosted-engine --vm-shutdown; hosted-engine
>> >>>> > --vm-poweroff
>> >>>>
>> >>>> Are you in global maintenance? I think you were in one of the
>> >>>> previous
>> >>>> emails, but worth checking.
>> >>>>
>> >>>> > I started ovirt-ha-broker with systemctl as root user but it does
>> >>>> > appear to be running under vdsm:
>> >>>>
>> >>>> That is the correct behavior.
>> >>>>
>> >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held
>> >>>> > by
>> >>>> > another host
>> >>>>
>> >>>> sanlock seems to think the VM runs somewhere and it is possible that
>> >>>> some other host tried to start the VM as well unless you are in
>> >>>> global
>> >>>> maintenance (that is why I asked the first question here).
>> >>>>
>> >>>> Martin
>> >>>>
>> >>>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef at gmail.com> wrote:
>> >>>> > Martin,
>> >>>> >
>> >>>> > Thanks so much for keeping with me, this is driving me crazy! I
>> >>>> > really do
>> >>>> > appreciate it, thanks again
>> >>>> >
>> >>>> > Let's go through this:
>> >>>> >
>> >>>> > HE VM is down - YES
>> >>>> >
>> >>>> >
>> >>>> > HE agent fails when opening metadata using the symlink - YES
>> >>>> >
>> >>>> >
>> >>>> > the symlink is there and readable by vdsm:kvm - it appears to be:
>> >>>> >
>> >>>> >
>> >>>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20
>> >>>> > 14a20941-1b84-4b82-be8f-ace38d7c037a
>> >>>> > ->
>> >>>> >
>> >>>> >
>> >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
>> >>>> >
>> >>>> >
>> >>>> > And the files in the linked directory exist and have vdsm:kvm perms
>> >>>> > as
>> >>>> > well:
>> >>>> >
>> >>>> >
>> >>>> > # cd
>> >>>> >
>> >>>> >
>> >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
>> >>>> >
>> >>>> > [root at cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al
>> >>>> >
>> >>>> > total 2040
>> >>>> >
>> >>>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 .
>> >>>> >
>> >>>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 ..
>> >>>> >
>> >>>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19
>> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >
>> >>>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016
>> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease
>> >>>> >
>> >>>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016
>> >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta
>> >>>> >
>> >>>> >
>> >>>> > I started ovirt-ha-broker with systemctl as root user but it does
>> >>>> > appear to
>> >>>> > be running under vdsm:
>> >>>> >
>> >>>> >
>> >>>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18
>> >>>> > /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Here is something I tried:
>> >>>> >
>> >>>> >
>> >>>> > - On all three hosts I ran hosted-engine --vm-shutdown;
>> >>>> > hosted-engine
>> >>>> > --vm-poweroff
>> >>>> >
>> >>>> > - On HOST0 (cultivar0) I disconnected and reconnected storage using
>> >>>> > hosted-engine
>> >>>> >
>> >>>> > - Tried starting up the hosted VM on cultivar0 while tailing the
>> >>>> > logs:
>> >>>> >
>> >>>> >
>> >>>> > # hosted-engine --vm-start
>> >>>> >
>> >>>> > VM exists and is down, cleaning up and restarting
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/vdsm/vdsm.log <==
>> >>>> >
>> >>>> > jsonrpc/2::ERROR::2018-01-12
>> >>>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats)
>> >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm
>> >>>> > stats
>> >>>> >
>> >>>> > Traceback (most recent call last):
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>> >>>> > 1762,
>> >>>> > in
>> >>>> > _getRunningVmStats
>> >>>> >
>> >>>> > vm_sample.interval)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
>> >>>> > line
>> >>>> > 45, in
>> >>>> > produce
>> >>>> >
>> >>>> > networks(vm, stats, first_sample, last_sample, interval)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
>> >>>> > line
>> >>>> > 322, in
>> >>>> > networks
>> >>>> >
>> >>>> > if nic.name.startswith('hostdev'):
>> >>>> >
>> >>>> > AttributeError: name
>> >>>> >
>> >>>> > jsonrpc/3::ERROR::2018-01-12
>> >>>> >
>> >>>> > 11:27:27,221::__init__::611::jsonrpc.JsonRpcServer::(_handle_request)
>> >>>> > Internal server error
>> >>>> >
>> >>>> > Traceback (most recent call last):
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
>> >>>> > line
>> >>>> > 606,
>> >>>> > in _handle_request
>> >>>> >
>> >>>> > res = method(**params)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line
>> >>>> > 201, in
>> >>>> > _dynamicMethod
>> >>>> >
>> >>>> > result = fn(*methodArgs)
>> >>>> >
>> >>>> > File "<string>", line 2, in getAllVmIoTunePolicies
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
>> >>>> > 48,
>> >>>> > in
>> >>>> > method
>> >>>> >
>> >>>> > ret = func(*args, **kwargs)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354,
>> >>>> > in
>> >>>> > getAllVmIoTunePolicies
>> >>>> >
>> >>>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line
>> >>>> > 524,
>> >>>> > in
>> >>>> > getAllVmIoTunePolicies
>> >>>> >
>> >>>> > 'current_values': v.getIoTune()}
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>> >>>> > 3481,
>> >>>> > in
>> >>>> > getIoTune
>> >>>> >
>> >>>> > result = self.getIoTuneResponse()
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>> >>>> > 3500,
>> >>>> > in
>> >>>> > getIoTuneResponse
>> >>>> >
>> >>>> > res = self._dom.blockIoTune(
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
>> >>>> > line
>> >>>> > 47,
>> >>>> > in __getattr__
>> >>>> >
>> >>>> > % self.vmid)
>> >>>> >
>> >>>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was
>> >>>> > not
>> >>>> > defined
>> >>>> > yet or was undefined
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/messages <==
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR
>> >>>> > Internal
>> >>>> > server error#012Traceback (most recent call last):#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606,
>> >>>> > in
>> >>>> > _handle_request#012 res = method(**params)#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
>> >>>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File
>> >>>> > "<string>",
>> >>>> > line 2,
>> >>>> > in getAllVmIoTunePolicies#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
>> >>>> > method#012 ret = func(*args, **kwargs)#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in
>> >>>> > getAllVmIoTunePolicies#012 io_tune_policies_dict =
>> >>>> > self._cif.getAllVmIoTunePolicies()#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in
>> >>>> > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012
>> >>>> > File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in
>> >>>> > getIoTune#012 result = self.getIoTuneResponse()#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in
>> >>>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
>> >>>> > in
>> >>>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM
>> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was
>> >>>> > undefined
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > blocking
>> >>>> > state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > disabled
>> >>>> > state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered promiscuous
>> >>>> > mode
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > blocking
>> >>>> > state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > forwarding state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No
>> >>>> > buffer
>> >>>> > space
>> >>>> > available
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
>> >>>> > [1515770847.4264]
>> >>>> > manager: (vnet4): new Tun device
>> >>>> > (/org/freedesktop/NetworkManager/Devices/135)
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
>> >>>> > [1515770847.4342]
>> >>>> > device (vnet4): state change: unmanaged -> unavailable (reason
>> >>>> > 'connection-assumed') [10 20 41]
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
>> >>>> > [1515770847.4353]
>> >>>> > device (vnet4): state change: unavailable -> disconnected (reason
>> >>>> > 'none')
>> >>>> > [20 30 0]
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >>>> >
>> >>>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: 3.2.0,
>> >>>> > package:
>> >>>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>,
>> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version:
>> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
>> >>>> > cultivar0.grove.silverorange.com
>> >>>> >
>> >>>> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> >>>> > guest=Cultivar,debug-threads=on -S -object
>> >>>> >
>> >>>> >
>> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-114-Cultivar/master-key.aes
>> >>>> > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off
>> >>>> > -cpu
>> >>>> > Conroe -m 8192 -realtime mlock=off -smp
>> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
>> >>>> >
>> >>>> >
>> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >>>> > -no-user-config -nodefaults -chardev
>> >>>> >
>> >>>> >
>> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-114-Cultivar/monitor.sock,server,nowait
>> >>>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> >>>> > base=2018-01-12T15:27:27,driftfix=slew -global
>> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on
>> >>>> > -device
>> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
>> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
>> >>>> >
>> >>>> >
>> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> >>>> > -device
>> >>>> >
>> >>>> >
>> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device
>> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
>> >>>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
>> >>>> >
>> >>>> >
>> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> >>>> > -chardev
>> >>>> >
>> >>>> >
>> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> >>>> > -device
>> >>>> >
>> >>>> >
>> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> >>>> > -chardev
>> >>>> >
>> >>>> >
>> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> >>>> > -device
>> >>>> >
>> >>>> >
>> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent -device
>> >>>> >
>> >>>> >
>> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> >>>> > -chardev
>> >>>> >
>> >>>> >
>> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> >>>> > -device
>> >>>> >
>> >>>> >
>> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> >>>> > -chardev pty,id=charconsole0 -device
>> >>>> > virtconsole,chardev=charconsole0,id=console0 -spice
>> >>>> >
>> >>>> >
>> >>>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object
>> >>>> > rng-random,id=objrng0,filename=/dev/urandom -device
>> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg
>> >>>> > timestamp=on
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/messages <==
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine
>> >>>> > qemu-114-Cultivar.
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine
>> >>>> > qemu-114-Cultivar.
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine
>> >>>> > qemu-114-Cultivar.
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >>>> >
>> >>>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev pty,id=charconsole0:
>> >>>> > char
>> >>>> > device redirected to /dev/pts/2 (label charconsole0)
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/messages <==
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >>>> >
>> >>>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/messages <==
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 15:27:27.773+0000:
>> >>>> > 1910:
>> >>>> > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed
>> >>>> > to
>> >>>> > acquire
>> >>>> > lock: Lease is held by another host
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >>>> >
>> >>>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 from
>> >>>> > pid 1773
>> >>>> > (/usr/sbin/libvirtd)
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/messages <==
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > disabled
>> >>>> > state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left promiscuous
>> >>>> > mode
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered
>> >>>> > disabled
>> >>>> > state
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
>> >>>> > [1515770847.7989]
>> >>>> > device (vnet4): state change: disconnected -> unmanaged (reason
>> >>>> > 'unmanaged')
>> >>>> > [30 10 3]
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info>
>> >>>> > [1515770847.7989]
>> >>>> > device (vnet4): released from master device ovirtmgmt
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active
>> >>>> >
>> >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine
>> >>>> > qemu-114-Cultivar
>> >>>> > terminated.
>> >>>> >
>> >>>> >
>> >>>> > ==> /var/log/vdsm/vdsm.log <==
>> >>>> >
>> >>>> > vm/4013c829::ERROR::2018-01-12
>> >>>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm)
>> >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process
>> >>>> > failed
>> >>>> >
>> >>>> > Traceback (most recent call last):
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>> >>>> > 843,
>> >>>> > in
>> >>>> > _startUnderlyingVm
>> >>>> >
>> >>>> > self._run()
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
>> >>>> > 2721,
>> >>>> > in
>> >>>> > _run
>> >>>> >
>> >>>> > dom.createWithFlags(flags)
>> >>>> >
>> >>>> > File
>> >>>> > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
>> >>>> > line
>> >>>> > 126, in wrapper
>> >>>> >
>> >>>> > ret = f(*args, **kwargs)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512,
>> >>>> > in
>> >>>> > wrapper
>> >>>> >
>> >>>> > return func(inst, *args, **kwargs)
>> >>>> >
>> >>>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069,
>> >>>> > in
>> >>>> > createWithFlags
>> >>>> >
>> >>>> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
>> >>>> > failed',
>> >>>> > dom=self)
>> >>>> >
>> >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held
>> >>>> > by
>> >>>> > another host
>> >>>> >
>> >>>> > periodic/47::ERROR::2018-01-12
>> >>>> > 11:27:32,858::periodic::215::virt.periodic.Operation::(__call__)
>> >>>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590>
>> >>>> > operation
>> >>>> > failed
>> >>>> >
>> >>>> > Traceback (most recent call last):
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py",
>> >>>> > line
>> >>>> > 213,
>> >>>> > in __call__
>> >>>> >
>> >>>> > self._func()
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py",
>> >>>> > line
>> >>>> > 522,
>> >>>> > in __call__
>> >>>> >
>> >>>> > self._send_metrics()
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py",
>> >>>> > line
>> >>>> > 538,
>> >>>> > in _send_metrics
>> >>>> >
>> >>>> > vm_sample.interval)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
>> >>>> > line
>> >>>> > 45, in
>> >>>> > produce
>> >>>> >
>> >>>> > networks(vm, stats, first_sample, last_sample, interval)
>> >>>> >
>> >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
>> >>>> > line
>> >>>> > 322, in
>> >>>> > networks
>> >>>> >
>> >>>> > if nic.name.startswith('hostdev'):
>> >>>> >
>> >>>> > AttributeError: name
>> >>>> >
>> >>>> >
>> >>>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak <msivak at redhat.com>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> Hmm that rules out most of NFS related permission issues.
>> >>>> >>
>> >>>> >> So the current status is (I need to sum it up to get the full
>> >>>> >> picture):
>> >>>> >>
>> >>>> >> - HE VM is down
>> >>>> >> - HE agent fails when opening metadata using the symlink
>> >>>> >> - the symlink is there
>> >>>> >> - the symlink is readable by vdsm:kvm
>> >>>> >>
>> >>>> >> Hmm can you check under which user is ovirt-ha-broker started?
>> >>>> >>
>> >>>> >> Martin
>> >>>> >>
>> >>>> >>
>> >>>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef at gmail.com> wrote:
>> >>>> >> > Same thing happens with data images of other VMs as well though,
>> >>>> >> > and
>> >>>> >> > those
>> >>>> >> > seem to be running ok so I'm not sure if it's the problem.
>> >>>> >> >
>> >>>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme <jaymef at gmail.com>
>> >>>> >> > wrote:
>> >>>> >> >>
>> >>>> >> >> Martin,
>> >>>> >> >>
>> >>>> >> >> I can as VDSM user but not as root . I get permission denied
>> >>>> >> >> trying to
>> >>>> >> >> touch one of the files as root, is that normal?
>> >>>> >> >>
>> >>>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak
>> >>>> >> >> <msivak at redhat.com>
>> >>>> >> >> wrote:
>> >>>> >> >>>
>> >>>> >> >>> Hmm, then it might be a permission issue indeed. Can you touch
>> >>>> >> >>> the
>> >>>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not
>> >>>> >> >>> prevent
>> >>>> >> >>> you from doing that.
>> >>>> >> >>>
>> >>>> >> >>> Martin
>> >>>> >> >>>
>> >>>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme <jaymef at gmail.com>
>> >>>> >> >>> wrote:
>> >>>> >> >>> > Sorry, I think we got confused about the symlink, there are
>> >>>> >> >>> > symlinks
>> >>>> >> >>> > in
>> >>>> >> >>> > /var/run that point the /rhev when I was doing an LS it was
>> >>>> >> >>> > listing
>> >>>> >> >>> > the
>> >>>> >> >>> > files in /rhev
>> >>>> >> >>> >
>> >>>> >> >>> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286
>> >>>> >> >>> >
>> >>>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a ->
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
>> >>>> >> >>> >
>> >>>> >> >>> > ls -al
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a
>> >>>> >> >>> > total 2040
>> >>>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 .
>> >>>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 ..
>> >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56
>> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016
>> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease
>> >>>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016
>> >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta
>> >>>> >> >>> >
>> >>>> >> >>> > Is it possible that this is the wrong image for hosted
>> >>>> >> >>> > engine?
>> >>>> >> >>> >
>> >>>> >> >>> > this is all I get in vdsm log when running hosted-engine
>> >>>> >> >>> > --connect-storage
>> >>>> >> >>> >
>> >>>> >> >>> > jsonrpc/4::ERROR::2018-01-12
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> > 10:52:53,019::__init__::611::jsonrpc.JsonRpcServer::(_handle_request)
>> >>>> >> >>> > Internal server error
>> >>>> >> >>> > Traceback (most recent call last):
>> >>>> >> >>> > File
>> >>>> >> >>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 606,
>> >>>> >> >>> > in _handle_request
>> >>>> >> >>> > res = method(**params)
>> >>>> >> >>> > File
>> >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 201,
>> >>>> >> >>> > in
>> >>>> >> >>> > _dynamicMethod
>> >>>> >> >>> > result = fn(*methodArgs)
>> >>>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies
>> >>>> >> >>> > File
>> >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 48,
>> >>>> >> >>> > in
>> >>>> >> >>> > method
>> >>>> >> >>> > ret = func(*args, **kwargs)
>> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line
>> >>>> >> >>> > 1354, in
>> >>>> >> >>> > getAllVmIoTunePolicies
>> >>>> >> >>> > io_tune_policies_dict =
>> >>>> >> >>> > self._cif.getAllVmIoTunePolicies()
>> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 524,
>> >>>> >> >>> > in
>> >>>> >> >>> > getAllVmIoTunePolicies
>> >>>> >> >>> > 'current_values': v.getIoTune()}
>> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 3481,
>> >>>> >> >>> > in
>> >>>> >> >>> > getIoTune
>> >>>> >> >>> > result = self.getIoTuneResponse()
>> >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 3500,
>> >>>> >> >>> > in
>> >>>> >> >>> > getIoTuneResponse
>> >>>> >> >>> > res = self._dom.blockIoTune(
>> >>>> >> >>> > File
>> >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
>> >>>> >> >>> > line
>> >>>> >> >>> > 47,
>> >>>> >> >>> > in __getattr__
>> >>>> >> >>> > % self.vmid)
>> >>>> >> >>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >>>> >> >>> > was not
>> >>>> >> >>> > defined
>> >>>> >> >>> > yet or was undefined
>> >>>> >> >>> >
>> >>>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak
>> >>>> >> >>> > <msivak at redhat.com>
>> >>>> >> >>> > wrote:
>> >>>> >> >>> >>
>> >>>> >> >>> >> Hi,
>> >>>> >> >>> >>
>> >>>> >> >>> >> what happens when you try hosted-engine --connect-storage?
>> >>>> >> >>> >> Do
>> >>>> >> >>> >> you
>> >>>> >> >>> >> see
>> >>>> >> >>> >> any errors in the vdsm log?
>> >>>> >> >>> >>
>> >>>> >> >>> >> Best regards
>> >>>> >> >>> >>
>> >>>> >> >>> >> Martin Sivak
>> >>>> >> >>> >>
>> >>>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme <jaymef at gmail.com>
>> >>>> >> >>> >> wrote:
>> >>>> >> >>> >> > Ok this is what I've done:
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > - All three hosts in global maintenance mode
>> >>>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop
>> >>>> >> >>> >> > ovirt-ha-broker --
>> >>>> >> >>> >> > on
>> >>>> >> >>> >> > all three hosts
>> >>>> >> >>> >> > - Moved ALL files in
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/
>> >>>> >> >>> >> > to
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup
>> >>>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl start
>> >>>> >> >>> >> > ovirt-ha-broker
>> >>>> >> >>> >> > --
>> >>>> >> >>> >> > on all three hosts
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0):
>> >>>> >> >>> >> > hosted-engine
>> >>>> >> >>> >> > --vm-start
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > Lots of errors in the logs still, it appears to be having
>> >>>> >> >>> >> > problems
>> >>>> >> >>> >> > with
>> >>>> >> >>> >> > that
>> >>>> >> >>> >> > directory still:
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker
>> >>>> >> >>> >> > ERROR
>> >>>> >> >>> >> > Failed
>> >>>> >> >>> >> > to
>> >>>> >> >>> >> > write metadata for host 1 to
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8#012Traceback
>> >>>> >> >>> >> > (most recent call last):#012 File
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> >>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path,
>> >>>> >> >>> >> > direct_flag
>> >>>> >> >>> >> > |
>> >>>> >> >>> >> > os.O_WRONLY |
>> >>>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or
>> >>>> >> >>> >> > directory:
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > There are no new files or symlinks in
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > - Jayme
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak
>> >>>> >> >>> >> > <msivak at redhat.com>
>> >>>> >> >>> >> > wrote:
>> >>>> >> >>> >> >>
>> >>>> >> >>> >> >> > Can you please stop all hosted engine tooling (
>> >>>> >> >>> >> >>
>> >>>> >> >>> >> >> On all hosts I should have added.
>> >>>> >> >>> >> >>
>> >>>> >> >>> >> >> Martin
>> >>>> >> >>> >> >>
>> >>>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak
>> >>>> >> >>> >> >> <msivak at redhat.com>
>> >>>> >> >>> >> >> wrote:
>> >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No
>> >>>> >> >>> >> >> >> such
>> >>>> >> >>> >> >> >> file
>> >>>> >> >>> >> >> >> or
>> >>>> >> >>> >> >> >> directory:
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> ls -al
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are
>> >>>> >> >>> >> >> >> referring to
>> >>>> >> >>> >> >> >> that
>> >>>> >> >>> >> >> >> was
>> >>>> >> >>> >> >> >> addressed in RC1 or something else?
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > No, this file is the symlink. It should point to
>> >>>> >> >>> >> >> > somewhere
>> >>>> >> >>> >> >> > inside
>> >>>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is
>> >>>> >> >>> >> >> > really
>> >>>> >> >>> >> >> > interesting.
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > Can you please stop all hosted engine tooling
>> >>>> >> >>> >> >> > (ovirt-ha-agent,
>> >>>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is not
>> >>>> >> >>> >> >> > important
>> >>>> >> >>> >> >> > when
>> >>>> >> >>> >> >> > services are stopped, but better safe than sorry) and
>> >>>> >> >>> >> >> > restart
>> >>>> >> >>> >> >> > all
>> >>>> >> >>> >> >> > services again?
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> >> Could there possibly be a permissions
>> >>>> >> >>> >> >> >> problem somewhere?
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > Maybe, but the file itself looks out of the ordinary.
>> >>>> >> >>> >> >> > I
>> >>>> >> >>> >> >> > wonder
>> >>>> >> >>> >> >> > how it
>> >>>> >> >>> >> >> > got there.
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > Best regards
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > Martin Sivak
>> >>>> >> >>> >> >> >
>> >>>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme
>> >>>> >> >>> >> >> > <jaymef at gmail.com>
>> >>>> >> >>> >> >> > wrote:
>> >>>> >> >>> >> >> >> Thanks for the help thus far. Storage could be
>> >>>> >> >>> >> >> >> related
>> >>>> >> >>> >> >> >> but
>> >>>> >> >>> >> >> >> all
>> >>>> >> >>> >> >> >> other
>> >>>> >> >>> >> >> >> VMs on
>> >>>> >> >>> >> >> >> same storage are running ok. The storage is mounted
>> >>>> >> >>> >> >> >> via
>> >>>> >> >>> >> >> >> NFS
>> >>>> >> >>> >> >> >> from
>> >>>> >> >>> >> >> >> within one
>> >>>> >> >>> >> >> >> of the three hosts, I realize this is not ideal.
>> >>>> >> >>> >> >> >> This
>> >>>> >> >>> >> >> >> was
>> >>>> >> >>> >> >> >> setup
>> >>>> >> >>> >> >> >> by
>> >>>> >> >>> >> >> >> a
>> >>>> >> >>> >> >> >> previous admin more as a proof of concept and VMs
>> >>>> >> >>> >> >> >> were
>> >>>> >> >>> >> >> >> put on
>> >>>> >> >>> >> >> >> there
>> >>>> >> >>> >> >> >> that
>> >>>> >> >>> >> >> >> should not have been placed in a proof of concept
>> >>>> >> >>> >> >> >> environment..
>> >>>> >> >>> >> >> >> it
>> >>>> >> >>> >> >> >> was
>> >>>> >> >>> >> >> >> intended to be rebuilt with proper storage down the
>> >>>> >> >>> >> >> >> road.
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts mount
>> >>>> >> >>> >> >> >> NFS
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/data
>> >>>> >> >>> >> >> >> 4861742080
>> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_data
>> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso
>> >>>> >> >>> >> >> >> 4861742080
>> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_iso
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/import_export
>> >>>> >> >>> >> >> >> 4861742080
>> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_import__export
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/hosted_engine
>> >>>> >> >>> >> >> >> 4861742080
>> >>>> >> >>> >> >> >> 1039352832 3822389248 22%
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> Like I said, the VM data storage itself seems to be
>> >>>> >> >>> >> >> >> working
>> >>>> >> >>> >> >> >> ok,
>> >>>> >> >>> >> >> >> as
>> >>>> >> >>> >> >> >> all
>> >>>> >> >>> >> >> >> other
>> >>>> >> >>> >> >> >> VMs appear to be running.
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> I'm curious why the broker log says this file is not
>> >>>> >> >>> >> >> >> found
>> >>>> >> >>> >> >> >> when
>> >>>> >> >>> >> >> >> it
>> >>>> >> >>> >> >> >> is
>> >>>> >> >>> >> >> >> correct and I can see the file at that path:
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No
>> >>>> >> >>> >> >> >> such
>> >>>> >> >>> >> >> >> file
>> >>>> >> >>> >> >> >> or
>> >>>> >> >>> >> >> >> directory:
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> ls -al
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are
>> >>>> >> >>> >> >> >> referring to
>> >>>> >> >>> >> >> >> that
>> >>>> >> >>> >> >> >> was
>> >>>> >> >>> >> >> >> addressed in RC1 or something else? Could there
>> >>>> >> >>> >> >> >> possibly be
>> >>>> >> >>> >> >> >> a
>> >>>> >> >>> >> >> >> permissions
>> >>>> >> >>> >> >> >> problem somewhere?
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms installed
>> >>>> >> >>> >> >> >> and the
>> >>>> >> >>> >> >> >> host
>> >>>> >> >>> >> >> >> engine
>> >>>> >> >>> >> >> >> will not start is it safe for me to update hosts to
>> >>>> >> >>> >> >> >> 4.2
>> >>>> >> >>> >> >> >> RC1
>> >>>> >> >>> >> >> >> rpms?
>> >>>> >> >>> >> >> >> Or
>> >>>> >> >>> >> >> >> perhaps install that repo and *only* update the ovirt
>> >>>> >> >>> >> >> >> HA
>> >>>> >> >>> >> >> >> packages?
>> >>>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates to
>> >>>> >> >>> >> >> >> the
>> >>>> >> >>> >> >> >> inaccessible
>> >>>> >> >>> >> >> >> hosted
>> >>>> >> >>> >> >> >> engine VM.
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> I should also mention one more thing. I originally
>> >>>> >> >>> >> >> >> upgraded
>> >>>> >> >>> >> >> >> the
>> >>>> >> >>> >> >> >> engine
>> >>>> >> >>> >> >> >> VM
>> >>>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed
>> >>>> >> >>> >> >> >> due
>> >>>> >> >>> >> >> >> to not
>> >>>> >> >>> >> >> >> being
>> >>>> >> >>> >> >> >> in
>> >>>> >> >>> >> >> >> global maintenance, so I set global maintenance and
>> >>>> >> >>> >> >> >> ran
>> >>>> >> >>> >> >> >> it
>> >>>> >> >>> >> >> >> again,
>> >>>> >> >>> >> >> >> which
>> >>>> >> >>> >> >> >> appeared to complete as intended but never came back
>> >>>> >> >>> >> >> >> up
>> >>>> >> >>> >> >> >> after.
>> >>>> >> >>> >> >> >> Just
>> >>>> >> >>> >> >> >> in
>> >>>> >> >>> >> >> >> case
>> >>>> >> >>> >> >> >> this might have anything at all to do with what could
>> >>>> >> >>> >> >> >> have
>> >>>> >> >>> >> >> >> happened.
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> Thanks very much again, I very much appreciate the
>> >>>> >> >>> >> >> >> help!
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> - Jayme
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi
>> >>>> >> >>> >> >> >> <stirabos at redhat.com>
>> >>>> >> >>> >> >> >> wrote:
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak
>> >>>> >> >>> >> >> >>> <msivak at redhat.com>
>> >>>> >> >>> >> >> >>> wrote:
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> Hi,
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by
>> >>>> >> >>> >> >> >>>> restarting
>> >>>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest
>> >>>> >> >>> >> >> >>>> ovirt-hosted-engine-ha
>> >>>> >> >>> >> >> >>>> and
>> >>>> >> >>> >> >> >>>> -setup. We improved handling of the missing
>> >>>> >> >>> >> >> >>>> symlink.
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> All the other issues seem to point to some storage
>> >>>> >> >>> >> >> >>>> problem
>> >>>> >> >>> >> >> >>>> I
>> >>>> >> >>> >> >> >>>> am
>> >>>> >> >>> >> >> >>>> afraid.
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> You said you started the VM, do you see it in virsh
>> >>>> >> >>> >> >> >>>> -r
>> >>>> >> >>> >> >> >>>> list?
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> Best regards
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> Martin Sivak
>> >>>> >> >>> >> >> >>>>
>> >>>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme
>> >>>> >> >>> >> >> >>>> <jaymef at gmail.com>
>> >>>> >> >>> >> >> >>>> wrote:
>> >>>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to try
>> >>>> >> >>> >> >> >>>> > at
>> >>>> >> >>> >> >> >>>> > this
>> >>>> >> >>> >> >> >>>> > point.
>> >>>> >> >>> >> >> >>>> > Thank
>> >>>> >> >>> >> >> >>>> > you
>> >>>> >> >>> >> >> >>>> > for reading!
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted
>> >>>> >> >>> >> >> >>>> > engine
>> >>>> >> >>> >> >> >>>> > running
>> >>>> >> >>> >> >> >>>> > after a
>> >>>> >> >>> >> >> >>>> > botched
>> >>>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from
>> >>>> >> >>> >> >> >>>> > within
>> >>>> >> >>> >> >> >>>> > one
>> >>>> >> >>> >> >> >>>> > of
>> >>>> >> >>> >> >> >>>> > the
>> >>>> >> >>> >> >> >>>> > hosts.
>> >>>> >> >>> >> >> >>>> > Right
>> >>>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully updated
>> >>>> >> >>> >> >> >>>> > with
>> >>>> >> >>> >> >> >>>> > yum
>> >>>> >> >>> >> >> >>>> > packages
>> >>>> >> >>> >> >> >>>> > from
>> >>>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with yum
>> >>>> >> >>> >> >> >>>> > packages
>> >>>> >> >>> >> >> >>>> > and
>> >>>> >> >>> >> >> >>>> > failed to
>> >>>> >> >>> >> >> >>>> > come
>> >>>> >> >>> >> >> >>>> > up after reboot. As of right now, everything
>> >>>> >> >>> >> >> >>>> > should
>> >>>> >> >>> >> >> >>>> > have
>> >>>> >> >>> >> >> >>>> > full
>> >>>> >> >>> >> >> >>>> > yum
>> >>>> >> >>> >> >> >>>> > updates
>> >>>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global
>> >>>> >> >>> >> >> >>>> > maintenance
>> >>>> >> >>> >> >> >>>> > mode
>> >>>> >> >>> >> >> >>>> > on
>> >>>> >> >>> >> >> >>>> > right
>> >>>> >> >>> >> >> >>>> > now
>> >>>> >> >>> >> >> >>>> > and
>> >>>> >> >>> >> >> >>>> > started hosted-engine on one of the three host
>> >>>> >> >>> >> >> >>>> > and
>> >>>> >> >>> >> >> >>>> > the
>> >>>> >> >>> >> >> >>>> > status is
>> >>>> >> >>> >> >> >>>> > currently:
>> >>>> >> >>> >> >> >>>> > Engine status : {"reason": "failed liveliness
>> >>>> >> >>> >> >> >>>> > check”;
>> >>>> >> >>> >> >> >>>> > "health":
>> >>>> >> >>> >> >> >>>> > "bad",
>> >>>> >> >>> >> >> >>>> > "vm":
>> >>>> >> >>> >> >> >>>> > "up", "detail": "Up"}
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > this is what I get when trying to enter hosted-vm
>> >>>> >> >>> >> >> >>>> > --console
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > The engine VM is running on this host
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine'
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > error: Domain not found: no domain with matching
>> >>>> >> >>> >> >> >>>> > name
>> >>>> >> >>> >> >> >>>> > 'HostedEngine'
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Here are logs from various sources when I start
>> >>>> >> >>> >> >> >>>> > the
>> >>>> >> >>> >> >> >>>> > VM on
>> >>>> >> >>> >> >> >>>> > HOST3:
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > hosted-engine --vm-start
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID':
>> >>>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} failed:
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not exist:
>> >>>> >> >>> >> >> >>>> > {'vmId':
>> >>>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'})
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: New
>> >>>> >> >>> >> >> >>>> > machine
>> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started
>> >>>> >> >>> >> >> >>>> > Virtual
>> >>>> >> >>> >> >> >>>> > Machine
>> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting
>> >>>> >> >>> >> >> >>>> > Virtual
>> >>>> >> >>> >> >> >>>> > Machine
>> >>>> >> >>> >> >> >>>> > qemu-110-Cultivar.
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now
>> >>>> >> >>> >> >> >>>> > active
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <==
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
>> >>>> >> >>> >> >> >>>> > line
>> >>>> >> >>> >> >> >>>> > 48,
>> >>>> >> >>> >> >> >>>> > in
>> >>>> >> >>> >> >> >>>> > method
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ret = func(*args, **kwargs)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>> >>>> >> >>> >> >> >>>> > line
>> >>>> >> >>> >> >> >>>> > 2718, in
>> >>>> >> >>> >> >> >>>> > getStorageDomainInfo
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py",
>> >>>> >> >>> >> >> >>>> > line
>> >>>> >> >>> >> >> >>>> > 304, in
>> >>>> >> >>> >> >> >>>> > validateSdUUID
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > sdDom.validate()
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py",
>> >>>> >> >>> >> >> >>>> > line
>> >>>> >> >>> >> >> >>>> > 515,
>> >>>> >> >>> >> >> >>>> > in validate
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > raise
>> >>>> >> >>> >> >> >>>> > se.StorageDomainAccessError(self.sdUUID)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either
>> >>>> >> >>> >> >> >>>> > partially
>> >>>> >> >>> >> >> >>>> > accessible
>> >>>> >> >>> >> >> >>>> > or
>> >>>> >> >>> >> >> >>>> > entirely
>> >>>> >> >>> >> >> >>>> > inaccessible:
>> >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82::storage.Dispatcher::(wrapper)
>> >>>> >> >>> >> >> >>>> > FINISH
>> >>>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either
>> >>>> >> >>> >> >> >>>> > partially
>> >>>> >> >>> >> >> >>>> > accessible
>> >>>> >> >>> >> >> >>>> > or
>> >>>> >> >>> >> >> >>>> > entirely
>> >>>> >> >>> >> >> >>>> > inaccessible:
>> >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <==
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > LC_ALL=C
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes
>> >>>> >> >>> >> >> >>>> > -machine
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off
>> >>>> >> >>> >> >> >>>> > -cpu
>> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
>> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait
>> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
>> >>>> >> >>> >> >> >>>> > -rtc
>> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global
>> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet
>> >>>> >> >>> >> >> >>>> > -no-reboot
>> >>>> >> >>> >> >> >>>> > -boot
>> >>>> >> >>> >> >> >>>> > strict=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4
>> >>>> >> >>> >> >> >>>> > -drive
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> >>>> >> >>> >> >> >>>> > -netdev
>> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
>> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
>> >>>> >> >>> >> >> >>>> > -spice
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
>> >>>> >> >>> >> >> >>>> > -object
>> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5
>> >>>> >> >>> >> >> >>>> > -msg
>> >>>> >> >>> >> >> >>>> > timestamp=on
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev
>> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
>> >>>> >> >>> >> >> >>>> > char
>> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
>> >>>> >> >>> >> >> >>>> > charconsole0)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down,
>> >>>> >> >>> >> >> >>>> > reason=shutdown
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up libvirt
>> >>>> >> >>> >> >> >>>> > version:
>> >>>> >> >>> >> >> >>>> > 3.2.0,
>> >>>> >> >>> >> >> >>>> > package:
>> >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem
>> >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>,
>> >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu
>> >>>> >> >>> >> >> >>>> > version:
>> >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
>> >>>> >> >>> >> >> >>>> > cultivar3
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > LC_ALL=C
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes
>> >>>> >> >>> >> >> >>>> > -machine
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off
>> >>>> >> >>> >> >> >>>> > -cpu
>> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
>> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait
>> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
>> >>>> >> >>> >> >> >>>> > -rtc
>> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global
>> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet
>> >>>> >> >>> >> >> >>>> > -no-reboot
>> >>>> >> >>> >> >> >>>> > -boot
>> >>>> >> >>> >> >> >>>> > strict=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4
>> >>>> >> >>> >> >> >>>> > -drive
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> >>>> >> >>> >> >> >>>> > -netdev
>> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
>> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
>> >>>> >> >>> >> >> >>>> > -spice
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
>> >>>> >> >>> >> >> >>>> > -object
>> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5
>> >>>> >> >>> >> >> >>>> > -msg
>> >>>> >> >>> >> >> >>>> > timestamp=on
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev
>> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
>> >>>> >> >>> >> >> >>>> > char
>> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
>> >>>> >> >>> >> >> >>>> > charconsole0)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down,
>> >>>> >> >>> >> >> >>>> > reason=shutdown
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up libvirt
>> >>>> >> >>> >> >> >>>> > version:
>> >>>> >> >>> >> >> >>>> > 3.2.0,
>> >>>> >> >>> >> >> >>>> > package:
>> >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem
>> >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>,
>> >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu
>> >>>> >> >>> >> >> >>>> > version:
>> >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname:
>> >>>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > LC_ALL=C
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
>> >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name
>> >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes
>> >>>> >> >>> >> >> >>>> > -machine
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off
>> >>>> >> >>> >> >> >>>> > -cpu
>> >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp
>> >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
>> >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios
>> >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c'
>> >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait
>> >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control
>> >>>> >> >>> >> >> >>>> > -rtc
>> >>>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global
>> >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet
>> >>>> >> >>> >> >> >>>> > -no-reboot
>> >>>> >> >>> >> >> >>>> > -boot
>> >>>> >> >>> >> >> >>>> > strict=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4
>> >>>> >> >>> >> >> >>>> > -drive
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> >>>> >> >>> >> >> >>>> > -netdev
>> >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
>> >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
>> >>>> >> >>> >> >> >>>> > -chardev
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0
>> >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device
>> >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0
>> >>>> >> >>> >> >> >>>> > -spice
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on
>> >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2
>> >>>> >> >>> >> >> >>>> > -object
>> >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom
>> >>>> >> >>> >> >> >>>> > -device
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5
>> >>>> >> >>> >> >> >>>> > -msg
>> >>>> >> >>> >> >> >>>> > timestamp=on
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev
>> >>>> >> >>> >> >> >>>> > pty,id=charconsole0:
>> >>>> >> >>> >> >> >>>> > char
>> >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label
>> >>>> >> >>> >> >> >>>> > charconsole0)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/broker.log
>> >>>> >> >>> >> >> >>>> > <==
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> >>>> >> >>> >> >> >>>> > line 151, in get_raw_stats
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | os.O_RDONLY |
>> >>>> >> >>> >> >> >>>> > os.O_SYNC)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or directory:
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 16:55:15,761::status_broker::92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
>> >>>> >> >>> >> >> >>>> > Failed to read state.
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Traceback (most recent call last):
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
>> >>>> >> >>> >> >> >>>> > line 88, in run
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats()
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> >>>> >> >>> >> >> >>>> > line 162, in get_raw_stats
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > .format(str(e)))
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno 2]
>> >>>> >> >>> >> >> >>>> > No
>> >>>> >> >>> >> >> >>>> > such
>> >>>> >> >>> >> >> >>>> > file or
>> >>>> >> >>> >> >> >>>> > directory:
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8'
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/agent.log <==
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > result = refresh_method()
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
>> >>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > content =
>> >>>> >> >>> >> >> >>>> > self._get_file_content_from_shared_storage(VM)
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
>> >>>> >> >>> >> >> >>>> > line 484, in
>> >>>> >> >>> >> >> >>>> > _get_file_content_from_shared_storage
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > config_volume_path =
>> >>>> >> >>> >> >> >>>> > self._get_config_volume_path()
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py",
>> >>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > conf_vol_uuid
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py",
>> >>>> >> >>> >> >> >>>> > line 358, in get_volume_path
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR,
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > RuntimeError: Path to volume
>> >>>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f
>> >>>> >> >>> >> >> >>>> > not
>> >>>> >> >>> >> >> >>>> > found
>> >>>> >> >>> >> >> >>>> > in /run/vdsm/storag
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <==
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260::virt.vmstats::(send_metrics)
>> >>>> >> >>> >> >> >>>> > VM
>> >>>> >> >>> >> >> >>>> > metrics
>> >>>> >> >>> >> >> >>>> > collection failed
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > Traceback (most recent call last):
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > File
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py",
>> >>>> >> >>> >> >> >>>> > line
>> >>>> >> >>> >> >> >>>> > 197, in
>> >>>> >> >>> >> >> >>>> > send_metrics
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] =
>> >>>> >> >>> >> >> >>>> > stat['cpuUsage']
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > KeyError: 'cpuUsage'
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> > _______________________________________________
>> >>>> >> >>> >> >> >>>> > Users mailing list
>> >>>> >> >>> >> >> >>>> > Users at ovirt.org
>> >>>> >> >>> >> >> >>>> > http://lists.ovirt.org/mailman/listinfo/users
>> >>>> >> >>> >> >> >>>> >
>> >>>> >> >>> >> >> >>>> _______________________________________________
>> >>>> >> >>> >> >> >>>> Users mailing list
>> >>>> >> >>> >> >> >>>> Users at ovirt.org
>> >>>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman/listinfo/users
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>>
>> >>>> >> >>> >> >> >>
>> >>>> >> >>> >> >
>> >>>> >> >>> >> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>
>> >
>
>
More information about the Users
mailing list