The lock space issue was an issue I needed to clear but I don't believe it has resolved the problem. I shutdown agent and broker on all hosts and disconnected hosted-storage then enabled broker/agent on just one host and connected storage. I started the VM and actually didn't get any errors in the logs barely at all which was good to see, however the VM is still not running: HOST3: Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} ==> /var/log/messages <== Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer space available Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 kvm: 3 guests now active Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535: error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU guest agent is not connected Interestingly though, now I'm seeing this in the logs which may be a new clue: ==> /var/log/vdsm/vdsm.log <== File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 126, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 116, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) jsonrpc/4::ERROR::2018-01-12 12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH getStorageDomainInfo error=Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? ==> /var/log/ovirt-hosted-engine-ha/broker.log <== File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 151, in get_raw_stats f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' StatusStorageThread::ERROR::2018-01-12 12:32:06,049::status_broker::92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to read state. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 88, in run self._storage_broker.get_raw_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 162, in get_raw_stats .format(str(e))) RequestError: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak(a)redhat.com&gt; wrote: > > The lock is the issue. > > - try running sanlock client status on all hosts > - also make sure you do not have some forgotten host still connected > to the lockspace, but without ha daemons running (and with the VM) > > I need to go to our president election now, I might check the email > later tonight. > > Martin > > On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef(a)gmail.com&gt; wrote: > > Here are the newest logs from me trying to start hosted vm: > > > > ==> /var/log/messages <== > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > blocking > > state > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered promiscuous mode > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > blocking > > state > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > forwarding state > > Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No buffer > > space > > available > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8715] > > manager: (vnet4): new Tun device > > (/org/freedesktop/NetworkManager/Devices/140) > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8795] > > device (vnet4): state change: unmanaged -> unavailable (reason > > 'connection-assumed') [10 20 41] > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0, > > package: > > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>;, > > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > > cultivar0.grove.silverorange.com > > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > > guest=Cultivar,debug-threads=on -S -object > > > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-119-Cultivar/master-key.aes > > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu > > Conroe -m 8192 -realtime mlock=off -smp > > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > > 'type=1,manufacturer=oVirt,product=oVirt > > > > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > > -no-user-config -nodefaults -chardev > > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-119-Cultivar/monitor.sock,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control -rtc > > base=2018-01-12T15:58:14,driftfix=slew -global > > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > > -device > > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > > > > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > > -device > > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > > -drive if=none,id=drive-ide0-1-0,readonly=on -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > > -chardev > > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > > -device > > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > > -chardev > > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > > -device > > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > > -chardev spicevmc,id=charchannel2,name=vdagent -device > > > > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > > -chardev > > > > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > > -device > > > > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > > -chardev pty,id=charconsole0 -device > > virtconsole,chardev=charconsole0,id=console0 -spice > > > > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > > rng-random,id=objrng0,filename=/dev/urandom -device > > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on > > > > ==> /var/log/messages <== > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8807] > > device (vnet4): state change: unavailable -> disconnected (reason > > 'none') > > [20 30 0] > > Jan 12 11:58:14 cultivar0 systemd-machined: New machine > > qemu-119-Cultivar. > > Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine > > qemu-119-Cultivar. > > Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine > > qemu-119-Cultivar. > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev pty,id=charconsole0: char > > device redirected to /dev/pts/1 (label charconsole0) > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 kvm: 5 guests now active > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12 15:58:15.217+0000: shutting down, reason=failed > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: 1908: > > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to > > acquire > > lock: Lease is held by another host > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from pid > > 1773 > > (/usr/sbin/libvirtd) > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode > > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > > [1515772695.2348] > > device (vnet4): state change: disconnected -> unmanaged (reason > > 'unmanaged') > > [30 10 3] > > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > > [1515772695.2349] > > device (vnet4): released from master device ovirtmgmt > > Jan 12 11:58:15 cultivar0 kvm: 4 guests now active > > Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar > > terminated. > > > > ==> /var/log/vdsm/vdsm.log <== > > vm/4013c829::ERROR::2018-01-12 > > 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm) > > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process > > failed > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, in > > _startUnderlyingVm > > self._run() > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, in > > _run > > dom.createWithFlags(flags) > > File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", > > line > > 126, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in > > wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in > > createWithFlags > > if ret == -1: raise libvirtError ('virDomainCreateWithFlags() > > failed', > > dom=self) > > libvirtError: resource busy: Failed to acquire lock: Lease is held by > > another host > > jsonrpc/6::ERROR::2018-01-12 > > 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_handle_request) > > Internal server error > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line > > 606, > > in _handle_request > > res = method(**params) > > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, > > in > > _dynamicMethod > > result = fn(*methodArgs) > > File "<string>", line 2, in getAllVmIoTunePolicies > > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, > > in > > method > > ret = func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > > getAllVmIoTunePolicies > > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() > > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in > > getAllVmIoTunePolicies > > 'current_values': v.getIoTune()} > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in > > getIoTune > > result = self.getIoTuneResponse() > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in > > getIoTuneResponse > > res = self._dom.blockIoTune( > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line > > 47, > > in __getattr__ > > % self.vmid) > > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not > > defined > > yet or was undefined > > > > ==> /var/log/messages <== > > Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR > > Internal > > server error#012Traceback (most recent call last):#012 File > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in > > _handle_request#012 res = method(**params)#012 File > > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in > > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>", > > line 2, > > in getAllVmIoTunePolicies#012 File > > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in > > method#012 ret = func(*args, **kwargs)#012 File > > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > > getAllVmIoTunePolicies#012 io_tune_policies_dict = > > self._cif.getAllVmIoTunePolicies()#012 File > > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in > > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 File > > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in > > getIoTune#012 result = self.getIoTuneResponse()#012 File > > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in > > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File > > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in > > __getattr__#012 % self.vmid)#012NotConnectedError: VM > > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was > > undefined > > > > On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef(a)gmail.com&gt; wrote: > >> > >> One other tidbit I noticed is that it seems like there are less errors > >> if > >> I started in paused mode: > >> > >> but still shows: Engine status : {"reason": "bad > >> vm > >> status", "health": "bad", "vm": "up", "detail": "Paused"} > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> blocking state > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> disabled state > >> Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered promiscuous mode > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> blocking state > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> forwarding state > >> Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No buffer > >> space available > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3625] > >> manager: (vnet4): new Tun device > >> (/org/freedesktop/NetworkManager/Devices/139) > >> > >> ==> /var/log/libvirt/qemu/Cultivar.log <== > >> 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0, > >> package: > >> 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>;, > >> 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > >> 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >> cultivar0.grove.silverorange.com > >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >> guest=Cultivar,debug-threads=on -S -object > >> > >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-118-Cultivar/master-key.aes > >> -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu > >> Conroe -m 8192 -realtime mlock=off -smp > >> 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >> 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >> 'type=1,manufacturer=oVirt,product=oVirt > >> > >> Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >> -no-user-config -nodefaults -chardev > >> > >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-118-Cultivar/monitor.sock,server,nowait > >> -mon chardev=charmonitor,id=monitor,mode=control -rtc > >> base=2018-01-12T15:55:05,driftfix=slew -global > >> kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > >> -device > >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > >> > >> file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >> -device > >> > >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >> -drive if=none,id=drive-ide0-1-0,readonly=on -device > >> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > >> tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > >> > >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >> -chardev > >> > >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >> -device > >> > >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >> -chardev > >> > >> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >> -device > >> > >> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >> -chardev spicevmc,id=charchannel2,name=vdagent -device > >> > >> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >> -chardev > >> > >> socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > >> -device > >> > >> virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >> -chardev pty,id=charconsole0 -device > >> virtconsole,chardev=charconsole0,id=console0 -spice > >> > >> tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > >> rng-random,id=objrng0,filename=/dev/urandom -device > >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3689] > >> device (vnet4): state change: unmanaged -> unavailable (reason > >> 'connection-assumed') [10 20 41] > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3702] > >> device (vnet4): state change: unavailable -> disconnected (reason > >> 'none') > >> [20 30 0] > >> Jan 12 11:55:05 cultivar0 systemd-machined: New machine > >> qemu-118-Cultivar. > >> Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine > >> qemu-118-Cultivar. > >> Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine > >> qemu-118-Cultivar. > >> > >> ==> /var/log/libvirt/qemu/Cultivar.log <== > >> 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev pty,id=charconsole0: > >> char > >> device redirected to /dev/pts/1 (label charconsole0) > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 kvm: 5 guests now active > >> > >> On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef(a)gmail.com&gt; wrote: > >>> > >>> Yeah I am in global maintenance: > >>> > >>> state=GlobalMaintenance > >>> > >>> host0: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> host2: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> host3: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> > >>> I understand the lock is an issue, I'll try to make sure it is fully > >>> stopped on all three before starting but I don't think that is the > >>> issue at > >>> hand either. What concerns me is mostly that it seems to be unable > >>> to read > >>> the meta data, I think that might be the heart of the problem but I'm > >>> not > >>> sure what is causing it. > >>> > >>> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak <msivak(a)redhat.com&gt; > >>> wrote: > >>>> > >>>> > On all three hosts I ran hosted-engine --vm-shutdown; hosted-engine > >>>> > --vm-poweroff > >>>> > >>>> Are you in global maintenance? I think you were in one of the > >>>> previous > >>>> emails, but worth checking. > >>>> > >>>> > I started ovirt-ha-broker with systemctl as root user but it does > >>>> > appear to be running under vdsm: > >>>> > >>>> That is the correct behavior. > >>>> > >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held > >>>> > by > >>>> > another host > >>>> > >>>> sanlock seems to think the VM runs somewhere and it is possible that > >>>> some other host tried to start the VM as well unless you are in > >>>> global > >>>> maintenance (that is why I asked the first question here). > >>>> > >>>> Martin > >>>> > >>>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef(a)gmail.com&gt; wrote: > >>>> > Martin, > >>>> > > >>>> > Thanks so much for keeping with me, this is driving me crazy! I > >>>> > really do > >>>> > appreciate it, thanks again > >>>> > > >>>> > Let's go through this: > >>>> > > >>>> > HE VM is down - YES > >>>> > > >>>> > > >>>> > HE agent fails when opening metadata using the symlink - YES > >>>> > > >>>> > > >>>> > the symlink is there and readable by vdsm:kvm - it appears to be: > >>>> > > >>>> > > >>>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20 > >>>> > 14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > -> > >>>> > > >>>> > > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > > >>>> > > >>>> > And the files in the linked directory exist and have vdsm:kvm perms > >>>> > as > >>>> > well: > >>>> > > >>>> > > >>>> > # cd > >>>> > > >>>> > > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > > >>>> > [root@cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al > >>>> > > >>>> > total 2040 > >>>> > > >>>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . > >>>> > > >>>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. > >>>> > > >>>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> > > >>>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease > >>>> > > >>>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta > >>>> > > >>>> > > >>>> > I started ovirt-ha-broker with systemctl as root user but it does > >>>> > appear to > >>>> > be running under vdsm: > >>>> > > >>>> > > >>>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18 > >>>> > /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker > >>>> > > >>>> > > >>>> > > >>>> > Here is something I tried: > >>>> > > >>>> > > >>>> > - On all three hosts I ran hosted-engine --vm-shutdown; > >>>> > hosted-engine > >>>> > --vm-poweroff > >>>> > > >>>> > - On HOST0 (cultivar0) I disconnected and reconnected storage using > >>>> > hosted-engine > >>>> > > >>>> > - Tried starting up the hosted VM on cultivar0 while tailing the > >>>> > logs: > >>>> > > >>>> > > >>>> > # hosted-engine --vm-start > >>>> > > >>>> > VM exists and is down, cleaning up and restarting > >>>> > > >>>> > > >>>> > > >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> > > >>>> > jsonrpc/2::ERROR::2018-01-12 > >>>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats) > >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm > >>>> > stats > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line > >>>> > 1762, > >>>> > in > >>>> > _getRunningVmStats > >>>> > > >>>> > vm_sample.interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 45, in > >>>> > produce > >>>> > > >>>> > networks(vm, stats, first_sample, last_sample, interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 322, in > >>>> > networks > >>>> > > >>>> > if nic.name.startswith('hostdev'): > >>>> > > >>>> > AttributeError: name > >>>> > > >>>> > jsonrpc/3::ERROR::2018-01-12 > >>>> > > >>>> > 11:27:27,221::__init__::611::jsonrpc.JsonRpcServer::(_handle_request) > >>>> > Internal server error > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", > >>>> > line > >>>> > 606, > >>>> > in _handle_request > >>>> > > >>>> > res = method(**params) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line > >>>> > 201, in > >>>> > _dynamicMethod > >>>> > > >>>> > result = fn(*methodArgs) > >>>> > > >>>> > File "<string>", line 2, in getAllVmIoTunePolicies > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line > >>>> > 48, > >>>> > in > >>>> > method > >>>> > > >>>> > ret = func(*args, **kwargs) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, > >>>> > in > >>>> > getAllVmIoTunePolicies > >>>> > > >>>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line > >>>> > 524, > >>>> > in > >>>> > getAllVmIoTunePolicies > >>>> > > >>>> > 'current_values': v.getIoTune()} > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line > >>>> > 3481, > >>>> > in > >>>> > getIoTune > >>>> > > >>>> > result = self.getIoTuneResponse() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line > >>>> > 3500, > >>>> > in > >>>> > getIoTuneResponse > >>>> > > >>>> > res = self._dom.blockIoTune( > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", > >>>> > line > >>>> > 47, > >>>> > in __getattr__ > >>>> > > >>>> > % self.vmid) > >>>> > > >>>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was > >>>> > not > >>>> > defined > >>>> > yet or was undefined > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR > >>>> > Internal > >>>> > server error#012Traceback (most recent call last):#012 File > >>>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, > >>>> > in > >>>> > _handle_request#012 res = method(**params)#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in > >>>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File > >>>> > "<string>", > >>>> > line 2, > >>>> > in getAllVmIoTunePolicies#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in > >>>> > method#012 ret = func(*args, **kwargs)#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > >>>> > getAllVmIoTunePolicies#012 io_tune_policies_dict = > >>>> > self._cif.getAllVmIoTunePolicies()#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in > >>>> > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 > >>>> > File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in > >>>> > getIoTune#012 result = self.getIoTuneResponse()#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in > >>>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, > >>>> > in > >>>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM > >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was > >>>> > undefined > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > blocking > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered promiscuous > >>>> > mode > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > blocking > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > forwarding state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No > >>>> > buffer > >>>> > space > >>>> > available > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4264] > >>>> > manager: (vnet4): new Tun device > >>>> > (/org/freedesktop/NetworkManager/Devices/135) > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4342] > >>>> > device (vnet4): state change: unmanaged -> unavailable (reason > >>>> > 'connection-assumed') [10 20 41] > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4353] > >>>> > device (vnet4): state change: unavailable -> disconnected (reason > >>>> > 'none') > >>>> > [20 30 0] > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: 3.2.0, > >>>> > package: > >>>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>;, > >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >>>> > cultivar0.grove.silverorange.com > >>>> > > >>>> > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> > > >>>> > > >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-114-Cultivar/master-key.aes > >>>> > -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off > >>>> > -cpu > >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> > > >>>> > > >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> > -no-user-config -nodefaults -chardev > >>>> > > >>>> > > >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-114-Cultivar/monitor.sock,server,nowait > >>>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc > >>>> > base=2018-01-12T15:27:27,driftfix=slew -global > >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > >>>> > -device > >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > >>>> > > >>>> > > >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> > -device > >>>> > > >>>> > > >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device > >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > >>>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > >>>> > > >>>> > > >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> > -chardev > >>>> > > >>>> > > >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> > -chardev > >>>> > > >>>> > > >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> > -chardev spicevmc,id=charchannel2,name=vdagent -device > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> > -chardev > >>>> > > >>>> > > >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >>>> > -chardev pty,id=charconsole0 -device > >>>> > virtconsole,chardev=charconsole0,id=console0 -spice > >>>> > > >>>> > > >>>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > >>>> > rng-random,id=objrng0,filename=/dev/urandom -device > >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > >>>> > timestamp=on > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev pty,id=charconsole0: > >>>> > char > >>>> > device redirected to /dev/pts/2 (label charconsole0) > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 15:27:27.773+0000: > >>>> > 1910: > >>>> > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed > >>>> > to > >>>> > acquire > >>>> > lock: Lease is held by another host > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 from > >>>> > pid 1773 > >>>> > (/usr/sbin/libvirtd) > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left promiscuous > >>>> > mode > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.7989] > >>>> > device (vnet4): state change: disconnected -> unmanaged (reason > >>>> > 'unmanaged') > >>>> > [30 10 3] > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.7989] > >>>> > device (vnet4): released from master device ovirtmgmt > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine > >>>> > qemu-114-Cultivar > >>>> > terminated. > >>>> > > >>>> > > >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> > > >>>> > vm/4013c829::ERROR::2018-01-12 > >>>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm) > >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process > >>>> > failed > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line > >>>> > 843, > >>>> > in > >>>> > _startUnderlyingVm > >>>> > > >>>> > self._run() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line > >>>> > 2721, > >>>> > in > >>>> > _run > >>>> > > >>>> > dom.createWithFlags(flags) > >>>> > > >>>> > File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", > >>>> > line > >>>> > 126, in wrapper > >>>> > > >>>> > ret = f(*args, **kwargs) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, > >>>> > in > >>>> > wrapper > >>>> > > >>>> > return func(inst, *args, **kwargs) > >>>> > > >>>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, > >>>> > in > >>>> > createWithFlags > >>>> > > >>>> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags() > >>>> > failed', > >>>> > dom=self) > >>>> > > >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is held > >>>> > by > >>>> > another host > >>>> > > >>>> > periodic/47::ERROR::2018-01-12 > >>>> > 11:27:32,858::periodic::215::virt.periodic.Operation::(__call__) > >>>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590> > >>>> > operation > >>>> > failed > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", > >>>> > line > >>>> > 213, > >>>> > in __call__ > >>>> > > >>>> > self._func() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", > >>>> > line > >>>> > 522, > >>>> > in __call__ > >>>> > > >>>> > self._send_metrics() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", > >>>> > line > >>>> > 538, > >>>> > in _send_metrics > >>>> > > >>>> > vm_sample.interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 45, in > >>>> > produce > >>>> > > >>>> > networks(vm, stats, first_sample, last_sample, interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 322, in > >>>> > networks > >>>> > > >>>> > if nic.name.startswith('hostdev'): > >>>> > > >>>> > AttributeError: name > >>>> > > >>>> > > >>>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak <msivak(a)redhat.com&gt; > >>>> > wrote: > >>>> >> > >>>> >> Hmm that rules out most of NFS related permission issues. > >>>> >> > >>>> >> So the current status is (I need to sum it up to get the full > >>>> >> picture): > >>>> >> > >>>> >> - HE VM is down > >>>> >> - HE agent fails when opening metadata using the symlink > >>>> >> - the symlink is there > >>>> >> - the symlink is readable by vdsm:kvm > >>>> >> > >>>> >> Hmm can you check under which user is ovirt-ha-broker started? > >>>> >> > >>>> >> Martin > >>>> >> > >>>> >> > >>>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef(a)gmail.com&gt; wrote: > >>>> >> > Same thing happens with data images of other VMs as well though, > >>>> >> > and > >>>> >> > those > >>>> >> > seem to be running ok so I'm not sure if it's the problem. > >>>> >> > > >>>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme <jaymef(a)gmail.com&gt; > >>>> >> > wrote: > >>>> >> >> > >>>> >> >> Martin, > >>>> >> >> > >>>> >> >> I can as VDSM user but not as root . I get permission denied > >>>> >> >> trying to > >>>> >> >> touch one of the files as root, is that normal? > >>>> >> >> > >>>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak > >>>> >> >> <msivak(a)redhat.com&gt; > >>>> >> >> wrote: > >>>> >> >>> > >>>> >> >>> Hmm, then it might be a permission issue indeed. Can you touch > >>>> >> >>> the > >>>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not > >>>> >> >>> prevent > >>>> >> >>> you from doing that. > >>>> >> >>> > >>>> >> >>> Martin > >>>> >> >>> > >>>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme <jaymef(a)gmail.com&gt; > >>>> >> >>> wrote: > >>>> >> >>> > Sorry, I think we got confused about the symlink, there are > >>>> >> >>> > symlinks > >>>> >> >>> > in > >>>> >> >>> > /var/run that point the /rhev when I was doing an LS it was > >>>> >> >>> > listing > >>>> >> >>> > the > >>>> >> >>> > files in /rhev > >>>> >> >>> > > >>>> >> >>> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286 > >>>> >> >>> > > >>>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a -> > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> >> >>> > > >>>> >> >>> > ls -al > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> >> >>> > total 2040 > >>>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . > >>>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. > >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease > >>>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta > >>>> >> >>> > > >>>> >> >>> > Is it possible that this is the wrong image for hosted > >>>> >> >>> > engine? > >>>> >> >>> > > >>>> >> >>> > this is all I get in vdsm log when running hosted-engine > >>>> >> >>> > --connect-storage > >>>> >> >>> > > >>>> >> >>> > jsonrpc/4::ERROR::2018-01-12 > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > 10:52:53,019::__init__::611::jsonrpc.JsonRpcServer::(_handle_request) > >>>> >> >>> > Internal server error > >>>> >> >>> > Traceback (most recent call last): > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", > >>>> >> >>> > line > >>>> >> >>> > 606, > >>>> >> >>> > in _handle_request > >>>> >> >>> > res = method(**params) > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", > >>>> >> >>> > line > >>>> >> >>> > 201, > >>>> >> >>> > in > >>>> >> >>> > _dynamicMethod > >>>> >> >>> > result = fn(*methodArgs) > >>>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", > >>>> >> >>> > line > >>>> >> >>> > 48, > >>>> >> >>> > in > >>>> >> >>> > method > >>>> >> >>> > ret = func(*args, **kwargs) > >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line > >>>> >> >>> > 1354, in > >>>> >> >>> > getAllVmIoTunePolicies > >>>> >> >>> > io_tune_policies_dict = > >>>> >> >>> > self._cif.getAllVmIoTunePolicies() > >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", > >>>> >> >>> > line > >>>> >> >>> > 524, > >>>> >> >>> > in > >>>> >> >>> > getAllVmIoTunePolicies > >>>> >> >>> > 'current_values': v.getIoTune()} > >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", > >>>> >> >>> > line > >>>> >> >>> > 3481, > >>>> >> >>> > in > >>>> >> >>> > getIoTune > >>>> >> >>> > result = self.getIoTuneResponse() > >>>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", > >>>> >> >>> > line > >>>> >> >>> > 3500, > >>>> >> >>> > in > >>>> >> >>> > getIoTuneResponse > >>>> >> >>> > res = self._dom.blockIoTune( > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", > >>>> >> >>> > line > >>>> >> >>> > 47, > >>>> >> >>> > in __getattr__ > >>>> >> >>> > % self.vmid) > >>>> >> >>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> > was not > >>>> >> >>> > defined > >>>> >> >>> > yet or was undefined > >>>> >> >>> > > >>>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak > >>>> >> >>> > <msivak(a)redhat.com&gt; > >>>> >> >>> > wrote: > >>>> >> >>> >> > >>>> >> >>> >> Hi, > >>>> >> >>> >> > >>>> >> >>> >> what happens when you try hosted-engine --connect-storage? > >>>> >> >>> >> Do > >>>> >> >>> >> you > >>>> >> >>> >> see > >>>> >> >>> >> any errors in the vdsm log? > >>>> >> >>> >> > >>>> >> >>> >> Best regards > >>>> >> >>> >> > >>>> >> >>> >> Martin Sivak > >>>> >> >>> >> > >>>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme <jaymef(a)gmail.com&gt; > >>>> >> >>> >> wrote: > >>>> >> >>> >> > Ok this is what I've done: > >>>> >> >>> >> > > >>>> >> >>> >> > - All three hosts in global maintenance mode > >>>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop > >>>> >> >>> >> > ovirt-ha-broker -- > >>>> >> >>> >> > on > >>>> >> >>> >> > all three hosts > >>>> >> >>> >> > - Moved ALL files in > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ > >>>> >> >>> >> > to > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup > >>>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl start > >>>> >> >>> >> > ovirt-ha-broker > >>>> >> >>> >> > -- > >>>> >> >>> >> > on all three hosts > >>>> >> >>> >> > > >>>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0): > >>>> >> >>> >> > hosted-engine > >>>> >> >>> >> > --vm-start > >>>> >> >>> >> > > >>>> >> >>> >> > Lots of errors in the logs still, it appears to be having > >>>> >> >>> >> > problems > >>>> >> >>> >> > with > >>>> >> >>> >> > that > >>>> >> >>> >> > directory still: > >>>> >> >>> >> > > >>>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker > >>>> >> >>> >> > > >>>> >> >>> >> > ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker > >>>> >> >>> >> > ERROR > >>>> >> >>> >> > Failed > >>>> >> >>> >> > to > >>>> >> >>> >> > write metadata for host 1 to > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8#012Traceback > >>>> >> >>> >> > (most recent call last):#012 File > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path, > >>>> >> >>> >> > direct_flag > >>>> >> >>> >> > | > >>>> >> >>> >> > os.O_WRONLY | > >>>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or > >>>> >> >>> >> > directory: > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> > > >>>> >> >>> >> > There are no new files or symlinks in > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ > >>>> >> >>> >> > > >>>> >> >>> >> > - Jayme > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak > >>>> >> >>> >> > <msivak(a)redhat.com&gt; > >>>> >> >>> >> > wrote: > >>>> >> >>> >> >> > >>>> >> >>> >> >> > Can you please stop all hosted engine tooling ( > >>>> >> >>> >> >> > >>>> >> >>> >> >> On all hosts I should have added. > >>>> >> >>> >> >> > >>>> >> >>> >> >> Martin > >>>> >> >>> >> >> > >>>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak > >>>> >> >>> >> >> <msivak(a)redhat.com&gt; > >>>> >> >>> >> >> wrote: > >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No > >>>> >> >>> >> >> >> such > >>>> >> >>> >> >> >> file > >>>> >> >>> >> >> >> or > >>>> >> >>> >> >> >> directory: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> ls -al > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are > >>>> >> >>> >> >> >> referring to > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> addressed in RC1 or something else? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > No, this file is the symlink. It should point to > >>>> >> >>> >> >> > somewhere > >>>> >> >>> >> >> > inside > >>>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is > >>>> >> >>> >> >> > really > >>>> >> >>> >> >> > interesting. > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Can you please stop all hosted engine tooling > >>>> >> >>> >> >> > (ovirt-ha-agent, > >>>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is not > >>>> >> >>> >> >> > important > >>>> >> >>> >> >> > when > >>>> >> >>> >> >> > services are stopped, but better safe than sorry) and > >>>> >> >>> >> >> > restart > >>>> >> >>> >> >> > all > >>>> >> >>> >> >> > services again? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> >> Could there possibly be a permissions > >>>> >> >>> >> >> >> problem somewhere? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Maybe, but the file itself looks out of the ordinary. > >>>> >> >>> >> >> > I > >>>> >> >>> >> >> > wonder > >>>> >> >>> >> >> > how it > >>>> >> >>> >> >> > got there. > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Best regards > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Martin Sivak > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme > >>>> >> >>> >> >> > <jaymef(a)gmail.com&gt; > >>>> >> >>> >> >> > wrote: > >>>> >> >>> >> >> >> Thanks for the help thus far. Storage could be > >>>> >> >>> >> >> >> related > >>>> >> >>> >> >> >> but > >>>> >> >>> >> >> >> all > >>>> >> >>> >> >> >> other > >>>> >> >>> >> >> >> VMs on > >>>> >> >>> >> >> >> same storage are running ok. The storage is mounted > >>>> >> >>> >> >> >> via > >>>> >> >>> >> >> >> NFS > >>>> >> >>> >> >> >> from > >>>> >> >>> >> >> >> within one > >>>> >> >>> >> >> >> of the three hosts, I realize this is not ideal. > >>>> >> >>> >> >> >> This > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> setup > >>>> >> >>> >> >> >> by > >>>> >> >>> >> >> >> a > >>>> >> >>> >> >> >> previous admin more as a proof of concept and VMs > >>>> >> >>> >> >> >> were > >>>> >> >>> >> >> >> put on > >>>> >> >>> >> >> >> there > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> should not have been placed in a proof of concept > >>>> >> >>> >> >> >> environment.. > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> intended to be rebuilt with proper storage down the > >>>> >> >>> >> >> >> road. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts mount > >>>> >> >>> >> >> >> NFS > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/data > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_data > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_iso > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/import_export > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_import__export > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/hosted_engine > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_exports_hosted__engine > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Like I said, the VM data storage itself seems to be > >>>> >> >>> >> >> >> working > >>>> >> >>> >> >> >> ok, > >>>> >> >>> >> >> >> as > >>>> >> >>> >> >> >> all > >>>> >> >>> >> >> >> other > >>>> >> >>> >> >> >> VMs appear to be running. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> I'm curious why the broker log says this file is not > >>>> >> >>> >> >> >> found > >>>> >> >>> >> >> >> when > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> is > >>>> >> >>> >> >> >> correct and I can see the file at that path: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] No > >>>> >> >>> >> >> >> such > >>>> >> >>> >> >> >> file > >>>> >> >>> >> >> >> or > >>>> >> >>> >> >> >> directory: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> ls -al > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are > >>>> >> >>> >> >> >> referring to > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> addressed in RC1 or something else? Could there > >>>> >> >>> >> >> >> possibly be > >>>> >> >>> >> >> >> a > >>>> >> >>> >> >> >> permissions > >>>> >> >>> >> >> >> problem somewhere? > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms installed > >>>> >> >>> >> >> >> and the > >>>> >> >>> >> >> >> host > >>>> >> >>> >> >> >> engine > >>>> >> >>> >> >> >> will not start is it safe for me to update hosts to > >>>> >> >>> >> >> >> 4.2 > >>>> >> >>> >> >> >> RC1 > >>>> >> >>> >> >> >> rpms? > >>>> >> >>> >> >> >> Or > >>>> >> >>> >> >> >> perhaps install that repo and *only* update the ovirt > >>>> >> >>> >> >> >> HA > >>>> >> >>> >> >> >> packages? > >>>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates to > >>>> >> >>> >> >> >> the > >>>> >> >>> >> >> >> inaccessible > >>>> >> >>> >> >> >> hosted > >>>> >> >>> >> >> >> engine VM. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> I should also mention one more thing. I originally > >>>> >> >>> >> >> >> upgraded > >>>> >> >>> >> >> >> the > >>>> >> >>> >> >> >> engine > >>>> >> >>> >> >> >> VM > >>>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed > >>>> >> >>> >> >> >> due > >>>> >> >>> >> >> >> to not > >>>> >> >>> >> >> >> being > >>>> >> >>> >> >> >> in > >>>> >> >>> >> >> >> global maintenance, so I set global maintenance and > >>>> >> >>> >> >> >> ran > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> again, > >>>> >> >>> >> >> >> which > >>>> >> >>> >> >> >> appeared to complete as intended but never came back > >>>> >> >>> >> >> >> up > >>>> >> >>> >> >> >> after. > >>>> >> >>> >> >> >> Just > >>>> >> >>> >> >> >> in > >>>> >> >>> >> >> >> case > >>>> >> >>> >> >> >> this might have anything at all to do with what could > >>>> >> >>> >> >> >> have > >>>> >> >>> >> >> >> happened. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Thanks very much again, I very much appreciate the > >>>> >> >>> >> >> >> help! > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> - Jayme > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi > >>>> >> >>> >> >> >> <stirabos(a)redhat.com&gt; > >>>> >> >>> >> >> >> wrote: > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak > >>>> >> >>> >> >> >>> <msivak(a)redhat.com&gt; > >>>> >> >>> >> >> >>> wrote: > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Hi, > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by > >>>> >> >>> >> >> >>>> restarting > >>>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest > >>>> >> >>> >> >> >>>> ovirt-hosted-engine-ha > >>>> >> >>> >> >> >>>> and > >>>> >> >>> >> >> >>>> -setup. We improved handling of the missing > >>>> >> >>> >> >> >>>> symlink. > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1 > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> All the other issues seem to point to some storage > >>>> >> >>> >> >> >>>> problem > >>>> >> >>> >> >> >>>> I > >>>> >> >>> >> >> >>>> am > >>>> >> >>> >> >> >>>> afraid. > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> You said you started the VM, do you see it in virsh > >>>> >> >>> >> >> >>>> -r > >>>> >> >>> >> >> >>>> list? > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Best regards > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Martin Sivak > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme > >>>> >> >>> >> >> >>>> <jaymef(a)gmail.com&gt; > >>>> >> >>> >> >> >>>> wrote: > >>>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to try > >>>> >> >>> >> >> >>>> > at > >>>> >> >>> >> >> >>>> > this > >>>> >> >>> >> >> >>>> > point. > >>>> >> >>> >> >> >>>> > Thank > >>>> >> >>> >> >> >>>> > you > >>>> >> >>> >> >> >>>> > for reading! > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted > >>>> >> >>> >> >> >>>> > engine > >>>> >> >>> >> >> >>>> > running > >>>> >> >>> >> >> >>>> > after a > >>>> >> >>> >> >> >>>> > botched > >>>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from > >>>> >> >>> >> >> >>>> > within > >>>> >> >>> >> >> >>>> > one > >>>> >> >>> >> >> >>>> > of > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > hosts. > >>>> >> >>> >> >> >>>> > Right > >>>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully updated > >>>> >> >>> >> >> >>>> > with > >>>> >> >>> >> >> >>>> > yum > >>>> >> >>> >> >> >>>> > packages > >>>> >> >>> >> >> >>>> > from > >>>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with yum > >>>> >> >>> >> >> >>>> > packages > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > failed to > >>>> >> >>> >> >> >>>> > come > >>>> >> >>> >> >> >>>> > up after reboot. As of right now, everything > >>>> >> >>> >> >> >>>> > should > >>>> >> >>> >> >> >>>> > have > >>>> >> >>> >> >> >>>> > full > >>>> >> >>> >> >> >>>> > yum > >>>> >> >>> >> >> >>>> > updates > >>>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global > >>>> >> >>> >> >> >>>> > maintenance > >>>> >> >>> >> >> >>>> > mode > >>>> >> >>> >> >> >>>> > on > >>>> >> >>> >> >> >>>> > right > >>>> >> >>> >> >> >>>> > now > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > started hosted-engine on one of the three host > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > status is > >>>> >> >>> >> >> >>>> > currently: > >>>> >> >>> >> >> >>>> > Engine status : {"reason": "failed liveliness > >>>> >> >>> >> >> >>>> > check”; > >>>> >> >>> >> >> >>>> > "health": > >>>> >> >>> >> >> >>>> > "bad", > >>>> >> >>> >> >> >>>> > "vm": > >>>> >> >>> >> >> >>>> > "up", "detail": "Up"} > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > this is what I get when trying to enter hosted-vm > >>>> >> >>> >> >> >>>> > --console > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > The engine VM is running on this host > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > error: Domain not found: no domain with matching > >>>> >> >>> >> >> >>>> > name > >>>> >> >>> >> >> >>>> > 'HostedEngine' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Here are logs from various sources when I start > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > VM on > >>>> >> >>> >> >> >>>> > HOST3: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > hosted-engine --vm-start > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID': > >>>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} failed: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not exist: > >>>> >> >>> >> >> >>>> > {'vmId': > >>>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'}) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: New > >>>> >> >>> >> >> >>>> > machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started > >>>> >> >>> >> >> >>>> > Virtual > >>>> >> >>> >> >> >>>> > Machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting > >>>> >> >>> >> >> >>>> > Virtual > >>>> >> >>> >> >> >>>> > Machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now > >>>> >> >>> >> >> >>>> > active > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 48, > >>>> >> >>> >> >> >>>> > in > >>>> >> >>> >> >> >>>> > method > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ret = func(*args, **kwargs) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 2718, in > >>>> >> >>> >> >> >>>> > getStorageDomainInfo > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 304, in > >>>> >> >>> >> >> >>>> > validateSdUUID > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > sdDom.validate() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 515, > >>>> >> >>> >> >> >>>> > in validate > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > raise > >>>> >> >>> >> >> >>>> > se.StorageDomainAccessError(self.sdUUID) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either > >>>> >> >>> >> >> >>>> > partially > >>>> >> >>> >> >> >>>> > accessible > >>>> >> >>> >> >> >>>> > or > >>>> >> >>> >> >> >>>> > entirely > >>>> >> >>> >> >> >>>> > inaccessible: > >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82::storage.Dispatcher::(wrapper) > >>>> >> >>> >> >> >>>> > FINISH > >>>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either > >>>> >> >>> >> >> >>>> > partially > >>>> >> >>> >> >> >>>> > accessible > >>>> >> >>> >> >> >>>> > or > >>>> >> >>> >> >> >>>> > entirely > >>>> >> >>> >> >> >>>> > inaccessible: > >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down, > >>>> >> >>> >> >> >>>> > reason=shutdown > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up libvirt > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 3.2.0, > >>>> >> >>> >> >> >>>> > package: > >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem > >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>;, > >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >>>> >> >>> >> >> >>>> > cultivar3 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down, > >>>> >> >>> >> >> >>>> > reason=shutdown > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up libvirt > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 3.2.0, > >>>> >> >>> >> >> >>>> > package: > >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem > >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>;, > >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >>>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon chardev=charmonitor,id=monitor,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name=vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsole0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/broker.log > >>>> >> >>> >> >> >>>> > <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> >> >>>> > line 151, in get_raw_stats > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | os.O_RDONLY | > >>>> >> >>> >> >> >>>> > os.O_SYNC) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or directory: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:55:15,761::status_broker::92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) > >>>> >> >>> >> >> >>>> > Failed to read state. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", > >>>> >> >>> >> >> >>>> > line 88, in run > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> >> >>>> > line 162, in get_raw_stats > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > .format(str(e))) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno 2] > >>>> >> >>> >> >> >>>> > No > >>>> >> >>> >> >> >>>> > such > >>>> >> >>> >> >> >>>> > file or > >>>> >> >>> >> >> >>>> > directory: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-ha/agent.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > result = refresh_method() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > content = > >>>> >> >>> >> >> >>>> > self._get_file_content_from_shared_storage(VM) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 484, in > >>>> >> >>> >> >> >>>> > _get_file_content_from_shared_storage > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > config_volume_path = > >>>> >> >>> >> >> >>>> > self._get_config_volume_path() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > conf_vol_uuid > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py", > >>>> >> >>> >> >> >>>> > line 358, in get_volume_path > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR, > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > RuntimeError: Path to volume > >>>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f > >>>> >> >>> >> >> >>>> > not > >>>> >> >>> >> >> >>>> > found > >>>> >> >>> >> >> >>>> > in /run/vdsm/storag > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260::virt.vmstats::(send_metrics) > >>>> >> >>> >> >> >>>> > VM > >>>> >> >>> >> >> >>>> > metrics > >>>> >> >>> >> >> >>>> > collection failed > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 197, in > >>>> >> >>> >> >> >>>> > send_metrics > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] = > >>>> >> >>> >> >> >>>> > stat['cpuUsage'] > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > KeyError: 'cpuUsage' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > _______________________________________________ > >>>> >> >>> >> >> >>>> > Users mailing list > >>>> >> >>> >> >> >>>> > Users(a)ovirt.org > >>>> >> >>> >> >> >>>> > http://lists.ovirt.org/mailman/listinfo/users > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> _______________________________________________ > >>>> >> >>> >> >> >>>> Users mailing list > >>>> >> >>> >> >> >>>> Users(a)ovirt.org > >>>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman/listinfo/users > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >> > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >> > >>>> >> >> > >>>> >> > > >>>> > > >>>> > > >>> > >>> > >> > >