Re: [ovirt-users] unable to bring up hosted engine after botched 4.2 upgrade
No luck I'm afraid. It's very odd that I wouldn't be able to get a console to it, if the status is up and seen by virsh. Any clue? Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} # virsh -r list Id Name State ---------------------------------------------------- 118 Cultivar running # hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' # hosted-engine --console 118 The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' # hosted-engine --console Cultivar The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' On Fri, Jan 12, 2018 at 2:05 PM, Martin Sivak <msivak@redhat.com> wrote:
Try listing the domains with
virsh -r list
maybe it just has some weird name...
Martin
I thought that it might be a good sign but unfortunately I cannot access it with console :( if I could get console access to it I might be able to fix the problem. But seeing is how the console is also not working leads me to believe there is a bigger issue at hand here.
hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
I really wonder if this is all a symlinking problem in some way. Is it possible for me to upgrade host to 4.2 RC2 without being able to upgrade
engine first or should I keep everything on 4.2 as it is?
On Fri, Jan 12, 2018 at 1:49 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
the VM is up according to the status (at least for a while). You should be able to use console and diagnose anything that happened inside (line the need for fsck and such) now.
Check the presence of those links again now, the metadata file content is not important, but the file has to exist (agents will populate it with status data). I have no new idea about what is wrong with that though.
Best regards
Martin
On Fri, Jan 12, 2018 at 5:47 PM, Jayme <jaymef@gmail.com> wrote:
The lock space issue was an issue I needed to clear but I don't
believe
it has resolved the problem. I shutdown agent and broker on all hosts and disconnected hosted-storage then enabled broker/agent on just one host and connected storage. I started the VM and actually didn't get any errors in the logs barely at all which was good to see, however the VM is still not running:
HOST3:
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
==> /var/log/messages <== Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer space available Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 kvm: 3 guests now active Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535: error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU guest agent is not connected
Interestingly though, now I'm seeing this in the logs which may be a new clue:
==> /var/log/vdsm/vdsm.log <== File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 126, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 116, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) jsonrpc/4::ERROR::2018-01-12 12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH getStorageDomainInfo error=Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished?
==> /var/log/ovirt-hosted-engine-ha/broker.log <== File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ ha/broker/storage_broker.py", line 151, in get_raw_stats f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8' StatusStorageThread::ERROR::2018-01-12
12:32:06,049::status_broker::92::ovirt_hosted_engine_ha. broker.status_broker.StatusBroker.Update::(run) Failed to read state. Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ ha/broker/status_broker.py", line 88, in run self._storage_broker.get_raw_stats() File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ ha/broker/storage_broker.py", line 162, in get_raw_stats .format(str(e))) RequestError: failed to read metadata: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8'
On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak@redhat.com> wrote:
The lock is the issue.
- try running sanlock client status on all hosts - also make sure you do not have some forgotten host still connected to the lockspace, but without ha daemons running (and with the VM)
I need to go to our president election now, I might check the email later tonight.
Martin
On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef@gmail.com> wrote:
Here are the newest logs from me trying to start hosted vm:
==> /var/log/messages <== Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered blocking state Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered disabled state Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered promiscuous mode Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered blocking state Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered forwarding state Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No buffer space available Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8715] manager: (vnet4): new Tun device (/org/freedesktop/NetworkManager/Devices/140) Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8795] device (vnet4): state change: unmanaged -> unavailable (reason 'connection-assumed') [10 20 41]
==> /var/log/libvirt/qemu/Cultivar.log <== 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: cultivar0.grove.silverorange.com LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=Cultivar,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
qemu/domain-119-Cultivar/master-key.aes
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m 8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios 'type=1,manufacturer=oVirt,product=oVirt
Node,version=7-4.1708.el7.centos,serial=44454C4C-3300- 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain- 119-Cultivar/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2018-01-12T15:58:14,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5- fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0, serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none, werror=stop,rerror=stop,aio=threads -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e: 7f:d6:83,bus=pci.0,addr=0x3 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev= charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait -device
virtserialport,bus=virtio-serial0.0,nr=2,chardev= charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev= charchannel2,id=channel2,name=com.redhat.spice.0 -chardev
socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted- engine-setup.0,server,nowait -device
virtserialport,bus=virtio-serial0.0,nr=4,chardev= charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice
tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice, tls-channel=default,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg timestamp=on
==> /var/log/messages <== Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> [1515772694.8807] device (vnet4): state change: unavailable -> disconnected (reason 'none') [20 30 0] Jan 12 11:58:14 cultivar0 systemd-machined: New machine qemu-119-Cultivar. Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine qemu-119-Cultivar. Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine qemu-119-Cultivar.
==> /var/log/libvirt/qemu/Cultivar.log <== 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev pty,id=charconsole0: char device redirected to /dev/pts/1 (label charconsole0)
==> /var/log/messages <== Jan 12 11:58:15 cultivar0 kvm: 5 guests now active
==> /var/log/libvirt/qemu/Cultivar.log <== 2018-01-12 15:58:15.217+0000: shutting down, reason=failed
==> /var/log/messages <== Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: 1908: error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to acquire lock: Lease is held by another host
==> /var/log/libvirt/qemu/Cultivar.log <== 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from pid 1773 (/usr/sbin/libvirtd)
==> /var/log/messages <== Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered disabled state Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered disabled state Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> [1515772695.2348] device (vnet4): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> [1515772695.2349] device (vnet4): released from master device ovirtmgmt Jan 12 11:58:15 cultivar0 kvm: 4 guests now active Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar terminated.
==> /var/log/vdsm/vdsm.log <== vm/4013c829::ERROR::2018-01-12 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm) (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start process failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, in _run dom.createWithFlags(flags) File "/usr/lib/python2.7/site-packages/vdsm/
line 126, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirtError: resource busy: Failed to acquire lock: Lease is held by another host jsonrpc/6::ERROR::2018-01-12 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_ handle_request) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmIoTunePolicies File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in getAllVmIoTunePolicies 'current_values': v.getIoTune()} File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in getIoTune result = self.getIoTuneResponse() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in getIoTuneResponse res = self._dom.blockIoTune( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was undefined
==> /var/log/messages <== Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server error#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request#012 res = method(**params)#012 File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>", line 2, in getAllVmIoTunePolicies#012 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method#012 ret = func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in getAllVmIoTunePolicies#012 io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()#012 File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in getIoTune#012 result = self.getIoTuneResponse()#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__#012 % self.vmid)#012NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was undefined
On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef@gmail.com> wrote: > > One other tidbit I noticed is that it seems like there are less > errors > if > I started in paused mode: > > but still shows: Engine status : {"reason": > "bad > vm > status", "health": "bad", "vm": "up", "detail": "Paused"} > > ==> /var/log/messages <== > Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > blocking state > Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > disabled state > Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered promiscuous > mode > Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > blocking state > Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > forwarding state > Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No > buffer > space available > Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > [1515772505.3625] > manager: (vnet4): new Tun device > (/org/freedesktop/NetworkManager/Devices/139) > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0, > package: > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > cultivar0.grove.silverorange.com > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > guest=Cultivar,debug-threads=on -S -object > > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/ qemu/domain-118-Cultivar/master-key.aes > -machine pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > -cpu > Conroe -m 8192 -realtime mlock=off -smp > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > 'type=1,manufacturer=oVirt,product=oVirt > > > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300- 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain- 118-Cultivar/monitor.sock,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=2018-01-12T15:55:05,driftfix=slew -global > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > > > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5- fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0, serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none, werror=stop,rerror=stop,aio=threads > -device > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 > -drive if=none,id=drive-ide0-1-0,readonly=on -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e: 7f:d6:83,bus=pci.0,addr=0x3 > -chardev > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev= charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > -chardev > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev= charchannel1,id=channel1,name=org.qemu.guest_agent.0 > -chardev spicevmc,id=charchannel2,name=vdagent -device > > > virtserialport,bus=virtio-serial0.0,nr=3,chardev= charchannel2,id=channel2,name=com.redhat.spice.0 > -chardev > > > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted- engine-setup.0,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=4,chardev= charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > -chardev pty,id=charconsole0 -device > virtconsole,chardev=charconsole0,id=console0 -spice > > > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice, tls-channel=default,seamless-migration=on > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > rng-random,id=objrng0,filename=/dev/urandom -device > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > timestamp=on > > ==> /var/log/messages <== > Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > [1515772505.3689] > device (vnet4): state change: unmanaged -> unavailable (reason > 'connection-assumed') [10 20 41] > Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > [1515772505.3702] > device (vnet4): state change: unavailable -> disconnected (reason > 'none') > [20 30 0] > Jan 12 11:55:05 cultivar0 systemd-machined: New machine > qemu-118-Cultivar. > Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine > qemu-118-Cultivar. > Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine > qemu-118-Cultivar. > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev
> char > device redirected to /dev/pts/1 (label charconsole0) > > ==> /var/log/messages <== > Jan 12 11:55:05 cultivar0 kvm: 5 guests now active > > On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef@gmail.com> wrote: >> >> Yeah I am in global maintenance: >> >> state=GlobalMaintenance >> >> host0: {"reason": "vm not running on this host", "health": "bad", >> "vm": >> "down", "detail": "unknown"} >> host2: {"reason": "vm not running on this host", "health": "bad", >> "vm": >> "down", "detail": "unknown"} >> host3: {"reason": "vm not running on this host", "health": "bad", >> "vm": >> "down", "detail": "unknown"} >> >> I understand the lock is an issue, I'll try to make sure it is >> fully >> stopped on all three before starting but I don't think that is
>> issue at >> hand either. What concerns me is mostly that it seems to be >> unable >> to read >> the meta data, I think that might be the heart of the problem but >> I'm >> not >> sure what is causing it. >> >> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak < msivak@redhat.com> >> wrote: >>> >>> > On all three hosts I ran hosted-engine --vm-shutdown; >>> > hosted-engine >>> > --vm-poweroff >>> >>> Are you in global maintenance? I think you were in one of the >>> previous >>> emails, but worth checking. >>> >>> > I started ovirt-ha-broker with systemctl as root user but it >>> > does >>> > appear to be running under vdsm: >>> >>> That is the correct behavior. >>> >>> > libvirtError: resource busy: Failed to acquire lock: Lease is >>> > held >>> > by >>> > another host >>> >>> sanlock seems to think the VM runs somewhere and it is possible >>> that >>> some other host tried to start the VM as well unless you are in >>> global >>> maintenance (that is why I asked the first question here). >>> >>> Martin >>> >>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef@gmail.com> wrote: >>> > Martin, >>> > >>> > Thanks so much for keeping with me, this is driving me crazy! I >>> > really do >>> > appreciate it, thanks again >>> > >>> > Let's go through this: >>> > >>> > HE VM is down - YES >>> > >>> > >>> > HE agent fails when opening metadata using the symlink - YES >>> > >>> > >>> > the symlink is there and readable by vdsm:kvm - it appears to >>> > be: >>> > >>> > >>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20 >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a >>> > -> >>> > >>> > >>> > >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange. com:_exports_hosted__engine/248f46f0-d793-4581-9810- c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>> > >>> > >>> > And the files in the linked directory exist and have vdsm:kvm >>> > perms >>> > as >>> > well: >>> > >>> > >>> > # cd >>> > >>> > >>> > >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange. com:_exports_hosted__engine/248f46f0-d793-4581-9810- c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>> > >>> > [root@cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al >>> > >>> > total 2040 >>> > >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . >>> > >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. >>> > >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19 >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>> > >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease >>> > >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta >>> > >>> > >>> > I started ovirt-ha-broker with systemctl as root user but it >>> > does >>> > appear to >>> > be running under vdsm: >>> > >>> > >>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18 >>> > /usr/bin/python >>> > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>> > >>> > >>> > >>> > Here is something I tried: >>> > >>> > >>> > - On all three hosts I ran hosted-engine --vm-shutdown; >>> > hosted-engine >>> > --vm-poweroff >>> > >>> > - On HOST0 (cultivar0) I disconnected and reconnected storage >>> > using >>> > hosted-engine >>> > >>> > - Tried starting up the hosted VM on cultivar0 while tailing
>>> > logs: >>> > >>> > >>> > # hosted-engine --vm-start >>> > >>> > VM exists and is down, cleaning up and restarting >>> > >>> > >>> > >>> > ==> /var/log/vdsm/vdsm.log <== >>> > >>> > jsonrpc/2::ERROR::2018-01-12 >>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats) >>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm >>> > stats >>> > >>> > Traceback (most recent call last): >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>> > 1762, >>> > in >>> > _getRunningVmStats >>> > >>> > vm_sample.interval) >>> > >>> > File "/usr/lib/python2.7/site-
>>> > line >>> > 45, in >>> > produce >>> > >>> > networks(vm, stats, first_sample, last_sample, interval) >>> > >>> > File "/usr/lib/python2.7/site-
>>> > line >>> > 322, in >>> > networks >>> > >>> > if nic.name.startswith('hostdev'): >>> > >>> > AttributeError: name >>> > >>> > jsonrpc/3::ERROR::2018-01-12 >>> > >>> > >>> > 11:27:27,221::__init__::611::jsonrpc.JsonRpcServer::(_ handle_request) >>> > Internal server error >>> > >>> > Traceback (most recent call last): >>> > >>> > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.
>>> > line >>> > 606, >>> > in _handle_request >>> > >>> > res = method(**params) >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", >>> > line >>> > 201, in >>> > _dynamicMethod >>> > >>> > result = fn(*methodArgs) >>> > >>> > File "<string>", line 2, in getAllVmIoTunePolicies >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", >>> > line >>> > 48, >>> > in >>> > method >>> > >>> > ret = func(*args, **kwargs) >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line >>> > 1354, >>> > in >>> > getAllVmIoTunePolicies >>> > >>> > io_tune_policies_dict = self._cif. getAllVmIoTunePolicies() >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
>>> > 524, >>> > in >>> > getAllVmIoTunePolicies >>> > >>> > 'current_values': v.getIoTune()} >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>> > 3481, >>> > in >>> > getIoTune >>> > >>> > result = self.getIoTuneResponse() >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>> > 3500, >>> > in >>> > getIoTuneResponse >>> > >>> > res = self._dom.blockIoTune( >>> > >>> > File >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", >>> > line >>> > 47, >>> > in __getattr__ >>> > >>> > % self.vmid) >>> > >>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was >>> > not >>> > defined >>> > yet or was undefined >>> > >>> > >>> > ==> /var/log/messages <== >>> > >>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer >>> > ERROR >>> > Internal >>> > server error#012Traceback (most recent call last):#012 File >>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
>>> > 606, >>> > in >>> > _handle_request#012 res = method(**params)#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, >>> > in >>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File >>> > "<string>", >>> > line 2, >>> > in getAllVmIoTunePolicies#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, >>> > in >>> > method#012 ret = func(*args, **kwargs)#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in >>> > getAllVmIoTunePolicies#012 io_tune_policies_dict = >>> > self._cif.getAllVmIoTunePolicies()#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, >>> > in >>> > getAllVmIoTunePolicies#012 'current_values': >>> > v.getIoTune()}#012 >>> > File >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, >>> > in >>> > getIoTune#012 result = self.getIoTuneResponse()#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, >>> > in >>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
>>> > 47, >>> > in >>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM >>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or >>> > was >>> > undefined >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > blocking >>> > state >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > disabled >>> > state >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered >>> > promiscuous >>> > mode >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > blocking >>> > state >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > forwarding state >>> > >>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No >>> > buffer >>> > space >>> > available >>> > >>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>> > [1515770847.4264] >>> > manager: (vnet4): new Tun device >>> > (/org/freedesktop/NetworkManager/Devices/135) >>> > >>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>> > [1515770847.4342] >>> > device (vnet4): state change: unmanaged -> unavailable (reason >>> > 'connection-assumed') [10 20 41] >>> > >>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>> > [1515770847.4353] >>> > device (vnet4): state change: unavailable -> disconnected >>> > (reason >>> > 'none') >>> > [20 30 0] >>> > >>> > >>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>> > >>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: >>> > 3.2.0, >>> > package: >>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, >>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: >>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: >>> > cultivar0.grove.silverorange.com >>> > >>> > LC_ALL=C PATH=/usr/local/sbin:/usr/ local/bin:/usr/sbin:/usr/bin >>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name >>> > guest=Cultivar,debug-threads=on -S -object >>> > >>> > >>> > >>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/ qemu/domain-114-Cultivar/master-key.aes >>> > -machine >>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off >>> > -cpu >>> > Conroe -m 8192 -realtime mlock=off -smp >>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid >>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>> > 'type=1,manufacturer=oVirt,product=oVirt >>> > >>> > >>> > >>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300- 1042-8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' >>> > -no-user-config -nodefaults -chardev >>> > >>> > >>> > >>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain- 114-Cultivar/monitor.sock,server,nowait >>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc >>> > base=2018-01-12T15:27:27,driftfix=slew -global >>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot >>> > strict=on >>> > -device >>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device >>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>> > >>> > >>> > >>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5- fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0, serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none, werror=stop,rerror=stop,aio=threads >>> > -device >>> > >>> > >>> > >>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 >>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device >>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev >>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device >>> > >>> > >>> > >>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e: 7f:d6:83,bus=pci.0,addr=0x3 >>> > -chardev >>> > >>> > >>> > >>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >>> > -device >>> > >>> > >>> > >>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev= charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>> > -chardev >>> > >>> > >>> > >>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait >>> > -device >>> > >>> > >>> > >>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev= charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>> > -chardev spicevmc,id=charchannel2,name=vdagent -device >>> > >>> > >>> > >>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev= charchannel2,id=channel2,name=com.redhat.spice.0 >>> > -chardev >>> > >>> > >>> > >>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/ 4013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted- engine-setup.0,server,nowait >>> > -device >>> > >>> > >>> > >>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev= charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 >>> > -chardev pty,id=charconsole0 -device >>> > virtconsole,chardev=charconsole0,id=console0 -spice >>> > >>> > >>> > >>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice, tls-channel=default,seamless-migration=on >>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object >>> > rng-random,id=objrng0,filename=/dev/urandom -device >>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg >>> > timestamp=on >>> > >>> > >>> > ==> /var/log/messages <== >>> > >>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine >>> > qemu-114-Cultivar. >>> > >>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine >>> > qemu-114-Cultivar. >>> > >>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine >>> > qemu-114-Cultivar. >>> > >>> > >>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>> > >>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev >>> > pty,id=charconsole0: >>> > char >>> > device redirected to /dev/pts/2 (label charconsole0) >>> > >>> > >>> > ==> /var/log/messages <== >>> > >>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active >>> > >>> > >>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>> > >>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed >>> > >>> > >>> > ==> /var/log/messages <== >>> > >>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 >>> > 15:27:27.773+0000: >>> > 1910: >>> > error : virLockManagerSanlockAcquire:1041 : resource busy: >>> > Failed >>> > to >>> > acquire >>> > lock: Lease is held by another host >>> > >>> > >>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>> > >>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 >>> > from >>> > pid 1773 >>> > (/usr/sbin/libvirtd) >>> > >>> > >>> > ==> /var/log/messages <== >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > disabled >>> > state >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left
>>> > mode >>> > >>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>> > entered >>> > disabled >>> > state >>> > >>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>> > [1515770847.7989] >>> > device (vnet4): state change: disconnected -> unmanaged (reason >>> > 'unmanaged') >>> > [30 10 3] >>> > >>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>> > [1515770847.7989] >>> > device (vnet4): released from master device ovirtmgmt >>> > >>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active >>> > >>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine >>> > qemu-114-Cultivar >>> > terminated. >>> > >>> > >>> > ==> /var/log/vdsm/vdsm.log <== >>> > >>> > vm/4013c829::ERROR::2018-01-12 >>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm) >>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start >>> > process >>> > failed >>> > >>> > Traceback (most recent call last): >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>> > 843, >>> > in >>> > _startUnderlyingVm >>> > >>> > self._run() >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>> > 2721, >>> > in >>> > _run >>> > >>> > dom.createWithFlags(flags) >>> > >>> > File >>> > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", >>> > line >>> > 126, in wrapper >>> > >>> > ret = f(*args, **kwargs) >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line >>> > 512, >>> > in >>> > wrapper >>> > >>> > return func(inst, *args, **kwargs) >>> > >>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line >>> > 1069, >>> > in >>> > createWithFlags >>> > >>> > if ret == -1: raise libvirtError >>> > ('virDomainCreateWithFlags() >>> > failed', >>> > dom=self) >>> > >>> > libvirtError: resource busy: Failed to acquire lock: Lease is >>> > held >>> > by >>> > another host >>> > >>> > periodic/47::ERROR::2018-01-12 >>> > 11:27:32,858::periodic::215::virt.periodic.Operation::(__ call__) >>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590> >>> > operation >>> > failed >>> > >>> > Traceback (most recent call last): >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.
>>> > line >>> > 213, >>> > in __call__ >>> > >>> > self._func() >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.
>>> > line >>> > 522, >>> > in __call__ >>> > >>> > self._send_metrics() >>> > >>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.
>>> > line >>> > 538, >>> > in _send_metrics >>> > >>> > vm_sample.interval) >>> > >>> > File "/usr/lib/python2.7/site-
>>> > line >>> > 45, in >>> > produce >>> > >>> > networks(vm, stats, first_sample, last_sample, interval) >>> > >>> > File "/usr/lib/python2.7/site-
>>> > line >>> > 322, in >>> > networks >>> > >>> > if nic.name.startswith('hostdev'): >>> > >>> > AttributeError: name >>> > >>> > >>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak >>> > <msivak@redhat.com> >>> > wrote: >>> >> >>> >> Hmm that rules out most of NFS related permission issues. >>> >> >>> >> So the current status is (I need to sum it up to get the full >>> >> picture): >>> >> >>> >> - HE VM is down >>> >> - HE agent fails when opening metadata using the symlink >>> >> - the symlink is there >>> >> - the symlink is readable by vdsm:kvm >>> >> >>> >> Hmm can you check under which user is ovirt-ha-broker started? >>> >> >>> >> Martin >>> >> >>> >> >>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef@gmail.com> >>> >> wrote: >>> >> > Same thing happens with data images of other VMs as well >>> >> > though, >>> >> > and >>> >> > those >>> >> > seem to be running ok so I'm not sure if it's the problem. >>> >> > >>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme <jaymef@gmail.com> >>> >> > wrote: >>> >> >> >>> >> >> Martin, >>> >> >> >>> >> >> I can as VDSM user but not as root . I get permission denied >>> >> >> trying to >>> >> >> touch one of the files as root, is that normal? >>> >> >> >>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak >>> >> >> <msivak@redhat.com> >>> >> >> wrote: >>> >> >>> >>> >> >>> Hmm, then it might be a permission issue indeed. Can you >>> >> >>> touch >>> >> >>> the >>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not >>> >> >>> prevent >>> >> >>> you from doing that. >>> >> >>> >>> >> >>> Martin >>> >> >>> >>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme <jaymef@gmail.com
>>> >> >>> wrote: >>> >> >>> > Sorry, I think we got confused about the symlink, there >>> >> >>> > are >>> >> >>> > symlinks >>> >> >>> > in >>> >> >>> > /var/run that point the /rhev when I was doing an LS it >>> >> >>> > was >>> >> >>> > listing >>> >> >>> > the >>> >> >>> > files in /rhev >>> >> >>> > >>> >> >>> > >>> >> >>> > /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286 >>> >> >>> > >>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a -> >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange. com:_exports_hosted__engine/248f46f0-d793-4581-9810- c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>> >> >>> > >>> >> >>> > ls -al >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > /rhev/data-center/mnt/cultivar0.grove.silverorange. com:_exports_hosted__engine/248f46f0-d793-4581-9810- c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>> >> >>> > total 2040 >>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . >>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. >>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56 >>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 >>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease >>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 >>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta >>> >> >>> > >>> >> >>> > Is it possible that this is the wrong image for hosted >>> >> >>> > engine? >>> >> >>> > >>> >> >>> > this is all I get in vdsm log when running hosted-engine >>> >> >>> > --connect-storage >>> >> >>> > >>> >> >>> > jsonrpc/4::ERROR::2018-01-12 >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > 10:52:53,019::__init__::611::jsonrpc.JsonRpcServer::(_ handle_request) >>> >> >>> > Internal server error >>> >> >>> > Traceback (most recent call last): >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.
>>> >> >>> > line >>> >> >>> > 606, >>> >> >>> > in _handle_request >>> >> >>> > res = method(**params) >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", >>> >> >>> > line >>> >> >>> > 201, >>> >> >>> > in >>> >> >>> > _dynamicMethod >>> >> >>> > result = fn(*methodArgs) >>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", >>> >> >>> > line >>> >> >>> > 48, >>> >> >>> > in >>> >> >>> > method >>> >> >>> > ret = func(*args, **kwargs) >>> >> >>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", >>> >> >>> > line >>> >> >>> > 1354, in >>> >> >>> > getAllVmIoTunePolicies >>> >> >>> > io_tune_policies_dict = >>> >> >>> > self._cif.getAllVmIoTunePolicies() >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", >>> >> >>> > line >>> >> >>> > 524, >>> >> >>> > in >>> >> >>> > getAllVmIoTunePolicies >>> >> >>> > 'current_values': v.getIoTune()} >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", >>> >> >>> > line >>> >> >>> > 3481, >>> >> >>> > in >>> >> >>> > getIoTune >>> >> >>> > result = self.getIoTuneResponse() >>> >> >>> > File >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", >>> >> >>> > line >>> >> >>> > 3500, >>> >> >>> > in >>> >> >>> > getIoTuneResponse >>> >> >>> > res = self._dom.blockIoTune( >>> >> >>> > File >>> >> >>> > >>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.
>>> >> >>> > line >>> >> >>> > 47, >>> >> >>> > in __getattr__ >>> >> >>> > % self.vmid) >>> >> >>> > NotConnectedError: VM >>> >> >>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' >>> >> >>> > was not >>> >> >>> > defined >>> >> >>> > yet or was undefined >>> >> >>> > >>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak >>> >> >>> > <msivak@redhat.com> >>> >> >>> > wrote: >>> >> >>> >> >>> >> >>> >> Hi, >>> >> >>> >> >>> >> >>> >> what happens when you try hosted-engine >>> >> >>> >> --connect-storage? >>> >> >>> >> Do >>> >> >>> >> you >>> >> >>> >> see >>> >> >>> >> any errors in the vdsm log? >>> >> >>> >> >>> >> >>> >> Best regards >>> >> >>> >> >>> >> >>> >> Martin Sivak >>> >> >>> >> >>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme >>> >> >>> >> <jaymef@gmail.com> >>> >> >>> >> wrote: >>> >> >>> >> > Ok this is what I've done: >>> >> >>> >> > >>> >> >>> >> > - All three hosts in global maintenance mode >>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop >>> >> >>> >> > ovirt-ha-broker -- >>> >> >>> >> > on >>> >> >>> >> > all three hosts >>> >> >>> >> > - Moved ALL files in >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ >>> >> >>> >> > to >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup >>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl >>> >> >>> >> > start >>> >> >>> >> > ovirt-ha-broker >>> >> >>> >> > -- >>> >> >>> >> > on all three hosts >>> >> >>> >> > >>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0): >>> >> >>> >> > hosted-engine >>> >> >>> >> > --vm-start >>> >> >>> >> > >>> >> >>> >> > Lots of errors in the logs still, it appears to be >>> >> >>> >> > having >>> >> >>> >> > problems >>> >> >>> >> > with >>> >> >>> >> > that >>> >> >>> >> > directory still: >>> >> >>> >> > >>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > ovirt_hosted_engine_ha.broker. storage_broker.StorageBroker >>> >> >>> >> > ERROR >>> >> >>> >> > Failed >>> >> >>> >> > to >>> >> >>> >> > write metadata for host 1 to >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8#012Traceback >>> >> >>> >> > (most recent call last):#012 File >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > "/usr/lib/python2.7/site-
>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path, >>> >> >>> >> > direct_flag >>> >> >>> >> > | >>> >> >>> >> > os.O_WRONLY | >>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or >>> >> >>> >> > directory: >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > '/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8' >>> >> >>> >> > >>> >> >>> >> > There are no new files or symlinks in >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ >>> >> >>> >> > >>> >> >>> >> > - Jayme >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak >>> >> >>> >> > <msivak@redhat.com> >>> >> >>> >> > wrote: >>> >> >>> >> >> >>> >> >>> >> >> > Can you please stop all hosted engine tooling ( >>> >> >>> >> >> >>> >> >>> >> >> On all hosts I should have added. >>> >> >>> >> >> >>> >> >>> >> >> Martin >>> >> >>> >> >> >>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak >>> >> >>> >> >> <msivak@redhat.com> >>> >> >>> >> >> wrote: >>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] >>> >> >>> >> >> >> No >>> >> >>> >> >> >> such >>> >> >>> >> >> >> file >>> >> >>> >> >> >> or >>> >> >>> >> >> >> directory: >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8' >>> >> >>> >> >> >> >>> >> >>> >> >> >> ls -al >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8 >>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8 >>> >> >>> >> >> >> >>> >> >>> >> >> >> Is this due to the symlink problem you guys are >>> >> >>> >> >> >> referring to >>> >> >>> >> >> >> that >>> >> >>> >> >> >> was >>> >> >>> >> >> >> addressed in RC1 or something else? >>> >> >>> >> >> > >>> >> >>> >> >> > No, this file is the symlink. It should point to >>> >> >>> >> >> > somewhere >>> >> >>> >> >> > inside >>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is >>> >> >>> >> >> > really >>> >> >>> >> >> > interesting. >>> >> >>> >> >> > >>> >> >>> >> >> > Can you please stop all hosted engine tooling >>> >> >>> >> >> > (ovirt-ha-agent, >>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is >>> >> >>> >> >> > not >>> >> >>> >> >> > important >>> >> >>> >> >> > when >>> >> >>> >> >> > services are stopped, but better safe than sorry) >>> >> >>> >> >> > and >>> >> >>> >> >> > restart >>> >> >>> >> >> > all >>> >> >>> >> >> > services again? >>> >> >>> >> >> > >>> >> >>> >> >> >> Could there possibly be a permissions >>> >> >>> >> >> >> problem somewhere? >>> >> >>> >> >> > >>> >> >>> >> >> > Maybe, but the file itself looks out of the >>> >> >>> >> >> > ordinary. >>> >> >>> >> >> > I >>> >> >>> >> >> > wonder >>> >> >>> >> >> > how it >>> >> >>> >> >> > got there. >>> >> >>> >> >> > >>> >> >>> >> >> > Best regards >>> >> >>> >> >> > >>> >> >>> >> >> > Martin Sivak >>> >> >>> >> >> > >>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme >>> >> >>> >> >> > <jaymef@gmail.com> >>> >> >>> >> >> > wrote: >>> >> >>> >> >> >> Thanks for the help thus far. Storage could be >>> >> >>> >> >> >> related >>> >> >>> >> >> >> but >>> >> >>> >> >> >> all >>> >> >>> >> >> >> other >>> >> >>> >> >> >> VMs on >>> >> >>> >> >> >> same storage are running ok. The storage is >>> >> >>> >> >> >> mounted >>> >> >>> >> >> >> via >>> >> >>> >> >> >> NFS >>> >> >>> >> >> >> from >>> >> >>> >> >> >> within one >>> >> >>> >> >> >> of the three hosts, I realize this is not ideal. >>> >> >>> >> >> >> This >>> >> >>> >> >> >> was >>> >> >>> >> >> >> setup >>> >> >>> >> >> >> by >>> >> >>> >> >> >> a >>> >> >>> >> >> >> previous admin more as a proof of concept and VMs >>> >> >>> >> >> >> were >>> >> >>> >> >> >> put on >>> >> >>> >> >> >> there >>> >> >>> >> >> >> that >>> >> >>> >> >> >> should not have been placed in a proof of concept >>> >> >>> >> >> >> environment.. >>> >> >>> >> >> >> it >>> >> >>> >> >> >> was >>> >> >>> >> >> >> intended to be rebuilt with proper storage down >>> >> >>> >> >> >> the >>> >> >>> >> >> >> road. >>> >> >>> >> >> >> >>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts >>> >> >>> >> >> >> mount >>> >> >>> >> >> >> NFS >>> >> >>> >> >> >> >>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/data >>> >> >>> >> >> >> 4861742080 >>> >> >>> >> >> >> 1039352832 3822389248 22% >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /rhev/data-center/mnt/ cultivar0.grove.silverorange.com:_exports_data >>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso >>> >> >>> >> >> >> 4861742080 >>> >> >>> >> >> >> 1039352832 3822389248 22% >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /rhev/data-center/mnt/ cultivar0.grove.silverorange.com:_exports_iso >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> cultivar0.grove.silverorange. com:/exports/import_export >>> >> >>> >> >> >> 4861742080 >>> >> >>> >> >> >> 1039352832 3822389248 22% >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /rhev/data-center/mnt/ cultivar0.grove.silverorange.com:_exports_import__export >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> cultivar0.grove.silverorange. com:/exports/hosted_engine >>> >> >>> >> >> >> 4861742080 >>> >> >>> >> >> >> 1039352832 3822389248 22% >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /rhev/data-center/mnt/ cultivar0.grove.silverorange.com:_exports_hosted__engine >>> >> >>> >> >> >> >>> >> >>> >> >> >> Like I said, the VM data storage itself seems to >>> >> >>> >> >> >> be >>> >> >>> >> >> >> working >>> >> >>> >> >> >> ok, >>> >> >>> >> >> >> as >>> >> >>> >> >> >> all >>> >> >>> >> >> >> other >>> >> >>> >> >> >> VMs appear to be running. >>> >> >>> >> >> >> >>> >> >>> >> >> >> I'm curious why the broker log says this file is >>> >> >>> >> >> >> not >>> >> >>> >> >> >> found >>> >> >>> >> >> >> when >>> >> >>> >> >> >> it >>> >> >>> >> >> >> is >>> >> >>> >> >> >> correct and I can see the file at that path: >>> >> >>> >> >> >> >>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] >>> >> >>> >> >> >> No >>> >> >>> >> >> >> such >>> >> >>> >> >> >> file >>> >> >>> >> >> >> or >>> >> >>> >> >> >> directory: >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8' >>> >> >>> >> >> >> >>> >> >>> >> >> >> ls -al >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8 >>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0-d793-4581-9810- c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e- f5b7ec1f1cf8 >>> >> >>> >> >> >> >>> >> >>> >> >> >> Is this due to the symlink problem you guys are >>> >> >>> >> >> >> referring to >>> >> >>> >> >> >> that >>> >> >>> >> >> >> was >>> >> >>> >> >> >> addressed in RC1 or something else? Could there >>> >> >>> >> >> >> possibly be >>> >> >>> >> >> >> a >>> >> >>> >> >> >> permissions >>> >> >>> >> >> >> problem somewhere? >>> >> >>> >> >> >> >>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms >>> >> >>> >> >> >> installed >>> >> >>> >> >> >> and the >>> >> >>> >> >> >> host >>> >> >>> >> >> >> engine >>> >> >>> >> >> >> will not start is it safe for me to update hosts >>> >> >>> >> >> >> to >>> >> >>> >> >> >> 4.2 >>> >> >>> >> >> >> RC1 >>> >> >>> >> >> >> rpms? >>> >> >>> >> >> >> Or >>> >> >>> >> >> >> perhaps install that repo and *only* update the >>> >> >>> >> >> >> ovirt >>> >> >>> >> >> >> HA >>> >> >>> >> >> >> packages? >>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates >>> >> >>> >> >> >> to >>> >> >>> >> >> >> the >>> >> >>> >> >> >> inaccessible >>> >> >>> >> >> >> hosted >>> >> >>> >> >> >> engine VM. >>> >> >>> >> >> >> >>> >> >>> >> >> >> I should also mention one more thing. I >>> >> >>> >> >> >> originally >>> >> >>> >> >> >> upgraded >>> >> >>> >> >> >> the >>> >> >>> >> >> >> engine >>> >> >>> >> >> >> VM >>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed >>> >> >>> >> >> >> due >>> >> >>> >> >> >> to not >>> >> >>> >> >> >> being >>> >> >>> >> >> >> in >>> >> >>> >> >> >> global maintenance, so I set global maintenance >>> >> >>> >> >> >> and >>> >> >>> >> >> >> ran >>> >> >>> >> >> >> it >>> >> >>> >> >> >> again, >>> >> >>> >> >> >> which >>> >> >>> >> >> >> appeared to complete as intended but never came >>> >> >>> >> >> >> back >>> >> >>> >> >> >> up >>> >> >>> >> >> >> after. >>> >> >>> >> >> >> Just >>> >> >>> >> >> >> in >>> >> >>> >> >> >> case >>> >> >>> >> >> >> this might have anything at all to do with what >>> >> >>> >> >> >> could >>> >> >>> >> >> >> have >>> >> >>> >> >> >> happened. >>> >> >>> >> >> >> >>> >> >>> >> >> >> Thanks very much again, I very much appreciate
>>> >> >>> >> >> >> help! >>> >> >>> >> >> >> >>> >> >>> >> >> >> - Jayme >>> >> >>> >> >> >> >>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi >>> >> >>> >> >> >> <stirabos@redhat.com> >>> >> >>> >> >> >> wrote: >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak >>> >> >>> >> >> >>> <msivak@redhat.com> >>> >> >>> >> >> >>> wrote: >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> Hi, >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by >>> >> >>> >> >> >>>> restarting >>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest >>> >> >>> >> >> >>>> ovirt-hosted-engine-ha >>> >> >>> >> >> >>>> and >>> >> >>> >> >> >>>> -setup. We improved handling of the missing >>> >> >>> >> >> >>>> symlink. >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1 >>> >> >>> >> >> >>> >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> All the other issues seem to point to some >>> >> >>> >> >> >>>> storage >>> >> >>> >> >> >>>> problem >>> >> >>> >> >> >>>> I >>> >> >>> >> >> >>>> am >>> >> >>> >> >> >>>> afraid. >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> You said you started the VM, do you see it in >>> >> >>> >> >> >>>> virsh >>> >> >>> >> >> >>>> -r >>> >> >>> >> >> >>>> list? >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> Best regards >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> Martin Sivak >>> >> >>> >> >> >>>> >>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme >>> >> >>> >> >> >>>> <jaymef@gmail.com> >>> >> >>> >> >> >>>> wrote: >>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to >>> >> >>> >> >> >>>> > try >>> >> >>> >> >> >>>> > at >>> >> >>> >> >> >>>> > this >>> >> >>> >> >> >>>> > point. >>> >> >>> >> >> >>>> > Thank >>> >> >>> >> >> >>>> > you >>> >> >>> >> >> >>>> > for reading! >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted >>> >> >>> >> >> >>>> > engine >>> >> >>> >> >> >>>> > running >>> >> >>> >> >> >>>> > after a >>> >> >>> >> >> >>>> > botched >>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from >>> >> >>> >> >> >>>> > within >>> >> >>> >> >> >>>> > one >>> >> >>> >> >> >>>> > of >>> >> >>> >> >> >>>> > the >>> >> >>> >> >> >>>> > hosts. >>> >> >>> >> >> >>>> > Right >>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully >>> >> >>> >> >> >>>> > updated >>> >> >>> >> >> >>>> > with >>> >> >>> >> >> >>>> > yum >>> >> >>> >> >> >>>> > packages >>> >> >>> >> >> >>>> > from >>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with >>> >> >>> >> >> >>>> > yum >>> >> >>> >> >> >>>> > packages >>> >> >>> >> >> >>>> > and >>> >> >>> >> >> >>>> > failed to >>> >> >>> >> >> >>>> > come >>> >> >>> >> >> >>>> > up after reboot. As of right now, everything >>> >> >>> >> >> >>>> > should >>> >> >>> >> >> >>>> > have >>> >> >>> >> >> >>>> > full >>> >> >>> >> >> >>>> > yum >>> >> >>> >> >> >>>> > updates >>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global >>> >> >>> >> >> >>>> > maintenance >>> >> >>> >> >> >>>> > mode >>> >> >>> >> >> >>>> > on >>> >> >>> >> >> >>>> > right >>> >> >>> >> >> >>>> > now >>> >> >>> >> >> >>>> > and >>> >> >>> >> >> >>>> > started hosted-engine on one of the three host >>> >> >>> >> >> >>>> > and >>> >> >>> >> >> >>>> > the >>> >> >>> >> >> >>>> > status is >>> >> >>> >> >> >>>> > currently: >>> >> >>> >> >> >>>> > Engine status : {"reason": "failed
>>> >> >>> >> >> >>>> > check”; >>> >> >>> >> >> >>>> > "health": >>> >> >>> >> >> >>>> > "bad", >>> >> >>> >> >> >>>> > "vm": >>> >> >>> >> >> >>>> > "up", "detail": "Up"} >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > this is what I get when trying to enter >>> >> >>> >> >> >>>> > hosted-vm >>> >> >>> >> >> >>>> > --console >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > The engine VM is running on this host >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine' >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > error: Domain not found: no domain with >>> >> >>> >> >> >>>> > matching >>> >> >>> >> >> >>>> > name >>> >> >>> >> >> >>>> > 'HostedEngine' >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Here are logs from various sources when I >>> >> >>> >> >> >>>> > start >>> >> >>> >> >> >>>> > the >>> >> >>> >> >> >>>> > VM on >>> >> >>> >> >> >>>> > HOST3: >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > hosted-engine --vm-start >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID': >>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} >>> >> >>> >> >> >>>> > failed: >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not >>> >> >>> >> >> >>>> > exist: >>> >> >>> >> >> >>>> > {'vmId': >>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'}) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: >>> >> >>> >> >> >>>> > New >>> >> >>> >> >> >>>> > machine >>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started >>> >> >>> >> >> >>>> > Virtual >>> >> >>> >> >> >>>> > Machine >>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting >>> >> >>> >> >> >>>> > Virtual >>> >> >>> >> >> >>>> > Machine >>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now >>> >> >>> >> >> >>>> > active >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line >>> >> >>> >> >> >>>> > 48, >>> >> >>> >> >> >>>> > in >>> >> >>> >> >> >>>> > method >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ret = func(*args, **kwargs) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line >>> >> >>> >> >> >>>> > 2718, in >>> >> >>> >> >> >>>> > getStorageDomainInfo >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line >>> >> >>> >> >> >>>> > 304, in >>> >> >>> >> >> >>>> > validateSdUUID >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > sdDom.validate() >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line >>> >> >>> >> >> >>>> > 515, >>> >> >>> >> >> >>>> > in validate >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > raise >>> >> >>> >> >> >>>> > se.StorageDomainAccessError(self.sdUUID) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either >>> >> >>> >> >> >>>> > partially >>> >> >>> >> >> >>>> > accessible >>> >> >>> >> >> >>>> > or >>> >> >>> >> >> >>>> > entirely >>> >> >>> >> >> >>>> > inaccessible: >>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11 >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82:: storage.Dispatcher::(wrapper) >>> >> >>> >> >> >>>> > FINISH >>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either >>> >> >>> >> >> >>>> > partially >>> >> >>> >> >> >>>> > accessible >>> >> >>> >> >> >>>> > or >>> >> >>> >> >> >>>> > entirely >>> >> >>> >> >> >>>> > inaccessible: >>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > LC_ALL=C >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/ local/bin:/usr/sbin:/usr/bin >>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>> >> >>> >> >> >>>> > -name >>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > secret,id=masterKey0,format= raw,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes >>> >> >>> >> >> >>>> > -machine >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>> >> >>> >> >> >>>> > -cpu >>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>> >> >>> >> >> >>>> > -uuid >>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7. centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid= 4013c829-c9d7-4b72-90d5-6fe58137504c' >>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/ var/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait >>> >> >>> >> >> >>>> > -mon >>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor,mode=control >>> >> >>> >> >> >>>> > -rtc >>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global >>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>> >> >>> >> >> >>>> > -no-reboot >>> >> >>> >> >> >>>> > -boot >>> >> >>> >> >> >>>> > strict=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio- serial0,bus=pci.0,addr=0x4 >>> >> >>> >> >> >>>> > -drive >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/ 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/ 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if= none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0, readonly=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>> >> >>> >> >> >>>> > -netdev >>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-net-pci,netdev= hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.qemu.guest_agent.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=2,chardev=charchannel1,id=channel1,name= org.qemu.guest_agent.0 >>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=4,chardev=charchannel3,id=channel3,name= org.ovirt.hosted-engine-setup.0 >>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>> >> >>> >> >> >>>> > virtconsole,chardev= charconsole0,id=console0 >>> >> >>> >> >> >>>> > -spice >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>> >> >>> >> >> >>>> > -object >>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>> >> >>> >> >> >>>> > -msg >>> >> >>> >> >> >>>> > timestamp=on >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev >>> >> >>> >> >> >>>> > pty,id=charconsole0: >>> >> >>> >> >> >>>> > char >>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>> >> >>> >> >> >>>> > charconsole0) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down, >>> >> >>> >> >> >>>> > reason=shutdown >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up >>> >> >>> >> >> >>>> > libvirt >>> >> >>> >> >> >>>> > version: >>> >> >>> >> >> >>>> > 3.2.0, >>> >> >>> >> >> >>>> > package: >>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem >>> >> >>> >> >> >>>> > <http://bugs.centos.org>, >>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), >>> >> >>> >> >> >>>> > qemu >>> >> >>> >> >> >>>> > version: >>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), >>> >> >>> >> >> >>>> > hostname: >>> >> >>> >> >> >>>> > cultivar3 >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > LC_ALL=C >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/ local/bin:/usr/sbin:/usr/bin >>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>> >> >>> >> >> >>>> > -name >>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > secret,id=masterKey0,format= raw,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes >>> >> >>> >> >> >>>> > -machine >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>> >> >>> >> >> >>>> > -cpu >>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>> >> >>> >> >> >>>> > -uuid >>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7. centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid= 4013c829-c9d7-4b72-90d5-6fe58137504c' >>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/ var/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait >>> >> >>> >> >> >>>> > -mon >>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor,mode=control >>> >> >>> >> >> >>>> > -rtc >>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global >>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>> >> >>> >> >> >>>> > -no-reboot >>> >> >>> >> >> >>>> > -boot >>> >> >>> >> >> >>>> > strict=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio- serial0,bus=pci.0,addr=0x4 >>> >> >>> >> >> >>>> > -drive >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/ 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/ 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if= none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0, readonly=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>> >> >>> >> >> >>>> > -netdev >>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-net-pci,netdev= hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.qemu.guest_agent.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=2,chardev=charchannel1,id=channel1,name= org.qemu.guest_agent.0 >>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=4,chardev=charchannel3,id=channel3,name= org.ovirt.hosted-engine-setup.0 >>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>> >> >>> >> >> >>>> > virtconsole,chardev= charconsole0,id=console0 >>> >> >>> >> >> >>>> > -spice >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>> >> >>> >> >> >>>> > -object >>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>> >> >>> >> >> >>>> > -msg >>> >> >>> >> >> >>>> > timestamp=on >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev >>> >> >>> >> >> >>>> > pty,id=charconsole0: >>> >> >>> >> >> >>>> > char >>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>> >> >>> >> >> >>>> > charconsole0) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down, >>> >> >>> >> >> >>>> > reason=shutdown >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up >>> >> >>> >> >> >>>> > libvirt >>> >> >>> >> >> >>>> > version: >>> >> >>> >> >> >>>> > 3.2.0, >>> >> >>> >> >> >>>> > package: >>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem >>> >> >>> >> >> >>>> > <http://bugs.centos.org>, >>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), >>> >> >>> >> >> >>>> > qemu >>> >> >>> >> >> >>>> > version: >>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), >>> >> >>> >> >> >>>> > hostname: >>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > LC_ALL=C >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/ local/bin:/usr/sbin:/usr/bin >>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>> >> >>> >> >> >>>> > -name >>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > secret,id=masterKey0,format= raw,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes >>> >> >>> >> >> >>>> > -machine >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>> >> >>> >> >> >>>> > -cpu >>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>> >> >>> >> >> >>>> > -uuid >>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7. centos,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid= 4013c829-c9d7-4b72-90d5-6fe58137504c' >>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/ var/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait >>> >> >>> >> >> >>>> > -mon >>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor,mode=control >>> >> >>> >> >> >>>> > -rtc >>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global >>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>> >> >>> >> >> >>>> > -no-reboot >>> >> >>> >> >> >>>> > -boot >>> >> >>> >> >> >>>> > strict=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio- serial0,bus=pci.0,addr=0x4 >>> >> >>> >> >> >>>> > -drive >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/ 248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8e04cf387f9/ 23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw,if= none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=
>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0, readonly=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>> >> >>> >> >> >>>> > -netdev >>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-net-pci,netdev= hostnet0,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.qemu.guest_agent.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=2,chardev=charchannel1,id=channel1,name= org.qemu.guest_agent.0 >>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>> >> >>> >> >> >>>> > -chardev >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/ var/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5- 6fe58137504c.org.ovirt.hosted-engine-setup.0,server,nowait >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtserialport,bus=virtio- serial0.0,nr=4,chardev=charchannel3,id=channel3,name= org.ovirt.hosted-engine-setup.0 >>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>> >> >>> >> >> >>>> > virtconsole,chardev= charconsole0,id=console0 >>> >> >>> >> >> >>>> > -spice >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>> >> >>> >> >> >>>> > -object >>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename=/dev/urandom >>> >> >>> >> >> >>>> > -device >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>> >> >>> >> >> >>>> > -msg >>> >> >>> >> >> >>>> > timestamp=on >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev >>> >> >>> >> >> >>>> > pty,id=charconsole0: >>> >> >>> >> >> >>>> > char >>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>> >> >>> >> >> >>>> > charconsole0) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine- ha/broker.log >>> >> >>> >> >> >>>> > <== >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 151, in get_raw_stats >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | >>> >> >>> >> >> >>>> > os.O_RDONLY | >>> >> >>> >> >> >>>> > os.O_SYNC) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or
>>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/ 248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11 >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 16:55:15,761::status_broker:: 92::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) >>> >> >>> >> >> >>>> > Failed to read state. >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Traceback (most recent call last): >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 88, in run >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats() >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 162, in get_raw_stats >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > .format(str(e))) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno >>> >> >>> >> >> >>>> > 2] >>> >> >>> >> >> >>>> > No >>> >> >>> >> >> >>>> > such >>> >> >>> >> >> >>>> > file or >>> >> >>> >> >> >>>> > directory: >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/ 248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine- ha/agent.log >>> >> >>> >> >> >>>> > <== >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > result = refresh_method() >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > content = >>> >> >>> >> >> >>>> > self._get_file_content_from_ shared_storage(VM) >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 484, in >>> >> >>> >> >> >>>> > _get_file_content_from_shared_storage >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > config_volume_path = >>> >> >>> >> >> >>>> > self._get_config_volume_path() >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > conf_vol_uuid >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
>>> >> >>> >> >> >>>> > line 358, in get_volume_path >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR, >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > RuntimeError: Path to volume >>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f >>> >> >>> >> >> >>>> > not >>> >> >>> >> >> >>>> > found >>> >> >>> >> >> >>>> > in /run/vdsm/storag >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11 >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260:: virt.vmstats::(send_metrics) >>> >> >>> >> >> >>>> > VM >>> >> >>> >> >> >>>> > metrics >>> >> >>> >> >> >>>> > collection failed >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > Traceback (most recent call last): >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > File >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-
On Fri, Jan 12, 2018 at 6:56 PM, Jayme <jaymef@gmail.com> wrote: the libvirtconnection.py", line pty,id=charconsole0: the the line packages/vdsm/virt/vmstats.py", packages/vdsm/virt/vmstats.py", py", line line line line line promiscuous line line py", py", py", packages/vdsm/virt/vmstats.py", packages/vdsm/virt/vmstats.py", py", py", packages/ovirt_hosted_engine_ha/broker/storage_broker.py", the liveliness packages/vdsm/common/api.py", packages/vdsm/storage/hsm.py", packages/vdsm/storage/hsm.py", packages/vdsm/storage/fileSD.py", pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 packages/ovirt_hosted_engine_ha/broker/storage_broker.py", directory: packages/ovirt_hosted_engine_ha/broker/status_broker.py", packages/ovirt_hosted_engine_ha/broker/storage_broker.py", packages/ovirt_hosted_engine_ha/env/config.py", packages/ovirt_hosted_engine_ha/env/config.py", packages/ovirt_hosted_engine_ha/env/config.py", packages/ovirt_hosted_engine_ha/lib/heconflib.py", packages/vdsm/virt/vmstats.py",
>>> >> >>> >> >> >>>> > line >>> >> >>> >> >> >>>> > 197, in >>> >> >>> >> >> >>>> > send_metrics >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] = >>> >> >>> >> >> >>>> > stat['cpuUsage'] >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > KeyError: 'cpuUsage' >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> > ______________________________
>>> >> >>> >> >> >>>> > Users mailing list >>> >> >>> >> >> >>>> > Users@ovirt.org >>> >> >>> >> >> >>>> > http://lists.ovirt.org/ mailman/listinfo/users >>> >> >>> >> >> >>>> > >>> >> >>> >> >> >>>> ______________________________
>>> >> >>> >> >> >>>> Users mailing list >>> >> >>> >> >> >>>> Users@ovirt.org >>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >> >>> >> >> >>> >>> >> >>> >> >> >>> >>> >> >>> >> >> >> >>> >> >>> >> > >>> >> >>> >> > >>> >> >>> > >>> >> >>> > >>> >> >> >>> >> >> >>> >> > >>> > >>> > >> >> >
Martin, actually might be some good news here. I could not get to console using hosted-engine console but I connected through virsh and got a console to the hosted VM and was able to login, this is a great start. Now to find out what is wrong with the VM. On Fri, Jan 12, 2018 at 2:11 PM, Jayme <jaymef@gmail.com> wrote:
No luck I'm afraid. It's very odd that I wouldn't be able to get a console to it, if the status is up and seen by virsh. Any clue?
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
# virsh -r list Id Name State ---------------------------------------------------- 118 Cultivar running
# hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
# hosted-engine --console 118 The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
# hosted-engine --console Cultivar The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
On Fri, Jan 12, 2018 at 2:05 PM, Martin Sivak <msivak@redhat.com> wrote:
Try listing the domains with
virsh -r list
maybe it just has some weird name...
Martin
I thought that it might be a good sign but unfortunately I cannot access it with console :( if I could get console access to it I might be able to fix the problem. But seeing is how the console is also not working leads me to believe there is a bigger issue at hand here.
hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
I really wonder if this is all a symlinking problem in some way. Is it possible for me to upgrade host to 4.2 RC2 without being able to upgrade the engine first or should I keep everything on 4.2 as it is?
On Fri, Jan 12, 2018 at 1:49 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
the VM is up according to the status (at least for a while). You should be able to use console and diagnose anything that happened inside (line the need for fsck and such) now.
Check the presence of those links again now, the metadata file content is not important, but the file has to exist (agents will populate it with status data). I have no new idea about what is wrong with that though.
Best regards
Martin
On Fri, Jan 12, 2018 at 5:47 PM, Jayme <jaymef@gmail.com> wrote:
The lock space issue was an issue I needed to clear but I don't
believe
it has resolved the problem. I shutdown agent and broker on all hosts and disconnected hosted-storage then enabled broker/agent on just one host and connected storage. I started the VM and actually didn't get any errors in the logs barely at all which was good to see, however the VM is still not running:
HOST3:
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
==> /var/log/messages <== Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer space available Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 kvm: 3 guests now active Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535: error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU guest agent is not connected
Interestingly though, now I'm seeing this in the logs which may be a new clue:
==> /var/log/vdsm/vdsm.log <== File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py",
126, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py",
116, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) jsonrpc/4::ERROR::2018-01-12 12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH getStorageDomainInfo error=Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished?
==> /var/log/ovirt-hosted-engine-ha/broker.log <== File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/ broker/storage_broker.py", line 151, in get_raw_stats f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/ 14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54- 47af-9f1e-f5b7ec1f1cf8' StatusStorageThread::ERROR::2018-01-12
12:32:06,049::status_broker::92::ovirt_hosted_engine_ha.brok er.status_broker.StatusBroker.Update::(run) Failed to read state. Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/ broker/status_broker.py", line 88, in run self._storage_broker.get_raw_stats() File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/ broker/storage_broker.py", line 162, in get_raw_stats .format(str(e))) RequestError: failed to read metadata: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/ 14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54- 47af-9f1e-f5b7ec1f1cf8'
On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak@redhat.com> wrote:
The lock is the issue.
- try running sanlock client status on all hosts - also make sure you do not have some forgotten host still connected to the lockspace, but without ha daemons running (and with the VM)
I need to go to our president election now, I might check the email later tonight.
Martin
On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef@gmail.com> wrote: > Here are the newest logs from me trying to start hosted vm: > > ==> /var/log/messages <== > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > blocking > state > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > disabled > state > Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered promiscuous > mode > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > blocking > state > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > forwarding state > Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No > buffer > space > available > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > [1515772694.8715] > manager: (vnet4): new Tun device > (/org/freedesktop/NetworkManager/Devices/140) > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > [1515772694.8795] > device (vnet4): state change: unmanaged -> unavailable (reason > 'connection-assumed') [10 20 41] > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0, > package: > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > cultivar0.grove.silverorange.com > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > guest=Cultivar,debug-threads=on -S -object > > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/
domain-119-Cultivar/master-key.aes
> -machine pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > -cpu > Conroe -m 8192 -realtime mlock=off -smp > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > 'type=1,manufacturer=oVirt,product=oVirt > > > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042- 8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-119- Cultivar/monitor.sock,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=2018-01-12T15:58:14,driftfix=slew -global > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > > > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f 286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c- 4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio- disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache= none,werror=stop,rerror=stop,aio=threads > -device > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti o-disk0,id=virtio-disk0,bootindex=1 > -drive if=none,id=drive-ide0-1-0,readonly=on -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6: 83,bus=pci.0,addr=0x3 > -chardev > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.rhevm.vdsm > -chardev > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel 1,id=channel1,name=org.qemu.guest_agent.0 > -chardev spicevmc,id=charchannel2,name=vdagent -device > > > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel 2,id=channel2,name=com.redhat.spice.0 > -chardev > > > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine- setup.0,server,nowait > -device > > > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel 3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > -chardev pty,id=charconsole0 -device > virtconsole,chardev=charconsole0,id=console0 -spice > > > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tl s-channel=default,seamless-migration=on > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > rng-random,id=objrng0,filename=/dev/urandom -device > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > timestamp=on > > ==> /var/log/messages <== > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > [1515772694.8807] > device (vnet4): state change: unavailable -> disconnected (reason > 'none') > [20 30 0] > Jan 12 11:58:14 cultivar0 systemd-machined: New machine > qemu-119-Cultivar. > Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine > qemu-119-Cultivar. > Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine > qemu-119-Cultivar. > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev
> char > device redirected to /dev/pts/1 (label charconsole0) > > ==> /var/log/messages <== > Jan 12 11:58:15 cultivar0 kvm: 5 guests now active > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12 15:58:15.217+0000: shutting down, reason=failed > > ==> /var/log/messages <== > Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: > 1908: > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to > acquire > lock: Lease is held by another host > > ==> /var/log/libvirt/qemu/Cultivar.log <== > 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from > pid > 1773 > (/usr/sbin/libvirtd) > > ==> /var/log/messages <== > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > disabled > state > Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > disabled > state > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > [1515772695.2348] > device (vnet4): state change: disconnected -> unmanaged (reason > 'unmanaged') > [30 10 3] > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > [1515772695.2349] > device (vnet4): released from master device ovirtmgmt > Jan 12 11:58:15 cultivar0 kvm: 4 guests now active > Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar > terminated. > > ==> /var/log/vdsm/vdsm.log <== > vm/4013c829::ERROR::2018-01-12 > 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm) > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start
> failed > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, > in > _startUnderlyingVm > self._run() > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, > in > _run > dom.createWithFlags(flags) > File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", > line > 126, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in > wrapper > return func(inst, *args, **kwargs) > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in > createWithFlags > if ret == -1: raise libvirtError ('virDomainCreateWithFlags() > failed', > dom=self) > libvirtError: resource busy: Failed to acquire lock: Lease is held by > another host > jsonrpc/6::ERROR::2018-01-12 > 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_handle _request) > Internal server error > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> 606, > in _handle_request > res = method(**params) > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
> 201, > in > _dynamicMethod > result = fn(*methodArgs) > File "<string>", line 2, in getAllVmIoTunePolicies > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
> 48, > in > method > ret = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > getAllVmIoTunePolicies > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, > in > getAllVmIoTunePolicies > 'current_values': v.getIoTune()} > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, > in > getIoTune > result = self.getIoTuneResponse() > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, > in > getIoTuneResponse > res = self._dom.blockIoTune( > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", > line > 47, > in __getattr__ > % self.vmid) > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not > defined > yet or was undefined > > ==> /var/log/messages <== > Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR > Internal > server error#012Traceback (most recent call last):#012 File > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, > in > _handle_request#012 res = method(**params)#012 File > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>", > line 2, > in getAllVmIoTunePolicies#012 File > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in > method#012 ret = func(*args, **kwargs)#012 File > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > getAllVmIoTunePolicies#012 io_tune_policies_dict = > self._cif.getAllVmIoTunePolicies()#012 File > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 > File > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in > getIoTune#012 result = self.getIoTuneResponse()#012 File > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, > in > __getattr__#012 % self.vmid)#012NotConnectedError: VM > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was > undefined > > On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef@gmail.com> wrote: >> >> One other tidbit I noticed is that it seems like there are less >> errors >> if >> I started in paused mode: >> >> but still shows: Engine status : {"reason": >> "bad >> vm >> status", "health": "bad", "vm": "up", "detail": "Paused"} >> >> ==> /var/log/messages <== >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered >> blocking state >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered >> disabled state >> Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered
>> mode >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered >> blocking state >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered >> forwarding state >> Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No >> buffer >> space available >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> >> [1515772505.3625] >> manager: (vnet4): new Tun device >> (/org/freedesktop/NetworkManager/Devices/139) >> >> ==> /var/log/libvirt/qemu/Cultivar.log <== >> 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0, >> package: >> 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, >> 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: >> 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: >> cultivar0.grove.silverorange.com >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin >> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name >> guest=Cultivar,debug-threads=on -S -object >> >> >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/ domain-118-Cultivar/master-key.aes >> -machine pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >> -cpu >> Conroe -m 8192 -realtime mlock=off -smp >> 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid >> 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >> 'type=1,manufacturer=oVirt,product=oVirt >> >> >> Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042- 8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' >> -no-user-config -nodefaults -chardev >> >> >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-118- Cultivar/monitor.sock,server,nowait >> -mon chardev=charmonitor,id=monitor,mode=control -rtc >> base=2018-01-12T15:55:05,driftfix=slew -global >> kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on >> -device >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >> >> >> file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f 286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c- 4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio- disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache= none,werror=stop,rerror=stop,aio=threads >> -device >> >> >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti o-disk0,id=virtio-disk0,bootindex=1 >> -drive if=none,id=drive-ide0-1-0,readonly=on -device >> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev >> tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device >> >> >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6: 83,bus=pci.0,addr=0x3 >> -chardev >> >> >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >> -device >> >> >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.rhevm.vdsm >> -chardev >> >> >> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait >> -device >> >> >> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel 1,id=channel1,name=org.qemu.guest_agent.0 >> -chardev spicevmc,id=charchannel2,name=vdagent -device >> >> >> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel 2,id=channel2,name=com.redhat.spice.0 >> -chardev >> >> >> socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine- setup.0,server,nowait >> -device >> >> >> virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel 3,id=channel3,name=org.ovirt.hosted-engine-setup.0 >> -chardev pty,id=charconsole0 -device >> virtconsole,chardev=charconsole0,id=console0 -spice >> >> >> tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tl s-channel=default,seamless-migration=on >> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object >> rng-random,id=objrng0,filename=/dev/urandom -device >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg >> timestamp=on >> >> ==> /var/log/messages <== >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> >> [1515772505.3689] >> device (vnet4): state change: unmanaged -> unavailable (reason >> 'connection-assumed') [10 20 41] >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> >> [1515772505.3702] >> device (vnet4): state change: unavailable -> disconnected (reason >> 'none') >> [20 30 0] >> Jan 12 11:55:05 cultivar0 systemd-machined: New machine >> qemu-118-Cultivar. >> Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine >> qemu-118-Cultivar. >> Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine >> qemu-118-Cultivar. >> >> ==> /var/log/libvirt/qemu/Cultivar.log <== >> 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev
>> char >> device redirected to /dev/pts/1 (label charconsole0) >> >> ==> /var/log/messages <== >> Jan 12 11:55:05 cultivar0 kvm: 5 guests now active >> >> On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef@gmail.com> wrote: >>> >>> Yeah I am in global maintenance: >>> >>> state=GlobalMaintenance >>> >>> host0: {"reason": "vm not running on this host", "health": "bad", >>> "vm": >>> "down", "detail": "unknown"} >>> host2: {"reason": "vm not running on this host", "health": "bad", >>> "vm": >>> "down", "detail": "unknown"} >>> host3: {"reason": "vm not running on this host", "health": "bad", >>> "vm": >>> "down", "detail": "unknown"} >>> >>> I understand the lock is an issue, I'll try to make sure it is >>> fully >>> stopped on all three before starting but I don't think that is
>>> issue at >>> hand either. What concerns me is mostly that it seems to be >>> unable >>> to read >>> the meta data, I think that might be the heart of the problem but >>> I'm >>> not >>> sure what is causing it. >>> >>> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak < msivak@redhat.com> >>> wrote: >>>> >>>> > On all three hosts I ran hosted-engine --vm-shutdown; >>>> > hosted-engine >>>> > --vm-poweroff >>>> >>>> Are you in global maintenance? I think you were in one of the >>>> previous >>>> emails, but worth checking. >>>> >>>> > I started ovirt-ha-broker with systemctl as root user but it >>>> > does >>>> > appear to be running under vdsm: >>>> >>>> That is the correct behavior. >>>> >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is >>>> > held >>>> > by >>>> > another host >>>> >>>> sanlock seems to think the VM runs somewhere and it is possible >>>> that >>>> some other host tried to start the VM as well unless you are in >>>> global >>>> maintenance (that is why I asked the first question here). >>>> >>>> Martin >>>> >>>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef@gmail.com> wrote: >>>> > Martin, >>>> > >>>> > Thanks so much for keeping with me, this is driving me crazy! I >>>> > really do >>>> > appreciate it, thanks again >>>> > >>>> > Let's go through this: >>>> > >>>> > HE VM is down - YES >>>> > >>>> > >>>> > HE agent fails when opening metadata using the symlink - YES >>>> > >>>> > >>>> > the symlink is there and readable by vdsm:kvm - it appears to >>>> > be: >>>> > >>>> > >>>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20 >>>> > 14a20941-1b84-4b82-be8f-ace38d7c037a >>>> > -> >>>> > >>>> > >>>> > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_ exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/ images/14a20941-1b84-4b82-be8f-ace38d7c037a >>>> > >>>> > >>>> > And the files in the linked directory exist and have vdsm:kvm >>>> > perms >>>> > as >>>> > well: >>>> > >>>> > >>>> > # cd >>>> > >>>> > >>>> > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com:_ exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/ images/14a20941-1b84-4b82-be8f-ace38d7c037a >>>> > >>>> > [root@cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al >>>> > >>>> > total 2040 >>>> > >>>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . >>>> > >>>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. >>>> > >>>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19 >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> > >>>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease >>>> > >>>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta >>>> > >>>> > >>>> > I started ovirt-ha-broker with systemctl as root user but it >>>> > does >>>> > appear to >>>> > be running under vdsm: >>>> > >>>> > >>>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18 >>>> > /usr/bin/python >>>> > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>>> > >>>> > >>>> > >>>> > Here is something I tried: >>>> > >>>> > >>>> > - On all three hosts I ran hosted-engine --vm-shutdown; >>>> > hosted-engine >>>> > --vm-poweroff >>>> > >>>> > - On HOST0 (cultivar0) I disconnected and reconnected storage >>>> > using >>>> > hosted-engine >>>> > >>>> > - Tried starting up the hosted VM on cultivar0 while tailing
>>>> > logs: >>>> > >>>> > >>>> > # hosted-engine --vm-start >>>> > >>>> > VM exists and is down, cleaning up and restarting >>>> > >>>> > >>>> > >>>> > ==> /var/log/vdsm/vdsm.log <== >>>> > >>>> > jsonrpc/2::ERROR::2018-01-12 >>>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats) >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm >>>> > stats >>>> > >>>> > Traceback (most recent call last): >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>>> > 1762, >>>> > in >>>> > _getRunningVmStats >>>> > >>>> > vm_sample.interval) >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", >>>> > line >>>> > 45, in >>>> > produce >>>> > >>>> > networks(vm, stats, first_sample, last_sample, interval) >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", >>>> > line >>>> > 322, in >>>> > networks >>>> > >>>> > if nic.name.startswith('hostdev'): >>>> > >>>> > AttributeError: name >>>> > >>>> > jsonrpc/3::ERROR::2018-01-12 >>>> > >>>> > >>>> > 11:27:27,221::__init__::611::jsonrpc.JsonRpcServer::(_handle _request) >>>> > Internal server error >>>> > >>>> > Traceback (most recent call last): >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/yajsonrpc/__init__.py", >>>> > line >>>> > 606, >>>> > in _handle_request >>>> > >>>> > res = method(**params) >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/rpc/Bridge.py", >>>> > line >>>> > 201, in >>>> > _dynamicMethod >>>> > >>>> > result = fn(*methodArgs) >>>> > >>>> > File "<string>", line 2, in getAllVmIoTunePolicies >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", >>>> > line >>>> > 48, >>>> > in >>>> > method >>>> > >>>> > ret = func(*args, **kwargs) >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line >>>> > 1354, >>>> > in >>>> > getAllVmIoTunePolicies >>>> > >>>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolici es() >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
>>>> > 524, >>>> > in >>>> > getAllVmIoTunePolicies >>>> > >>>> > 'current_values': v.getIoTune()} >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>>> > 3481, >>>> > in >>>> > getIoTune >>>> > >>>> > result = self.getIoTuneResponse() >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>>> > 3500, >>>> > in >>>> > getIoTuneResponse >>>> > >>>> > res = self._dom.blockIoTune( >>>> > >>>> > File >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", >>>> > line >>>> > 47, >>>> > in __getattr__ >>>> > >>>> > % self.vmid) >>>> > >>>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was >>>> > not >>>> > defined >>>> > yet or was undefined >>>> > >>>> > >>>> > ==> /var/log/messages <== >>>> > >>>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer >>>> > ERROR >>>> > Internal >>>> > server error#012Traceback (most recent call last):#012 File >>>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
>>>> > 606, >>>> > in >>>> > _handle_request#012 res = method(**params)#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, >>>> > in >>>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File >>>> > "<string>", >>>> > line 2, >>>> > in getAllVmIoTunePolicies#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, >>>> > in >>>> > method#012 ret = func(*args, **kwargs)#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in >>>> > getAllVmIoTunePolicies#012 io_tune_policies_dict = >>>> > self._cif.getAllVmIoTunePolicies()#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, >>>> > in >>>> > getAllVmIoTunePolicies#012 'current_values': >>>> > v.getIoTune()}#012 >>>> > File >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, >>>> > in >>>> > getIoTune#012 result = self.getIoTuneResponse()#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, >>>> > in >>>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
>>>> > 47, >>>> > in >>>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or >>>> > was >>>> > undefined >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > blocking >>>> > state >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > disabled >>>> > state >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered >>>> > promiscuous >>>> > mode >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > blocking >>>> > state >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > forwarding state >>>> > >>>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No >>>> > buffer >>>> > space >>>> > available >>>> > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>>> > [1515770847.4264] >>>> > manager: (vnet4): new Tun device >>>> > (/org/freedesktop/NetworkManager/Devices/135) >>>> > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>>> > [1515770847.4342] >>>> > device (vnet4): state change: unmanaged -> unavailable (reason >>>> > 'connection-assumed') [10 20 41] >>>> > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>>> > [1515770847.4353] >>>> > device (vnet4): state change: unavailable -> disconnected >>>> > (reason >>>> > 'none') >>>> > [20 30 0] >>>> > >>>> > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>>> > >>>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: >>>> > 3.2.0, >>>> > package: >>>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: >>>> > cultivar0.grove.silverorange.com >>>> > >>>> > LC_ALL=C PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name >>>> > guest=Cultivar,debug-threads=on -S -object >>>> > >>>> > >>>> > >>>> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/ domain-114-Cultivar/master-key.aes >>>> > -machine >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off >>>> > -cpu >>>> > Conroe -m 8192 -realtime mlock=off -smp >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>>> > >>>> > >>>> > >>>> > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042- 8031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' >>>> > -no-user-config -nodefaults -chardev >>>> > >>>> > >>>> > >>>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-114- Cultivar/monitor.sock,server,nowait >>>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc >>>> > base=2018-01-12T15:27:27,driftfix=slew -global >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot >>>> > strict=on >>>> > -device >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive >>>> > >>>> > >>>> > >>>> > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f 286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c- 4967-a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio- disk0,serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache= none,werror=stop,rerror=stop,aio=threads >>>> > -device >>>> > >>>> > >>>> > >>>> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti o-disk0,id=virtio-disk0,bootindex=1 >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev >>>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device >>>> > >>>> > >>>> > >>>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6: 83,bus=pci.0,addr=0x3 >>>> > -chardev >>>> > >>>> > >>>> > >>>> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait >>>> > -device >>>> > >>>> > >>>> > >>>> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.rhevm.vdsm >>>> > -chardev >>>> > >>>> > >>>> > >>>> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait >>>> > -device >>>> > >>>> > >>>> > >>>> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel 1,id=channel1,name=org.qemu.guest_agent.0 >>>> > -chardev spicevmc,id=charchannel2,name=vdagent -device >>>> > >>>> > >>>> > >>>> > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel 2,id=channel2,name=com.redhat.spice.0 >>>> > -chardev >>>> > >>>> > >>>> > >>>> > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine- setup.0,server,nowait >>>> > -device >>>> > >>>> > >>>> > >>>> > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel 3,id=channel3,name=org.ovirt.hosted-engine-setup.0 >>>> > -chardev pty,id=charconsole0 -device >>>> > virtconsole,chardev=charconsole0,id=console0 -spice >>>> > >>>> > >>>> > >>>> > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tl s-channel=default,seamless-migration=on >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object >>>> > rng-random,id=objrng0,filename=/dev/urandom -device >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg >>>> > timestamp=on >>>> > >>>> > >>>> > ==> /var/log/messages <== >>>> > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine >>>> > qemu-114-Cultivar. >>>> > >>>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine >>>> > qemu-114-Cultivar. >>>> > >>>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine >>>> > qemu-114-Cultivar. >>>> > >>>> > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>>> > >>>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev >>>> > pty,id=charconsole0: >>>> > char >>>> > device redirected to /dev/pts/2 (label charconsole0) >>>> > >>>> > >>>> > ==> /var/log/messages <== >>>> > >>>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active >>>> > >>>> > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>>> > >>>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed >>>> > >>>> > >>>> > ==> /var/log/messages <== >>>> > >>>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 >>>> > 15:27:27.773+0000: >>>> > 1910: >>>> > error : virLockManagerSanlockAcquire:1041 : resource busy: >>>> > Failed >>>> > to >>>> > acquire >>>> > lock: Lease is held by another host >>>> > >>>> > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>>> > >>>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 >>>> > from >>>> > pid 1773 >>>> > (/usr/sbin/libvirtd) >>>> > >>>> > >>>> > ==> /var/log/messages <== >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > disabled >>>> > state >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left
>>>> > mode >>>> > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) >>>> > entered >>>> > disabled >>>> > state >>>> > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>>> > [1515770847.7989] >>>> > device (vnet4): state change: disconnected -> unmanaged (reason >>>> > 'unmanaged') >>>> > [30 10 3] >>>> > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> >>>> > [1515770847.7989] >>>> > device (vnet4): released from master device ovirtmgmt >>>> > >>>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active >>>> > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine >>>> > qemu-114-Cultivar >>>> > terminated. >>>> > >>>> > >>>> > ==> /var/log/vdsm/vdsm.log <== >>>> > >>>> > vm/4013c829::ERROR::2018-01-12 >>>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm) >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start >>>> > process >>>> > failed >>>> > >>>> > Traceback (most recent call last): >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>>> > 843, >>>> > in >>>> > _startUnderlyingVm >>>> > >>>> > self._run() >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
>>>> > 2721, >>>> > in >>>> > _run >>>> > >>>> > dom.createWithFlags(flags) >>>> > >>>> > File >>>> > "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py" , >>>> > line >>>> > 126, in wrapper >>>> > >>>> > ret = f(*args, **kwargs) >>>> > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py",
>>>> > 512, >>>> > in >>>> > wrapper >>>> > >>>> > return func(inst, *args, **kwargs) >>>> > >>>> > File "/usr/lib64/python2.7/site-packages/libvirt.py", line >>>> > 1069, >>>> > in >>>> > createWithFlags >>>> > >>>> > if ret == -1: raise libvirtError >>>> > ('virDomainCreateWithFlags() >>>> > failed', >>>> > dom=self) >>>> > >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is >>>> > held >>>> > by >>>> > another host >>>> > >>>> > periodic/47::ERROR::2018-01-12 >>>> > 11:27:32,858::periodic::215::virt.periodic.Operation::(__cal l__) >>>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590> >>>> > operation >>>> > failed >>>> > >>>> > Traceback (most recent call last): >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/periodic.py", >>>> > line >>>> > 213, >>>> > in __call__ >>>> > >>>> > self._func() >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/sampling.py", >>>> > line >>>> > 522, >>>> > in __call__ >>>> > >>>> > self._send_metrics() >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/sampling.py", >>>> > line >>>> > 538, >>>> > in _send_metrics >>>> > >>>> > vm_sample.interval) >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", >>>> > line >>>> > 45, in >>>> > produce >>>> > >>>> > networks(vm, stats, first_sample, last_sample, interval) >>>> > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", >>>> > line >>>> > 322, in >>>> > networks >>>> > >>>> > if nic.name.startswith('hostdev'): >>>> > >>>> > AttributeError: name >>>> > >>>> > >>>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak >>>> > <msivak@redhat.com> >>>> > wrote: >>>> >> >>>> >> Hmm that rules out most of NFS related permission issues. >>>> >> >>>> >> So the current status is (I need to sum it up to get the full >>>> >> picture): >>>> >> >>>> >> - HE VM is down >>>> >> - HE agent fails when opening metadata using the symlink >>>> >> - the symlink is there >>>> >> - the symlink is readable by vdsm:kvm >>>> >> >>>> >> Hmm can you check under which user is ovirt-ha-broker started? >>>> >> >>>> >> Martin >>>> >> >>>> >> >>>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef@gmail.com> >>>> >> wrote: >>>> >> > Same thing happens with data images of other VMs as well >>>> >> > though, >>>> >> > and >>>> >> > those >>>> >> > seem to be running ok so I'm not sure if it's the problem. >>>> >> > >>>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme <jaymef@gmail.com
>>>> >> > wrote: >>>> >> >> >>>> >> >> Martin, >>>> >> >> >>>> >> >> I can as VDSM user but not as root . I get permission denied >>>> >> >> trying to >>>> >> >> touch one of the files as root, is that normal? >>>> >> >> >>>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak >>>> >> >> <msivak@redhat.com> >>>> >> >> wrote: >>>> >> >>> >>>> >> >>> Hmm, then it might be a permission issue indeed. Can you >>>> >> >>> touch >>>> >> >>> the >>>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not >>>> >> >>> prevent >>>> >> >>> you from doing that. >>>> >> >>> >>>> >> >>> Martin >>>> >> >>> >>>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme < jaymef@gmail.com> >>>> >> >>> wrote: >>>> >> >>> > Sorry, I think we got confused about the symlink,
>>>> >> >>> > are >>>> >> >>> > symlinks >>>> >> >>> > in >>>> >> >>> > /var/run that point the /rhev when I was doing an LS it >>>> >> >>> > was >>>> >> >>> > listing >>>> >> >>> > the >>>> >> >>> > files in /rhev >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286 >>>> >> >>> > >>>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a -> >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine/248f46f 0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>>> >> >>> > >>>> >> >>> > ls -al >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine/248f46f 0-d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a >>>> >> >>> > total 2040 >>>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . >>>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56 >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease >>>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta >>>> >> >>> > >>>> >> >>> > Is it possible that this is the wrong image for hosted >>>> >> >>> > engine? >>>> >> >>> > >>>> >> >>> > this is all I get in vdsm log when running hosted-engine >>>> >> >>> > --connect-storage >>>> >> >>> > >>>> >> >>> > jsonrpc/4::ERROR::2018-01-12 >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > >>>> >> >>> > 10:52:53,019::__init__::611::j sonrpc.JsonRpcServer::(_handle_request) >>>> >> >>> > Internal server error >>>> >> >>> > Traceback (most recent call last): >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/yajsonrpc/__init__.py", >>>> >> >>> > line >>>> >> >>> > 606, >>>> >> >>> > in _handle_request >>>> >> >>> > res = method(**params) >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/rpc/Bridge.py", >>>> >> >>> > line >>>> >> >>> > 201, >>>> >> >>> > in >>>> >> >>> > _dynamicMethod >>>> >> >>> > result = fn(*methodArgs) >>>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", >>>> >> >>> > line >>>> >> >>> > 48, >>>> >> >>> > in >>>> >> >>> > method >>>> >> >>> > ret = func(*args, **kwargs) >>>> >> >>> > File "/usr/lib/python2.7/site-packa ges/vdsm/API.py", >>>> >> >>> > line >>>> >> >>> > 1354, in >>>> >> >>> > getAllVmIoTunePolicies >>>> >> >>> > io_tune_policies_dict = >>>> >> >>> > self._cif.getAllVmIoTunePolicies() >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", >>>> >> >>> > line >>>> >> >>> > 524, >>>> >> >>> > in >>>> >> >>> > getAllVmIoTunePolicies >>>> >> >>> > 'current_values': v.getIoTune()} >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", >>>> >> >>> > line >>>> >> >>> > 3481, >>>> >> >>> > in >>>> >> >>> > getIoTune >>>> >> >>> > result = self.getIoTuneResponse() >>>> >> >>> > File >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", >>>> >> >>> > line >>>> >> >>> > 3500, >>>> >> >>> > in >>>> >> >>> > getIoTuneResponse >>>> >> >>> > res = self._dom.blockIoTune( >>>> >> >>> > File >>>> >> >>> > >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/virt/virdomain.py", >>>> >> >>> > line >>>> >> >>> > 47, >>>> >> >>> > in __getattr__ >>>> >> >>> > % self.vmid) >>>> >> >>> > NotConnectedError: VM >>>> >> >>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' >>>> >> >>> > was not >>>> >> >>> > defined >>>> >> >>> > yet or was undefined >>>> >> >>> > >>>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak >>>> >> >>> > <msivak@redhat.com> >>>> >> >>> > wrote: >>>> >> >>> >> >>>> >> >>> >> Hi, >>>> >> >>> >> >>>> >> >>> >> what happens when you try hosted-engine >>>> >> >>> >> --connect-storage? >>>> >> >>> >> Do >>>> >> >>> >> you >>>> >> >>> >> see >>>> >> >>> >> any errors in the vdsm log? >>>> >> >>> >> >>>> >> >>> >> Best regards >>>> >> >>> >> >>>> >> >>> >> Martin Sivak >>>> >> >>> >> >>>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme >>>> >> >>> >> <jaymef@gmail.com> >>>> >> >>> >> wrote: >>>> >> >>> >> > Ok this is what I've done: >>>> >> >>> >> > >>>> >> >>> >> > - All three hosts in global maintenance mode >>>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop >>>> >> >>> >> > ovirt-ha-broker -- >>>> >> >>> >> > on >>>> >> >>> >> > all three hosts >>>> >> >>> >> > - Moved ALL files in >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ >>>> >> >>> >> > to >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup >>>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl >>>> >> >>> >> > start >>>> >> >>> >> > ovirt-ha-broker >>>> >> >>> >> > -- >>>> >> >>> >> > on all three hosts >>>> >> >>> >> > >>>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0): >>>> >> >>> >> > hosted-engine >>>> >> >>> >> > --vm-start >>>> >> >>> >> > >>>> >> >>> >> > Lots of errors in the logs still, it appears to be >>>> >> >>> >> > having >>>> >> >>> >> > problems >>>> >> >>> >> > with >>>> >> >>> >> > that >>>> >> >>> >> > directory still: >>>> >> >>> >> > >>>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > ovirt_hosted_engine_ha.broker. storage_broker.StorageBroker >>>> >> >>> >> > ERROR >>>> >> >>> >> > Failed >>>> >> >>> >> > to >>>> >> >>> >> > write metadata for host 1 to >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8#012Traceback >>>> >> >>> >> > (most recent call last):#012 File >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", >>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path, >>>> >> >>> >> > direct_flag >>>> >> >>> >> > | >>>> >> >>> >> > os.O_WRONLY | >>>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or >>>> >> >>> >> > directory: >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>>> >> >>> >> > >>>> >> >>> >> > There are no new files or symlinks in >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ >>>> >> >>> >> > >>>> >> >>> >> > - Jayme >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak >>>> >> >>> >> > <msivak@redhat.com> >>>> >> >>> >> > wrote: >>>> >> >>> >> >> >>>> >> >>> >> >> > Can you please stop all hosted engine tooling ( >>>> >> >>> >> >> >>>> >> >>> >> >> On all hosts I should have added. >>>> >> >>> >> >> >>>> >> >>> >> >> Martin >>>> >> >>> >> >> >>>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak >>>> >> >>> >> >> <msivak@redhat.com> >>>> >> >>> >> >> wrote: >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] >>>> >> >>> >> >> >> No >>>> >> >>> >> >> >> such >>>> >> >>> >> >> >> file >>>> >> >>> >> >> >> or >>>> >> >>> >> >> >> directory: >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> ls -al >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are >>>> >> >>> >> >> >> referring to >>>> >> >>> >> >> >> that >>>> >> >>> >> >> >> was >>>> >> >>> >> >> >> addressed in RC1 or something else? >>>> >> >>> >> >> > >>>> >> >>> >> >> > No, this file is the symlink. It should point to >>>> >> >>> >> >> > somewhere >>>> >> >>> >> >> > inside >>>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is >>>> >> >>> >> >> > really >>>> >> >>> >> >> > interesting. >>>> >> >>> >> >> > >>>> >> >>> >> >> > Can you please stop all hosted engine tooling >>>> >> >>> >> >> > (ovirt-ha-agent, >>>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is >>>> >> >>> >> >> > not >>>> >> >>> >> >> > important >>>> >> >>> >> >> > when >>>> >> >>> >> >> > services are stopped, but better safe than sorry) >>>> >> >>> >> >> > and >>>> >> >>> >> >> > restart >>>> >> >>> >> >> > all >>>> >> >>> >> >> > services again? >>>> >> >>> >> >> > >>>> >> >>> >> >> >> Could there possibly be a permissions >>>> >> >>> >> >> >> problem somewhere? >>>> >> >>> >> >> > >>>> >> >>> >> >> > Maybe, but the file itself looks out of the >>>> >> >>> >> >> > ordinary. >>>> >> >>> >> >> > I >>>> >> >>> >> >> > wonder >>>> >> >>> >> >> > how it >>>> >> >>> >> >> > got there. >>>> >> >>> >> >> > >>>> >> >>> >> >> > Best regards >>>> >> >>> >> >> > >>>> >> >>> >> >> > Martin Sivak >>>> >> >>> >> >> > >>>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme >>>> >> >>> >> >> > <jaymef@gmail.com> >>>> >> >>> >> >> > wrote: >>>> >> >>> >> >> >> Thanks for the help thus far. Storage could be >>>> >> >>> >> >> >> related >>>> >> >>> >> >> >> but >>>> >> >>> >> >> >> all >>>> >> >>> >> >> >> other >>>> >> >>> >> >> >> VMs on >>>> >> >>> >> >> >> same storage are running ok. The storage is >>>> >> >>> >> >> >> mounted >>>> >> >>> >> >> >> via >>>> >> >>> >> >> >> NFS >>>> >> >>> >> >> >> from >>>> >> >>> >> >> >> within one >>>> >> >>> >> >> >> of the three hosts, I realize this is not ideal. >>>> >> >>> >> >> >> This >>>> >> >>> >> >> >> was >>>> >> >>> >> >> >> setup >>>> >> >>> >> >> >> by >>>> >> >>> >> >> >> a >>>> >> >>> >> >> >> previous admin more as a proof of concept and VMs >>>> >> >>> >> >> >> were >>>> >> >>> >> >> >> put on >>>> >> >>> >> >> >> there >>>> >> >>> >> >> >> that >>>> >> >>> >> >> >> should not have been placed in a proof of concept >>>> >> >>> >> >> >> environment.. >>>> >> >>> >> >> >> it >>>> >> >>> >> >> >> was >>>> >> >>> >> >> >> intended to be rebuilt with proper storage down >>>> >> >>> >> >> >> the >>>> >> >>> >> >> >> road. >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts >>>> >> >>> >> >> >> mount >>>> >> >>> >> >> >> NFS >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/data >>>> >> >>> >> >> >> 4861742080 >>>> >> >>> >> >> >> 1039352832 3822389248 22% >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_data >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso >>>> >> >>> >> >> >> 4861742080 >>>> >> >>> >> >> >> 1039352832 3822389248 22% >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_iso >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.c om:/exports/import_export >>>> >> >>> >> >> >> 4861742080 >>>> >> >>> >> >> >> 1039352832 3822389248 22% >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_import__export >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> cultivar0.grove.silverorange.c om:/exports/hosted_engine >>>> >> >>> >> >> >> 4861742080 >>>> >> >>> >> >> >> 1039352832 3822389248 22% >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> Like I said, the VM data storage itself seems to >>>> >> >>> >> >> >> be >>>> >> >>> >> >> >> working >>>> >> >>> >> >> >> ok, >>>> >> >>> >> >> >> as >>>> >> >>> >> >> >> all >>>> >> >>> >> >> >> other >>>> >> >>> >> >> >> VMs appear to be running. >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> I'm curious why the broker log says this file is >>>> >> >>> >> >> >> not >>>> >> >>> >> >> >> found >>>> >> >>> >> >> >> when >>>> >> >>> >> >> >> it >>>> >> >>> >> >> >> is >>>> >> >>> >> >> >> correct and I can see the file at that path: >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] >>>> >> >>> >> >> >> No >>>> >> >>> >> >> >> such >>>> >> >>> >> >> >> file >>>> >> >>> >> >> >> or >>>> >> >>> >> >> >> directory: >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> ls -al >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are >>>> >> >>> >> >> >> referring to >>>> >> >>> >> >> >> that >>>> >> >>> >> >> >> was >>>> >> >>> >> >> >> addressed in RC1 or something else? Could
>>>> >> >>> >> >> >> possibly be >>>> >> >>> >> >> >> a >>>> >> >>> >> >> >> permissions >>>> >> >>> >> >> >> problem somewhere? >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms >>>> >> >>> >> >> >> installed >>>> >> >>> >> >> >> and the >>>> >> >>> >> >> >> host >>>> >> >>> >> >> >> engine >>>> >> >>> >> >> >> will not start is it safe for me to update hosts >>>> >> >>> >> >> >> to >>>> >> >>> >> >> >> 4.2 >>>> >> >>> >> >> >> RC1 >>>> >> >>> >> >> >> rpms? >>>> >> >>> >> >> >> Or >>>> >> >>> >> >> >> perhaps install that repo and *only* update the >>>> >> >>> >> >> >> ovirt >>>> >> >>> >> >> >> HA >>>> >> >>> >> >> >> packages? >>>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates >>>> >> >>> >> >> >> to >>>> >> >>> >> >> >> the >>>> >> >>> >> >> >> inaccessible >>>> >> >>> >> >> >> hosted >>>> >> >>> >> >> >> engine VM. >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> I should also mention one more thing. I >>>> >> >>> >> >> >> originally >>>> >> >>> >> >> >> upgraded >>>> >> >>> >> >> >> the >>>> >> >>> >> >> >> engine >>>> >> >>> >> >> >> VM >>>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed >>>> >> >>> >> >> >> due >>>> >> >>> >> >> >> to not >>>> >> >>> >> >> >> being >>>> >> >>> >> >> >> in >>>> >> >>> >> >> >> global maintenance, so I set global maintenance >>>> >> >>> >> >> >> and >>>> >> >>> >> >> >> ran >>>> >> >>> >> >> >> it >>>> >> >>> >> >> >> again, >>>> >> >>> >> >> >> which >>>> >> >>> >> >> >> appeared to complete as intended but never came >>>> >> >>> >> >> >> back >>>> >> >>> >> >> >> up >>>> >> >>> >> >> >> after. >>>> >> >>> >> >> >> Just >>>> >> >>> >> >> >> in >>>> >> >>> >> >> >> case >>>> >> >>> >> >> >> this might have anything at all to do with what >>>> >> >>> >> >> >> could >>>> >> >>> >> >> >> have >>>> >> >>> >> >> >> happened. >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> Thanks very much again, I very much appreciate
>>>> >> >>> >> >> >> help! >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> - Jayme >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi >>>> >> >>> >> >> >> <stirabos@redhat.com> >>>> >> >>> >> >> >> wrote: >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak >>>> >> >>> >> >> >>> <msivak@redhat.com> >>>> >> >>> >> >> >>> wrote: >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> Hi, >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by >>>> >> >>> >> >> >>>> restarting >>>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest >>>> >> >>> >> >> >>>> ovirt-hosted-engine-ha >>>> >> >>> >> >> >>>> and >>>> >> >>> >> >> >>>> -setup. We improved handling of the missing >>>> >> >>> >> >> >>>> symlink. >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1 >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> All the other issues seem to point to some >>>> >> >>> >> >> >>>> storage >>>> >> >>> >> >> >>>> problem >>>> >> >>> >> >> >>>> I >>>> >> >>> >> >> >>>> am >>>> >> >>> >> >> >>>> afraid. >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> You said you started the VM, do you see it in >>>> >> >>> >> >> >>>> virsh >>>> >> >>> >> >> >>>> -r >>>> >> >>> >> >> >>>> list? >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> Best regards >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> Martin Sivak >>>> >> >>> >> >> >>>> >>>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme >>>> >> >>> >> >> >>>> <jaymef@gmail.com> >>>> >> >>> >> >> >>>> wrote: >>>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to >>>> >> >>> >> >> >>>> > try >>>> >> >>> >> >> >>>> > at >>>> >> >>> >> >> >>>> > this >>>> >> >>> >> >> >>>> > point. >>>> >> >>> >> >> >>>> > Thank >>>> >> >>> >> >> >>>> > you >>>> >> >>> >> >> >>>> > for reading! >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted >>>> >> >>> >> >> >>>> > engine >>>> >> >>> >> >> >>>> > running >>>> >> >>> >> >> >>>> > after a >>>> >> >>> >> >> >>>> > botched >>>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from >>>> >> >>> >> >> >>>> > within >>>> >> >>> >> >> >>>> > one >>>> >> >>> >> >> >>>> > of >>>> >> >>> >> >> >>>> > the >>>> >> >>> >> >> >>>> > hosts. >>>> >> >>> >> >> >>>> > Right >>>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully >>>> >> >>> >> >> >>>> > updated >>>> >> >>> >> >> >>>> > with >>>> >> >>> >> >> >>>> > yum >>>> >> >>> >> >> >>>> > packages >>>> >> >>> >> >> >>>> > from >>>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with >>>> >> >>> >> >> >>>> > yum >>>> >> >>> >> >> >>>> > packages >>>> >> >>> >> >> >>>> > and >>>> >> >>> >> >> >>>> > failed to >>>> >> >>> >> >> >>>> > come >>>> >> >>> >> >> >>>> > up after reboot. As of right now, everything >>>> >> >>> >> >> >>>> > should >>>> >> >>> >> >> >>>> > have >>>> >> >>> >> >> >>>> > full >>>> >> >>> >> >> >>>> > yum >>>> >> >>> >> >> >>>> > updates >>>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global >>>> >> >>> >> >> >>>> > maintenance >>>> >> >>> >> >> >>>> > mode >>>> >> >>> >> >> >>>> > on >>>> >> >>> >> >> >>>> > right >>>> >> >>> >> >> >>>> > now >>>> >> >>> >> >> >>>> > and >>>> >> >>> >> >> >>>> > started hosted-engine on one of the three host >>>> >> >>> >> >> >>>> > and >>>> >> >>> >> >> >>>> > the >>>> >> >>> >> >> >>>> > status is >>>> >> >>> >> >> >>>> > currently: >>>> >> >>> >> >> >>>> > Engine status : {"reason": "failed
>>>> >> >>> >> >> >>>> > check”; >>>> >> >>> >> >> >>>> > "health": >>>> >> >>> >> >> >>>> > "bad", >>>> >> >>> >> >> >>>> > "vm": >>>> >> >>> >> >> >>>> > "up", "detail": "Up"} >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > this is what I get when trying to enter >>>> >> >>> >> >> >>>> > hosted-vm >>>> >> >>> >> >> >>>> > --console >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > The engine VM is running on this host >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine' >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > error: Domain not found: no domain with >>>> >> >>> >> >> >>>> > matching >>>> >> >>> >> >> >>>> > name >>>> >> >>> >> >> >>>> > 'HostedEngine' >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Here are logs from various sources when I >>>> >> >>> >> >> >>>> > start >>>> >> >>> >> >> >>>> > the >>>> >> >>> >> >> >>>> > VM on >>>> >> >>> >> >> >>>> > HOST3: >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > hosted-engine --vm-start >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID': >>>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} >>>> >> >>> >> >> >>>> > failed: >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not >>>> >> >>> >> >> >>>> > exist: >>>> >> >>> >> >> >>>> > {'vmId': >>>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'}) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: >>>> >> >>> >> >> >>>> > New >>>> >> >>> >> >> >>>> > machine >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started >>>> >> >>> >> >> >>>> > Virtual >>>> >> >>> >> >> >>>> > Machine >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting >>>> >> >>> >> >> >>>> > Virtual >>>> >> >>> >> >> >>>> > Machine >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now >>>> >> >>> >> >> >>>> > active >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", >>>> >> >>> >> >> >>>> > line >>>> >> >>> >> >> >>>> > 48, >>>> >> >>> >> >> >>>> > in >>>> >> >>> >> >> >>>> > method >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ret = func(*args, **kwargs) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/hsm.py", >>>> >> >>> >> >> >>>> > line >>>> >> >>> >> >> >>>> > 2718, in >>>> >> >>> >> >> >>>> > getStorageDomainInfo >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/hsm.py", >>>> >> >>> >> >> >>>> > line >>>> >> >>> >> >> >>>> > 304, in >>>> >> >>> >> >> >>>> > validateSdUUID >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > sdDom.validate() >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/fileSD.py", >>>> >> >>> >> >> >>>> > line >>>> >> >>> >> >> >>>> > 515, >>>> >> >>> >> >> >>>> > in validate >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > raise >>>> >> >>> >> >> >>>> > se.StorageDomainAccessError(self.sdUUID) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either >>>> >> >>> >> >> >>>> > partially >>>> >> >>> >> >> >>>> > accessible >>>> >> >>> >> >> >>>> > or >>>> >> >>> >> >> >>>> > entirely >>>> >> >>> >> >> >>>> > inaccessible: >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11 >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82:: storage.Dispatcher::(wrapper) >>>> >> >>> >> >> >>>> > FINISH >>>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either >>>> >> >>> >> >> >>>> > partially >>>> >> >>> >> >> >>>> > accessible >>>> >> >>> >> >> >>>> > or >>>> >> >>> >> >> >>>> > entirely >>>> >> >>> >> >> >>>> > inaccessible: >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d965e2f286',) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > LC_ALL=C >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>>> >> >>> >> >> >>>> > -name >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes >>>> >> >>> >> >> >>>> > -machine >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>>> >> >>> >> >> >>>> > -cpu >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>>> >> >>> >> >> >>>> > -uuid >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c82 9-c9d7-4b72-90d5-6fe58137504c' >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait >>>> >> >>> >> >> >>>> > -mon >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control >>>> >> >>> >> >> >>>> > -rtc >>>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>>> >> >>> >> >> >>>> > -no-reboot >>>> >> >>> >> >> >>>> > -boot >>>> >> >>> >> >> >>>> > strict=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 >>>> >> >>> >> >> >>>> > -drive >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421- c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format= raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978- 4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>>> >> >>> >> >> >>>> > -netdev >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt. hosted-engine-setup.0 >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 >>>> >> >>> >> >> >>>> > -spice >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>>> >> >>> >> >> >>>> > -object >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>>> >> >>> >> >> >>>> > -msg >>>> >> >>> >> >> >>>> > timestamp=on >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev >>>> >> >>> >> >> >>>> > pty,id=charconsole0: >>>> >> >>> >> >> >>>> > char >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>>> >> >>> >> >> >>>> > charconsole0) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down, >>>> >> >>> >> >> >>>> > reason=shutdown >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up >>>> >> >>> >> >> >>>> > libvirt >>>> >> >>> >> >> >>>> > version: >>>> >> >>> >> >> >>>> > 3.2.0, >>>> >> >>> >> >> >>>> > package: >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>, >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org ), >>>> >> >>> >> >> >>>> > qemu >>>> >> >>> >> >> >>>> > version: >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), >>>> >> >>> >> >> >>>> > hostname: >>>> >> >>> >> >> >>>> > cultivar3 >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > LC_ALL=C >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>>> >> >>> >> >> >>>> > -name >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes >>>> >> >>> >> >> >>>> > -machine >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>>> >> >>> >> >> >>>> > -cpu >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>>> >> >>> >> >> >>>> > -uuid >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c82 9-c9d7-4b72-90d5-6fe58137504c' >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait >>>> >> >>> >> >> >>>> > -mon >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control >>>> >> >>> >> >> >>>> > -rtc >>>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>>> >> >>> >> >> >>>> > -no-reboot >>>> >> >>> >> >> >>>> > -boot >>>> >> >>> >> >> >>>> > strict=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 >>>> >> >>> >> >> >>>> > -drive >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421- c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format= raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978- 4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>>> >> >>> >> >> >>>> > -netdev >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt. hosted-engine-setup.0 >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 >>>> >> >>> >> >> >>>> > -spice >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>>> >> >>> >> >> >>>> > -object >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>>> >> >>> >> >> >>>> > -msg >>>> >> >>> >> >> >>>> > timestamp=on >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev >>>> >> >>> >> >> >>>> > pty,id=charconsole0: >>>> >> >>> >> >> >>>> > char >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>>> >> >>> >> >> >>>> > charconsole0) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down, >>>> >> >>> >> >> >>>> > reason=shutdown >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up >>>> >> >>> >> >> >>>> > libvirt >>>> >> >>> >> >> >>>> > version: >>>> >> >>> >> >> >>>> > 3.2.0, >>>> >> >>> >> >> >>>> > package: >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>, >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org ), >>>> >> >>> >> >> >>>> > qemu >>>> >> >>> >> >> >>>> > version: >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), >>>> >> >>> >> >> >>>> > hostname: >>>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > LC_ALL=C >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm >>>> >> >>> >> >> >>>> > -name >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes >>>> >> >>> >> >> >>>> > -machine >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off >>>> >> >>> >> >> >>>> > -cpu >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 >>>> >> >>> >> >> >>>> > -uuid >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c82 9-c9d7-4b72-90d5-6fe58137504c' >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait >>>> >> >>> >> >> >>>> > -mon >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control >>>> >> >>> >> >> >>>> > -rtc >>>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet >>>> >> >>> >> >> >>>> > -no-reboot >>>> >> >>> >> >> >>>> > -boot >>>> >> >>> >> >> >>>> > strict=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 >>>> >> >>> >> >> >>>> > -drive >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421- c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format= raw,if=none,id=drive-virtio-disk0,serial=c2dde892-f978- 4dfc-a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 >>>> >> >>> >> >> >>>> > -netdev >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 >>>> >> >>> >> >> >>>> > -chardev >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt. hosted-engine-setup.0 >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 >>>> >> >>> >> >> >>>> > -spice >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 >>>> >> >>> >> >> >>>> > -object >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom >>>> >> >>> >> >> >>>> > -device >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 >>>> >> >>> >> >> >>>> > -msg >>>> >> >>> >> >> >>>> > timestamp=on >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev >>>> >> >>> >> >> >>>> > pty,id=charconsole0: >>>> >> >>> >> >> >>>> > char >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label >>>> >> >>> >> >> >>>> > charconsole0) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-h a/broker.log >>>> >> >>> >> >> >>>> > <== >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", >>>> >> >>> >> >> >>>> > line 151, in get_raw_stats >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | >>>> >> >>> >> >> >>>> > os.O_RDONLY | >>>> >> >>> >> >> >>>> > os.O_SYNC) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or
On Fri, Jan 12, 2018 at 6:56 PM, Jayme <jaymef@gmail.com> wrote: line line pty,id=charconsole0: process line line line promiscuous pty,id=charconsole0: the the line line line line line line promiscuous line line line there there the liveliness directory:
>>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11 >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 16:55:15,761::status_broker::9 2::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) >>>> >> >>> >> >> >>>> > Failed to read state. >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/status_broker.py", >>>> >> >>> >> >> >>>> > line 88, in run >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats() >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", >>>> >> >>> >> >> >>>> > line 162, in get_raw_stats >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > .format(str(e))) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno >>>> >> >>> >> >> >>>> > 2] >>>> >> >>> >> >> >>>> > No >>>> >> >>> >> >> >>>> > such >>>> >> >>> >> >> >>>> > file or >>>> >> >>> >> >> >>>> > directory: >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f- ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-h a/agent.log >>>> >> >>> >> >> >>>> > <== >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > result = refresh_method() >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", >>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > content = >>>> >> >>> >> >> >>>> > self._get_file_content_from_sh ared_storage(VM) >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", >>>> >> >>> >> >> >>>> > line 484, in >>>> >> >>> >> >> >>>> > _get_file_content_from_shared_storage >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > config_volume_path = >>>> >> >>> >> >> >>>> > self._get_config_volume_path() >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", >>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > conf_vol_uuid >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/lib/heconflib.py", >>>> >> >>> >> >> >>>> > line 358, in get_volume_path >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR, >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > RuntimeError: Path to volume >>>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f >>>> >> >>> >> >> >>>> > not >>>> >> >>> >> >> >>>> > found >>>> >> >>> >> >> >>>> > in /run/vdsm/storag >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11 >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260::vi rt.vmstats::(send_metrics) >>>> >> >>> >> >> >>>> > VM >>>> >> >>> >> >> >>>> > metrics >>>> >> >>> >> >> >>>> > collection failed >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > File >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", >>>> >> >>> >> >> >>>> > line >>>> >> >>> >> >> >>>> > 197, in >>>> >> >>> >> >> >>>> > send_metrics >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] = >>>> >> >>> >> >> >>>> > stat['cpuUsage'] >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > KeyError: 'cpuUsage' >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > ______________________________
>>>> >> >>> >> >> >>>> > Users mailing list >>>> >> >>> >> >> >>>> > Users@ovirt.org >>>> >> >>> >> >> >>>> > http://lists.ovirt.org/mailman /listinfo/users >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> ______________________________
>>>> >> >>> >> >> >>>> Users mailing list >>>> >> >>> >> >> >>>> Users@ovirt.org >>>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman /listinfo/users >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >>> >>>> >> >>> >> >> >> >>>> >> >>> >> > >>>> >> >>> >> > >>>> >> >>> > >>>> >> >>> > >>>> >> >> >>>> >> >> >>>> >> > >>>> > >>>> > >>> >>> >> >
I GOT IT WORKING!!!! On Fri, Jan 12, 2018 at 2:16 PM, Jayme <jaymef@gmail.com> wrote:
Martin, actually might be some good news here. I could not get to console using hosted-engine console but I connected through virsh and got a console to the hosted VM and was able to login, this is a great start. Now to find out what is wrong with the VM.
On Fri, Jan 12, 2018 at 2:11 PM, Jayme <jaymef@gmail.com> wrote:
No luck I'm afraid. It's very odd that I wouldn't be able to get a console to it, if the status is up and seen by virsh. Any clue?
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
# virsh -r list Id Name State ---------------------------------------------------- 118 Cultivar running
# hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
# hosted-engine --console 118 The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
# hosted-engine --console Cultivar The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
On Fri, Jan 12, 2018 at 2:05 PM, Martin Sivak <msivak@redhat.com> wrote:
Try listing the domains with
virsh -r list
maybe it just has some weird name...
Martin
I thought that it might be a good sign but unfortunately I cannot access it with console :( if I could get console access to it I might be able to fix the problem. But seeing is how the console is also not working leads me to believe there is a bigger issue at hand here.
hosted-engine --console The engine VM is running on this host error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine'
I really wonder if this is all a symlinking problem in some way. Is it possible for me to upgrade host to 4.2 RC2 without being able to upgrade the engine first or should I keep everything on 4.2 as it is?
On Fri, Jan 12, 2018 at 1:49 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
the VM is up according to the status (at least for a while). You should be able to use console and diagnose anything that happened inside (line the need for fsck and such) now.
Check the presence of those links again now, the metadata file content is not important, but the file has to exist (agents will populate it with status data). I have no new idea about what is wrong with that though.
Best regards
Martin
On Fri, Jan 12, 2018 at 5:47 PM, Jayme <jaymef@gmail.com> wrote:
The lock space issue was an issue I needed to clear but I don't
believe
it has resolved the problem. I shutdown agent and broker on all hosts and disconnected hosted-storage then enabled broker/agent on just one host and connected storage. I started the VM and actually didn't get any errors in the logs barely at all which was good to see, however the VM is still not running:
HOST3:
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}
==> /var/log/messages <== Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Jan 12 12:42:57 cultivar3 kernel: device vnet0 entered promiscuous mode Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Jan 12 12:42:57 cultivar3 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Jan 12 12:42:57 cultivar3 lldpad: recvfrom(Event interface): No buffer space available Jan 12 12:42:57 cultivar3 systemd-machined: New machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Started Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 systemd: Starting Virtual Machine qemu-111-Cultivar. Jan 12 12:42:57 cultivar3 kvm: 3 guests now active Jan 12 12:44:38 cultivar3 libvirtd: 2018-01-12 16:44:38.737+0000: 1535: error : qemuDomainAgentAvailable:6010 : Guest agent is not responding: QEMU guest agent is not connected
Interestingly though, now I'm seeing this in the logs which may be a new clue:
==> /var/log/vdsm/vdsm.log <== File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py",
126, in findDomain return NfsStorageDomain(NfsStorageDom ain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py",
116, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) jsonrpc/4::ERROR::2018-01-12 12:40:30,380::dispatcher::82::storage.Dispatcher::(wrapper) FINISH getStorageDomainInfo error=Storage domain does not exist: (u'248f46f0-d793-4581-9810-c9d965e2f286',) periodic/42::ERROR::2018-01-12 12:40:35,430::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:40:50,473::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/40::ERROR::2018-01-12 12:41:05,519::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? periodic/43::ERROR::2018-01-12 12:41:20,566::api::196::root::(_getHaInfo) failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished?
==> /var/log/ovirt-hosted-engine-ha/broker.log <== File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/bro ker/storage_broker.py", line 151, in get_raw_stats f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/ 14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af- 9f1e-f5b7ec1f1cf8' StatusStorageThread::ERROR::2018-01-12
12:32:06,049::status_broker::92::ovirt_hosted_engine_ha.brok er.status_broker.StatusBroker.Update::(run) Failed to read state. Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/bro ker/status_broker.py", line 88, in run self._storage_broker.get_raw_stats() File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/bro ker/storage_broker.py", line 162, in get_raw_stats .format(str(e))) RequestError: failed to read metadata: [Errno 2] No such file or directory:
'/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/ 14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af- 9f1e-f5b7ec1f1cf8'
On Fri, Jan 12, 2018 at 12:02 PM, Martin Sivak <msivak@redhat.com> wrote: > > The lock is the issue. > > - try running sanlock client status on all hosts > - also make sure you do not have some forgotten host still connected > to the lockspace, but without ha daemons running (and with the VM) > > I need to go to our president election now, I might check the email > later tonight. > > Martin > > On Fri, Jan 12, 2018 at 4:59 PM, Jayme <jaymef@gmail.com> wrote: > > Here are the newest logs from me trying to start hosted vm: > > > > ==> /var/log/messages <== > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > blocking > > state > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:14 cultivar0 kernel: device vnet4 entered
> > mode > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > blocking > > state > > Jan 12 11:58:14 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > forwarding state > > Jan 12 11:58:14 cultivar0 lldpad: recvfrom(Event interface): No > > buffer > > space > > available > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8715] > > manager: (vnet4): new Tun device > > (/org/freedesktop/NetworkManager/Devices/140) > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8795] > > device (vnet4): state change: unmanaged -> unavailable (reason > > 'connection-assumed') [10 20 41] > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12 15:58:14.879+0000: starting up libvirt version: 3.2.0, > > package: > > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, > > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > > cultivar0.grove.silverorange.com > > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > > guest=Cultivar,debug-threads=on -S -object > > > > > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/d omain-119-Cultivar/master-key.aes > > -machine pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > > -cpu > > Conroe -m 8192 -realtime mlock=off -smp > > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > > 'type=1,manufacturer=oVirt,product=oVirt > > > > > > Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8 031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > > -no-user-config -nodefaults -chardev > > > > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-119- Cultivar/monitor.sock,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control -rtc > > base=2018-01-12T15:58:14,driftfix=slew -global > > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > > -device > > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > > > > > > file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f 286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967- a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0, serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none, werror=stop,rerror=stop,aio=threads > > -device > > > > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti o-disk0,id=virtio-disk0,bootindex=1 > > -drive if=none,id=drive-ide0-1-0,readonly=on -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > > > > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83 ,bus=pci.0,addr=0x3 > > -chardev > > > > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > > -device > > > > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.rhevm.vdsm > > -chardev > > > > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > > -device > > > > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel 1,id=channel1,name=org.qemu.guest_agent.0 > > -chardev spicevmc,id=charchannel2,name=vdagent -device > > > > > > virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel 2,id=channel2,name=com.redhat.spice.0 > > -chardev > > > > > > socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine- setup.0,server,nowait > > -device > > > > > > virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel 3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > > -chardev pty,id=charconsole0 -device > > virtconsole,chardev=charconsole0,id=console0 -spice > > > > > > tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tl s-channel=default,seamless-migration=on > > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > > rng-random,id=objrng0,filename=/dev/urandom -device > > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > > timestamp=on > > > > ==> /var/log/messages <== > > Jan 12 11:58:14 cultivar0 NetworkManager[1092]: <info> > > [1515772694.8807] > > device (vnet4): state change: unavailable -> disconnected (reason > > 'none') > > [20 30 0] > > Jan 12 11:58:14 cultivar0 systemd-machined: New machine > > qemu-119-Cultivar. > > Jan 12 11:58:14 cultivar0 systemd: Started Virtual Machine > > qemu-119-Cultivar. > > Jan 12 11:58:14 cultivar0 systemd: Starting Virtual Machine > > qemu-119-Cultivar. > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12T15:58:15.094002Z qemu-kvm: -chardev
> > char > > device redirected to /dev/pts/1 (label charconsole0) > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 kvm: 5 guests now active > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12 15:58:15.217+0000: shutting down, reason=failed > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 libvirtd: 2018-01-12 15:58:15.217+0000: > > 1908: > > error : virLockManagerSanlockAcquire:1041 : resource busy: Failed to > > acquire > > lock: Lease is held by another host > > > > ==> /var/log/libvirt/qemu/Cultivar.log <== > > 2018-01-12T15:58:15.219934Z qemu-kvm: terminating on signal 15 from > > pid > > 1773 > > (/usr/sbin/libvirtd) > > > > ==> /var/log/messages <== > > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:15 cultivar0 kernel: device vnet4 left promiscuous mode > > Jan 12 11:58:15 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > > disabled > > state > > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > > [1515772695.2348] > > device (vnet4): state change: disconnected -> unmanaged (reason > > 'unmanaged') > > [30 10 3] > > Jan 12 11:58:15 cultivar0 NetworkManager[1092]: <info> > > [1515772695.2349] > > device (vnet4): released from master device ovirtmgmt > > Jan 12 11:58:15 cultivar0 kvm: 4 guests now active > > Jan 12 11:58:15 cultivar0 systemd-machined: Machine qemu-119-Cultivar > > terminated. > > > > ==> /var/log/vdsm/vdsm.log <== > > vm/4013c829::ERROR::2018-01-12 > > 11:58:15,444::vm::914::virt.vm::(_startUnderlyingVm) > > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start
> > failed > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 843, > > in > > _startUnderlyingVm > > self._run() > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2721, > > in > > _run > > dom.createWithFlags(flags) > > File "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", > > line > > 126, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in > > wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in > > createWithFlags > > if ret == -1: raise libvirtError ('virDomainCreateWithFlags() > > failed', > > dom=self) > > libvirtError: resource busy: Failed to acquire lock: Lease is held by > > another host > > jsonrpc/6::ERROR::2018-01-12 > > 11:58:16,421::__init__::611::jsonrpc.JsonRpcServer::(_handle _request) > > Internal server error > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> > 606, > > in _handle_request > > res = method(**params) > > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
> > 201, > > in > > _dynamicMethod > > result = fn(*methodArgs) > > File "<string>", line 2, in getAllVmIoTunePolicies > > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
> > 48, > > in > > method > > ret = func(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > > getAllVmIoTunePolicies > > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() > > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
> > in > > getAllVmIoTunePolicies > > 'current_values': v.getIoTune()} > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, > > in > > getIoTune > > result = self.getIoTuneResponse() > > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, > > in > > getIoTuneResponse > > res = self._dom.blockIoTune( > > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/virdomain.py", > > line > > 47, > > in __getattr__ > > % self.vmid) > > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was not > > defined > > yet or was undefined > > > > ==> /var/log/messages <== > > Jan 12 11:58:16 cultivar0 journal: vdsm jsonrpc.JsonRpcServer ERROR > > Internal > > server error#012Traceback (most recent call last):#012 File > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, > > in > > _handle_request#012 res = method(**params)#012 File > > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in > > _dynamicMethod#012 result = fn(*methodArgs)#012 File "<string>", > > line 2, > > in getAllVmIoTunePolicies#012 File > > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in > > method#012 ret = func(*args, **kwargs)#012 File > > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > > getAllVmIoTunePolicies#012 io_tune_policies_dict = > > self._cif.getAllVmIoTunePolicies()#012 File > > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, in > > getAllVmIoTunePolicies#012 'current_values': v.getIoTune()}#012 > > File > > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, in > > getIoTune#012 result = self.getIoTuneResponse()#012 File > > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, in > > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File > > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, > > in > > __getattr__#012 % self.vmid)#012NotConnectedError: VM > > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or was > > undefined > > > > On Fri, Jan 12, 2018 at 11:55 AM, Jayme <jaymef@gmail.com> wrote: > >> > >> One other tidbit I noticed is that it seems like there are less > >> errors > >> if > >> I started in paused mode: > >> > >> but still shows: Engine status : {"reason": > >> "bad > >> vm > >> status", "health": "bad", "vm": "up", "detail": "Paused"} > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> blocking state > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> disabled state > >> Jan 12 11:55:05 cultivar0 kernel: device vnet4 entered
> >> mode > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> blocking state > >> Jan 12 11:55:05 cultivar0 kernel: ovirtmgmt: port 6(vnet4) entered > >> forwarding state > >> Jan 12 11:55:05 cultivar0 lldpad: recvfrom(Event interface): No > >> buffer > >> space available > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3625] > >> manager: (vnet4): new Tun device > >> (/org/freedesktop/NetworkManager/Devices/139) > >> > >> ==> /var/log/libvirt/qemu/Cultivar.log <== > >> 2018-01-12 15:55:05.370+0000: starting up libvirt version: 3.2.0, > >> package: > >> 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, > >> 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > >> 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >> cultivar0.grove.silverorange.com > >> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin > >> QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >> guest=Cultivar,debug-threads=on -S -object > >> > >> > >> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/d omain-118-Cultivar/master-key.aes > >> -machine pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > >> -cpu > >> Conroe -m 8192 -realtime mlock=off -smp > >> 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >> 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >> 'type=1,manufacturer=oVirt,product=oVirt > >> > >> > >> Node,version=7-4.1708.el7.centos,serial=44454C4C-3300-1042-8 031-B4C04F4B4831,uuid=4013c829-c9d7-4b72-90d5-6fe58137504c' > >> -no-user-config -nodefaults -chardev > >> > >> > >> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-118- Cultivar/monitor.sock,server,nowait > >> -mon chardev=charmonitor,id=monitor,mode=control -rtc > >> base=2018-01-12T15:55:05,driftfix=slew -global > >> kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on > >> -device > >> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > >> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > >> > >> > >> file=/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f 286/c2dde892-f978-4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967- a1e5-fbe47c0cd705,format=raw,if=none,id=drive-virtio-disk0, serial=c2dde892-f978-4dfc-a421-c8e04cf387f9,cache=none, werror=stop,rerror=stop,aio=threads > >> -device > >> > >> > >> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virti o-disk0,id=virtio-disk0,bootindex=1 > >> -drive if=none,id=drive-ide0-1-0,readonly=on -device > >> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > >> tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > >> > >> > >> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:7f:d6:83 ,bus=pci.0,addr=0x3 > >> -chardev > >> > >> > >> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.com.redhat.rhevm.vdsm,server,nowait > >> -device > >> > >> > >> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel 0,id=channel0,name=com.redhat.rhevm.vdsm > >> -chardev > >> > >> > >> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.qemu.guest_agent.0,server,nowait > >> -device > >> > >> > >> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel 1,id=channel1,name=org.qemu.guest_agent.0 > >> -chardev spicevmc,id=charchannel2,name=vdagent -device > >> > >> > >> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel 2,id=channel2,name=com.redhat.spice.0 > >> -chardev > >> > >> > >> socket,id=charchannel3,path=/var/lib/libvirt/qemu/channels/4 013c829-c9d7-4b72-90d5-6fe58137504c.org.ovirt.hosted-engine- setup.0,server,nowait > >> -device > >> > >> > >> virtserialport,bus=virtio-serial0.0,nr=4,chardev=charchannel 3,id=channel3,name=org.ovirt.hosted-engine-setup.0 > >> -chardev pty,id=charconsole0 -device > >> virtconsole,chardev=charconsole0,id=console0 -spice > >> > >> > >> tls-port=5904,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tl s-channel=default,seamless-migration=on > >> -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > >> rng-random,id=objrng0,filename=/dev/urandom -device > >> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > >> timestamp=on > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3689] > >> device (vnet4): state change: unmanaged -> unavailable (reason > >> 'connection-assumed') [10 20 41] > >> Jan 12 11:55:05 cultivar0 NetworkManager[1092]: <info> > >> [1515772505.3702] > >> device (vnet4): state change: unavailable -> disconnected (reason > >> 'none') > >> [20 30 0] > >> Jan 12 11:55:05 cultivar0 systemd-machined: New machine > >> qemu-118-Cultivar. > >> Jan 12 11:55:05 cultivar0 systemd: Started Virtual Machine > >> qemu-118-Cultivar. > >> Jan 12 11:55:05 cultivar0 systemd: Starting Virtual Machine > >> qemu-118-Cultivar. > >> > >> ==> /var/log/libvirt/qemu/Cultivar.log <== > >> 2018-01-12T15:55:05.586827Z qemu-kvm: -chardev
> >> char > >> device redirected to /dev/pts/1 (label charconsole0) > >> > >> ==> /var/log/messages <== > >> Jan 12 11:55:05 cultivar0 kvm: 5 guests now active > >> > >> On Fri, Jan 12, 2018 at 11:36 AM, Jayme <jaymef@gmail.com> wrote: > >>> > >>> Yeah I am in global maintenance: > >>> > >>> state=GlobalMaintenance > >>> > >>> host0: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> host2: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> host3: {"reason": "vm not running on this host", "health": "bad", > >>> "vm": > >>> "down", "detail": "unknown"} > >>> > >>> I understand the lock is an issue, I'll try to make sure it is > >>> fully > >>> stopped on all three before starting but I don't think that is
> >>> issue at > >>> hand either. What concerns me is mostly that it seems to be > >>> unable > >>> to read > >>> the meta data, I think that might be the heart of the problem but > >>> I'm > >>> not > >>> sure what is causing it. > >>> > >>> On Fri, Jan 12, 2018 at 11:33 AM, Martin Sivak < msivak@redhat.com> > >>> wrote: > >>>> > >>>> > On all three hosts I ran hosted-engine --vm-shutdown; > >>>> > hosted-engine > >>>> > --vm-poweroff > >>>> > >>>> Are you in global maintenance? I think you were in one of the > >>>> previous > >>>> emails, but worth checking. > >>>> > >>>> > I started ovirt-ha-broker with systemctl as root user but it > >>>> > does > >>>> > appear to be running under vdsm: > >>>> > >>>> That is the correct behavior. > >>>> > >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is > >>>> > held > >>>> > by > >>>> > another host > >>>> > >>>> sanlock seems to think the VM runs somewhere and it is
> >>>> that > >>>> some other host tried to start the VM as well unless you are in > >>>> global > >>>> maintenance (that is why I asked the first question here). > >>>> > >>>> Martin > >>>> > >>>> On Fri, Jan 12, 2018 at 4:28 PM, Jayme <jaymef@gmail.com> wrote: > >>>> > Martin, > >>>> > > >>>> > Thanks so much for keeping with me, this is driving me crazy! I > >>>> > really do > >>>> > appreciate it, thanks again > >>>> > > >>>> > Let's go through this: > >>>> > > >>>> > HE VM is down - YES > >>>> > > >>>> > > >>>> > HE agent fails when opening metadata using the symlink - YES > >>>> > > >>>> > > >>>> > the symlink is there and readable by vdsm:kvm - it appears to > >>>> > be: > >>>> > > >>>> > > >>>> > lrwxrwxrwx. 1 vdsm kvm 159 Jan 10 21:20 > >>>> > 14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > -> > >>>> > > >>>> > > >>>> > > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com: _exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/im ages/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > > >>>> > > >>>> > And the files in the linked directory exist and have vdsm:kvm > >>>> > perms > >>>> > as > >>>> > well: > >>>> > > >>>> > > >>>> > # cd > >>>> > > >>>> > > >>>> > > >>>> > /rhev/data-center/mnt/cultivar0.grove.silverorange.com: _exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/im ages/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> > > >>>> > [root@cultivar0 14a20941-1b84-4b82-be8f-ace38d7c037a]# ls -al > >>>> > > >>>> > total 2040 > >>>> > > >>>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . > >>>> > > >>>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. > >>>> > > >>>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 11:19 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> > > >>>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease > >>>> > > >>>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 > >>>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta > >>>> > > >>>> > > >>>> > I started ovirt-ha-broker with systemctl as root user but it > >>>> > does > >>>> > appear to > >>>> > be running under vdsm: > >>>> > > >>>> > > >>>> > vdsm 16928 0.6 0.0 1618244 43328 ? Ssl 10:33 0:18 > >>>> > /usr/bin/python > >>>> > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker > >>>> > > >>>> > > >>>> > > >>>> > Here is something I tried: > >>>> > > >>>> > > >>>> > - On all three hosts I ran hosted-engine --vm-shutdown; > >>>> > hosted-engine > >>>> > --vm-poweroff > >>>> > > >>>> > - On HOST0 (cultivar0) I disconnected and reconnected storage > >>>> > using > >>>> > hosted-engine > >>>> > > >>>> > - Tried starting up the hosted VM on cultivar0 while tailing the > >>>> > logs: > >>>> > > >>>> > > >>>> > # hosted-engine --vm-start > >>>> > > >>>> > VM exists and is down, cleaning up and restarting > >>>> > > >>>> > > >>>> > > >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> > > >>>> > jsonrpc/2::ERROR::2018-01-12 > >>>> > 11:27:27,194::vm::1766::virt.vm::(_getRunningVmStats) > >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') Error fetching vm > >>>> > stats > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> >>>> > 1762, > >>>> > in > >>>> > _getRunningVmStats > >>>> > > >>>> > vm_sample.interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 45, in > >>>> > produce > >>>> > > >>>> > networks(vm, stats, first_sample, last_sample, interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 322, in > >>>> > networks > >>>> > > >>>> > if nic.name.startswith('hostdev'): > >>>> > > >>>> > AttributeError: name > >>>> > > >>>> > jsonrpc/3::ERROR::2018-01-12 > >>>> > > >>>> > > >>>> > 11:27:27,221::__init__::611::j sonrpc.JsonRpcServer::(_handle_request) > >>>> > Internal server error > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/yajsonrpc/__init__.py", > >>>> > line > >>>> > 606, > >>>> > in _handle_request > >>>> > > >>>> > res = method(**params) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/rpc/Bridge.py", > >>>> > line > >>>> > 201, in > >>>> > _dynamicMethod > >>>> > > >>>> > result = fn(*methodArgs) > >>>> > > >>>> > File "<string>", line 2, in getAllVmIoTunePolicies > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", > >>>> > line > >>>> > 48, > >>>> > in > >>>> > method > >>>> > > >>>> > ret = func(*args, **kwargs) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line > >>>> > 1354, > >>>> > in > >>>> > getAllVmIoTunePolicies > >>>> > > >>>> > io_tune_policies_dict = self._cif.getAllVmIoTunePolici es() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py",
> >>>> > 524, > >>>> > in > >>>> > getAllVmIoTunePolicies > >>>> > > >>>> > 'current_values': v.getIoTune()} > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> >>>> > 3481, > >>>> > in > >>>> > getIoTune > >>>> > > >>>> > result = self.getIoTuneResponse() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> >>>> > 3500, > >>>> > in > >>>> > getIoTuneResponse > >>>> > > >>>> > res = self._dom.blockIoTune( > >>>> > > >>>> > File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", > >>>> > line > >>>> > 47, > >>>> > in __getattr__ > >>>> > > >>>> > % self.vmid) > >>>> > > >>>> > NotConnectedError: VM '4013c829-c9d7-4b72-90d5-6fe58137504c' was > >>>> > not > >>>> > defined > >>>> > yet or was undefined > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 journal: vdsm jsonrpc.JsonRpcServer > >>>> > ERROR > >>>> > Internal > >>>> > server error#012Traceback (most recent call last):#012 File > >>>> > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> >>>> > 606, > >>>> > in > >>>> > _handle_request#012 res = method(**params)#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
> >>>> > in > >>>> > _dynamicMethod#012 result = fn(*methodArgs)#012 File > >>>> > "<string>", > >>>> > line 2, > >>>> > in getAllVmIoTunePolicies#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
> >>>> > in > >>>> > method#012 ret = func(*args, **kwargs)#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1354, in > >>>> > getAllVmIoTunePolicies#012 io_tune_policies_dict = > >>>> > self._cif.getAllVmIoTunePolicies()#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 524, > >>>> > in > >>>> > getAllVmIoTunePolicies#012 'current_values': > >>>> > v.getIoTune()}#012 > >>>> > File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3481, > >>>> > in > >>>> > getIoTune#012 result = self.getIoTuneResponse()#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3500, > >>>> > in > >>>> > getIoTuneResponse#012 res = self._dom.blockIoTune(#012 File > >>>> > "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
> >>>> > 47, > >>>> > in > >>>> > __getattr__#012 % self.vmid)#012NotConnectedError: VM > >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' was not defined yet or > >>>> > was > >>>> > undefined > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > blocking > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 entered > >>>> > promiscuous > >>>> > mode > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > blocking > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > forwarding state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 lldpad: recvfrom(Event interface): No > >>>> > buffer > >>>> > space > >>>> > available > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4264] > >>>> > manager: (vnet4): new Tun device > >>>> > (/org/freedesktop/NetworkManager/Devices/135) > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4342] > >>>> > device (vnet4): state change: unmanaged -> unavailable (reason > >>>> > 'connection-assumed') [10 20 41] > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.4353] > >>>> > device (vnet4): state change: unavailable -> disconnected > >>>> > (reason > >>>> > 'none') > >>>> > [20 30 0] > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12 15:27:27.435+0000: starting up libvirt version: > >>>> > 3.2.0, > >>>> > package: > >>>> > 14.el7_4.7 (CentOS BuildSystem <http://bugs.centos.org>, > >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org), qemu version: > >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), hostname: > >>>> > cultivar0.grove.silverorange.com > >>>> > > >>>> > LC_ALL=C PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin > >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name > >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> > > >>>> > > >>>> > > >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-114-Cultivar/master-key.aes > >>>> > -machine > >>>> > pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off > >>>> > -cpu > >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> > > >>>> > > >>>> > > >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-3300-1042-8031-B4C04F4B4831,uuid=4013c829 -c9d7-4b72-90d5-6fe58137504c' > >>>> > -no-user-config -nodefaults -chardev > >>>> > > >>>> > > >>>> > > >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-114-Cultivar/monitor.sock,server,nowait > >>>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc > >>>> > base=2018-01-12T15:27:27,driftfix=slew -global > >>>> > kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot > >>>> > strict=on > >>>> > -device > >>>> > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > >>>> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive > >>>> > > >>>> > > >>>> > > >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8 e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw, if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> > -device > >>>> > > >>>> > > >>>> > > >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> > -drive if=none,id=drive-ide0-1-0,readonly=on -device > >>>> > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev > >>>> > tap,fd=35,id=hostnet0,vhost=on,vhostfd=38 -device > >>>> > > >>>> > > >>>> > > >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> > -chardev > >>>> > > >>>> > > >>>> > > >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> > -chardev > >>>> > > >>>> > > >>>> > > >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> > -chardev spicevmc,id=charchannel2,name=vdagent -device > >>>> > > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> > -chardev > >>>> > > >>>> > > >>>> > > >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> > -device > >>>> > > >>>> > > >>>> > > >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.h osted-engine-setup.0 > >>>> > -chardev pty,id=charconsole0 -device > >>>> > virtconsole,chardev=charconsole0,id=console0 -spice > >>>> > > >>>> > > >>>> > > >>>> > tls-port=5904,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -object > >>>> > rng-random,id=objrng0,filename=/dev/urandom -device > >>>> > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x5 -msg > >>>> > timestamp=on > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: New machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd: Started Virtual Machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd: Starting Virtual Machine > >>>> > qemu-114-Cultivar. > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12T15:27:27.651669Z qemu-kvm: -chardev > >>>> > pty,id=charconsole0: > >>>> > char > >>>> > device redirected to /dev/pts/2 (label charconsole0) > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kvm: 5 guests now active > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12 15:27:27.773+0000: shutting down, reason=failed > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 libvirtd: 2018-01-12 > >>>> > 15:27:27.773+0000: > >>>> > 1910: > >>>> > error : virLockManagerSanlockAcquire:1041 : resource busy: > >>>> > Failed > >>>> > to > >>>> > acquire > >>>> > lock: Lease is held by another host > >>>> > > >>>> > > >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> > > >>>> > 2018-01-12T15:27:27.776135Z qemu-kvm: terminating on signal 15 > >>>> > from > >>>> > pid 1773 > >>>> > (/usr/sbin/libvirtd) > >>>> > > >>>> > > >>>> > ==> /var/log/messages <== > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: device vnet4 left
> >>>> > mode > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kernel: ovirtmgmt: port 6(vnet4) > >>>> > entered > >>>> > disabled > >>>> > state > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.7989] > >>>> > device (vnet4): state change: disconnected -> unmanaged (reason > >>>> > 'unmanaged') > >>>> > [30 10 3] > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 NetworkManager[1092]: <info> > >>>> > [1515770847.7989] > >>>> > device (vnet4): released from master device ovirtmgmt > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 kvm: 4 guests now active > >>>> > > >>>> > Jan 12 11:27:27 cultivar0 systemd-machined: Machine > >>>> > qemu-114-Cultivar > >>>> > terminated. > >>>> > > >>>> > > >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> > > >>>> > vm/4013c829::ERROR::2018-01-12 > >>>> > 11:27:28,001::vm::914::virt.vm::(_startUnderlyingVm) > >>>> > (vmId='4013c829-c9d7-4b72-90d5-6fe58137504c') The vm start > >>>> > process > >>>> > failed > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> >>>> > 843, > >>>> > in > >>>> > _startUnderlyingVm > >>>> > > >>>> > self._run() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
> >>>> > 2721, > >>>> > in > >>>> > _run > >>>> > > >>>> > dom.createWithFlags(flags) > >>>> > > >>>> > File > >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/libvirtconnection.py", > >>>> > line > >>>> > 126, in wrapper > >>>> > > >>>> > ret = f(*args, **kwargs) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packages/vdsm/utils.py",
> >>>> > 512, > >>>> > in > >>>> > wrapper > >>>> > > >>>> > return func(inst, *args, **kwargs) > >>>> > > >>>> > File "/usr/lib64/python2.7/site-packages/libvirt.py",
> >>>> > 1069, > >>>> > in > >>>> > createWithFlags > >>>> > > >>>> > if ret == -1: raise libvirtError > >>>> > ('virDomainCreateWithFlags() > >>>> > failed', > >>>> > dom=self) > >>>> > > >>>> > libvirtError: resource busy: Failed to acquire lock: Lease is > >>>> > held > >>>> > by > >>>> > another host > >>>> > > >>>> > periodic/47::ERROR::2018-01-12 > >>>> > 11:27:32,858::periodic::215::v irt.periodic.Operation::(__call__) > >>>> > <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x3692590> > >>>> > operation > >>>> > failed > >>>> > > >>>> > Traceback (most recent call last): > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/periodic.py", > >>>> > line > >>>> > 213, > >>>> > in __call__ > >>>> > > >>>> > self._func() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/sampling.py", > >>>> > line > >>>> > 522, > >>>> > in __call__ > >>>> > > >>>> > self._send_metrics() > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/sampling.py", > >>>> > line > >>>> > 538, > >>>> > in _send_metrics > >>>> > > >>>> > vm_sample.interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 45, in > >>>> > produce > >>>> > > >>>> > networks(vm, stats, first_sample, last_sample, interval) > >>>> > > >>>> > File "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", > >>>> > line > >>>> > 322, in > >>>> > networks > >>>> > > >>>> > if nic.name.startswith('hostdev'): > >>>> > > >>>> > AttributeError: name > >>>> > > >>>> > > >>>> > On Fri, Jan 12, 2018 at 11:14 AM, Martin Sivak > >>>> > <msivak@redhat.com> > >>>> > wrote: > >>>> >> > >>>> >> Hmm that rules out most of NFS related permission issues. > >>>> >> > >>>> >> So the current status is (I need to sum it up to get the full > >>>> >> picture): > >>>> >> > >>>> >> - HE VM is down > >>>> >> - HE agent fails when opening metadata using the symlink > >>>> >> - the symlink is there > >>>> >> - the symlink is readable by vdsm:kvm > >>>> >> > >>>> >> Hmm can you check under which user is ovirt-ha-broker started? > >>>> >> > >>>> >> Martin > >>>> >> > >>>> >> > >>>> >> On Fri, Jan 12, 2018 at 4:10 PM, Jayme <jaymef@gmail.com> > >>>> >> wrote: > >>>> >> > Same thing happens with data images of other VMs as well > >>>> >> > though, > >>>> >> > and > >>>> >> > those > >>>> >> > seem to be running ok so I'm not sure if it's the
> >>>> >> > > >>>> >> > On Fri, Jan 12, 2018 at 11:08 AM, Jayme < jaymef@gmail.com> > >>>> >> > wrote: > >>>> >> >> > >>>> >> >> Martin, > >>>> >> >> > >>>> >> >> I can as VDSM user but not as root . I get permission denied > >>>> >> >> trying to > >>>> >> >> touch one of the files as root, is that normal? > >>>> >> >> > >>>> >> >> On Fri, Jan 12, 2018 at 11:03 AM, Martin Sivak > >>>> >> >> <msivak@redhat.com> > >>>> >> >> wrote: > >>>> >> >>> > >>>> >> >>> Hmm, then it might be a permission issue indeed. Can you > >>>> >> >>> touch > >>>> >> >>> the > >>>> >> >>> file? Open it? (try hexdump) Just to make sure NFS does not > >>>> >> >>> prevent > >>>> >> >>> you from doing that. > >>>> >> >>> > >>>> >> >>> Martin > >>>> >> >>> > >>>> >> >>> On Fri, Jan 12, 2018 at 3:57 PM, Jayme < jaymef@gmail.com> > >>>> >> >>> wrote: > >>>> >> >>> > Sorry, I think we got confused about the symlink,
> >>>> >> >>> > are > >>>> >> >>> > symlinks > >>>> >> >>> > in > >>>> >> >>> > /var/run that point the /rhev when I was doing an LS it > >>>> >> >>> > was > >>>> >> >>> > listing > >>>> >> >>> > the > >>>> >> >>> > files in /rhev > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286 > >>>> >> >>> > > >>>> >> >>> > 14a20941-1b84-4b82-be8f-ace38d7c037a -> > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine/248f46f0- d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> >> >>> > > >>>> >> >>> > ls -al > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine/248f46f0- d793-4581-9810-c9d965e2f286/images/14a20941-1b84-4b82-be8f-ace38d7c037a > >>>> >> >>> > total 2040 > >>>> >> >>> > drwxr-xr-x. 2 vdsm kvm 4096 Jan 12 10:51 . > >>>> >> >>> > drwxr-xr-x. 8 vdsm kvm 4096 Feb 3 2016 .. > >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1028096 Jan 12 10:56 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> > -rw-rw----. 1 vdsm kvm 1048576 Feb 3 2016 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.lease > >>>> >> >>> > -rw-r--r--. 1 vdsm kvm 283 Feb 3 2016 > >>>> >> >>> > 8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8.meta > >>>> >> >>> > > >>>> >> >>> > Is it possible that this is the wrong image for hosted > >>>> >> >>> > engine? > >>>> >> >>> > > >>>> >> >>> > this is all I get in vdsm log when running hosted-engine > >>>> >> >>> > --connect-storage > >>>> >> >>> > > >>>> >> >>> > jsonrpc/4::ERROR::2018-01-12 > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >>> > 10:52:53,019::__init__::611::j sonrpc.JsonRpcServer::(_handle_request) > >>>> >> >>> > Internal server error > >>>> >> >>> > Traceback (most recent call last): > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/yajsonrpc/__init__.py", > >>>> >> >>> > line > >>>> >> >>> > 606, > >>>> >> >>> > in _handle_request > >>>> >> >>> > res = method(**params) > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/rpc/Bridge.py", > >>>> >> >>> > line > >>>> >> >>> > 201, > >>>> >> >>> > in > >>>> >> >>> > _dynamicMethod > >>>> >> >>> > result = fn(*methodArgs) > >>>> >> >>> > File "<string>", line 2, in getAllVmIoTunePolicies > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", > >>>> >> >>> > line > >>>> >> >>> > 48, > >>>> >> >>> > in > >>>> >> >>> > method > >>>> >> >>> > ret = func(*args, **kwargs) > >>>> >> >>> > File "/usr/lib/python2.7/site-packa ges/vdsm/API.py", > >>>> >> >>> > line > >>>> >> >>> > 1354, in > >>>> >> >>> > getAllVmIoTunePolicies > >>>> >> >>> > io_tune_policies_dict = > >>>> >> >>> > self._cif.getAllVmIoTunePolicies() > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", > >>>> >> >>> > line > >>>> >> >>> > 524, > >>>> >> >>> > in > >>>> >> >>> > getAllVmIoTunePolicies > >>>> >> >>> > 'current_values': v.getIoTune()} > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", > >>>> >> >>> > line > >>>> >> >>> > 3481, > >>>> >> >>> > in > >>>> >> >>> > getIoTune > >>>> >> >>> > result = self.getIoTuneResponse() > >>>> >> >>> > File > >>>> >> >>> > "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", > >>>> >> >>> > line > >>>> >> >>> > 3500, > >>>> >> >>> > in > >>>> >> >>> > getIoTuneResponse > >>>> >> >>> > res = self._dom.blockIoTune( > >>>> >> >>> > File > >>>> >> >>> > > >>>> >> >>> > "/usr/lib/python2.7/site-packa ges/vdsm/virt/virdomain.py", > >>>> >> >>> > line > >>>> >> >>> > 47, > >>>> >> >>> > in __getattr__ > >>>> >> >>> > % self.vmid) > >>>> >> >>> > NotConnectedError: VM > >>>> >> >>> > '4013c829-c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> > was not > >>>> >> >>> > defined > >>>> >> >>> > yet or was undefined > >>>> >> >>> > > >>>> >> >>> > On Fri, Jan 12, 2018 at 10:48 AM, Martin Sivak > >>>> >> >>> > <msivak@redhat.com> > >>>> >> >>> > wrote: > >>>> >> >>> >> > >>>> >> >>> >> Hi, > >>>> >> >>> >> > >>>> >> >>> >> what happens when you try hosted-engine > >>>> >> >>> >> --connect-storage? > >>>> >> >>> >> Do > >>>> >> >>> >> you > >>>> >> >>> >> see > >>>> >> >>> >> any errors in the vdsm log? > >>>> >> >>> >> > >>>> >> >>> >> Best regards > >>>> >> >>> >> > >>>> >> >>> >> Martin Sivak > >>>> >> >>> >> > >>>> >> >>> >> On Fri, Jan 12, 2018 at 3:41 PM, Jayme > >>>> >> >>> >> <jaymef@gmail.com> > >>>> >> >>> >> wrote: > >>>> >> >>> >> > Ok this is what I've done: > >>>> >> >>> >> > > >>>> >> >>> >> > - All three hosts in global maintenance mode > >>>> >> >>> >> > - Ran: systemctl stop ovirt-ha-broker; systemctl stop > >>>> >> >>> >> > ovirt-ha-broker -- > >>>> >> >>> >> > on > >>>> >> >>> >> > all three hosts > >>>> >> >>> >> > - Moved ALL files in > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ > >>>> >> >>> >> > to > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/backup > >>>> >> >>> >> > - Ran: systemctl start ovirt-ha-broker; systemctl > >>>> >> >>> >> > start > >>>> >> >>> >> > ovirt-ha-broker > >>>> >> >>> >> > -- > >>>> >> >>> >> > on all three hosts > >>>> >> >>> >> > > >>>> >> >>> >> > - attempt start of engine vm from HOST0 (cultivar0): > >>>> >> >>> >> > hosted-engine > >>>> >> >>> >> > --vm-start > >>>> >> >>> >> > > >>>> >> >>> >> > Lots of errors in the logs still, it appears to be > >>>> >> >>> >> > having > >>>> >> >>> >> > problems > >>>> >> >>> >> > with > >>>> >> >>> >> > that > >>>> >> >>> >> > directory still: > >>>> >> >>> >> > > >>>> >> >>> >> > Jan 12 10:40:13 cultivar0 journal: ovirt-ha-broker > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > ovirt_hosted_engine_ha.broker. storage_broker.StorageBroker > >>>> >> >>> >> > ERROR > >>>> >> >>> >> > Failed > >>>> >> >>> >> > to > >>>> >> >>> >> > write metadata for host 1 to > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7 c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8#012Traceback > >>>> >> >>> >> > (most recent call last):#012 File > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> > line 202, in put_stats#012 f = os.open(path, > >>>> >> >>> >> > direct_flag > >>>> >> >>> >> > | > >>>> >> >>> >> > os.O_WRONLY | > >>>> >> >>> >> > os.O_SYNC)#012OSError: [Errno 2] No such file or > >>>> >> >>> >> > directory: > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d 7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> > > >>>> >> >>> >> > There are no new files or symlinks in > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/ > >>>> >> >>> >> > > >>>> >> >>> >> > - Jayme > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> >> > On Fri, Jan 12, 2018 at 10:23 AM, Martin Sivak > >>>> >> >>> >> > <msivak@redhat.com> > >>>> >> >>> >> > wrote: > >>>> >> >>> >> >> > >>>> >> >>> >> >> > Can you please stop all hosted engine tooling ( > >>>> >> >>> >> >> > >>>> >> >>> >> >> On all hosts I should have added. > >>>> >> >>> >> >> > >>>> >> >>> >> >> Martin > >>>> >> >>> >> >> > >>>> >> >>> >> >> On Fri, Jan 12, 2018 at 3:22 PM, Martin Sivak > >>>> >> >>> >> >> <msivak@redhat.com> > >>>> >> >>> >> >> wrote: > >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] > >>>> >> >>> >> >> >> No > >>>> >> >>> >> >> >> such > >>>> >> >>> >> >> >> file > >>>> >> >>> >> >> >> or > >>>> >> >>> >> >> >> directory: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d 7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> ls -al > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7 c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7 c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are > >>>> >> >>> >> >> >> referring to > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> addressed in RC1 or something else? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > No, this file is the symlink. It should point to > >>>> >> >>> >> >> > somewhere > >>>> >> >>> >> >> > inside > >>>> >> >>> >> >> > /rhev/. I see it is a 1G file in your case. That is > >>>> >> >>> >> >> > really > >>>> >> >>> >> >> > interesting. > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Can you please stop all hosted engine tooling > >>>> >> >>> >> >> > (ovirt-ha-agent, > >>>> >> >>> >> >> > ovirt-ha-broker), move the file (metadata file is > >>>> >> >>> >> >> > not > >>>> >> >>> >> >> > important > >>>> >> >>> >> >> > when > >>>> >> >>> >> >> > services are stopped, but better safe than sorry) > >>>> >> >>> >> >> > and > >>>> >> >>> >> >> > restart > >>>> >> >>> >> >> > all > >>>> >> >>> >> >> > services again? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> >> Could there possibly be a permissions > >>>> >> >>> >> >> >> problem somewhere? > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Maybe, but the file itself looks out of the > >>>> >> >>> >> >> > ordinary. > >>>> >> >>> >> >> > I > >>>> >> >>> >> >> > wonder > >>>> >> >>> >> >> > how it > >>>> >> >>> >> >> > got there. > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Best regards > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > Martin Sivak > >>>> >> >>> >> >> > > >>>> >> >>> >> >> > On Fri, Jan 12, 2018 at 3:09 PM, Jayme > >>>> >> >>> >> >> > <jaymef@gmail.com> > >>>> >> >>> >> >> > wrote: > >>>> >> >>> >> >> >> Thanks for the help thus far. Storage could be > >>>> >> >>> >> >> >> related > >>>> >> >>> >> >> >> but > >>>> >> >>> >> >> >> all > >>>> >> >>> >> >> >> other > >>>> >> >>> >> >> >> VMs on > >>>> >> >>> >> >> >> same storage are running ok. The storage is > >>>> >> >>> >> >> >> mounted > >>>> >> >>> >> >> >> via > >>>> >> >>> >> >> >> NFS > >>>> >> >>> >> >> >> from > >>>> >> >>> >> >> >> within one > >>>> >> >>> >> >> >> of the three hosts, I realize this is not ideal. > >>>> >> >>> >> >> >> This > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> setup > >>>> >> >>> >> >> >> by > >>>> >> >>> >> >> >> a > >>>> >> >>> >> >> >> previous admin more as a proof of concept and VMs > >>>> >> >>> >> >> >> were > >>>> >> >>> >> >> >> put on > >>>> >> >>> >> >> >> there > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> should not have been placed in a proof of concept > >>>> >> >>> >> >> >> environment.. > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> intended to be rebuilt with proper storage down > >>>> >> >>> >> >> >> the > >>>> >> >>> >> >> >> road. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> So the storage is on HOST0 and the other hosts > >>>> >> >>> >> >> >> mount > >>>> >> >>> >> >> >> NFS > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.c om:/exports/data > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_data > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.com:/exports/iso > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_iso > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.c om:/exports/import_export > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_import__export > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> cultivar0.grove.silverorange.c om:/exports/hosted_engine > >>>> >> >>> >> >> >> 4861742080 > >>>> >> >>> >> >> >> 1039352832 3822389248 22% > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /rhev/data-center/mnt/cultivar 0.grove.silverorange.com:_exports_hosted__engine > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Like I said, the VM data storage itself seems to > >>>> >> >>> >> >> >> be > >>>> >> >>> >> >> >> working > >>>> >> >>> >> >> >> ok, > >>>> >> >>> >> >> >> as > >>>> >> >>> >> >> >> all > >>>> >> >>> >> >> >> other > >>>> >> >>> >> >> >> VMs appear to be running. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> I'm curious why the broker log says this file is > >>>> >> >>> >> >> >> not > >>>> >> >>> >> >> >> found > >>>> >> >>> >> >> >> when > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> is > >>>> >> >>> >> >> >> correct and I can see the file at that path: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> RequestError: failed to read metadata: [Errno 2] > >>>> >> >>> >> >> >> No > >>>> >> >>> >> >> >> such > >>>> >> >>> >> >> >> file > >>>> >> >>> >> >> >> or > >>>> >> >>> >> >> >> directory: > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d 7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> ls -al > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7 c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> -rw-rw----. 1 vdsm kvm 1028096 Jan 12 09:59 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> /var/run/vdsm/storage/248f46f0 -d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7 c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8 > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Is this due to the symlink problem you guys are > >>>> >> >>> >> >> >> referring to > >>>> >> >>> >> >> >> that > >>>> >> >>> >> >> >> was > >>>> >> >>> >> >> >> addressed in RC1 or something else? Could
> >>>> >> >>> >> >> >> possibly be > >>>> >> >>> >> >> >> a > >>>> >> >>> >> >> >> permissions > >>>> >> >>> >> >> >> problem somewhere? > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Assuming that all three hosts have 4.2 rpms > >>>> >> >>> >> >> >> installed > >>>> >> >>> >> >> >> and the > >>>> >> >>> >> >> >> host > >>>> >> >>> >> >> >> engine > >>>> >> >>> >> >> >> will not start is it safe for me to update hosts > >>>> >> >>> >> >> >> to > >>>> >> >>> >> >> >> 4.2 > >>>> >> >>> >> >> >> RC1 > >>>> >> >>> >> >> >> rpms? > >>>> >> >>> >> >> >> Or > >>>> >> >>> >> >> >> perhaps install that repo and *only* update
> >>>> >> >>> >> >> >> ovirt > >>>> >> >>> >> >> >> HA > >>>> >> >>> >> >> >> packages? > >>>> >> >>> >> >> >> Assuming that I cannot yet apply the same updates > >>>> >> >>> >> >> >> to > >>>> >> >>> >> >> >> the > >>>> >> >>> >> >> >> inaccessible > >>>> >> >>> >> >> >> hosted > >>>> >> >>> >> >> >> engine VM. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> I should also mention one more thing. I > >>>> >> >>> >> >> >> originally > >>>> >> >>> >> >> >> upgraded > >>>> >> >>> >> >> >> the > >>>> >> >>> >> >> >> engine > >>>> >> >>> >> >> >> VM > >>>> >> >>> >> >> >> first using new RPMS then engine-setup. It failed > >>>> >> >>> >> >> >> due > >>>> >> >>> >> >> >> to not > >>>> >> >>> >> >> >> being > >>>> >> >>> >> >> >> in > >>>> >> >>> >> >> >> global maintenance, so I set global
> >>>> >> >>> >> >> >> and > >>>> >> >>> >> >> >> ran > >>>> >> >>> >> >> >> it > >>>> >> >>> >> >> >> again, > >>>> >> >>> >> >> >> which > >>>> >> >>> >> >> >> appeared to complete as intended but never came > >>>> >> >>> >> >> >> back > >>>> >> >>> >> >> >> up > >>>> >> >>> >> >> >> after. > >>>> >> >>> >> >> >> Just > >>>> >> >>> >> >> >> in > >>>> >> >>> >> >> >> case > >>>> >> >>> >> >> >> this might have anything at all to do with what > >>>> >> >>> >> >> >> could > >>>> >> >>> >> >> >> have > >>>> >> >>> >> >> >> happened. > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> Thanks very much again, I very much appreciate the > >>>> >> >>> >> >> >> help! > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> - Jayme > >>>> >> >>> >> >> >> > >>>> >> >>> >> >> >> On Fri, Jan 12, 2018 at 8:44 AM, Simone Tiraboschi > >>>> >> >>> >> >> >> <stirabos@redhat.com> > >>>> >> >>> >> >> >> wrote: > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> On Fri, Jan 12, 2018 at 11:11 AM, Martin Sivak > >>>> >> >>> >> >> >>> <msivak@redhat.com> > >>>> >> >>> >> >> >>> wrote: > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Hi, > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> the hosted engine agent issue might be fixed by > >>>> >> >>> >> >> >>>> restarting > >>>> >> >>> >> >> >>>> ovirt-ha-broker or updating to newest > >>>> >> >>> >> >> >>>> ovirt-hosted-engine-ha > >>>> >> >>> >> >> >>>> and > >>>> >> >>> >> >> >>>> -setup. We improved handling of the missing > >>>> >> >>> >> >> >>>> symlink. > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> Available just in oVirt 4.2.1 RC1 > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> All the other issues seem to point to some > >>>> >> >>> >> >> >>>> storage > >>>> >> >>> >> >> >>>> problem > >>>> >> >>> >> >> >>>> I > >>>> >> >>> >> >> >>>> am > >>>> >> >>> >> >> >>>> afraid. > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> You said you started the VM, do you see it in > >>>> >> >>> >> >> >>>> virsh > >>>> >> >>> >> >> >>>> -r > >>>> >> >>> >> >> >>>> list? > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Best regards > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> Martin Sivak > >>>> >> >>> >> >> >>>> > >>>> >> >>> >> >> >>>> On Thu, Jan 11, 2018 at 10:00 PM, Jayme > >>>> >> >>> >> >> >>>> <jaymef@gmail.com> > >>>> >> >>> >> >> >>>> wrote: > >>>> >> >>> >> >> >>>> > Please help, I'm really not sure what else to > >>>> >> >>> >> >> >>>> > try > >>>> >> >>> >> >> >>>> > at > >>>> >> >>> >> >> >>>> > this > >>>> >> >>> >> >> >>>> > point. > >>>> >> >>> >> >> >>>> > Thank > >>>> >> >>> >> >> >>>> > you > >>>> >> >>> >> >> >>>> > for reading! > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > I'm still working on trying to get my hosted > >>>> >> >>> >> >> >>>> > engine > >>>> >> >>> >> >> >>>> > running > >>>> >> >>> >> >> >>>> > after a > >>>> >> >>> >> >> >>>> > botched > >>>> >> >>> >> >> >>>> > upgrade to 4.2. Storage is NFS mounted from > >>>> >> >>> >> >> >>>> > within > >>>> >> >>> >> >> >>>> > one > >>>> >> >>> >> >> >>>> > of > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > hosts. > >>>> >> >>> >> >> >>>> > Right > >>>> >> >>> >> >> >>>> > now I have 3 centos7 hosts that are fully > >>>> >> >>> >> >> >>>> > updated > >>>> >> >>> >> >> >>>> > with > >>>> >> >>> >> >> >>>> > yum > >>>> >> >>> >> >> >>>> > packages > >>>> >> >>> >> >> >>>> > from > >>>> >> >>> >> >> >>>> > ovirt 4.2, the engine was fully updated with > >>>> >> >>> >> >> >>>> > yum > >>>> >> >>> >> >> >>>> > packages > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > failed to > >>>> >> >>> >> >> >>>> > come > >>>> >> >>> >> >> >>>> > up after reboot. As of right now, everything > >>>> >> >>> >> >> >>>> > should > >>>> >> >>> >> >> >>>> > have > >>>> >> >>> >> >> >>>> > full > >>>> >> >>> >> >> >>>> > yum > >>>> >> >>> >> >> >>>> > updates > >>>> >> >>> >> >> >>>> > and all having 4.2 rpms. I have global > >>>> >> >>> >> >> >>>> > maintenance > >>>> >> >>> >> >> >>>> > mode > >>>> >> >>> >> >> >>>> > on > >>>> >> >>> >> >> >>>> > right > >>>> >> >>> >> >> >>>> > now > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > started hosted-engine on one of the three host > >>>> >> >>> >> >> >>>> > and > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > status is > >>>> >> >>> >> >> >>>> > currently: > >>>> >> >>> >> >> >>>> > Engine status : {"reason": "failed
> >>>> >> >>> >> >> >>>> > check”; > >>>> >> >>> >> >> >>>> > "health": > >>>> >> >>> >> >> >>>> > "bad", > >>>> >> >>> >> >> >>>> > "vm": > >>>> >> >>> >> >> >>>> > "up", "detail": "Up"} > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > this is what I get when trying to enter > >>>> >> >>> >> >> >>>> > hosted-vm > >>>> >> >>> >> >> >>>> > --console > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > The engine VM is running on this host > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > error: failed to get domain 'HostedEngine' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > error: Domain not found: no domain with > >>>> >> >>> >> >> >>>> > matching > >>>> >> >>> >> >> >>>> > name > >>>> >> >>> >> >> >>>> > 'HostedEngine' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Here are logs from various sources when I > >>>> >> >>> >> >> >>>> > start > >>>> >> >>> >> >> >>>> > the > >>>> >> >>> >> >> >>>> > VM on > >>>> >> >>> >> >> >>>> > HOST3: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > hosted-engine --vm-start > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Command VM.getStats with args {'vmID': > >>>> >> >>> >> >> >>>> > '4013c829-c9d7-4b72-90d5-6fe58137504c'} > >>>> >> >>> >> >> >>>> > failed: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > (code=1, message=Virtual machine does not > >>>> >> >>> >> >> >>>> > exist: > >>>> >> >>> >> >> >>>> > {'vmId': > >>>> >> >>> >> >> >>>> > u'4013c829-c9d7-4b72-90d5-6fe58137504c'}) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd-machined: > >>>> >> >>> >> >> >>>> > New > >>>> >> >>> >> >> >>>> > machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Started > >>>> >> >>> >> >> >>>> > Virtual > >>>> >> >>> >> >> >>>> > Machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 systemd: Starting > >>>> >> >>> >> >> >>>> > Virtual > >>>> >> >>> >> >> >>>> > Machine > >>>> >> >>> >> >> >>>> > qemu-110-Cultivar. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Jan 11 16:55:57 cultivar3 kvm: 3 guests now > >>>> >> >>> >> >> >>>> > active > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/common/api.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 48, > >>>> >> >>> >> >> >>>> > in > >>>> >> >>> >> >> >>>> > method > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ret = func(*args, **kwargs) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/hsm.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 2718, in > >>>> >> >>> >> >> >>>> > getStorageDomainInfo > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > dom = self.validateSdUUID(sdUUID) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/hsm.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 304, in > >>>> >> >>> >> >> >>>> > validateSdUUID > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > sdDom.validate() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/storage/fileSD.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 515, > >>>> >> >>> >> >> >>>> > in validate > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > raise > >>>> >> >>> >> >> >>>> > se.StorageDomainAccessError(self.sdUUID) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > StorageDomainAccessError: Domain is either > >>>> >> >>> >> >> >>>> > partially > >>>> >> >>> >> >> >>>> > accessible > >>>> >> >>> >> >> >>>> > or > >>>> >> >>> >> >> >>>> > entirely > >>>> >> >>> >> >> >>>> > inaccessible: > >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d 965e2f286',) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > jsonrpc/2::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:55:16,144::dispatcher::82:: storage.Dispatcher::(wrapper) > >>>> >> >>> >> >> >>>> > FINISH > >>>> >> >>> >> >> >>>> > getStorageDomainInfo error=Domain is either > >>>> >> >>> >> >> >>>> > partially > >>>> >> >>> >> >> >>>> > accessible > >>>> >> >>> >> >> >>>> > or > >>>> >> >>> >> >> >>>> > entirely > >>>> >> >>> >> >> >>>> > inaccessible: > >>>> >> >>> >> >> >>>> > (u'248f46f0-d793-4581-9810-c9d 965e2f286',) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/libvirt/qemu/Cultivar.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm > >>>> >> >>> >> >> >>>> > -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-108-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 > >>>> >> >>> >> >> >>>> > -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829 -c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-108-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon > >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:33:19,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8 e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw, if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.h osted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:33:19.699999Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:38:11.640+0000: shutting down, > >>>> >> >>> >> >> >>>> > reason=shutdown > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:39:02.122+0000: starting up > >>>> >> >>> >> >> >>>> > libvirt > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 3.2.0, > >>>> >> >>> >> >> >>>> > package: > >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem > >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>, > >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org ), > >>>> >> >>> >> >> >>>> > qemu > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), > >>>> >> >>> >> >> >>>> > hostname: > >>>> >> >>> >> >> >>>> > cultivar3 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm > >>>> >> >>> >> >> >>>> > -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-109-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 > >>>> >> >>> >> >> >>>> > -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829 -c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-109-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon > >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:39:02,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8 e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw, if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.h osted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:39:02.380773Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:53:11.407+0000: shutting down, > >>>> >> >>> >> >> >>>> > reason=shutdown > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11 20:55:57.210+0000: starting up > >>>> >> >>> >> >> >>>> > libvirt > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 3.2.0, > >>>> >> >>> >> >> >>>> > package: > >>>> >> >>> >> >> >>>> > 14.el7_4.7 (CentOS BuildSystem > >>>> >> >>> >> >> >>>> > <http://bugs.centos.org>, > >>>> >> >>> >> >> >>>> > 2018-01-04-19:31:34, c1bm.rdu2.centos.org ), > >>>> >> >>> >> >> >>>> > qemu > >>>> >> >>> >> >> >>>> > version: > >>>> >> >>> >> >> >>>> > 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1), > >>>> >> >>> >> >> >>>> > hostname: > >>>> >> >>> >> >> >>>> > cultivar3.grove.silverorange.com > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > LC_ALL=C > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > PATH=/usr/local/sbin:/usr/loca l/bin:/usr/sbin:/usr/bin > >>>> >> >>> >> >> >>>> > QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm > >>>> >> >>> >> >> >>>> > -name > >>>> >> >>> >> >> >>>> > guest=Cultivar,debug-threads=on -S -object > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > secret,id=masterKey0,format=ra w,file=/var/lib/libvirt/qemu/domain-110-Cultivar/master-key.aes > >>>> >> >>> >> >> >>>> > -machine > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > pc-i440fx-rhel7.3.0,accel=kvm, usb=off,dump-guest-core=off > >>>> >> >>> >> >> >>>> > -cpu > >>>> >> >>> >> >> >>>> > Conroe -m 8192 -realtime mlock=off -smp > >>>> >> >>> >> >> >>>> > 2,maxcpus=16,sockets=16,cores=1,threads=1 > >>>> >> >>> >> >> >>>> > -uuid > >>>> >> >>> >> >> >>>> > 4013c829-c9d7-4b72-90d5-6fe58137504c -smbios > >>>> >> >>> >> >> >>>> > 'type=1,manufacturer=oVirt,product=oVirt > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Node,version=7-4.1708.el7.cent os,serial=44454C4C-4300-1034-8035-CAC04F424331,uuid=4013c829 -c9d7-4b72-90d5-6fe58137504c' > >>>> >> >>> >> >> >>>> > -no-user-config -nodefaults -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charmonitor,path=/va r/lib/libvirt/qemu/domain-110-Cultivar/monitor.sock,server,nowait > >>>> >> >>> >> >> >>>> > -mon > >>>> >> >>> >> >> >>>> > chardev=charmonitor,id=monitor ,mode=control > >>>> >> >>> >> >> >>>> > -rtc > >>>> >> >>> >> >> >>>> > base=2018-01-11T20:55:57,driftfix=slew -global > >>>> >> >>> >> >> >>>> > kvm-pit.lost_tick_policy=delay -no-hpet > >>>> >> >>> >> >> >>>> > -no-reboot > >>>> >> >>> >> >> >>>> > -boot > >>>> >> >>> >> >> >>>> > strict=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > piix3-usb-uhci,id=usb,bus=pci. 0,addr=0x1.0x2 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-serial-pci,id=virtio-se rial0,bus=pci.0,addr=0x4 > >>>> >> >>> >> >> >>>> > -drive > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > file=/var/run/vdsm/storage/248 f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-4dfc-a421-c8 e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705,format=raw, if=none,id=drive-virtio-disk0,serial=c2dde892-f978-4dfc- a421-c8e04cf387f9,cache=none,werror=stop,rerror=stop,aio=threads > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-blk-pci,scsi=off,bus=pc i.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > >>>> >> >>> >> >> >>>> > -drive if=none,id=drive-ide0-1-0,read only=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ide-cd,bus=ide.1,unit=0,drive= drive-ide0-1-0,id=ide0-1-0 > >>>> >> >>> >> >> >>>> > -netdev > >>>> >> >>> >> >> >>>> > tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-net-pci,netdev=hostnet0 ,id=net0,mac=00:16:3e:7f:d6:83,bus=pci.0,addr=0x3 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel0,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.com.redhat.rhevm.vdsm,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel1,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.qemu.guest_agent.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 > >>>> >> >>> >> >> >>>> > -chardev spicevmc,id=charchannel2,name= vdagent > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 > >>>> >> >>> >> >> >>>> > -chardev > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > socket,id=charchannel3,path=/v ar/lib/libvirt/qemu/channels/4013c829-c9d7-4b72-90d5-6fe5813 7504c.org.ovirt.hosted-engine-setup.0,server,nowait > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtserialport,bus=virtio-seri al0.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.h osted-engine-setup.0 > >>>> >> >>> >> >> >>>> > -chardev pty,id=charconsole0 -device > >>>> >> >>> >> >> >>>> > virtconsole,chardev=charconsol e0,id=console0 > >>>> >> >>> >> >> >>>> > -spice > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > tls-port=5900,addr=0,x509-dir= /etc/pki/vdsm/libvirt-spice,tls-channel=default,seamless-migration=on > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 > >>>> >> >>> >> >> >>>> > -object > >>>> >> >>> >> >> >>>> > rng-random,id=objrng0,filename =/dev/urandom > >>>> >> >>> >> >> >>>> > -device > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > virtio-rng-pci,rng=objrng0,id= rng0,bus=pci.0,addr=0x5 > >>>> >> >>> >> >> >>>> > -msg > >>>> >> >>> >> >> >>>> > timestamp=on > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 2018-01-11T20:55:57.468037Z qemu-kvm: -chardev > >>>> >> >>> >> >> >>>> > pty,id=charconsole0: > >>>> >> >>> >> >> >>>> > char > >>>> >> >>> >> >> >>>> > device redirected to /dev/pts/2 (label > >>>> >> >>> >> >> >>>> > charconsole0) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-h a/broker.log > >>>> >> >>> >> >> >>>> > <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> >> >>>> > line 151, in get_raw_stats > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > f = os.open(path, direct_flag | > >>>> >> >>> >> >> >>>> > os.O_RDONLY | > >>>> >> >>> >> >> >>>> > os.O_SYNC) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > OSError: [Errno 2] No such file or
On Fri, Jan 12, 2018 at 6:56 PM, Jayme <jaymef@gmail.com> wrote: line line promiscuous pty,id=charconsole0: process line line line line 524, promiscuous pty,id=charconsole0: the possible line line line line line line 201, line 48, line promiscuous line line line line problem. there there the maintenance liveliness directory:
> >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d 7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > StatusStorageThread::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:55:15,761::status_broker::9 2::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker. Update::(run) > >>>> >> >>> >> >> >>>> > Failed to read state. > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/status_broker.py", > >>>> >> >>> >> >> >>>> > line 88, in run > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > self._storage_broker.get_raw_stats() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/broker/storage_broker.py", > >>>> >> >>> >> >> >>>> > line 162, in get_raw_stats > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > .format(str(e))) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > RequestError: failed to read metadata: [Errno > >>>> >> >>> >> >> >>>> > 2] > >>>> >> >>> >> >> >>>> > No > >>>> >> >>> >> >> >>>> > such > >>>> >> >>> >> >> >>>> > file or > >>>> >> >>> >> >> >>>> > directory: > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > '/var/run/vdsm/storage/248f46f 0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d 7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/ovirt-hosted-engine-h a/agent.log > >>>> >> >>> >> >> >>>> > <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > result = refresh_method() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 519, in refresh_vm_conf > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > content = > >>>> >> >>> >> >> >>>> > self._get_file_content_from_sh ared_storage(VM) > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 484, in > >>>> >> >>> >> >> >>>> > _get_file_content_from_shared_storage > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > config_volume_path = > >>>> >> >>> >> >> >>>> > self._get_config_volume_path() > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/env/config.py", > >>>> >> >>> >> >> >>>> > line 188, in _get_config_volume_path > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > conf_vol_uuid > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/ovirt_hosted_engine_ha/lib/heconflib.py", > >>>> >> >>> >> >> >>>> > line 358, in get_volume_path > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > root=envconst.SD_RUN_DIR, > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > RuntimeError: Path to volume > >>>> >> >>> >> >> >>>> > 4838749f-216d-406b-b245-98d0343fcf7f > >>>> >> >>> >> >> >>>> > not > >>>> >> >>> >> >> >>>> > found > >>>> >> >>> >> >> >>>> > in /run/vdsm/storag > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ==> /var/log/vdsm/vdsm.log <== > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > periodic/42::ERROR::2018-01-11 > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > 16:56:11,446::vmstats::260::vi rt.vmstats::(send_metrics) > >>>> >> >>> >> >> >>>> > VM > >>>> >> >>> >> >> >>>> > metrics > >>>> >> >>> >> >> >>>> > collection failed > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > Traceback (most recent call last): > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > File > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > "/usr/lib/python2.7/site-packa ges/vdsm/virt/vmstats.py", > >>>> >> >>> >> >> >>>> > line > >>>> >> >>> >> >> >>>> > 197, in > >>>> >> >>> >> >> >>>> > send_metrics > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > data[prefix + '.cpu.usage'] = > >>>> >> >>> >> >> >>>> > stat['cpuUsage'] > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > KeyError: 'cpuUsage' > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> > ______________________________
> >>>> >> >>> >> >> >>>> > Users mailing list > >>>> >> >>> >> >> >>>> > Users@ovirt.org > >>>> >> >>> >> >> >>>> > http://lists.ovirt.org/mailman /listinfo/users > >>>> >> >>> >> >> >>>> > > >>>> >> >>> >> >> >>>> ______________________________
> >>>> >> >>> >> >> >>>> Users mailing list > >>>> >> >>> >> >> >>>> Users@ovirt.org > >>>> >> >>> >> >> >>>> http://lists.ovirt.org/mailman /listinfo/users > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >>> > >>>> >> >>> >> >> >> > >>>> >> >>> >> > > >>>> >> >>> >> > > >>>> >> >>> > > >>>> >> >>> > > >>>> >> >> > >>>> >> >> > >>>> >> > > >>>> > > >>>> > > >>> > >>> > >> > >
participants (1)
-
Jayme