<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Jan 12, 2018 11:43 AM, "Jayme" <<a href="mailto:jaymef@gmail.com">jaymef@gmail.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Please help, I'm really not sure what else to try at this point. Thank you for reading!</p><p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><br></p><p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">I'm still working on trying to get my hosted engine running after a botched upgrade to 4.2. Storage is NFS mounted from within one of the hosts.<span class="m_2105291413396759553gmail-Apple-converted-space"> </span>Right now I have 3 centos7 hosts that are fully </p></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">That's not a good idea. What happens when the host fails? </div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">updated with yum packages from ovirt 4.2, the engine was fully updated with yum packages and failed to come up after reboot. As of right now, everything should have full yum updates and all having 4.2 rpms.<span class="m_2105291413396759553gmail-Apple-converted-space"> </span>I have global maintenance mode on right now and started hosted-engine on one of the three host and the status is currently: Engine status : {"reason": "failed liveliness check”; "health": "bad", "vm": "up", "detail": "Up"}</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">this is what I get when trying to enter hosted-vm --console</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">The engine VM is running on this host</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">error: failed to get domain 'HostedEngine'</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">error: Domain not found: no domain with matching name 'HostedEngine'</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Here are logs from various sources when I start the VM on HOST3:</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">hosted-engine --vm-start</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Command VM.getStats with args {'vmID': '4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c'} failed:</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">(code=1, message=Virtual machine does not exist: {'vmId': u'4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c'})</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Jan 11 16:55:57 cultivar3 systemd-machined: New machine qemu-110-Cultivar.</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Jan 11 16:55:57 cultivar3 systemd: Started Virtual Machine qemu-110-Cultivar.</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Jan 11 16:55:57 cultivar3 systemd: Starting Virtual Machine qemu-110-Cultivar.</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Jan 11 16:55:57 cultivar3 kvm: 3 guests now active</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">==> /var/log/vdsm/vdsm.log <==</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/common/api.py", line 48, in method</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>ret = func(*args, **kwargs)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py", line 2718, in getStorageDomainInfo</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>dom = self.validateSdUUID(sdUUID)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/hsm.py", line 304, in validateSdUUID</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>sdDom.validate()</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/fileSD.<wbr>py", line 515, in validate</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>raise se.StorageDomainAccessError(<wbr>self.sdUUID)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">StorageDomainAccessError: Domain is either partially accessible or entirely inaccessible: (u'248f46f0-d793-4581-9810-<wbr>c9d965e2f286',)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">jsonrpc/2::ERROR::2018-01-11 16:55:16,144::dispatcher::82::<wbr>storage.Dispatcher::(wrapper) FINISH getStorageDomainInfo error=Domain is either partially accessible or entirely inaccessible: (u'248f46f0-d793-4581-9810-<wbr>c9d965e2f286',)</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">==> /var/log/libvirt/qemu/<wbr>Cultivar.log <==</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">LC_ALL=C PATH=/usr/local/sbin:/usr/<wbr>local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=Cultivar,debug-threads=<wbr>on -S -object secret,id=masterKey0,format=<wbr>raw,file=/var/lib/libvirt/<wbr>qemu/domain-108-Cultivar/<wbr>master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,<wbr>usb=off,dump-guest-core=off -cpu Conroe -m 8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=<wbr>1,threads=1 -uuid 4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c -smbios 'type=1,manufacturer=oVirt,<wbr>product=oVirt Node,version=7-4.1708.el7.<wbr>centos,serial=44454C4C-4300-<wbr>1034-8035-CAC04F424331,uuid=<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/<wbr>var/lib/libvirt/qemu/domain-<wbr>108-Cultivar/monitor.sock,<wbr>server,nowait -mon chardev=charmonitor,id=<wbr>monitor,mode=control -rtc base=2018-01-11T20:33:19,<wbr>driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.<wbr>0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-<wbr>serial0,bus=pci.0,addr=0x4 -drive file=/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/c2dde892-f978-<wbr>4dfc-a421-c8e04cf387f9/<wbr>23aa0a66-fa6c-4967-a1e5-<wbr>fbe47c0cd705,format=raw,if=<wbr>none,id=drive-virtio-disk0,<wbr>serial=c2dde892-f978-4dfc-<wbr>a421-c8e04cf387f9,cache=none,<wbr>werror=stop,rerror=stop,aio=<wbr>threads -device virtio-blk-pci,scsi=off,bus=<wbr>pci.0,addr=0x6,drive=drive-<wbr>virtio-disk0,id=virtio-disk0,<wbr>bootindex=1 -drive if=none,id=drive-ide0-1-0,<wbr>readonly=on -device ide-cd,bus=ide.1,unit=0,drive=<wbr>drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=30,id=hostnet0,vhost=<wbr>on,vhostfd=32 -device virtio-net-pci,netdev=<wbr>hostnet0,id=net0,mac=00:16:3e:<wbr>7f:d6:83,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.com.redhat.rhevm.<wbr>vdsm,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=1,chardev=<wbr>charchannel0,id=channel0,name=<wbr>com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.qemu.guest_<wbr>agent.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=2,chardev=<wbr>charchannel1,id=channel1,name=<wbr>org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=<wbr>vdagent -device virtserialport,bus=virtio-<wbr>serial0.0,nr=3,chardev=<wbr>charchannel2,id=channel2,name=<wbr>com.redhat.spice.0 -chardev socket,id=charchannel3,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.ovirt.hosted-<wbr>engine-setup.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=4,chardev=<wbr>charchannel3,id=channel3,name=<wbr>org.ovirt.hosted-engine-setup.<wbr>0 -chardev pty,id=charconsole0 -device virtconsole,chardev=<wbr>charconsole0,id=console0 -spice tls-port=5900,addr=0,x509-dir=<wbr>/etc/pki/vdsm/libvirt-spice,<wbr>tls-channel=default,seamless-<wbr>migration=on -device cirrus-vga,id=video0,bus=pci.<wbr>0,addr=0x2 -object rng-random,id=objrng0,<wbr>filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=<wbr>rng0,bus=pci.0,addr=0x5 -msg timestamp=on</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11T20:33:19.699999Z qemu-kvm: -chardev pty,id=charconsole0: char device redirected to /dev/pts/2 (label charconsole0)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11 20:38:11.640+0000: shutting down, reason=shutdown</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11 20:39:02.122+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.7 (CentOS BuildSystem <<a href="http://bugs.centos.org" target="_blank">http://bugs.centos.org</a>>, 2018-01-04-19:31:34, <a href="http://c1bm.rdu2.centos.org" target="_blank">c1bm.rdu2.centos.org</a>), qemu version: 2.9.0(qemu-kvm-ev-2.9.0-16.<wbr>el7_4.13.1), hostname: cultivar3</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">LC_ALL=C PATH=/usr/local/sbin:/usr/<wbr>local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=Cultivar,debug-threads=<wbr>on -S -object secret,id=masterKey0,format=<wbr>raw,file=/var/lib/libvirt/<wbr>qemu/domain-109-Cultivar/<wbr>master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,<wbr>usb=off,dump-guest-core=off -cpu Conroe -m 8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=<wbr>1,threads=1 -uuid 4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c -smbios 'type=1,manufacturer=oVirt,<wbr>product=oVirt Node,version=7-4.1708.el7.<wbr>centos,serial=44454C4C-4300-<wbr>1034-8035-CAC04F424331,uuid=<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/<wbr>var/lib/libvirt/qemu/domain-<wbr>109-Cultivar/monitor.sock,<wbr>server,nowait -mon chardev=charmonitor,id=<wbr>monitor,mode=control -rtc base=2018-01-11T20:39:02,<wbr>driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.<wbr>0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-<wbr>serial0,bus=pci.0,addr=0x4 -drive file=/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/c2dde892-f978-<wbr>4dfc-a421-c8e04cf387f9/<wbr>23aa0a66-fa6c-4967-a1e5-<wbr>fbe47c0cd705,format=raw,if=<wbr>none,id=drive-virtio-disk0,<wbr>serial=c2dde892-f978-4dfc-<wbr>a421-c8e04cf387f9,cache=none,<wbr>werror=stop,rerror=stop,aio=<wbr>threads -device virtio-blk-pci,scsi=off,bus=<wbr>pci.0,addr=0x6,drive=drive-<wbr>virtio-disk0,id=virtio-disk0,<wbr>bootindex=1 -drive if=none,id=drive-ide0-1-0,<wbr>readonly=on -device ide-cd,bus=ide.1,unit=0,drive=<wbr>drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=30,id=hostnet0,vhost=<wbr>on,vhostfd=32 -device virtio-net-pci,netdev=<wbr>hostnet0,id=net0,mac=00:16:3e:<wbr>7f:d6:83,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.com.redhat.rhevm.<wbr>vdsm,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=1,chardev=<wbr>charchannel0,id=channel0,name=<wbr>com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.qemu.guest_<wbr>agent.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=2,chardev=<wbr>charchannel1,id=channel1,name=<wbr>org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=<wbr>vdagent -device virtserialport,bus=virtio-<wbr>serial0.0,nr=3,chardev=<wbr>charchannel2,id=channel2,name=<wbr>com.redhat.spice.0 -chardev socket,id=charchannel3,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.ovirt.hosted-<wbr>engine-setup.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=4,chardev=<wbr>charchannel3,id=channel3,name=<wbr>org.ovirt.hosted-engine-setup.<wbr>0 -chardev pty,id=charconsole0 -device virtconsole,chardev=<wbr>charconsole0,id=console0 -spice tls-port=5900,addr=0,x509-dir=<wbr>/etc/pki/vdsm/libvirt-spice,<wbr>tls-channel=default,seamless-<wbr>migration=on -device cirrus-vga,id=video0,bus=pci.<wbr>0,addr=0x2 -object rng-random,id=objrng0,<wbr>filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=<wbr>rng0,bus=pci.0,addr=0x5 -msg timestamp=on</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11T20:39:02.380773Z qemu-kvm: -chardev pty,id=charconsole0: char device redirected to /dev/pts/2 (label charconsole0)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11 20:53:11.407+0000: shutting down, reason=shutdown</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11 20:55:57.210+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.7 (CentOS BuildSystem <<a href="http://bugs.centos.org" target="_blank">http://bugs.centos.org</a>>, 2018-01-04-19:31:34, <a href="http://c1bm.rdu2.centos.org" target="_blank">c1bm.rdu2.centos.org</a>), qemu version: 2.9.0(qemu-kvm-ev-2.9.0-16.<wbr>el7_4.13.1), hostname: <a href="http://cultivar3.grove.silverorange.com" target="_blank">cultivar3.grove.silverorange.<wbr>com</a></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">LC_ALL=C PATH=/usr/local/sbin:/usr/<wbr>local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=Cultivar,debug-threads=<wbr>on -S -object secret,id=masterKey0,format=<wbr>raw,file=/var/lib/libvirt/<wbr>qemu/domain-110-Cultivar/<wbr>master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,<wbr>usb=off,dump-guest-core=off -cpu Conroe -m 8192 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=<wbr>1,threads=1 -uuid 4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c -smbios 'type=1,manufacturer=oVirt,<wbr>product=oVirt Node,version=7-4.1708.el7.<wbr>centos,serial=44454C4C-4300-<wbr>1034-8035-CAC04F424331,uuid=<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/<wbr>var/lib/libvirt/qemu/domain-<wbr>110-Cultivar/monitor.sock,<wbr>server,nowait -mon chardev=charmonitor,id=<wbr>monitor,mode=control -rtc base=2018-01-11T20:55:57,<wbr>driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.<wbr>0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-<wbr>serial0,bus=pci.0,addr=0x4 -drive file=/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/c2dde892-f978-<wbr>4dfc-a421-c8e04cf387f9/<wbr>23aa0a66-fa6c-4967-a1e5-<wbr>fbe47c0cd705,format=raw,if=<wbr>none,id=drive-virtio-disk0,<wbr>serial=c2dde892-f978-4dfc-<wbr>a421-c8e04cf387f9,cache=none,<wbr>werror=stop,rerror=stop,aio=<wbr>threads -device virtio-blk-pci,scsi=off,bus=<wbr>pci.0,addr=0x6,drive=drive-<wbr>virtio-disk0,id=virtio-disk0,<wbr>bootindex=1 -drive if=none,id=drive-ide0-1-0,<wbr>readonly=on -device ide-cd,bus=ide.1,unit=0,drive=<wbr>drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=30,id=hostnet0,vhost=<wbr>on,vhostfd=32 -device virtio-net-pci,netdev=<wbr>hostnet0,id=net0,mac=00:16:3e:<wbr>7f:d6:83,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.com.redhat.rhevm.<wbr>vdsm,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=1,chardev=<wbr>charchannel0,id=channel0,name=<wbr>com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.qemu.guest_<wbr>agent.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=2,chardev=<wbr>charchannel1,id=channel1,name=<wbr>org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=<wbr>vdagent -device virtserialport,bus=virtio-<wbr>serial0.0,nr=3,chardev=<wbr>charchannel2,id=channel2,name=<wbr>com.redhat.spice.0 -chardev socket,id=charchannel3,path=/<wbr>var/lib/libvirt/qemu/channels/<wbr>4013c829-c9d7-4b72-90d5-<wbr>6fe58137504c.org.ovirt.hosted-<wbr>engine-setup.0,server,nowait -device virtserialport,bus=virtio-<wbr>serial0.0,nr=4,chardev=<wbr>charchannel3,id=channel3,name=<wbr>org.ovirt.hosted-engine-setup.<wbr>0 -chardev pty,id=charconsole0 -device virtconsole,chardev=<wbr>charconsole0,id=console0 -spice tls-port=5900,addr=0,x509-dir=<wbr>/etc/pki/vdsm/libvirt-spice,<wbr>tls-channel=default,seamless-<wbr>migration=on -device cirrus-vga,id=video0,bus=pci.<wbr>0,addr=0x2 -object rng-random,id=objrng0,<wbr>filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=<wbr>rng0,bus=pci.0,addr=0x5 -msg timestamp=on</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">2018-01-11T20:55:57.468037Z qemu-kvm: -chardev pty,id=charconsole0: char device redirected to /dev/pts/2 (label charconsole0)</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">==> /var/log/ovirt-hosted-engine-<wbr>ha/broker.log <==</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/storage_broker.py", line 151, in get_raw_stats</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8'</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">StatusStorageThread::ERROR::<wbr>2018-01-11 16:55:15,761::status_broker::<wbr>92::ovirt_hosted_engine_ha.<wbr>broker.status_broker.<wbr>StatusBroker.Update::(run) Failed to read state.</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Traceback (most recent call last):</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/status_broker.py", line 88, in run</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>self._storage_broker.get_raw_<wbr>stats()</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/broker/storage_broker.py", line 162, in get_raw_stats</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>.format(str(e)))</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">RequestError: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/<wbr>248f46f0-d793-4581-9810-<wbr>c9d965e2f286/14a20941-1b84-<wbr>4b82-be8f-ace38d7c037a/<wbr>8582bdfc-ef54-47af-9f1e-<wbr>f5b7ec1f1cf8'</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">==> /var/log/ovirt-hosted-engine-<wbr>ha/agent.log <==</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>result = refresh_method()</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/env/config.py", line 519, in refresh_vm_conf</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>content = self._get_file_content_from_<wbr>shared_storage(VM)</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/env/config.py", line 484, in _get_file_content_from_shared_<wbr>storage</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>config_volume_path = self._get_config_volume_path()</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/env/config.py", line 188, in _get_config_volume_path</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>conf_vol_uuid</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/lib/heconflib.py", line 358, in get_volume_path</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>root=envconst.SD_RUN_DIR,</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">RuntimeError: Path to volume 4838749f-216d-406b-b245-<wbr>98d0343fcf7f not found in /run/vdsm/storag</p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p2" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69);min-height:14px"><br></p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">==> /var/log/vdsm/vdsm.log <==</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">periodic/42::ERROR::2018-01-11 16:56:11,446::vmstats::260::<wbr>virt.vmstats::(send_metrics) VM metrics collection failed</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">Traceback (most recent call last):</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>File "/usr/lib/python2.7/site-<wbr>packages/vdsm/virt/vmstats.py"<wbr>, line 197, in send_metrics</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)"><span class="m_2105291413396759553gmail-Apple-converted-space"> </span>data[prefix + '.cpu.usage'] = stat['cpuUsage']</p>
<p class="m_2105291413396759553gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";color:rgb(69,69,69)">KeyError: 'cpuUsage'</p></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div></div>