<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 13, 2016 at 1:56 PM, Paolo Bonzini <span dir="ltr"><<a href="mailto:pbonzini@redhat.com" target="_blank">pbonzini@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
<br>
On 13/12/2016 12:38, Gianluca Cecchi wrote:<br>
> flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat<br>
> pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb<br>
> rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology<br>
> nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx<br>
> est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt<br>
> tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch<br>
> ida arat epb pln pts dtherm hwp hwp_noitfy hwp_act_window hwp_epp<br>
> tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep<br>
> bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1<br>
> xsaves<br>
> bogomips: 3600.06<br>
> clflush size: 64<br>
> cache_alignment: 64<br>
> address sizes: 39 bits physical, 48 bits virtual<br>
> power management:<br>
><br>
> . . .<br>
><br>
> What is the flag to check?<br>
<br>
</span>It's erms, which is there. But it's not the culprit.<br>
<br>
Sorry if you have already tested it, but have you tried using 7.2 kernel<br>
with QEMU 2.6, and then 7.3 kernel with QEMU 2.3? That would allow<br>
finding the guilty component more easily.<br>
<br>
Thanks,<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br></div></div></blockquote><div><br></div><div>No problem.</div><div><br></div><div>- 7.3 kernel with qemu 2.3 seems ok</div><div>It is the configuration used for deploying self hosted engine vm</div><div><br></div><div>[root@ovirt41 ~]# rpm -q qemu-kvm-ev</div><div>qemu-kvm-ev-2.3.0-31.el7_2.21.1.x86_64</div><div><br></div><div>[root@ovirt41 ~]# uname -r</div><div>3.10.0-514.el7.x86_64</div><div>[root@ovirt41 ~]#</div><div>( it seems it is the 7.3 kernel, based on <a href="https://access.redhat.com/articles/3078">https://access.redhat.com/articles/3078</a> )</div><div><br></div><div><div>[root@ovirt41 ~]# ps -ef| grep qemu-kvm</div><div>qemu 53257 1 3 Dec07 ? 05:56:52 /usr/libexec/qemu-kvm -name guest=HostedEngine,debug-threads=on -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Broadwell,+rtm,+hle -m 6184 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 2a262cdc-9102-4061-841f-ec64333cdad2 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-2.1511.el7.centos.2.10,serial=564D3726-E55D-5C11-DC45-CA1A50480E83,uuid=2a262cdc-9102-4061-841f-ec64333cdad2 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-HostedEngine/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2016-12-07T10:16:42,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/run/vdsm/storage/3e7d4336-c2e1-4fdc-99e7-81a0e69cf3a3/286a8fda-b77d-48b8-80a9-15b63e5321a2/63bfeca6-dc92-4145-845d-e785a18de949,format=raw,if=none,id=drive-virtio-disk0,serial=286a8fda-b77d-48b8-80a9-15b63e5321a2,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:08:cc:5a,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev socket,id=charchannel2,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.org.ovirt.hosted-engine-setup.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.ovirt.hosted-engine-setup.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -vnc 0:0,password -device VGA,id=video0,vgamem_mb=32,bus=pci.0,addr=0x2 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on</div></div><div><br></div><div><br></div><div>the VM is ok, I can ssh to it:</div><div><br></div><div><div>[root@ovirt41 ~]# ssh ovirt41she.localdomain.local</div><div>The authenticity of host 'ovirt41she.localdomain.local (192.168.150.122)' can't be established.</div><div>ECDSA key fingerprint is 24:fc:fa:07:14:4e:b3:ea:3e:9b:bc:8a:6a:3e:a7:76.</div><div>Are you sure you want to continue connecting (yes/no)? yes</div><div>Warning: Permanently added 'ovirt41she.localdomain.local,192.168.150.122' (ECDSA) to the list of known hosts.</div><div>root@ovirt41she.localdomain.local's password:</div><div>[root@ovirt41she ~]#</div></div><div><br></div><div><br></div><div>- 7.2 kernel with qemu-2.6</div><div><br></div><div>it seems it works too... see below</div><div><br></div><div>for qemu 2.6, I don't yet find in my repos:</div><div><div>[root@ovirt41 ~]# yum update qemu-kvm-ev</div><div>Loaded plugins: fastestmirror, langpacks</div><div>Loading mirror speeds from cached hostfile</div><div> * base: <a href="http://repo.de.bigstepcloud.com">repo.de.bigstepcloud.com</a></div><div> * epel: <a href="http://ftp.nluug.nl">ftp.nluug.nl</a></div><div> * extras: <a href="http://mirror.crazynetwork.it">mirror.crazynetwork.it</a></div><div> * ovirt-4.1: <a href="http://ftp.nluug.nl">ftp.nluug.nl</a></div><div> * ovirt-4.1-epel: <a href="http://ftp.nluug.nl">ftp.nluug.nl</a></div><div>No packages marked for update</div></div><div><br></div><div>What I tested, based on Sandro initial mail in the thread was this that I reproduce now:</div><div><br></div><div><div>- put oVirt in maintenance and shutdown self hosted engine VM</div><div>[root@ovirt41 ~]# hosted-engine --vm-status</div><div><br></div><div><br></div><div>--== Host 1 status ==--</div><div><br></div><div>Status up-to-date : True</div><div>Hostname : ovirt41.localdomain.local</div><div>Host ID : 1</div><div>Engine status : {"health": "good", "vm": "up", "detail": "up"}</div><div>Score : 3400</div><div>stopped : False</div><div>Local maintenance : False</div><div>crc32 : 8d9d58c2</div><div>Host timestamp : 609397</div><div>Extra metadata (valid at timestamp):</div><div> metadata_parse_version=1</div><div> metadata_feature_version=1</div><div> timestamp=609397 (Tue Dec 13 17:53:41 2016)</div><div> host-id=1</div><div> score=3400</div><div> maintenance=False</div><div> state=EngineUp</div><div> stopped=False</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div>[root@ovirt41 ~]# hosted-engine --set-maintenance --mode=global</div></div><div><br></div><div>ssh to the VM and shutdown</div><div><br></div><div><div>[root@ovirt41she ~]# shutdown -h now</div></div><div><br></div><div><br class="gmail-Apple-interchange-newline">coming back to host:<br></div><div><div>[root@ovirt41 ~]# ps -ef|grep qemu</div><div>root 101759 1459 0 17:56 pts/0 00:00:00 grep --color=auto qemu</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div>[root@ovirt41 ~]# hosted-engine --vm-status</div><div><br></div><div><br></div><div>!! Cluster is in GLOBAL MAINTENANCE mode !!</div><div><br></div><div><br></div><div><br></div><div>--== Host 1 status ==--</div><div><br></div><div>Status up-to-date : True</div><div>Hostname : ovirt41.localdomain.local</div><div>Host ID : 1</div><div>Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"}</div><div>Score : 3000</div><div>stopped : False</div><div>Local maintenance : False</div><div>crc32 : a246ec87</div><div>Host timestamp : 609544</div><div>Extra metadata (valid at timestamp):</div><div> metadata_parse_version=1</div><div> metadata_feature_version=1</div><div> timestamp=609544 (Tue Dec 13 17:56:09 2016)</div><div> host-id=1</div><div> score=3000</div><div> maintenance=False</div><div> state=GlobalMaintenance</div><div> stopped=False</div><div><br></div><div><br></div><div>!! Cluster is in GLOBAL MAINTENANCE mode !!</div><div><br></div><div>[root@ovirt41 ~]#</div></div><div><br></div><div>- install qemu-kvm-ev 2.6</div><div><br></div><div><div>[root@ovirt41 ~]# yum update <a href="http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-ev-2.6.0-27.1.el7.x86_64.rpm">http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-ev-2.6.0-27.1.el7.x86_64.rpm</a> <a href="http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-img-ev-2.6.0-27.1.el7.x86_64.rpm">http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-img-ev-2.6.0-27.1.el7.x86_64.rpm</a> <a href="http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-common-ev-2.6.0-27.1.el7.x86_64.rpm">http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-common-ev-2.6.0-27.1.el7.x86_64.rpm</a> <a href="http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64.rpm">http://buildlogs.centos.org/centos/7/virt/x86_64/kvm-common/qemu-kvm-tools-ev-2.6.0-27.1.el7.x86_64.rpm</a></div></div><div><br></div><div>- set kernel 3.10.0-327.36.3.el7 as the default boot one</div><div><br></div><div><div>* Mon Oct 24 2016 CentOS Sources <<a href="mailto:bugs@centos.org">bugs@centos.org</a>> - 3.10.0-327.36.3.el7</div><div>- Apply debranding changes</div></div><div><br></div><div><div>[root@ovirt41 ~]# grub2-editenv list</div><div>saved_entry=CentOS Linux (3.10.0-514.el7.x86_64) 7 (Core)</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div>[root@ovirt41 ~]# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg</div><div>CentOS Linux (3.10.0-514.el7.x86_64) 7 (Core)</div><div>CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)</div><div>CentOS Linux (3.10.0-327.36.1.el7.x86_64) 7 (Core)</div><div>CentOS Linux (3.10.0-327.22.2.el7.x86_64) 7 (Core)</div><div>CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core)</div><div>CentOS Linux (0-rescue-65496e25d5a842b58090b6a9f4246e68) 7 (Core)</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div>[root@ovirt41 ~]# grub2-set-default 'CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)'</div><div>[root@ovirt41 ~]# grub2-editenv list</div><div>saved_entry=CentOS Linux (3.10.0-327.36.3.el7.x86_64) 7 (Core)</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div>[root@ovirt41 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg</div><div>Generating grub configuration file ...</div><div>Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64</div><div>Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img</div><div>Found linux image: /boot/vmlinuz-3.10.0-327.36.3.el7.x86_64</div><div>Found initrd image: /boot/initramfs-3.10.0-327.36.3.el7.x86_64.img</div><div>Found linux image: /boot/vmlinuz-3.10.0-327.36.1.el7.x86_64</div><div>Found initrd image: /boot/initramfs-3.10.0-327.36.1.el7.x86_64.img</div><div>Found linux image: /boot/vmlinuz-3.10.0-327.22.2.el7.x86_64</div><div>Found initrd image: /boot/initramfs-3.10.0-327.22.2.el7.x86_64.img</div><div>Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64</div><div>Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img</div><div>Found linux image: /boot/vmlinuz-0-rescue-65496e25d5a842b58090b6a9f4246e68</div><div>Found initrd image: /boot/initramfs-0-rescue-65496e25d5a842b58090b6a9f4246e68.img</div><div>done</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div><div><br></div><div>[root@ovirt41 ~]# hosted-engine --set-maintenance --mode=local</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div>- reboot host</div><div><br></div><div>- after reboot</div><div><br></div><div><div>[root@ovirt41 ~]# uname -r</div><div>3.10.0-327.36.3.el7.x86_64</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div>exit from maintenance and see if the hosted engine vm starts</div><div><br></div><div><div>[root@ovirt41 ~]# hosted-engine --set-maintenance --mode=none</div><div>[root@ovirt41 ~]#</div></div><div><br></div><div>It seems the VM starts....</div><div><br></div><div><div>[root@ovirt41 qemu]# ps -ef|grep qemu</div><div>qemu 3485 1 59 18:21 ? 00:00:41 /usr/libexec/qemu-kvm -name guest=HostedEngine,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-HostedEngine/master-key.aes -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Broadwell,+rtm,+hle -m 6184 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 2a262cdc-9102-4061-841f-ec64333cdad2 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-2.1511.el7.centos.2.10,serial=564D3726-E55D-5C11-DC45-CA1A50480E83,uuid=2a262cdc-9102-4061-841f-ec64333cdad2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-HostedEngine/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2016-12-13T17:21:33,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/run/vdsm/storage/3e7d4336-c2e1-4fdc-99e7-81a0e69cf3a3/286a8fda-b77d-48b8-80a9-15b63e5321a2/63bfeca6-dc92-4145-845d-e785a18de949,format=raw,if=none,id=drive-virtio-disk0,serial=286a8fda-b77d-48b8-80a9-15b63e5321a2,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=30,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:08:cc:5a,bus=pci.0,addr=0x2 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev socket,id=charchannel2,path=/var/lib/libvirt/qemu/channels/2a262cdc-9102-4061-841f-ec64333cdad2.org.ovirt.hosted-engine-setup.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.ovirt.hosted-engine-setup.0 -msg timestamp=on</div><div>root 3845 1943 0 18:22 pts/0 00:00:00 grep --color=auto qemu</div></div><div><br></div><div>and I'm able to connect to it via ssh too</div><div><br></div><div><div>[root@ovirt41 qemu]# ssh ovirt41she.localdomain.local</div><div>root@ovirt41she.localdomain.local's password:</div><div>Last login: Tue Dec 13 17:55:37 2016 from ovirt41.localdomain.local</div><div>[root@ovirt41she ~]#</div></div><div><br></div><div>- So I have to try the mix of 7.3 kernel and qemu 2.6, correct?</div><div><br></div><div>Perhaps it was a problem only during install and not happening now that the VM has been deployed?</div><div>Gianluca</div></div></div></div>