oVirt 4.4 HE on Copy local VM disk to shared storage (NFS) failing
Deploy via cockpit fails after copy local VM disk to shared storage, Exit HE maintenance mode, Check engine VM health. The HE VM does not start from shared storage with hosted-engine --vm-status returning down_unexpected. I've also tried with iSCSI, with similar results. I'm using CentOS 8.1 now, but also tried with Node 4.4 with same results. VDSM log attached. If any other logs are needed, please let me know and I'll add. I've used the same NFS server to deploy 4.3 without issue, and I ran the nfs-check.py script to verify that the NFS share was correctly provisioned. [ INFO ] TASK [ovirt.hosted_engine_setup : Copy local VM disk to shared storage] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Start ovirt-ha-broker service on the host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Initialize lockspace volume] [ INFO ] TASK [ovirt.hosted_engine_setup : Workaround for ovirt-ha-broker start failures] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Initialize lockspace volume] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Start ovirt-ha-agent service on the host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Exit HE maintenance mode] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Check engine VM health] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 180, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.186082", "end": "2020-05-26 20:05:40.063450", "rc": 0, "start": "2020-05-26 20:05:39.877368", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"host-id\": 1, \"host-ts\": 2500, \"score\": 0, \"engine-status\": {\"vm\": \"down_unexpected\", \"health\": \"bad\", \"detail\": \"Down\", \"reason\": \"bad vm status\"}, \"hostname\": \"ovirthost02.po-lite.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"3b77b8b8\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 2500, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2500 (Tue May 26 20:05:36 2020)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2500 (Tue May 26 20:05:36 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Wed Dec 31 19:47:59 1969\\n\", \"live-data\": true}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"host-id\": 1, \"host-ts\": 2500, \"score\": 0, \"engine-status\": {\"vm\": \"down_unexpected\", \"health\": \"bad\", \"detail\": \"Down\", \"reason\": \"bad vm status\"}, \"hostname\": \"ovirthost02.po-lite.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"3b77b8b8\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 2500, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=2500 (Tue May 26 20:05:36 2020)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=2500 (Tue May 26 20:05:36 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Wed Dec 31 19:47:59 1969\\n\", \"live-data\": true}, \"global_maintenance\": false}"]} [ INFO ] TASK [ovirt.hosted_engine_setup : Check VM status at virt level] [ INFO ] TASK [ovirt.hosted_engine_setup : Fail if engine VM is not running] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"}
I think the problem with a storage connection. Verify your IP addresses on the storage adapter port. Are they connected? After install my ports for storage was deactivated by defaut. Try to connect your iSCSI manually with iscsiadm command and then check your storage connection in storage tab using a host admin portal.
From your vdsm log, it seems as occurrence of an issue discussed at https://lists.ovirt.org/archives/list/users@ovirt.org/message/JE76KK2WFDCDJL... 2020-05-26 19:51:53,837-0400 ERROR (vm/90f09a7c) [virt.vm] (vmId='90f09a7c-3af1-45a9-a210-78a9b0cd4c3d') The vm start process failed (vm:871) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 801, in _startUnderlyingVm self._run() File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2608, in _run dom.createWithFlags(flags) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1166, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirt.libvirtError: unsupported configuration: unknown CPU feature: tsx-ctrl On Wed, May 27, 2020 at 9:15 AM Patrick Lomakin <patrick.lomakin@gmail.com> wrote:
I think the problem with a storage connection. Verify your IP addresses on the storage adapter port. Are they connected? After install my ports for storage was deactivated by defaut. Try to connect your iSCSI manually with iscsiadm command and then check your storage connection in storage tab using a host admin portal. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KNQTADDDEIK2V3...
On Wed, May 27, 2020 at 10:21 AM Amit Bawer <abawer@redhat.com> wrote:
From your vdsm log, it seems as occurrence of an issue discussed at https://lists.ovirt.org/archives/list/users@ovirt.org/message/JE76KK2WFDCDJL...
2020-05-26 19:51:53,837-0400 ERROR (vm/90f09a7c) [virt.vm] (vmId='90f09a7c-3af1-45a9-a210-78a9b0cd4c3d') The vm start process failed (vm:871) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 801, in _startUnderlyingVm self._run() File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2608, in _run dom.createWithFlags(flags) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1166, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirt.libvirtError: unsupported configuration: unknown CPU feature: tsx-ctrl
On Wed, May 27, 2020 at 9:15 AM Patrick Lomakin <patrick.lomakin@gmail.com> wrote:
I think the problem with a storage connection. Verify your IP addresses on the storage adapter port. Are they connected? After install my ports for storage was deactivated by defaut. Try to connect your iSCSI manually with iscsiadm command and then check your storage connection in storage tab using a host admin portal.
Yes, in that thread I asked if possible to specify in some way an alternate cpu type and/or flags to be setup for self hosted engine vm, without setup to automatically detect/configure it, but I didn't receive an answer. Because the cpu flag is there also in 4.3.9 but doesn't create problems with self hosted engine or host. Gianluca
On Wed, May 27, 2020 at 11:26 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Wed, May 27, 2020 at 10:21 AM Amit Bawer <abawer@redhat.com> wrote:
From your vdsm log, it seems as occurrence of an issue discussed at https://lists.ovirt.org/archives/list/users@ovirt.org/message/JE76KK2WFDCDJL...
2020-05-26 19:51:53,837-0400 ERROR (vm/90f09a7c) [virt.vm] (vmId='90f09a7c-3af1-45a9-a210-78a9b0cd4c3d') The vm start process failed (vm:871) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 801, in _startUnderlyingVm self._run() File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2608, in _run dom.createWithFlags(flags) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1166, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirt.libvirtError: unsupported configuration: unknown CPU feature: tsx-ctrl
On Wed, May 27, 2020 at 9:15 AM Patrick Lomakin < patrick.lomakin@gmail.com> wrote:
I think the problem with a storage connection. Verify your IP addresses on the storage adapter port. Are they connected? After install my ports for storage was deactivated by defaut. Try to connect your iSCSI manually with iscsiadm command and then check your storage connection in storage tab using a host admin portal.
Yes, in that thread I asked if possible to specify in some way an alternate cpu type and/or flags to be setup for self hosted engine vm, without setup to automatically detect/configure it, but I didn't receive an answer. Because the cpu flag is there also in 4.3.9 but doesn't create problems with self hosted engine or host.
Gianluca
BTW: is there anything special we can do to help to have this cpu as supported? model name : Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz So that others could have better and wider options and also RHV benefit from it? Also, I now find here below that Cascadelake is supposed to be supported: https://ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engine_usi... Thanks, Gianluca
Maybe someone from virt team could refer to this. On Wed, May 27, 2020 at 2:06 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Wed, May 27, 2020 at 11:26 AM Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
On Wed, May 27, 2020 at 10:21 AM Amit Bawer <abawer@redhat.com> wrote:
From your vdsm log, it seems as occurrence of an issue discussed at https://lists.ovirt.org/archives/list/users@ovirt.org/message/JE76KK2WFDCDJL...
2020-05-26 19:51:53,837-0400 ERROR (vm/90f09a7c) [virt.vm] (vmId='90f09a7c-3af1-45a9-a210-78a9b0cd4c3d') The vm start process failed (vm:871) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 801, in _startUnderlyingVm self._run() File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 2608, in _run dom.createWithFlags(flags) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1166, in createWithFlags if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) libvirt.libvirtError: unsupported configuration: unknown CPU feature: tsx-ctrl
On Wed, May 27, 2020 at 9:15 AM Patrick Lomakin < patrick.lomakin@gmail.com> wrote:
I think the problem with a storage connection. Verify your IP addresses on the storage adapter port. Are they connected? After install my ports for storage was deactivated by defaut. Try to connect your iSCSI manually with iscsiadm command and then check your storage connection in storage tab using a host admin portal.
Yes, in that thread I asked if possible to specify in some way an alternate cpu type and/or flags to be setup for self hosted engine vm, without setup to automatically detect/configure it, but I didn't receive an answer. Because the cpu flag is there also in 4.3.9 but doesn't create problems with self hosted engine or host.
Gianluca
BTW: is there anything special we can do to help to have this cpu as supported?
model name : Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz
So that others could have better and wider options and also RHV benefit from it? Also, I now find here below that Cascadelake is supposed to be supported:
https://ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engine_usi...
Thanks, Gianluca
I wonder if this is something that needs QEMU 4.2 as the release notes specifically mention handling of TSX for x86 as seen here: https://wiki.qemu.org/ChangeLog/4.2#x86. Looks like QEMU 4.1.0 is in use in oVirt 4.4.0.
On Thu, May 28, 2020 at 1:29 AM Anton Gonzalez <anzigo@gmail.com> wrote:
I wonder if this is something that needs QEMU 4.2 as the release notes specifically mention handling of TSX for x86 as seen here: https://wiki.qemu.org/ChangeLog/4.2#x86. Looks like QEMU 4.1.0 is in use in oVirt 4.4.0.
See my latest comments on this thread below: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/KZHDCDE6JYADDMF... I was able to reach state with hosted engine up and gluster storage domains active. HIH testing, Gianluca
participants (4)
-
Amit Bawer -
Anton Gonzalez -
Gianluca Cecchi -
Patrick Lomakin