- Devel - oVirt List Archives

ISSUE with hosted engine: Could not find server cpu for server 'lago-he-basic-suite-master-host-0'
by Galit Rosenthal 04 Feb '20

04 Feb '20

Hi All, We have encountered an issue [1] when installing hosted engine on centos-8 Engine error [2]: 2020-01-30 03:22:27,367-05 ERROR [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ForkJoinPool-1-worker-19) [2010dcf3] Could not find server cpu for server 'lago-he-basic-suite-master-host-0' (c68e366d-f43f-43f9-a014-05bce8be23e6), flags: 'aes,clflush,tsc_adjust,constant_tsc,pse,sep,flexpriority,tpr_shadow,pae,pge,popcnt,vmx,hypervisor,xsaveopt,tsc,fpu,rep_good,cx16,tsc_deadline_timer,ept,mce,syscall,x2apic,pclmulqdq,fxsr,tsc_known_freq,de,pse36,lahf_lm,cmov,umip,vnmi,xsave,vpid,mmx,avx,mca,apic,rdtscp,xtopology,arat,skip-l1dfl-vmentry,pni,mtrr,ssse3,nx,msr,pat,sse,vme,cpuid,sse4_1,arch-capabilities,sse4_2,sse2,cx8,lm,nopl,pti,model_Conroe,model_486,model_pentium3,model_kvm64,model_Opteron_G2,model_qemu32,model_pentium,model_Opteron_G1,model_pentium2,model_Penryn,model_qemu64,model_kvm32' [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7791/ [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7791/… Regards, Galit -- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)redhat.com T: 972-9-7692230 <https://red.ht/sig>

2 3

vdsmd memory leak
by Chris Adams 04 Feb '20

04 Feb '20

I've seen vdsmd leak memory (RSS increasing) for a while (brought it up on the lists and opened a BZ ticket), and never gotten anywhere with diagnosing or resolving it. I reinstalled my dev setup Friday with up-to-date CentOS 7 (minimal install) and oVirt 4.3, with a hosted engine on iSCSI (multipath if it matters). In just 3 days, vdsmd on the host with the engine has gone up to an RSS of 481 MB. It just continues to steadily increase. Watching with a script, I see (this is VmRSS from /proc/$(pidof -x vdsmd)/status): 12:26:32.892 482076 +20 12:26:35.300 482096 +20 12:26:38.927 482112 +16 12:26:40.034 482128 +16 12:26:47.534 482132 +4 12:26:48.887 482144 +12 12:26:49.133 482156 +12 12:26:50.955 482172 +16 12:26:53.062 482176 +4 12:26:53.092 482204 +28 12:26:59.065 482212 +8 12:26:59.075 482228 +16 12:26:59.361 482244 +16 12:27:03.131 482252 +8 12:27:07.370 482256 +4 12:27:10.091 482272 +16 12:27:13.205 482296 +24 12:27:18.770 482308 +12 12:27:20.437 482332 +24 12:27:23.313 482340 +8 12:27:23.324 482364 +24 12:27:26.667 482372 +8 12:27:26.687 482376 +4 12:27:28.873 482388 +12 12:27:28.883 482392 +4 12:27:28.976 482396 +4 12:27:29.190 482408 +12 That's an increase of 352 kB in a minute. There's got to be some way to diagnose this, but I don't know python well enough. -- Chris Adams <cma(a)cmadams.net>

4 6

glance.ovirt.org down?
by Miguel Duarte de Mora Barroso 03 Feb '20

03 Feb '20

Hi, I'm seeing the following error whenever I attempt to import an image via glance. 2020-02-03 09:24:02,275+0000 ERROR (tasks/0) [storage.TaskManager.Task] (Task='c6c 4140b-1ddf-4273-b170-a0dd1589832b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in w rapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1778, in downlo adImage return img.download(methodArgs, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/image.py", line 1427, in dow nload vol.extend(imageSharing.getSize(methodArgs)) File "/usr/lib/python2.7/site-packages/vdsm/storage/imageSharing.py", line 172, in getSize return getSizeImpl(methodArgs) File "/usr/lib/python2.7/site-packages/vdsm/storage/imageSharing.py", line 44, i n httpGetSize methodArgs.get("headers", {})) File "/usr/lib/python2.7/site-packages/vdsm/storage/curlImgWrap.py", line 60, in head raise CurlError(rc, out, err) CurlError: ecode=6, stdout=[], stderr=['curl: (6) Could not resolve host: glance.o virt.org; Unknown error'], message=None Not sure where I could report this. Thanks in advance, Miguel

2 1

Ovirt System Tests "invalid syntax" for yaml import
by raymond.francis＠oracle.com 03 Feb '20

03 Feb '20

Seeing this when i run setup of System Tests for ovirt. Is there something in setup im missing 2020-01-31 10:20:53,167 021036:117 INFO: cd ovirt-system-tests && ./run_suite.sh basic-suite-4.3 + source common/helpers/logger.sh + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() + RPMS_TO_INSTALL=() + COVERAGE=false ++ getopt -o ho:e:n:b:cs:r:l:i --long help,output:,engine:,node:,boot-iso:,cleanup,images --long extra-rpm-source,reposync-config:,local-rpms: --long only-verify-requirements,ignore-requirements --long coverage -n run_suite.sh -- basic-suite-4.3 + options=' -- '\''basic-suite-4.3'\''' + [[ 0 != \0 ]] + eval set -- ' -- '\''basic-suite-4.3'\''' ++ set -- -- basic-suite-4.3 + true + case $1 in + shift + break + [[ -z basic-suite-4.3 ]] + export OST_REPO_ROOT=/ovirt-system-tests + OST_REPO_ROOT=/ovirt-system-tests ++ realpath --no-symlinks basic-suite-4.3 + export SUITE=/ovirt-system-tests/basic-suite-4.3 + SUITE=/ovirt-system-tests/basic-suite-4.3 + [[ -z '' ]] + PREFIX=/ovirt-system-tests/deployment-basic-suite-4.3 + export PREFIX + false + [[ -e /ovirt-system-tests/deployment-basic-suite-4.3 ]] + mkdir -p /ovirt-system-tests/deployment-basic-suite-4.3 + [[ -n '' ]] + verify_system_requirements /ovirt-system-tests/deployment-basic-suite-4.3 + local prefix=/ovirt-system-tests/deployment-basic-suite-4.3 + /ovirt-system-tests/common/scripts/verify_system_requirements.py --prefix-path /ovirt-system-tests/deployment-basic-suite-4.3 /ovirt-system-tests/basic-suite-4.3/vars/main.yml Traceback (most recent call last): File "/ovirt-system-tests/common/scripts/verify_system_requirements.py", line 17, in <module> import yaml File "/usr/local/lib64/python3.6/site-packages/yaml/__init__.py", line 399 class YAMLObject(metaclass=YAMLObjectMetaclass): ^ SyntaxError: invalid syntax

2 1

FOSDEM 2020 - AMENDMENT In Memory of our Friend, Lars Kurth
by Doron Fediuck 31 Jan '20

31 Jan '20

Hi all, For those of you who are unaware of this, Lars Kurth from the Xen project has passed away [1]. Lars was one of the people who truly practiced open source and worked with the oVirt project on running FOSDEM's Virtualization and IaaS room in the past years. In Lars' memory there will be a special session [2] on Sunday morning, which will be broadcasted live. You're all welcomed to join physically or via the live stream. Thanks, Doron [1] https://lists.xenproject.org/archives/html/xen-announce/2020-01/msg00004.ht… [2] https://fosdem.org/2020/schedule/event/vai_memory_of_lars_kurth/

1 0

Missing java-client-kubevirt dependency for ovirt-engine
by Andrej Krejcir 30 Jan '20

30 Jan '20

Hi, Installing the latest ovirt-engine master requires package: java-client-kubevirt >= 0.2.0 But the master nightly snapshot contains only version: java-client-kubevirt-0.1.0-1.20200102130005.git0e3be11 On the patch that bumped the version, there is a comment from the CI, that the change was not included in nightly builds: https://gerrit.ovirt.org/#/c/106499/ Best regards, Andrej

3 3

RFC: Not Failing OST on Missing Artifact During Collection
by Anton Marchukov 30 Jan '20

30 Jan '20

Hello All. I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test. I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead. Wdyt? -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat

4 3

running podman in mock
by Miguel Duarte de Mora Barroso 29 Jan '20

29 Jan '20

Hi, When attempting to make the ovirt-provider-ovn integration tests run in el8 (which requires podman instead of docker), I'm getting trouble even running it (it being podman). It fails with the following error - e.g. on a podman info command: Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported Has anyone faced anything like this / is able to provide some pointers ? Thanks in advance, Miguel

3 5

ovirt-engine has been tagged (ovirt-engine-4.3.9.0)
by Tal Nisan 29 Jan '20

29 Jan '20

1 0

host-deploy failed at Reconfigure vdsm tool
by Yedidyah Bar David 29 Jan '20

29 Jan '20

Hi all, Tried yesterday to deploy hosted-engine on current master snapshot, it failed during host-deploy. host-deploy log ends with: 2020-01-27 15:25:11 IST - TASK [ovirt-host-deploy-vdsm : Reconfigure vdsm tool] ************************** /var/log/messages on the host has: Jan 27 15:25:24 didi-centos8-host python3[12352]: detected unhandled Python exception in '/usr/bin/vdsm-tool' /var/log/httpd/ansible_runner_error_log on the engine has (I have to get used to searching there, grepped all of /var/log to find it): [Mon Jan 27 15:25:25.692397 2020] [wsgi:error] [pid 30029] cb_event_handler event_data={u'event': u'runner_on_failed', u'uuid': u'874e5fcc-12e8-419c-b010-d958adfc4aec', 'stdout': u'fatal: [d idi-centos8-host.lab.eng.tlv2.redhat.com] FAILED! => {"changed": true, "cmd": "vdsm-tool configure --force", "delta": "0:00:15.833532", "end": "2020-01-27 15:25:25.296212", "msg": "non-zero return code", "rc": 1, "start": "2020-01-27 15:25:09.462680", "stderr": "Error: Traceback (most recent call last):\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 209, in main\\\\n return tool_command[cmd][\\\\"command\\\\"](*args)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\", line 40, in wrapper\\\\n func(*args, **kwargs)\\\\n File \\\\"/u sr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\", line 149, in configure\\\\n service.service_start(s)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\ \\", line 193, in service_start\\\\n return _runAlts(_srvStartAlts, srvName)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 172, in _runAlts\\\\n \\\\ "%s failed\\\\" % alt.__name__, out, err)\\\\nvdsm.tool.service.ServiceOperationError: <exception str() failed>\\\\n\\\\nDuring handling of the above exception, another exception occurred:\\ \\n\\\\nTraceback (most recent call last):\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>\\\\n sys.exit(main())\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 214, in main\ \\\n print(\\'Error: \\', e, \\'\\\\\\\\n\\', file=sys.stderr)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 75, in __str__\\\\n return \\'\\\\\\\\n\ \'.join(s)\\\\nTypeError: sequence item 1: expected str instance, bytes found", "stderr_lines": ["Error: Traceback (most recent call last):", " File \\\\"/usr/bin/vdsm-tool\\\\", line 209, in main", " return tool_command[cmd][\\\\"command\\\\"](*args)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\", line 40, in wrapper", " func(*args, **kwargs )", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\", line 149, in configure", " service.service_start(s)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/ tool/service.py\\\\", line 193, in service_start", " return _runAlts(_srvStartAlts, srvName)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 172, in _runAl ts", " \\\\"%s failed\\\\" % alt.__name__, out, err)", "vdsm.tool.service.ServiceOperationError: <exception str() failed>", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>", " sys.exit(main())", " File \\\\"/usr/bin/vdsm-tool\\\\", line 214, in main", " print(\\'Error: \\', e, \\'\\\\\\\\n\\', file=sys.stderr)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 75, in __str__", " return \\'\\\\\\\ \n\\'.join(s)", "TypeError: sequence item 1: expected str instance, bytes found"], "stdout": "\\\\nChecking configuration status...\\\\n\\\\nCurrent revision of multipath.conf detected, pres erving\\\\nabrt is not configured for vdsm\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\' , \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names \\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nlibvirt is not configured for vdsm yet\\\\nFAILED: conflicting vdsm and libvirt-qemu tls configuration.\\\\nvdsm.conf with ssl=True requires the following changes:\\\\nlibvirtd.conf: listen_tcp=0, auth_tcp=\\\\"sasl\\\\", listen_tls=1\\\\nqemu.conf: spice_tls=1.\\\\nlvm is configured for vdsm\\\\nManaged volume database is already configured\\\\n\\\\nRunning configure...\\\\nReconfiguration of passwd is done.\\\\nReconfiguration of abrt is done.\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nReconfiguration of libvirt is done.", "stdout_lines": ["", "Checking configuration status...", "", "Current revision of multipath.conf detected, preserving", "abrt is not configured for vdsm", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "libvirt is not configured for vdsm yet", "FAILED: conflicting vdsm and libvirt-qemu tls configuration.", "vdsm.conf with ssl=True requires the following changes:", "libvirtd.conf: listen_tcp=0, auth_tcp=\\\\"sasl\\\\", listen_tls=1", "qemu.conf: spice_tls=1.", "lvm is configured for vdsm", "Managed volume database is already configured", "", "Running configure...", "Reconfiguration of passwd is done.", "Reconfiguration of abrt is done.", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "Reconfiguration of libvirt is done."]}', 'counter': 40, u'pid': 31685, u'created': u'2020-01-27T13:25:25.683262', 'end_line': 35, 'runner_ident': '5b95ec42-4108-11ea-8afe-001a4a16027a', 'start_line': 34, u'event_data': {u'play_pattern': u'all', u'play': u'all', u'event_loop': None, u'task_args': u'', u'remote_addr': u' didi-centos8-host.lab.eng.tlv2.redhat.com', u'res': {u'stderr_lines': [u'Error: Traceback (most recent call last):', u' File "/usr/bin/vdsm-tool", line 209, in main', u' return tool_command[cmd]["command"](*args)', u' File "/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py", line 40, in wrapper', u' func(*args, **kwargs)', u' File "/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py", line 149, in configure', u' service.service_start(s)', u' File "/usr/lib/python3.6/site- [Mon Jan 27 15:25:25.693655 2020] [wsgi:error] [pid 30029] cb_event_handler event_data={u'event': u'playbook_on_stats', u'uuid': u'2c8d843c-c1ad-41c0-940e-88354387b725', 'stdout': u'\\r\\nPLAY RECAP *********************************************************************\\r\\ ndidi-centos8-host.lab.eng.tlv2.redhat.com : ok=11 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 \\r\\n', 'counter': 41, u'pid': 31685, u'created': u'2020-01-27T13:25:25.688541', 'end_line': 39, 'runner_ident': '5b95ec42-4108-11ea-8afe-001a4a16027a', 'start_line': 35, u'event_data': {u'ignored': {}, u'skipped': {u'didi-centos8-host.lab.eng.tlv2.redhat.com': 1}, u'ok': {u'didi-centos8-host.lab.eng.tlv2.redhat.com': 11}, u'artifact_data': {}, u'rescued': {}, u'changed': {}, u'pid': 31685, u'dark': {}, u'playbook_uuid': u'ff508986-9930-47a2-b17b-61a2f2dae905', u'playbook': u'ovirt-host-deploy.yml', u'failures': {u' didi-centos8-host.lab.eng.tlv2.redhat.com': 1}, u'processed': {u' didi-centos8-host.lab.eng.tlv2.redhat.com': 1}}, u'parent_uuid': u'ff508986-9930-47a2-b17b-61a2f2dae905'} If I now manually run on the host 'vdsm-tool configure --force', it does not fail. I am now going to try deploy again after reverting to a snapshot I took before deploy. Best regards, -- Didi

4 5