January 2020 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

Purging inactive maintainers from vdsm-master-maintainers
by Nir Soffer 21 Jun '21

21 Jun '21

I want to remove inactive contributors from vdsm-master-maintainers. I suggest the simple rule of 2 years of inactivity for removing from this group, based on git log. See the list below for current status: https://gerrit.ovirt.org/#/admin/groups/106,members Thoughts? Nir

6 11

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

Why is vdsm enabled by default?
by Yedidyah Bar David 02 Apr '20

02 Apr '20

If I do e.g.: 1. Install CentOS 2. yum install ovirt-releaseSOMETHING 3. yum install vdsm Then reboot the machine, vdsm starts, and for this, it does all kinds of things to the system (such as configure various services using vdsm-tool etc.). Are we sure we want/need this? Why would we want vdsm configured/running at all at this stage, before being added to an engine? In particular, if (especially during development) we have a bug in this configuration process, and then fix it, it might not be enough to upgrade vdsm - the tooling will then also have to fix the changes done by the buggy previous version, or require a full machine reinstall. Thanks and best regards, -- Didi

7 13

Container-based CI backend is now available for use
by Barak Korren 26 Mar '20

26 Mar '20

A little less then a month ago I sent an email to this list telling you all about ongoing work to create a new container-based backend for the oVirt CI system. I'm pleased to announce that we have managed to finally merged an initial set of patches implementing that backend yesterday, and it is now available for general use. *What? Were? How do I use it?* Documentation about how to use the new backend is now available in read the docs <https://ovirt-infra-docs.readthedocs.io/en/latest/CI/STDCI-Containers/index…> . *Wait! I needed it to do X which it doesn't!* For the time being the new backend lacks some features that some may consider to be essential, such as automated cloning of patch source code and build artifact collection. We already have implemented patches providing a substantial amount of additional functionality, and hopefully we will be able to merge them soon. Following is a list of those patches and the features they implement: 1. Automated source cloning support: - 104213 <https://gerrit.ovirt.org/104213>: Implement STDCI DSL support for initContainers - 104590 <https://gerrit.ovirt.org/104590>: STDCI DSL: Add the `decorate` option - 104668 <https://gerrit.ovirt.org/104668>: Document source cloning extension for containers 2. Artifact collection support - 104690 <https://gerrit.ovirt.org/104690>: Added NFS server container image - 104273 <https://gerrit.ovirt.org/104273>: STDCI PODS: Unique UID for each job build's POD - 104756 <https://gerrit.ovirt.org/104756>: pipeline-loader: refactor: separate podspec func - 104757 <https://gerrit.ovirt.org/104757>: pipeline-loader: refactor: Use podspec struct def - 104766 <https://gerrit.ovirt.org/104766>: STDCI PODs: Add artifact collection logic - 105522 <https://gerrit.ovirt.org/105522>: Documented artifact collection in containers 3. Extended log collection - 104842 <https://gerrit.ovirt.org/104842>: STDCI PODs: Add POD log collection - 105523 <https://gerrit.ovirt.org/105523>: Documented log collection in containers 4. Privileged container support - 104786 <https://gerrit.ovirt.org/104786>: STDCI DSL: Enable privileged containers 5. Support for using containers in gating jobs: - 104804 <https://gerrit.ovirt.org/104804>: standard-stage: refactor: move DSL to a library - 104811 <https://gerrit.ovirt.org/104811>: gate: Support getting suits from STDCI DSL 6. Providing the `extra_sources` file to OST suit containers: - 104843 <https://gerrit.ovirt.org/104843>: stdci_runner: Create extra_sources for PODs 7. Support for mirror injection and upstream source cloning - 104917 <https://gerrit.ovirt.org/104917>: Added a container with STDCI tools - 104918 <https://gerrit.ovirt.org/104918>: decorate.py: Add script - 104989 <https://gerrit.ovirt.org/104989>: STDCI DSL: Use `tools` container for `decorate` - 104994 <https://gerrit.ovirt.org/104994>: stdci_runner: Inject mirrors in PODs As you can see, we have quite a big pile of reviews to do, as always, help is very welcome... Regards, Barak. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

2 4

Create VM snapshot with no disk
by scott.fitzgerald＠oracle.com 13 Feb '20

13 Feb '20

Hi, A VM Snapshot can be made using SnapshotBuilder builder = new SnapshotBuilder().vm(VM).name("Snap1").description("Test"); sysService.vmsService().vmService(VM.id()).snapshotsService().add().snapshot(builder).send(); It can be made with no memory by using a builder with persistMemorystate(false) How can one be made with no disk? .diskAttachments(emptyList) isnt working. TIA

3 5

vdsmd memory leak
by Chris Adams 04 Feb '20

04 Feb '20

I've seen vdsmd leak memory (RSS increasing) for a while (brought it up on the lists and opened a BZ ticket), and never gotten anywhere with diagnosing or resolving it. I reinstalled my dev setup Friday with up-to-date CentOS 7 (minimal install) and oVirt 4.3, with a hosted engine on iSCSI (multipath if it matters). In just 3 days, vdsmd on the host with the engine has gone up to an RSS of 481 MB. It just continues to steadily increase. Watching with a script, I see (this is VmRSS from /proc/$(pidof -x vdsmd)/status): 12:26:32.892 482076 +20 12:26:35.300 482096 +20 12:26:38.927 482112 +16 12:26:40.034 482128 +16 12:26:47.534 482132 +4 12:26:48.887 482144 +12 12:26:49.133 482156 +12 12:26:50.955 482172 +16 12:26:53.062 482176 +4 12:26:53.092 482204 +28 12:26:59.065 482212 +8 12:26:59.075 482228 +16 12:26:59.361 482244 +16 12:27:03.131 482252 +8 12:27:07.370 482256 +4 12:27:10.091 482272 +16 12:27:13.205 482296 +24 12:27:18.770 482308 +12 12:27:20.437 482332 +24 12:27:23.313 482340 +8 12:27:23.324 482364 +24 12:27:26.667 482372 +8 12:27:26.687 482376 +4 12:27:28.873 482388 +12 12:27:28.883 482392 +4 12:27:28.976 482396 +4 12:27:29.190 482408 +12 That's an increase of 352 kB in a minute. There's got to be some way to diagnose this, but I don't know python well enough. -- Chris Adams <cma(a)cmadams.net>

4 6

Ovirt System Tests "invalid syntax" for yaml import
by raymond.francis＠oracle.com 03 Feb '20

03 Feb '20

Seeing this when i run setup of System Tests for ovirt. Is there something in setup im missing 2020-01-31 10:20:53,167 021036:117 INFO: cd ovirt-system-tests && ./run_suite.sh basic-suite-4.3 + source common/helpers/logger.sh + CLI=lago + DO_CLEANUP=false + RECOMMENDED_RAM_IN_MB=8196 + EXTRA_SOURCES=() + RPMS_TO_INSTALL=() + COVERAGE=false ++ getopt -o ho:e:n:b:cs:r:l:i --long help,output:,engine:,node:,boot-iso:,cleanup,images --long extra-rpm-source,reposync-config:,local-rpms: --long only-verify-requirements,ignore-requirements --long coverage -n run_suite.sh -- basic-suite-4.3 + options=' -- '\''basic-suite-4.3'\''' + [[ 0 != \0 ]] + eval set -- ' -- '\''basic-suite-4.3'\''' ++ set -- -- basic-suite-4.3 + true + case $1 in + shift + break + [[ -z basic-suite-4.3 ]] + export OST_REPO_ROOT=/ovirt-system-tests + OST_REPO_ROOT=/ovirt-system-tests ++ realpath --no-symlinks basic-suite-4.3 + export SUITE=/ovirt-system-tests/basic-suite-4.3 + SUITE=/ovirt-system-tests/basic-suite-4.3 + [[ -z '' ]] + PREFIX=/ovirt-system-tests/deployment-basic-suite-4.3 + export PREFIX + false + [[ -e /ovirt-system-tests/deployment-basic-suite-4.3 ]] + mkdir -p /ovirt-system-tests/deployment-basic-suite-4.3 + [[ -n '' ]] + verify_system_requirements /ovirt-system-tests/deployment-basic-suite-4.3 + local prefix=/ovirt-system-tests/deployment-basic-suite-4.3 + /ovirt-system-tests/common/scripts/verify_system_requirements.py --prefix-path /ovirt-system-tests/deployment-basic-suite-4.3 /ovirt-system-tests/basic-suite-4.3/vars/main.yml Traceback (most recent call last): File "/ovirt-system-tests/common/scripts/verify_system_requirements.py", line 17, in <module> import yaml File "/usr/local/lib64/python3.6/site-packages/yaml/__init__.py", line 399 class YAMLObject(metaclass=YAMLObjectMetaclass): ^ SyntaxError: invalid syntax

2 1

FOSDEM 2020 - AMENDMENT In Memory of our Friend, Lars Kurth
by Doron Fediuck 31 Jan '20

31 Jan '20

Hi all, For those of you who are unaware of this, Lars Kurth from the Xen project has passed away [1]. Lars was one of the people who truly practiced open source and worked with the oVirt project on running FOSDEM's Virtualization and IaaS room in the past years. In Lars' memory there will be a special session [2] on Sunday morning, which will be broadcasted live. You're all welcomed to join physically or via the live stream. Thanks, Doron [1] https://lists.xenproject.org/archives/html/xen-announce/2020-01/msg00004.ht… [2] https://fosdem.org/2020/schedule/event/vai_memory_of_lars_kurth/

1 0

Missing java-client-kubevirt dependency for ovirt-engine
by Andrej Krejcir 30 Jan '20

30 Jan '20

Hi, Installing the latest ovirt-engine master requires package: java-client-kubevirt >= 0.2.0 But the master nightly snapshot contains only version: java-client-kubevirt-0.1.0-1.20200102130005.git0e3be11 On the patch that bumped the version, there is a comment from the CI, that the change was not included in nightly builds: https://gerrit.ovirt.org/#/c/106499/ Best regards, Andrej

3 3

RFC: Not Failing OST on Missing Artifact During Collection
by Anton Marchukov 30 Jan '20

30 Jan '20

Hello All. I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test. I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead. Wdyt? -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat

4 3

running podman in mock
by Miguel Duarte de Mora Barroso 29 Jan '20

29 Jan '20

Hi, When attempting to make the ovirt-provider-ovn integration tests run in el8 (which requires podman instead of docker), I'm getting trouble even running it (it being podman). It fails with the following error - e.g. on a podman info command: Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported Has anyone faced anything like this / is able to provide some pointers ? Thanks in advance, Miguel

3 5

ovirt-engine has been tagged (ovirt-engine-4.3.9.0)
by Tal Nisan 29 Jan '20

29 Jan '20

1 0

host-deploy failed at Reconfigure vdsm tool
by Yedidyah Bar David 29 Jan '20

29 Jan '20

Hi all, Tried yesterday to deploy hosted-engine on current master snapshot, it failed during host-deploy. host-deploy log ends with: 2020-01-27 15:25:11 IST - TASK [ovirt-host-deploy-vdsm : Reconfigure vdsm tool] ************************** /var/log/messages on the host has: Jan 27 15:25:24 didi-centos8-host python3[12352]: detected unhandled Python exception in '/usr/bin/vdsm-tool' /var/log/httpd/ansible_runner_error_log on the engine has (I have to get used to searching there, grepped all of /var/log to find it): [Mon Jan 27 15:25:25.692397 2020] [wsgi:error] [pid 30029] cb_event_handler event_data={u'event': u'runner_on_failed', u'uuid': u'874e5fcc-12e8-419c-b010-d958adfc4aec', 'stdout': u'fatal: [d idi-centos8-host.lab.eng.tlv2.redhat.com] FAILED! => {"changed": true, "cmd": "vdsm-tool configure --force", "delta": "0:00:15.833532", "end": "2020-01-27 15:25:25.296212", "msg": "non-zero return code", "rc": 1, "start": "2020-01-27 15:25:09.462680", "stderr": "Error: Traceback (most recent call last):\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 209, in main\\\\n return tool_command[cmd][\\\\"command\\\\"](*args)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\", line 40, in wrapper\\\\n func(*args, **kwargs)\\\\n File \\\\"/u sr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\", line 149, in configure\\\\n service.service_start(s)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\ \\", line 193, in service_start\\\\n return _runAlts(_srvStartAlts, srvName)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 172, in _runAlts\\\\n \\\\ "%s failed\\\\" % alt.__name__, out, err)\\\\nvdsm.tool.service.ServiceOperationError: <exception str() failed>\\\\n\\\\nDuring handling of the above exception, another exception occurred:\\ \\n\\\\nTraceback (most recent call last):\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>\\\\n sys.exit(main())\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line 214, in main\ \\\n print(\\'Error: \\', e, \\'\\\\\\\\n\\', file=sys.stderr)\\\\n File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 75, in __str__\\\\n return \\'\\\\\\\\n\ \'.join(s)\\\\nTypeError: sequence item 1: expected str instance, bytes found", "stderr_lines": ["Error: Traceback (most recent call last):", " File \\\\"/usr/bin/vdsm-tool\\\\", line 209, in main", " return tool_command[cmd][\\\\"command\\\\"](*args)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\", line 40, in wrapper", " func(*args, **kwargs )", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\", line 149, in configure", " service.service_start(s)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/ tool/service.py\\\\", line 193, in service_start", " return _runAlts(_srvStartAlts, srvName)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 172, in _runAl ts", " \\\\"%s failed\\\\" % alt.__name__, out, err)", "vdsm.tool.service.ServiceOperationError: <exception str() failed>", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>", " sys.exit(main())", " File \\\\"/usr/bin/vdsm-tool\\\\", line 214, in main", " print(\\'Error: \\', e, \\'\\\\\\\\n\\', file=sys.stderr)", " File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\", line 75, in __str__", " return \\'\\\\\\\ \n\\'.join(s)", "TypeError: sequence item 1: expected str instance, bytes found"], "stdout": "\\\\nChecking configuration status...\\\\n\\\\nCurrent revision of multipath.conf detected, pres erving\\\\nabrt is not configured for vdsm\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\' , \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names \\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nlibvirt is not configured for vdsm yet\\\\nFAILED: conflicting vdsm and libvirt-qemu tls configuration.\\\\nvdsm.conf with ssl=True requires the following changes:\\\\nlibvirtd.conf: listen_tcp=0, auth_tcp=\\\\"sasl\\\\", listen_tls=1\\\\nqemu.conf: spice_tls=1.\\\\nlvm is configured for vdsm\\\\nManaged volume database is already configured\\\\n\\\\nRunning configure...\\\\nReconfiguration of passwd is done.\\\\nReconfiguration of abrt is done.\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nlibvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked\\\\nReconfiguration of libvirt is done.", "stdout_lines": ["", "Checking configuration status...", "", "Current revision of multipath.conf detected, preserving", "abrt is not configured for vdsm", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "libvirt is not configured for vdsm yet", "FAILED: conflicting vdsm and libvirt-qemu tls configuration.", "vdsm.conf with ssl=True requires the following changes:", "libvirtd.conf: listen_tcp=0, auth_tcp=\\\\"sasl\\\\", listen_tls=1", "qemu.conf: spice_tls=1.", "lvm is configured for vdsm", "Managed volume database is already configured", "", "Running configure...", "Reconfiguration of passwd is done.", "Reconfiguration of abrt is done.", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "libvirtd socket units status: [{\\'Names\\': \\'libvirtd-tls.socket\\', \\'LoadState\\': \\'loaded\\'}, {\\'Names\\': \\'libvirtd-tcp.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd-admin.socket\\', \\'LoadState\\': \\'masked\\'}, {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\': \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket activation - one or more of its socket units have been masked", "Reconfiguration of libvirt is done."]}', 'counter': 40, u'pid': 31685, u'created': u'2020-01-27T13:25:25.683262', 'end_line': 35, 'runner_ident': '5b95ec42-4108-11ea-8afe-001a4a16027a', 'start_line': 34, u'event_data': {u'play_pattern': u'all', u'play': u'all', u'event_loop': None, u'task_args': u'', u'remote_addr': u' didi-centos8-host.lab.eng.tlv2.redhat.com', u'res': {u'stderr_lines': [u'Error: Traceback (most recent call last):', u' File "/usr/bin/vdsm-tool", line 209, in main', u' return tool_command[cmd]["command"](*args)', u' File "/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py", line 40, in wrapper', u' func(*args, **kwargs)', u' File "/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py", line 149, in configure', u' service.service_start(s)', u' File "/usr/lib/python3.6/site- [Mon Jan 27 15:25:25.693655 2020] [wsgi:error] [pid 30029] cb_event_handler event_data={u'event': u'playbook_on_stats', u'uuid': u'2c8d843c-c1ad-41c0-940e-88354387b725', 'stdout': u'\\r\\nPLAY RECAP *********************************************************************\\r\\ ndidi-centos8-host.lab.eng.tlv2.redhat.com : ok=11 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 \\r\\n', 'counter': 41, u'pid': 31685, u'created': u'2020-01-27T13:25:25.688541', 'end_line': 39, 'runner_ident': '5b95ec42-4108-11ea-8afe-001a4a16027a', 'start_line': 35, u'event_data': {u'ignored': {}, u'skipped': {u'didi-centos8-host.lab.eng.tlv2.redhat.com': 1}, u'ok': {u'didi-centos8-host.lab.eng.tlv2.redhat.com': 11}, u'artifact_data': {}, u'rescued': {}, u'changed': {}, u'pid': 31685, u'dark': {}, u'playbook_uuid': u'ff508986-9930-47a2-b17b-61a2f2dae905', u'playbook': u'ovirt-host-deploy.yml', u'failures': {u' didi-centos8-host.lab.eng.tlv2.redhat.com': 1}, u'processed': {u' didi-centos8-host.lab.eng.tlv2.redhat.com': 1}}, u'parent_uuid': u'ff508986-9930-47a2-b17b-61a2f2dae905'} If I now manually run on the host 'vdsm-tool configure --force', it does not fail. I am now going to try deploy again after reverting to a snapshot I took before deploy. Best regards, -- Didi

4 5

experimental CLI to upload images from URL
by Roy Golan 28 Jan '20

28 Jan '20

For a long time I think that ovirt is missing some CLI tools to help run quick tasks or just to automate stuff (not everything is ansible). Also I had to upload disks using code for the terraform provider support[1] and I thought I should place all the knowledge of how to upload a disk into a go module. So, on the way to achieve both I created this[2]. A typical usage is: ovirt-image-upload \ -s http://fqdn/image -d <domain-id> For now it is under my GH user, but if it will all go well I will port under ovirt-imageio, because that is probably the right place. (/cc @Nir Soffer <nsoffer(a)redhat.com> ) [1] https://github.com/oVirt/terraform-provider-ovirt/commit/d48bbca52bbb0c7c68… [2] https://github.com/rgolangh/ovirt-image-upload Regards, Roy

2 1

Implicit Affinity Label checkbox
by scott.fitzgerald＠oracle.com 27 Jan '20

27 Jan '20

Hi, Is it possible to reach the Implicit Affinity Label checkbox through the SDK? When unchecked, the label seems to have zero effect. Kind regards, Scott

2 2

Task Start and enable services failed to execute
by Yedidyah Bar David 27 Jan '20

27 Jan '20

Hi all, Tried now 'hosted-engine --deploy' on fully updated CentOS 8/ovirt-master-snapshot machine. It failed during adding the host to the engine. engine.log has: 2020-01-26 10:41:47,825+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [11ba00a7] Host installation failed for host 'efd6cb8a-935d-4812-b35c-3fbde5651b5a', 'didi-centos8-host.lab.eng.tlv2.redhat.com': Task Start and enable services failed to execute: 2020-01-26 10:41:47,836+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [11ba00a7] START, SetVdsStatusVDSCommand(HostName = didi-centos8-host.lab.eng.tlv2.redhat.com, SetVdsStatusVDSCommandParameters:{hostId='efd6cb8a-935d-4812-b35c-3fbde5651b5a', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4f107d5d 2020-01-26 10:41:47,901+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [11ba00a7] FINISH, SetVdsStatusVDSCommand, return: , log id: 4f107d5d 2020-01-26 10:41:48,002+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [11ba00a7] EVENT_ID: VDS_INSTALL_FAILED(505), Host didi-centos8-host.lab.eng.tlv2.redhat.com installation failed. Task Start and enable services failed to execute: . The code emitting this error seems to be in backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/common/utils/ansible/AnsibleRunnerHTTPClient.java : String.format("Task %1$s failed to execute: %2$s", task, "") // stdout, stderr? Seems like someone considered logging also stdout/stderr but failed to make up their minds. Is this tracked somewhere? ovirt-host-deploy-ansible-20200126103902-didi-centos8-host.lab.eng.tlv2.redhat.com-11ba00a7.log has: 2020-01-26 10:41:38 IST - TASK [ovirt-host-deploy-vdsm : Start and enable services] ********************** 2020-01-26 10:41:47 IST - 2020-01-26 10:41:47 IST - { "status" : "OK", "msg" : "", "data" : { "event" : "runner_on_failed", ... "msg" : "Unable to start service vdsmd.service: Failed to start vdsmd.service: Unit libvirtd-tcp.socket is masked.\n", "_ansible_item_label" : "vdsmd.service" systemctl status libvirtd-tcp.socket indeed still says it's masked. Package is: # rpm -qif /usr/lib/systemd/system/libvirtd-tcp.socket Name : libvirt-daemon Version : 5.6.0 Release : 6.el8 Architecture: x86_64 Install Date: Mon 20 Jan 2020 08:23:12 AM IST Group : Unspecified Size : 1320922 License : LGPLv2+ Signature : RSA/SHA1, Wed 08 Jan 2020 11:06:38 AM IST, Key ID 695b5f7eff3e3445 Source RPM : libvirt-5.6.0-6.el8.src.rpm Build Date : Wed 08 Jan 2020 11:06:04 AM IST Build Host : copr-builder-156909441.novalocal Relocations : (not relocatable) URL : https://libvirt.org/ Summary : Server side daemon and supporting files for libvirt library Description : Server side daemon required to manage the virtualization capabilities of recent versions of Linux. Requires a hypervisor specific sub-RPM for specific drivers. Known issue? Thanks and best regards, -- Didi

2 4

Download Disk via SDK
by scott.fitzgerald＠oracle.com 23 Jan '20

23 Jan '20

The diskService in the SDK has access to Move, Export, Remove, etc. However, I dont see a way to send a download request on the Disk. Does this functionality exist?

3 2

type object 'LinuxBridge' has no attribute 'STP' (was: [CQ]: 66276a7 (ovirt-ansible-hosted-engine-setup) failed "ovirt-master" system tests)
by Yedidyah Bar David 22 Jan '20

22 Jan '20

Resending and adding devel. This now happened to me again. I suspect this affects other runs. Any clue? https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7692/ On Tue, Jan 21, 2020 at 12:34 PM Yedidyah Bar David <didi(a)redhat.com> wrote: > > On Tue, Jan 21, 2020 at 1:18 AM oVirt Jenkins <jenkins(a)ovirt.org> wrote: > > > > Change 66276a7 (ovirt-ansible-hosted-engine-setup) is probably the reason > > behind recent system test failures in the "ovirt-master" change queue and needs > > to be fixed. > > > > This change had been removed from the testing queue. Artifacts build from this > > change will not be released until it is fixed. > > > > For further details about the change see: > > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/66276a7b4… > > Above change is unrelated to the failure below. How can I make CQ look > at it again? > > > > > For failed test results see: > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/ > > This failed in basic suite, in 002_bootstrap.verify_add_hosts. > > engine.log [1] has (e.g.): > > 2020-01-20 17:56:33,198-05 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [66755eba] EVENT_ID: > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM > lago-basic-suite-master-host-0 command HostSetupNetworksVDS failed: > Internal JSON-RPC error: {'reason': "type object 'LinuxBridge' has no > attribute 'STP'"} > > supervdsm log [2] has: > > MainProcess|jsonrpc/4::INFO::2020-01-20 > 17:56:32,430::configurator::190::root::(_setup_nmstate) Processing > setup through nmstate > MainProcess|jsonrpc/4::ERROR::2020-01-20 > 17:56:32,695::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) > Error in setupNetworks > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", > line 95, in wrapper > res = func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 240, in setupNetworks > _setup_networks(networks, bondings, options, net_info) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 265, in _setup_networks > networks, bondings, options, net_info, in_rollback > File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 154, in setup > _setup_nmstate(networks, bondings, options, in_rollback, net_info) > File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 195, in _setup_nmstate > desired_state = nmstate.generate_state(networks, bondings) > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 73, in generate_state > networks, rconfig.networks, current_ifaces_state > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 603, in generate_state > for netname, netattrs in six.viewitems(networks) > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 603, in <listcomp> > for netname, netattrs in six.viewitems(networks) > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 339, in __init__ > self._create_interfaces_state() > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 430, in _create_interfaces_state > sb_iface, vlan_iface, bridge_iface = self._create_ifaces() > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 444, in _create_ifaces > options=self._create_bridge_options(), > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 492, in _create_bridge_options > LinuxBridge.STP.ENABLED: self._netconf.stp > AttributeError: type object 'LinuxBridge' has no attribute 'STP' > > Perhaps that's related to recent changes adding/updating nmstate? > > [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/artifa… > [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/artifa… > > > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/JTIBSBDL5IYJT… > > > > -- > Didi -- Didi

5 11

basic suite 4.3 failed in live_storage_migration
by Yedidyah Bar David 22 Jan '20

22 Jan '20

Hi all, Please see [1][2]. lago.log [3]: 2020-01-20 11:55:16,647::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-72 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_standard-check-patch/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 514, in live_storage_migration lambda: api.follow_link(disk_service.get().storage_domains[0]).name == SD_ISCSI_NAME File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds Not sure, but this might be related: engine.log [4]: 2020-01-20 06:45:13,991-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [ae1529f8-3e05-4bc9-b3bf-d058f45dfb2b] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = General Exception: ('Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainGetBlockInfo)',), code = 100 vdsm.log [5]: 2020-01-20 06:45:13,940-0500 ERROR (jsonrpc/1) [api] FINISH diskReplicateFinish error=Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainGetBlockInfo) (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 580, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4650, in diskReplicateFinish blkJobInfo = self._dom.blockJobInfo(drive.name, 0) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 108, in f raise toe TimeoutError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainGetBlockInfo) Looked a bit, and can't find the root cause. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7684/ [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7684/… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7684/… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7684/… [5] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7684/… -- Didi

3 2

migration on migration network failed
by Dominik Holler 22 Jan '20

22 Jan '20

Hello, is live migration on ovirt-master via migration network currently expected to work? Currently, it fails in OST network-suite-master with 2020-01-21 22:49:20,991-0500 INFO (jsonrpc/2) [api.virt] START migrate(params={'abortOnError': 'true', 'autoConverge': 'true', 'dst': ' 192.168.201.3:54321', 'method': 'online', 'vmId': '736dea3b-64be-427f-9ebf-d1e758b6f68e', 'src': '192.168.201.4', 'dstqemu': '192.0.3.1', 'convergenceSchedule': {'init': [{'name': 'setDowntime', 'params': ['100']}], 'stalling': [{'limit': 1, 'action': {'name': 'setDowntime', 'params': ['150']}}, {'limit': 2, 'action': {'name': 'setDowntime', 'params': ['200']}}, {'limit': 3, 'action': {'name': 'setDowntime', 'params': ['300']}}, {'limit': 4, 'action': {'name': 'setDowntime', 'params': ['400']}}, {'limit': 6, 'action': {'name': 'setDowntime', 'params': ['500']}}, {'limit': -1, 'action': {'name': 'abort', 'params': []}}]}, 'outgoingLimit': 2, 'enableGuestEvents': True, 'tunneled': 'false', 'encrypted': False, 'compressed': 'false', 'incomingLimit': 2}) from=::ffff:192.168.201.2,43782, flow_id=5c0e0e0a-8d5f-4b66-bda3-acca1e626a41, vmId=736dea3b-64be-427f-9ebf-d1e758b6f68e (api:48) 2020-01-21 22:49:20,997-0500 INFO (jsonrpc/2) [api.virt] FINISH migrate return={'status': {'code': 0, 'message': 'Migration in progress'}, 'progress': 0} from=::ffff:192.168.201.2,43782, flow_id=5c0e0e0a-8d5f-4b66-bda3-acca1e626a41, vmId=736dea3b-64be-427f-9ebf-d1e758b6f68e (api:54) 2020-01-21 22:49:20,997-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.01 seconds (__init__:312) 2020-01-21 22:49:21,099-0500 INFO (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') Migration semaphore: acquiring (migration:405) 2020-01-21 22:49:21,099-0500 INFO (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') Migration semaphore: acquired (migration:407) 2020-01-21 22:49:21,837-0500 INFO (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') Creation of destination VM took: 0 seconds (migration:459) 2020-01-21 22:49:21,838-0500 INFO (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') starting migration to qemu+tls://192.168.201.3/system with miguri tcp://192.0.3.1 (migration:525) 2020-01-21 22:49:21,870-0500 ERROR (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') operation failed: Failed to connect to remote libvirt URI qemu+tls://192.168.201.3/system: unable to connect to server at '192.168.201.3:16514': Connection refused (migration:278) 2020-01-21 22:49:22,816-0500 ERROR (migsrc/736dea3b) [virt.vm] (vmId='736dea3b-64be-427f-9ebf-d1e758b6f68e') Failed to migrate (migration:441) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 422, in _regular_run time.time(), machineParams File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 528, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 609, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 545, in _perform_migration self._migration_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1838, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://192.168.201.3/system: unable to connect to server at ' 192.168.201.3:16514': Connection refused 2020-01-21 22:49:22,989-0500 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:192.168.201.2,43782 (api:48) 2020-01-21 22:49:22,992-0500 INFO (jsonrpc/7) [throttled] Current getAllVmStats: {'736dea3b-64be-427f-9ebf-d1e758b6f68e': 'Up'} (throttledlog:104) Please find details in https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-master/1245/ . Dominik

2 1

Failed to add host
by Ritesh Chikatwar 21 Jan '20

21 Jan '20

Hello, Ovirt-engine env: OS- Fedora-30 ovirt-engine running master When i try to adding host: VM OS- Centos-8 Repo enabled- http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm I attached a log file for host deploy. https://pastebin.com/HLwbK39w

4 5

Promotion of Eyal Shenitzky to RHV Storage team lead
by Tal Nisan 20 Jan '20

20 Jan '20

Hello, Today I have the pleasure of announcing Eyal Shenitzky's promotion to team lead in the RHV Storage team. Eyal joined our team in August 2017 and since then became an integral part of the team and had a key role in features such as DR, Cinberlib, Incremental Backup, LSM and among others. For the last months Eyal also took the role of the team's scrum master and maintaining of our team's Kanban board and tasks. Please join me in congratulating Eyal!

6 5

'dnf update qemu-kvm' fails: libspice-server, libseccomp
by Yedidyah Bar David 20 Jan '20

20 Jan '20

Hi all, I have a CentOS8 VM, last updated (successfully) a week ago. Now, 'dnf update' fails with many errors. 'dnf update ovirt-release-master' worked but didn't help. Trying to isolate most/all of them by 'dnf update qemu-kvm', I get: Problem: package qemu-kvm-15:4.1.0-16.el8.x86_64 requires qemu-kvm-core = 15:4.1.0-16.el8, but none of the providers can be installed - cannot install the best update candidate for package qemu-kvm-15:2.12.0-65.module_el8.0.0+189+f9babebb.5.x86_64 - nothing provides libspice-server.so.1(SPICE_SERVER_0.14.2)(64bit) needed by qemu-kvm-core-15:4.1.0-16.el8.x86_64 - nothing provides libseccomp >= 2.4.0 needed by qemu-kvm-core-15:4.1.0-16.el8.x86_64 Known issue? 'dnf update --nobest' does seem to work (posting this email while it still downloads the appliance), and says: ===================================================================== Skipping packages with conflicts: (add '--best --allowerasing' to command line to force their upgrade): libguestfs x86_64 1:1.40.2-14.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 2.7 M libguestfs-tools-c x86_64 1:1.40.2-14.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 5.6 M qemu-img x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 1.8 M qemu-kvm-block-curl x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 93 k qemu-kvm-block-gluster x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 95 k qemu-kvm-block-iscsi x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 101 k qemu-kvm-block-rbd x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 95 k qemu-kvm-block-ssh x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 96 k qemu-kvm-common x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 1.2 M vdsm-http noarch 4.40.0-1513.git0adaae655.el8 ovirt-master-snapshot 14 k vdsm-http noarch 4.40.0-1518.gitfde24a6b5.el8 ovirt-master-snapshot 14 k Skipping packages with broken dependencies: ovirt-imageio-daemon noarch 1.6.3-0.el8 ovirt-master-snapshot 39 k qemu-kvm x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 82 k qemu-kvm-core x86_64 15:4.1.0-16.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 3.3 M vdsm x86_64 4.40.0-1513.git0adaae655.el8 ovirt-master-snapshot 1.3 M vdsm x86_64 4.40.0-1518.gitfde24a6b5.el8 ovirt-master-snapshot 1.3 M vdsm-hook-ethtool-options noarch 4.40.0-1518.gitfde24a6b5.el8 ovirt-master-snapshot 9.2 k vdsm-hook-fcoe noarch 4.40.0-1518.gitfde24a6b5.el8 ovirt-master-snapshot 9.6 k vdsm-hook-vmfex-dev noarch 4.40.0-1518.gitfde24a6b5.el8 ovirt-master-snapshot 10 k virt-v2v x86_64 1:1.40.2-14.el8 copr:copr.fedorainfracloud.org:sbonazzo:AdvancedVirtualization 13 M ===================================================================== Best regards, -- Didi

2 1

OST CI: Time issue?
by Yedidyah Bar David 20 Jan '20

20 Jan '20

Hi all, I looked at failure of [1]. It timed out while waiting for the engine to be up, after restarting the engine VM. lago.log [2] has: 2020-01-20 04:16:06,026::008_restart_he_vm.py::_start_he_vm::173::root::INFO::Starting VM... ... 2020-01-20 04:16:09,823::008_restart_he_vm.py::_start_he_vm::178::root::INFO::Waiting for VM to be UP... ... 2020-01-20 04:16:25,113::008_restart_he_vm.py::_start_he_vm::186::root::INFO::VM is UP. 2020-01-20 04:16:25,113::008_restart_he_vm.py::_wait_for_engine_health::190::root::INFO::Waiting for engine to start... ... Then there is a loop of running 'hosted-engine --vm-status --json' and parsing the status, specifically checking the health. The health status does not change until the timeout (after 10 minutes), but after 'vm', there is 'detail', which does change. First one is 'Powering up': 2020-01-20 04:16:25,113::ssh.py::ssh::89::lago.ssh::DEBUG::Command 9ef791c0 on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5199 (Sun Jan 19 23:16:24 2020)\nhost-id=1\nscore=2400\nvm_conf_refresh_time=5199 (Sun Jan 19 23:16:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "Powering up"}, "score": 2400, "stopped": false, "maintenance": false, "crc32": "e7f42f2a", "local_conf_timestamp": 5199, "host-ts": 5199}, "2": {"conf_on_shared_storage": true, "live-data": false, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5190 (Sun Jan 19 23:16:15 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5190 (Sun Jan 19 23:16:15 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c3657e8b", "local_conf_timestamp": 5190, "host-ts": 5190}, "global_maintenance": true} ... Then a few similar ones, last one is from '2020-01-20 04:17:22,980', and then it changes to 'Up': 2020-01-20 04:17:27,517::ssh.py::ssh::89::lago.ssh::DEBUG::Command c426f53a on lago-he-basic-role-remote-suite-4-3-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5259 (Sun Jan 19 23:17:24 2020)\nhost-id=1\nscore=2928\nvm_conf_refresh_time=5259 (Sun Jan 19 23:17:24 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"}, "score": 2928, "stopped": false, "maintenance": false, "crc32": "08d4bfe1", "local_conf_timestamp": 5259, "host-ts": 5259}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5250 (Sun Jan 19 23:17:16 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5251 (Sun Jan 19 23:17:16 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ab277249", "local_conf_timestamp": 5251, "host-ts": 5250}, "global_maintenance": true} /var/log/messages of the engine vm [3] has: Jan 19 23:17:57 lago-he-basic-role-remote-suite-4-3-engine kernel: Initializing cgroup subsys cpuset Meaning, the first line after the reboot has a timestamp that's almost 2 minutes after starting the VM. That's a long time. That's half a minute after detail changed to 'Up', above (no idea where we take this 'detail' from, though). Then, first line that is not from the kernel: Jan 19 23:17:58 lago-he-basic-role-remote-suite-4-3-engine systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) then it starts lots of services, and then: Jan 19 23:20:31 lago-he-basic-role-remote-suite-4-3-engine systemd: Started oVirt Engine websockets proxy. Jan 19 23:20:33 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 45.33.103.94 Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock wrong by 3.919225 seconds, adjustment started Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine systemd: Time has been changed Jan 19 23:20:37 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: System clock was stepped by 3.919225 seconds Jan 19 23:20:38 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:20:43 lago-he-basic-role-remote-suite-4-3-engine cloud-init: Cloud-init v. 18.5 running 'modules:config' at Mon, 20 Jan 2020 04:20:39 +0000. Up 182.82 seconds. and later: Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Started Update UTMP about System Runlevel Changes. Jan 19 23:21:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Startup finished in 22.344s (kernel) + 32.237s (initrd) + 3min 815ms (userspace) = 3min 55.397s. Jan 19 23:21:43 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 206.55.191.142 Jan 19 23:23:52 lago-he-basic-role-remote-suite-4-3-engine chronyd[872]: Selected source 129.250.35.250 Jan 19 23:26:32 lago-he-basic-role-remote-suite-4-3-engine systemd: Created slice User Slice of root. engine.log's [4] first line after the reboot is: 2020-01-19 23:26:18,251-05 INFO [org.ovirt.engine.core.uutils.config.ShellLikeConfd] (ServerService Thread Pool -- 103) [] Loaded file '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf'. which is a mere 40 seconds before the timeout, and server.log's [5] first line is: 2020-01-19 23:23:13,392-05 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "apidoc.war" (runtime-name: "apidoc.war") which is a 3 minutes before that. I do not know much about what the engine does during its startup, and how the logs should look, and how long it should take between startup and a working health page. If this all looks normal, then perhaps we should simply give it more than 10 minutes. Otherwise, there is either heavy load on the infra, or perhaps some NTP problems. Anyone has a clue? Anyway, now pushed a patch [6] to allow up to 20 minutes, at least to reduce the noise. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [5] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… [6] https://gerrit.ovirt.org/106411 -- Didi

1 0

CI: Multiple jobs hang on "Archiving Artifacts"
by Amit Bawer 19 Jan '20

19 Jan '20

For the last 40 minutes, multiple CI jobs are hanging on "Archiving Artifacts" stage: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/ could be a false alarm, but none seem to progress.

2 2

maven central brings some surprises
by Fedor Gavrilov 17 Jan '20

17 Jan '20

Hi, Maven central seems to be not in the mood today for me: [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.2: Could not transfer artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.2 from/to central (http://repo1.maven.org/maven2) Failed to transfer file: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2…. Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1] Does anyone know if that's somehow a proxy thing or something else? I'm asking because I had a joy of dealing with lying proxy before. Also, iirc jcenter had a reputation of being more reliable, can we use it instead somehow? Thanks, Fedor

3 2

Restore fails whether engine is on or off
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

Wondering whether this is an issue; I had an engine where i created a backup and stored elsewhere. This engine was reinstalled and the backup file (called file_name) used to restore below. However i get 2 different errors based on whether engine is on.... [root@dub-mgrfarm113 ~]# service ovirt-engine status Redirecting to /bin/systemctl status ovirt-engine.service ● ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2020-01-16 16:13:41 GMT; 2h 5min ago Main PID: 5621 (ovirt-engine.py) CGroup: /system.slice/ovirt-engine.service ├─5621 /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify st... └─5661 ovirt-engine -server -XX:+TieredCompilation -Xms1991M -Xmx1991M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.clien... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com systemd[1]: Starting oVirt Engine... Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:40,816+0000 ovirt-engine: INFO _detectJBossV...l=36 Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16 16:13:41,795+0000 ovirt-engine: INFO _detectJBossV...'[]' Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com systemd[1]: Started oVirt Engine. Hint: Some lines were ellipsized, use -l to show in full. [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181932.log Preparing to restore: FATAL: Engine service is active - can not restore backup or off....... [root@dub-mgrfarm113 ~]# service ovirt-engine stop Redirecting to /bin/systemctl stop ovirt-engine.service [root@dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name --restore-permissions Start of engine-backup with mode 'restore' scope: all archive file: file_name log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181952.log Preparing to restore: - Unpacking file 'file_name' Restoring: - Files FATAL: Can't connect to database 'engine'. Please see '/usr/bin/engine-backup --help'. In addition, possibly related, the web page for the server will no longer launch even when ovirt-engine and httpd services are running while. Any troubleshooting tips or has anyone seen same before?

1 0

FYI for Intellij IDEA & CentOS 7 users
by Fedor Gavrilov 16 Jan '20

16 Jan '20

Hi, Just wanted to share in case someone will have same issue as me. Recent version of IDEA requires new libgcc which is not available for CentOS 7, which has 2.17: Exception in thread "JavaFX Application Thread" java.lang.UnsatisfiedLinkError: /home/fgavrilo/Software/idea-IC-193.5662.53/jbr/lib/libjfxmedia.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found The simplest solution to this is probably using flatpak distribution: sudo yum install flatpak flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo flatpak install flathub com.jetbrains.IntelliJ-IDEA-Community flatpak run com.jetbrains.IntelliJ-IDEA-Community Fedor

1 0

REST and Python APIs -- jobs and disk updates
by Martin Kletzander 16 Jan '20

16 Jan '20

Hello everyone, so I am trying to do some work related to oVirt and I am handling disks using the ovirtsdk4 python API. However I am not sure about how to do two things. First one is about avoiding an active wait or at least minimizing the guesswork. When creating a new disk I have to then wait in a loop until it is ready, polling it and checking if it is in 'OK' status. However the creation returns a job which has way better information on what is happening (is it still running? did it fail?) and I was wondering if there is a way how to get the job information using the Python SDK. Another thing is that, apparently, updating a disk only works for updating the qcow_version and nothing else. And I need to update its shareable attribute, which is completely possible using the web interface. What is the proper way to do that using REST APIs or the Python SDK? Thanks for any info, Martin

3 3

How to create IP address for a VM in Ovirt
by raymond.francis＠oracle.com 16 Jan '20

16 Jan '20

How can i configure an IP address on a NIC interface for a VM. Need to set it to DHCP to do it but im not spotting the option?

1 0

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls
by Sandro Bonazzola 15 Jan '20

15 Jan '20

Please review https://github.com/oVirt/ovirt-ansible-shutdown-env/pulls Thanks, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: [PATCH 0/4][RFC] Drop media update support in virDomainAttachDevice*
by Daniel P. Berrangé 15 Jan '20

15 Jan '20

On Wed, Jan 15, 2020 at 04:24:54PM +0800, Han Han wrote: > https://bugzilla.redhat.com/show_bug.cgi?id=1788793 > > Before virDomainUpdateDeviceFlags, virDomainAttachDevice* were used for > media update. Now we have introduced virDomainUpdateDeviceFlags for 9 > years(since commit 46a2ea3689) but keep that compatibility of > virDomainAttachDevice* until now. > > I think it is time to remove the compatibility since it doesn't work as > well as virDomainUpdateDeviceFlags on disk xml validation. > > I wonder if any uplayer software relying on this compatibility... NACK, this is explicitly the kind of thing we will never do. Providing a long term stable API means keeping existing methods operating unchanged, even if we have since introduced a new method which can do the same thing in a better way. The cost of keeping this existing code is negligible in context to the maint work of libvirt in general. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

1 0

planned Jenkins restart
by Evgheni Dereveanchin 14 Jan '20

14 Jan '20

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is back online. -- Regards, Evgheni Dereveanchin

1 1

OST manual suite fails for more than a month
by Tal Nisan 14 Jan '20

14 Jan '20

Ran it myself now on master just to have a baseline [1], it fails with: *17:34:58* # verify_add_all_hosts: ERROR (in 0:20:07) <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…>*17:34:58* @ Run test: 002_bootstrap.py: ERROR (in 0:27:56) <https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…>*17:34:58* Error occured, aborting Who can have a look? [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-test…

3 2

[ ERROR ] Failed to execute stage 'Setup validation': Aborted by user
by raymond.francis＠oracle.com 14 Jan '20

14 Jan '20

Not seen this message googling, im the only user too. Not sure whats up here engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213536-k7u9m1.log Version: otopi-1.8.3 (otopi-1.8.3-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization Do you want to remove all components? (Yes, No) [Yes]: --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213536-k7u9m1.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200113213548-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed [root@dub-mgrfarm113 ~]# engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213556-ao8uv5.log Version: otopi-1.8.3 (otopi-1.8.3-1.el7) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization Do you want to remove all components? (Yes, No) [Yes]: --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20200113213556-ao8uv5.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200113213601-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed

2 1

OST: Is it still possible to run it without mock?
by Milan Zamazal 13 Jan '20

13 Jan '20

Hi, when trying to run basic-suite-master using run_suite.sh on my machine, I get the following error: /home/pdm/ovirt-system-tests/run_suite.sh: line 532: /var/lib/ci_toolbox/safe_download.sh: No such file or directory ci_toolbox/safe_download.sh is in Jenkins repo -- is mock now required to run OST now or is it just a mistake (either in OST or on my side)? Thanks, Milan

2 4

Cannot delete disk despite all disk permissions given to user
by raymond.francis＠oracle.com 13 Jan '20

13 Jan '20

Possible bug i found in testing (or could be my own understanding). If i give a user the the following capabiities, wouldnt it stand to reason that he can then delete a disk? StorageAdmin DiskCreator DiskProfileUser SuperUser DiskOperator I get a permissions issue if i then try to delete a disk. Let me know if any logs etc needed. Thanks Raymond

1 1

openstack glace images credentials
by Dana Elfassy 12 Jan '20

12 Jan '20

Hi, I'm trying to get the credentials for several images on ovirt-image-repository (OpenStack Glance) on storage domains in the webadmin Who can help me with this? Thanks, Dana

3 3

oVirt and Gluster Hyperconverged - VM's Bad volume specification
by thomas.rockey＠datasphere.com 09 Jan '20

09 Jan '20

I am new to this forum so I apologize before hand if I don’t present the right content you are looking for or miss content you need. Background: By no means am I an expert with Ovirt and glusterfs. That said I have been using, managing, building out Ovirt (single hosts) and oVirt with Gluster Hyperconverged environments for 5 years or more. I started building out Ovirt environments with oVirt Engine Version: 3.6.7.5-1.el6 and earlier and now I’m using the latest oVirt with Gluster Hyperconverged. Current hardware and software layout: For the last 8 months I have been using a oVirt with Gluster Hyperconverged to host in total about 100 VM’s. My hardware layout in one environment is 5 Dell R410 3 of them configured with Gluster Hyperconverged and the other 2 are just added hosts. Below is a detailed list! Manufacturer: Dell Inc. PowerEdge R410 CPU Model Name: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz CPU Cores per Socket: 4 CPU Type: Intel Westmere IBRS SSBD Family Dell PERC H700 4 SAS Seagate 4 TB drives 7.2k 2 one gig links – NIC 1 for frontend and NIC 2 for gluster backend My software layout is: OS Version: RHEL - 7 - 7.1908.0.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 1062.9.1.el7.x86_64 KVM Version: 2.12.0 - 33.1.el7_7.4 LIBVIRT Version: libvirt-4.5.0-23.el7_7.3 VDSM Version: vdsm-4.30.38-1.el7 SPICE Version: 0.14.0 - 7.el7 GlusterFS Version: glusterfs-6.6-1.el7 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.11.0-4.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 VNC Encryption: Disabled My network layout is: 3 HP 3800-48G-4SFP+ Switch (J9576A) running FULL MESH Issue/timeline: • All 3 of the HP 3800 were rebooted at the same time and were down for 5 to 10 seconds before they came back up (meaning pingable and responsive). • A little more than 85% (36 or so) of the VM’s I had running all went into a pause state, do to and unknow storage error. • The gluster volume heal state went all the way up to 2300 on vmstore (OS data location) • After heal completed on the vmstore (took about an hour) 85% of the VM’s failed to launch with an error (see below). VM broadsort is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'b1bf3f56-a453-4383-a350-288bee06445b', 'index': 0, 'iface': 'virtio', 'apparentsize': '274877906944', 'specParams': {}, 'cache': 'none', 'imageID': 'b1bf3f56-a453-4383-a350-288bee06445b', 'truesize': '106767498240', 'type': 'disk', 'domainID': 'a7119613-a5ba-4a97-802b-0a985c647381', 'reqsize': '0', 'format': 'raw', 'poolID': '699fd2d6-c461-11e9-8b83-00163e18a045', 'device': 'disk', 'path': 'vmstore/a7119613-a5ba-4a97-802b-0a985c647381/images/b1bf3f56-a453-4383-a350-288bee06445b/25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'diskType': 'network', 'alias': 'ua-b1bf3f56-a453-4383-a350-288bee06445b', 'hosts': [{'name': 'glust01.mydomain.local', 'port': '0'}], 'discard': False}. Everyone of the VM’s had this same error and I had to find backups and old images to bring them back online. I deleted some of the corrupted VM’s that I had current images of, to get them back up. You shouldn’t be afraid to reboot 1, 2, 3, or even all of your switches at once because of a human error, power outage, or a simple update. Then have to worry about your VM’s getting corrupted because of this concerns me greatly. I am now thinking that I didn't setup oVirt with Gluster Hyperconverged correctly because of this issue. Have I missed something in the documentation or network layout/setup that would prevent this from happening again? I have searched the web for a few days now trying to find threads related to my situation with no luck. I want to thank you for your time and it is greatly appreciated!

1 0

oVirt and Gluster Hyperconverged - VM's Bad volume specification
by thomas.rockey＠datasphere.com 09 Jan '20

09 Jan '20

I am new to this forum so I apologize before hand if I don’t present right content correctly or miss the content you need. Background: By no means am I an expert with Ovirt and glusterfs. That said I have been using, managing, building out Ovirt (single hosts) and oVirt with Gluster Hyperconverged environments for 5 years or more. I started building out Ovirt environments with oVirt Engine Version: 3.6.7.5-1.el6 and earlier and now I’m using the latest oVirt with Gluster Hyperconverged. Current hardware and software layout: For the last 8 months I have been using a oVirt with Gluster Hyperconverged to host in total about 100 VM’s. My hardware layout in one environment is 5 Dell R410 3 of them configured with Gluster Hyperconverged and the other 2 are just added hosts. Below is a detailed list! Manufacturer: Dell Inc. PowerEdge R410 CPU Model Name: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz CPU Cores per Socket: 4 CPU Type: Intel Westmere IBRS SSBD Family Dell PERC H700 4 SAS Seagate 4 TB drives 7.2k 2 one gig links – NIC 1 for frontend and NIC 2 for gluster backend My software layout is: OS Version: RHEL - 7 - 7.1908.0.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 1062.9.1.el7.x86_64 KVM Version: 2.12.0 - 33.1.el7_7.4 LIBVIRT Version: libvirt-4.5.0-23.el7_7.3 VDSM Version: vdsm-4.30.38-1.el7 SPICE Version: 0.14.0 - 7.el7 GlusterFS Version: glusterfs-6.6-1.el7 CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.11.0-4.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 VNC Encryption: Disabled My network layout is: 3 HP 3800-48G-4SFP+ Switch (J9576A) running FULL MESH Issue/timeline: • All 3 of the HP 3800 were rebooted at the same time and were down for 5 to 10 seconds before they came back up (meaning pingable and responsive). • A little more than 85% (36 or so) of the VM’s I had running all went into a pause state, do to and unknow storage error. • The gluster volume heal state went all the way up to 2300 on vmstore (OS data location) • After heal completed on the vmstore (took about an hour) 85% of the VM’s failed to launch with an error (see below). VM broadsort is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'b1bf3f56-a453-4383-a350-288bee06445b', 'index': 0, 'iface': 'virtio', 'apparentsize': '274877906944', 'specParams': {}, 'cache': 'none', 'imageID': 'b1bf3f56-a453-4383-a350-288bee06445b', 'truesize': '106767498240', 'type': 'disk', 'domainID': 'a7119613-a5ba-4a97-802b-0a985c647381', 'reqsize': '0', 'format': 'raw', 'poolID': '699fd2d6-c461-11e9-8b83-00163e18a045', 'device': 'disk', 'path': 'vmstore/a7119613-a5ba-4a97-802b-0a985c647381/images/b1bf3f56-a453-4383-a350-288bee06445b/25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '25b0ab77-8f4c-42a1-9416-27db4cd25b39', 'diskType': 'network', 'alias': 'ua-b1bf3f56-a453-4383-a350-288bee06445b', 'hosts': [{'name': 'glust01.mydomain.local', 'port': '0'}], 'discard': False}. Everyone of the VM’s had this same error and I had to find backups and old images to bring them back online. I deleted some of the VM’s that I had current images of to get them back up. You shouldn’t be afraid to reboot 1, 2, 3, or even all of your switches at once because of a human error, power outage, or a simple update. Then have to worry about your VM’s getting corrupted because of this concerns me greatly that I didn’t setup oVirt with Gluster Hyperconverged correctly. Have I missed something in the documentation or network layout that would prevent this for happing? I want to thank you for your time and it is greatly appreciated!

1 0

Failed to add host using ansible runner
by Eyal Shenitzky 09 Jan '20

09 Jan '20

Hi, I am failing to add a new host to my env with the following message: 2019-12-25 10:57:17,587+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-63) [41ec72c1-88e2-402b-8bb9-f38c678d0bf0] EVENT_ID: VDS_INSTALL_FAILED(505), Host 10.35.0.158 installation failed. Failed to execute Ansible host-deploy role: null. Please check logs for more details: /home/engine/ovirt-engine/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20191225105714-10.35.0.158-41ec72c1-88e2-402b-8bb9-f38c678d0bf0.log. The host is Fedora-30 Ansible version is - 2.9.1 Ansible runner version - 1.3.4 /etc/ansible-runner-service/config.yaml: --- version: 1 target_user: root playbooks_root_dir: '/home/engine/ovirt-engine/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '/home/engine/ovirt-engine/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root There are no logs at all at the specified location in the error message. Did someone encounter that issue? Thanks, -- Regards, Eyal Shenitzky

6 13

New dependency for development environment
by Ondra Machacek 09 Jan '20

09 Jan '20

Hello, we are going to merge a series of patches to master branch, which integrates ansible-runner with oVirt engine. When the patches will be merged you will need to install new package called ansible-runner- service-dev, and follow instructions so your dev-env will keep working smoothly(all relevant info will be also in README.adoc): 1) sudo dnf update ovirt-release-master 2) sudo dnf install -y ansible-runner-service-dev 3) Edit `/etc/ansible-runner-service/config.yaml` file as follows: --- playbooks_root_dir: '$PREFIX/share/ovirt-engine/ansible-runner-service-project' ssh_private_key: '$PREFIX/etc/pki/ovirt-engine/keys/engine_id_rsa' port: 50001 target_user: root Where `$PREFIX` is the prefix of your development environment prefix, which you've specified during the compilation of the engine. 4) Restart and enable ansible-runner-service: # systemctl restart ansible-runner-service # systemctl enable ansible-runner-service That's it, your dev-env should start using the ansible-runner-service for host-deployment etc. Please note that only Fedora 30/31 and Centos7 was packaged, and are natively supported! Thanks, Ondra

9 24

Failed to install ovirt-imageio-proxy for fc30
by Shani Leviim 08 Jan '20

08 Jan '20

Hi All, I ran into a dependencies issue while trying to install imageio over fc30. I currently have a ovirt-imageio-common-1.6.2-0.202001051603.git77cf282.fc30.x86_64 but when I'm trying to install ovirt-imageio-proxy, I'm getting the following error: Error: Problem: conflicting requests - nothing provides ovirt-imageio-common = 1.5.0 needed by ovirt-imageio-proxy-1.5.0-0.201812112104.gitf7d8234.fc30.noarch I've also noticed that the available version in the repository for fc30 is ovirt-imageio-proxy-1.5.0 [1] While version 1.6.2 is available for rhel7 [2]. Please assist. [1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc30/noarch/ovirt… [2] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ovirt-… *Regards,* *Shani Leviim*

2 2

ovirt-engine has been tagged (ovirt-engine-4.3.8.2)
by Tal Nisan 07 Jan '20

07 Jan '20

1 0

OST is failing - Last successful run was Dec-13-2019
by Amit Bawer 06 Jan '20

06 Jan '20

Seems we have NFS permissions issue for el8 vdsm in some of the runs. Example from https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/arti… : 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1 (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)... 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata (monitor:501) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 499, in _pathChecked delay = result.delay() File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391, in delay raise exception.MiscFileReadException(self.path, self.rc, self.err) vdsm.storage.exception.MiscFileReadException: Internal file read failure: ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata', 1, bytearray(b"/usr/bin/dd: failed to open \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\': Operation not permitted\n")) 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer] Calling 'StoragePool.disconnect' in bridge with {'storagepoolID': 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey': 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (__init__:329) 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439', hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786, flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48) Probably need to set NFS server export options as in https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7

2 2

Unexpected exception when trying to add a storage domain
by Dana Elfassy 05 Jan '20

05 Jan '20

Hi, When trying to add a storage domain to a 4.4 host I'm getting this error message: Error while executing action New NFS Storage Domain: Unexpected exception The errors from vdsm.log: 2020-01-02 09:38:33,578-0500 ERROR (jsonrpc/0) [storage.initSANLock] Cannot initialize SANLock for domain 6ca1e203-5595-47e5-94b8-82a7e69d99a9 (clusterlock:259) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') 2020-01-02 09:38:33,579-0500 INFO (jsonrpc/0) [vdsm.api] FINISH createStorageDomain error=Could not initialize cluster lock: () from=::ffff:192.168.100.1,36452, flow_id=17dc614 e, task_id=05c2107a-4d59-48d0-a2f7-0938f051c9ab (api:52) 2020-01-02 09:38:33,582-0500 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') Unexpected error (task:874) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 250, in initSANLock lockspace_name, idsPath, align=alignment, sector=block_size) sanlock.SanlockException: (-202, 'Sanlock lockspace write failure', 'IO timeout') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 881, in _run return fn(*args, **kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2622, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 120, in create fsd.initSPMlease() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 974, in initSPMlease return self._manifest.initDomainLock() File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 620, in initDomainLock self._domainLock.initLock(self.getDomainLease()) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 308, in initLock block_size=self._block_size) File "/usr/lib/python3.6/site-packages/vdsm/storage/clusterlock.py", line 260, in initSANLock raise se.ClusterLockInitError() vdsm.storage.exception.ClusterLockInitError: Could not initialize cluster lock: () 2020-01-02 09:38:33,583-0500 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='05c2107a-4d59-48d0-a2f7-0938f051c9ab') aborting: Task is aborted: 'value=Could not initialize cluster lock: () abortedcode=701' (task:1184) 2020-01-02 09:38:33,584-0500 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Could not initialize cluster lock: () (dispatcher:83) 2020-01-02 09:38:33,584-0500 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 701) in 18.70 seconds (__init__:312) 2020-01-02 09:38:33,730-0500 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'protocol_version': 'auto', 'port': '', 'iqn': '', 'connection': 'vserver-spider.eng.lab.tlv.redhat.com:/pub/delfassy/nfs1', 'ipv6_enabled': 'false', 'id': 'bf075967-060b-4e60-9b0b-bb170fe073f9', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:192.168.100.1,36452, flow_id=f4f79738-b361-4688-95c5-3454f52b505d, task_id=dd7266ed-69ec-4e5b-9216-850d9db8ea3b (api:48) Can someone help me with it? Thanks, Dana

2 4

Who's the owner of "vdsm_hooks/checkimages/before_vm_start.py"?
by Pavel Bar 03 Jan '20

03 Jan '20

Hi, Can someone please point me to a relevant person? It looks like there is a potential issue with "*getImageSize()*" function there: *'image_bytes' might be referenced before assignment*. There is also a suspicion that this code is not used at all, so instead of fixing it might be a good idea to just delete it. See the code below: def getImageSize(disk_image, driver_type): ''' Obtain qcow2 image size in GiBs ''' if driver_type == 'block': dev_buffer = ' ' * 8 with open(disk_image) as device: dev_buffer = fcntl.ioctl(device.fileno(), BLKGETSIZE64, dev_buffer) image_bytes = struct.unpack(FORMAT, dev_buffer)[0] elif driver_type == 'file': image_bytes = os.stat(disk_image).st_size return float(image_bytes / GIB) Thank you in advance! Pavel

2 1

OST Fails for missing glusterfs mirrors at host-deploy
by Amit Bawer 02 Jan '20

02 Jan '20

Snippet From: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6293/console 23:31:25 + cd /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master 23:31:25 + lago ovirt deploy 23:31:26 @ Deploy oVirt environment: 23:31:26 # Deploy environment: 23:31:26 * [Thread-2] Deploy VM lago-basic-suite-master-host-0: 23:31:26 * [Thread-3] Deploy VM lago-basic-suite-master-host-1: 23:31:26 * [Thread-4] Deploy VM lago-basic-suite-master-engine: 23:32:15 * [Thread-3] Deploy VM lago-basic-suite-master-host-1: Success (in 0:00:49) 23:32:39 STDERR 23:32:39 + yum -y install ovirt-host 23:32:39 Error: Error downloading packages: 23:32:39 Cannot download glusterfs-6.6-1.el8.x86_64.rpm: All mirrors were tried 23:32:39 23:32:39 - STDERR 23:32:39 + yum -y install ovirt-host 23:32:39 Error: Error downloading packages: 23:32:39 Cannot download glusterfs-6.6-1.el8.x86_64.rpm: All mirrors were tried 23:32:39 23:32:39 * [Thread-2] Deploy VM lago-basic-suite-master-host-0: ERROR (in 0:01:13) 23:38:05 * [Thread-4] Deploy VM lago-basic-suite-master-engine: ERROR (in 0:06:39) 23:38:05 # Deploy environment: ERROR (in 0:06:39) 23:38:06 @ Deploy oVirt environment: ERROR (in 0:06:39) 23:38:06 Error occured, aborting 23:38:06 Traceback (most recent call last): 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 383, in do_run 23:38:06 self.cli_plugins[args.ovirtverb].do_run(args) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run 23:38:06 self._do_run(**vars(args)) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 573, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 584, in wrapper 23:38:06 return func(*args, prefix=prefix, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 181, in do_deploy 23:38:06 prefix.deploy() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 127, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/ovirtlago/prefix.py", line 284, in deploy 23:38:06 return super(OvirtPrefix, self).deploy() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper 23:38:06 return func(*args, **kwargs) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1671, in deploy 23:38:06 self.virt_env.get_vms().values() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 104, in invoke_in_parallel 23:38:06 return vt.join_all() 23:38:06 File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue 23:38:06 queue.put({'return': func()}) 23:38:06 File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1662, in _deploy_host 23:38:06 host.name(), 23:38:06 LagoDeployError: /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master/default/scripts/_home_jenkins_agent_workspace_ovirt-system-tests_manual_ovirt-system-tests_basic-suite-master_deploy-scripts_setup_1st_host_el7.sh failed with status 1 on lago-basic-suite-master-host-0 23:38:06 + res=1 23:38:06 + cd - 23:38:06 /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests 23:38:06 + return 1 23:38:06 + env_collect /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:06 + local tests_out_dir=/home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:06 + [[ -e /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master ]] 23:38:06 + mkdir -p /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master 23:38:06 + cd /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/deployment-basic-suite-master/current 23:38:06 + lago collect --output /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy 23:38:08 @ Collect artifacts: 23:38:08 # [Thread-1] lago-basic-suite-master-host-0: 23:38:08 # [Thread-2] lago-basic-suite-master-host-1: 23:38:08 # [Thread-3] lago-basic-suite-master-engine: 23:38:10 # [Thread-1] lago-basic-suite-master-host-0: Success (in 0:00:02) 23:38:10 # [Thread-2] lago-basic-suite-master-host-1: Success (in 0:00:02) 23:38:16 # [Thread-3] lago-basic-suite-master-engine: Success (in 0:00:07) 23:38:16 @ Collect artifacts: Success (in 0:00:07) 23:38:16 + cp -a logs /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests/test_logs/basic-suite-master/post-000_deploy/lago_logs 23:38:16 + cd - 23:38:16 /home/jenkins/agent/workspace/ovirt-system-tests_manual/ovirt-system-tests 23:38:16 + echo '@@@ ERROR: Failed in deploy stage' 23:38:16 @@@ ERROR: Failed in deploy stage

2 1