December 2019 - Devel - oVirt List Archives

CI system outage
by Evgheni Dereveanchin 17 Dec '19

17 Dec '19

Hello everyone, We received several reports of Jenkins jobs not running properly. The reason turned out to be the Jenkins Master which did not start jobs properly. It has been restarted and is working properly now. If you have any jobs that failed or did not start at all today - please re-trigger them or reach out to me for assistance. -- Regards, Evgheni Dereveanchin

1 0

published master engine is 13 days old
by Yedidyah Bar David 17 Dec '19

17 Dec '19

Hi all, $subject. [1] has ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch . Tried to look around, and I have a few notes/questions: 1. Last successful run of [2] is 3 days old, but apparently it wasn't published. Any idea why? 2. Failed runs of [2] are reported to infra, with emails such as: [CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause Is anyone monitoring these? Is this the only alerting that CI generates on such failures? If first is No and second is Yes, then we need someone/something to start monitoring. This was discussed a lot, but I do not see any change. Ideally, such alerts should be To'ed or Cc'ed to the author and reviewers of the patch that CI found to be guilty (which might be wrong, that's not the point). Do we plan to have something like this? Any idea when it will be ready? 3. I looked at a few recent failures of [2], specifically [3][4]. Both seem to have been killed after a timeout, while running 'engine-config'. For [3] that's clear, see [5]: 2019-12-16 17:11:44,766::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fb6611dc-55bb-4251-aeda-2578b2ec83a2:Get ssh client for lago-basic-suite-master-engine: 2019-12-16 17:11:44,931::ssh.py::ssh::58::lago.ssh::DEBUG::Running 22e2b6b6 on lago-basic-suite-master-engine: engine-config --set VdsmUseNmstate=true 2019-12-16 19:55:21,965::cmd.py::exit_handler::921::cli::DEBUG::signal 15 was caught Can't find stdout/stderr of engine-config, so it's hard to tell if it outputted anything helpful to understand why it was stuck. It's hard to tell that about [4], because it has very few artifacts collected, no idea why, notably no lago.log, but [6] does show: [36m # initialize_engine: [32mSuccess [0m (in 0:04:00) [0m [36m # engine_config: [0m [0m [0m [36m * Collect artifacts: [0m [0m [0m [36m - [Thread-34] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:04) [0m [36m * Collect artifacts: [31mERROR [0m (in 0:00:04) [0m [36m # engine_config: [31mERROR [0m (in 2:42:57) [0m /bin/bash: line 31: 5225 Killed ${_STDCI_TIMEOUT_CMD} "3h" "$script_path" < /dev/null If I run 'engine-config --set VdsmUseNmstate=true' on my 20191204120550.git04d5d05 engine, it returns quickly. Tried also adding a repo pointing at last successful run of [7], which is currently [8], and it prompts me to input a version, probably as a result of [9]. Ales/Martin, can you please have a look? Thanks. [1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/ [2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/ [3] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/ [4] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/ [5] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/artifa… [6] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/artifa… [7] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/ [8] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/384/ [9] https://gerrit.ovirt.org/105440 -- Didi

3 3

Failed to start Vagrant
by Ritesh Chikatwar 16 Dec '19

16 Dec '19

Hello, [root@localhost vagrant]# vagrant reload Error while connecting to libvirt: Error making a connection to libvirt URI qemu:///system?no_verify=1&keyfile=/root/.ssh/id_rsa: Call to virConnectOpen failed: authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) Status Of libvirtd https://pastebin.com/H7yNqXU7 (It is active) I cannot see any logs in /var/log/libvirt/qemu/vagrant_default.log My Vagrant file: https://pastebin.com/JXBd5ci1 Any help would be appreciated. Ritesh CHikatwar

1 0

oVirt 4.4.0 Alpha release is now available for testing
by Sandro Bonazzola 13 Dec '19

13 Dec '19

oVirt 4.4.0 Alpha release is now available for testing The oVirt Project is excited to announce the availability of the alpha release of oVirt 4.4.0 for testing, as of December 13th, 2019 This release unleashes an altogether more powerful and flexible open source virtualization solution that encompasses hundreds of individual changes and a wide range of enhancements across the engine, storage, network, user interface, and analytics on top of oVirt 4.3. Important notes before you try it Please note this is an Alpha release. The oVirt Project makes no guarantees as to its suitability or usefulness. This pre-release must not to be used in production, and it is not feature complete. In particular, please note that upgrades from 4.3 and future upgrades from this alpha to the final 4.4 release from this version are not supported. Some of the features included in oVirt 4.4.0 Alpha require content that will be available in CentOS Linux 8.1 which is currently being built from Red Hat Enterprise Linux 8.1. If you want to have a better experience you can test oVirt 4.4.0 Alpha on Red Hat Enterprise Linux 8.1. Known Issues - After installation open the Default cluster and hit “Save”, for any other new Cluster using CPU autodetection the dialog needs to be explicitly saved after the detection happens, after first host is added. (bug https://bugzilla.redhat.com/1770697) - Migrate, Cluster Upgrade, Export VM buttons cannot be clicked. Fixed in ovirt-engine-ui-extensions-1.0.11 Installation instructions For the engine: either use appliance or: - Install CentOS Linux 7.7 minimal from http://centos.mirror.garr.it/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Mi… - yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm - yum update (reboot if needed) - yum install ovirt-engine - engine-setup For the nodes: Either use oVirt Node ISO or: - Install CentOS Linux 8.0 from http://centos.mirror.garr.it/centos/8.0.1905/isos/x86_64/CentOS-8-x86_64-19… ; select minimal installation - yum config-manager --set-enabled PowerTools - yum install https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm - yum update (reboot if needed) - Attach the host to engine and let it be deployed. What’s new in oVirt 4.4.0 Alpha? - Hypervisors based on CentOS Linux 8 (rebuilt from award winning RHEL8), for both oVirt Node and standalone CentOS Linux hosts - Easier network management and configuration flexibility with NetworkManager - VMs based on a more modern Q35 chipset with legacy seabios and UEFI firmware - Support for direct passthrough of local host disks to VMs - Live migration improvements for High Performance guests. - New Windows Guest tools installer based on WiX framework now moved to VirtioWin project - Dropped support for cluster level prior to 4.2 - Dropped SDK3 support - 4K disks support - Exporting a VM to a data domain - Editing of floating disks - Integrating ansible-runner into engine, which allows a more detailed monitoring of playbooks executed from engine - Adding/reinstalling hosts are now completely based on Ansible - The OpenStack Neutron Agent cannot be configured by oVirt anymore, it should be configured by TripleO instead This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but < 8.0) * CentOS Linux (or similar) 7.7 or later (but < 8.0) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 8.0 or later * CentOS Linux (or similar) 8.0 or later * oVirt Node 4.4 based on CentOS Linux 8.0 (available for x86_64 only) See the release notes [1] for installation instructions and a list of new features and bugs fixed. If you manage more than one oVirt instance, OKD or RDO we also recommend to try ManageIQ <http://manageiq.org/>. In such a case, please be sure to take the qc2 image and not the ova image. Notes: - oVirt Appliance is already available for CentOS Linux 8 - oVirt Node NG is already available for CentOS Linux 8 Additional Resources: * Read more about the oVirt 4.4.0 release highlights: http://www.ovirt.org/release/4.4.0/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.0/ [2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

2 2

CI: jsonrpcserver test fails
by Amit Bawer 11 Dec '19

11 Dec '19

Hi devel, We have (also) frequent connectivity/timeout failures on CI tests for jsonrpcserver tests. Example: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/15642//artifact/che… Proposing the following patches: https://gerrit.ovirt.org/#/c/105518/ https://gerrit.ovirt.org/#/c/105519/ You are most welcome to review or suggest alternative ones. Thanks.

2 3

Vdsm/CI: Failed to synchronize cache for repo 'epel-el8'
by Milan Zamazal 11 Dec '19

11 Dec '19

Hi, I've seen this error more than once in Jenkins runs on Vdsm patches posted to gerrit: + python3 tests/profile debuginfo-install debuginfo-install -y python3 Error: Failed to synchronize cache for repo 'epel-el8' PROFILE {"command": ["debuginfo-install", "-y", "python3"], "cpu": 25.8319700558958, "elapsed": 39.049031019210815, "idrss": 0, "inblock": 0, "isrss": 0, "ixrss": 0, "majflt": 0, "maxrss": 92900, "minflt": 35674, "msgrcv": 0, "msgsnd": 0, "name": "debuginfo-install", "nivcsw": 1096, "nsignals": 0, "nswap": 0, "nvcsw": 904, "oublock": 64848, "start": 1575653575.116399, "status": 1, "stime": 1.135238, "utime": 8.951896} + teardown + res=1 + '[' 1 -ne 0 ']' + echo '*** err: 1' *** err: 1 See e.g. https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… Does anybody know what's going on and how to remedy it? Thanks, Milan

4 9

Vdsm/CI: TimeoutError in VdsmClientTests
by Milan Zamazal 11 Dec '19

11 Dec '19

Hi, this seems to be a frequent error in Jenkins run recently: =================================== FAILURES =================================== ______________________ VdsmClientTests.test_failing_call _______________________ self = <stomprpcclient_test.VdsmClientTests testMethod=test_failing_call> def test_failing_call(self): with self._create_client() as client: with self.assertRaises(ServerError) as ex: > client.Test.failingCall() lib/yajsonrpc/stomprpcclient_test.py:144: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def _call(self, namespace, method_name, **kwargs): """ Client call method, executes a given command Args: namespace (string): namespace name method_name (string): method name **kwargs: Arbitrary keyword arguments Returns: method result Raises: ClientError: in case of an error in the protocol. TimeoutError: if there is no response after a pre configured time. ServerError: in case of an error while executing the command """ method = namespace + "." + method_name timeout = kwargs.pop("_timeout", self._default_timeout) req = yajsonrpc.JsonRpcRequest( method, kwargs, reqId=str(uuid.uuid4())) try: responses = self._client.call( req, timeout=timeout, flow_id=self._flow_id) except EnvironmentError as e: raise ClientError(method, kwargs, e) if not responses: > raise TimeoutError(method, kwargs, timeout) E vdsm.client.TimeoutError: Request Test.failingCall with args {} timed out after 3 seconds ../lib/vdsm/client.py:294: TimeoutError ------------------------------ Captured log call ------------------------------- ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 47428, 0, 0) at 0x7f48ddc47d10> (<class 'ValueError'>:'b'ept-version:1.2'' contains illegal character ':' [/usr/lib64/python3.7/asyncore.py|readwrite|108] [/usr/lib64/python3.7/asyncore.py|handle_read_event|422] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645] [/usr/lib64/python3.7/asyncore.py|handle_read_event|422] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167]) WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc (StompReactor), started daemon 139950158767872)> failed Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py", line 260, in run ret = func(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py", line 393, in process_requests self._reactor.process_requests() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 238, in process_requests timeout=self._get_timeout(self._map), File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 253, in _get_timeout interval = disp.next_check_interval() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 99, in next_check_interval return getattr(self.__impl, "next_check_interval", default_func)() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 486, in next_check_interval self.handle_timeout() File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 451, in handle_timeout self._frame_handler.handle_timeout(self) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py", line 130, in handle_timeout dispatcher._on_timeout) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 630, in reconnect AsyncDispatcher(self, self._async_client, count=count)) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 271, in reconnect dispatcher.create_socket(address, sslctx) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 112, in create_socket sock = sslctx.wrapSocket(sock) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py", line 113, in wrapSocket ca_certs=self.ca_certs) File "/usr/lib64/python3.7/ssl.py", line 1230, in wrap_socket context.load_verify_locations(ca_certs) FileNotFoundError: [Errno 2] No such file or directory _____________________ VdsmClientTests.test_missing_method ______________________ self = <stomprpcclient_test.VdsmClientTests testMethod=test_missing_method> def test_missing_method(self): with self._create_client() as client: with self.assertRaises(ServerError) as ex: > client.Test.missingMethod() lib/yajsonrpc/stomprpcclient_test.py:155: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def _call(self, namespace, method_name, **kwargs): """ Client call method, executes a given command Args: namespace (string): namespace name method_name (string): method name **kwargs: Arbitrary keyword arguments Returns: method result Raises: ClientError: in case of an error in the protocol. TimeoutError: if there is no response after a pre configured time. ServerError: in case of an error while executing the command """ method = namespace + "." + method_name timeout = kwargs.pop("_timeout", self._default_timeout) req = yajsonrpc.JsonRpcRequest( method, kwargs, reqId=str(uuid.uuid4())) try: responses = self._client.call( req, timeout=timeout, flow_id=self._flow_id) except EnvironmentError as e: raise ClientError(method, kwargs, e) if not responses: > raise TimeoutError(method, kwargs, timeout) E vdsm.client.TimeoutError: Request Test.missingMethod with args {} timed out after 3 seconds ../lib/vdsm/client.py:294: TimeoutError

4 5

ovirt-engine has been tagged (retagging of 4.3.8.1)
by Tal Nisan 11 Dec '19

11 Dec '19

Due to a problem in the build data I had to retag ovirt-engine-4.3.8.1, the same tag now points to a new HEAD

1 0

[ACTION REQUIRED] Update db before running engine-setup
by Ales Musil 11 Dec '19

11 Dec '19

Hello, I made some changes to the db configuration table. After [0] is merged, you'll need to run SQL command, before next engine-setup: SELECT fn_db_delete_config_value_all_versions('VdsmUseNmstate'); If you have any questions let me know. Thanks, Ales [0] https://gerrit.ovirt.org/c/105440/ -- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

5 13

vdsm check-patch CI failing on missing dependency
by Ales Musil 11 Dec '19

11 Dec '19

Hi, CI is failing for some time now with error: Error: Unable to find a match: python3-ioprocess-1.3.0 Can anyone please take a look? Thanks you. Regards, Ales -- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

2 1