November 2020 - Devel - oVirt List Archives

Breaking change introduced in vdsm-jsonrpc-java-1.6.0
by Artur Socha 30 Nov '20

30 Nov '20

Hi, We have recently introduced the breaking change in vdsm-jsonrpc-java [1]. Reactive streams package has been replaced with java.util.concurrent.Flow (shipped with JDK). This change is released with vdsm-jsonrpc-java-1.6.0. This change had to be reflected in ovirt-engine as well (otherwise it would not compile). I was handled by this patch [2]. Unfortunately these two changes must be in sync. Please update vdsm-jsonrpc-java to 1.6.0 if you are already working on ovirt engine revision that includes [2]. [1] https://gerrit.ovirt.org/#/c/vdsm-jsonrpc-java/+/109916/ [2] https://gerrit.ovirt.org/#/c/ovirt-engine/+/112347/ Please accept my apologies for not letting you know earlier, ideally before these patches got merged. -- Artur Socha Senior Software Engineer, RHV Red Hat

1 0

Health check endpoint in the engine hanging forever on CheckDBConnection
by Marcin Sobczyk 27 Nov '20

27 Nov '20

Hi, I'm trying out (I think) the latest build of ovirt-engine in OST [1] and the basic suite fails when we do engine reconfiguration and then restart the service [2]. After restarting we wait on the health check endpoint status here [3]. This however ends with a timeout. I tried running manually: curl -D - http://engine/ovirt-engine/services/health and that command also hangs forever. In the engine log the last related entry seem to be: 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.utils.servlet.LocaleFilter] (default task-7) [] Incoming locale 'en-US'. Filter determined locale to be 'en-US' 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Health Status servlet: entry 2020-11-26 16:20:15,761+01 DEBUG [org.ovirt.engine.core.services.HealthStatus] (default task-7) [] Calling CheckDBConnection query Has anyone else also encountered that? Regards, Marcin [1] ovirt-engine-4.4.4.3-0.0.master.20201126133903.gitc2c805a2662.el8.noarch [2] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9… [3] https://github.com/oVirt/ovirt-system-tests/blob/3e2fc267b376a12eda131fa0e9…

4 5

ovirt-engine has been tagged (ovirt-engine-4.4.4.2)
by Tal Nisan 25 Nov '20

25 Nov '20

1 0

Failure on checkInterfaces: uncategorized SQLException
by Yedidyah Bar David 25 Nov '20

25 Nov '20

Hi all, I ran OST manually with suite he-basic-master, and it failed at a rather late stage (010_local_maintenance_cli. My changes were for 008_restart_he_vm, which passed). It failed because the hosted-engine VM was already on host-1, which wasn't expected. I spent quite some time trying to understand this, and I *think* the root cause was quite before that. engine.log: 2020-11-25 09:48:50,864+01 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-273) [2e1072c7] Host setup networks finished. Lock released. Monitoring can run now for host 'lago-he-basic-suite-master-host-0.lago.local' from data-center 'Default' 2020-11-25 09:48:50,866+01 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-273) [2e1072c7] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORKfba14f41-7ece-46ca-af65-7476e9fd2018=HOST_NETWORK]', sharedLocks=''}' 2020-11-25 09:48:50,923+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-273) [2e1072c7] EVENT_ID: PERSIST_NETWORK_ON_HOST_FINISHED(1,146), (1/2): Successfully applied changes for network(s) Labeled_Network on host lago-he-basic-suite-master-host-0.lago.local. (User: admin@internal-authz) 2020-11-25 09:48:53,883+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-47) [2e1072c7] Command 'PropagateNetworksToClusterHosts' id: '6e61f53b-c2b7-4a35-a840-67076e6d66cb' child commands '[fe7f022e-c13b-4924-a6ea-cc56407e6969, ca990d6e-199b-473d-928d-0a7e6d8cf073]' executions were completed, status 'SUCCEEDED' 2020-11-25 09:48:54,891+01 INFO [org.ovirt.engine.core.bll.network.cluster.PropagateNetworksToClusterHostsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [2e1072c7] Ending command 'org.ovirt.engine.core.bll.network.cluster.PropagateNetworksToClusterHostsCommand' successfully. 2020-11-25 09:49:24,749+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] Failure on checkInterfaces on update runtime info for host 'lago-he-basic-suite-master-host-0.lago.local': PreparedStatementCallback; uncategorized SQLException for SQL [select * from getallnetworkbyclusterid(?, ?, ?)]; SQL state [null]; error code [0]; IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7d88842; nested exception is java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7d88842 2020-11-25 09:49:24,752+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] ResourceManager::refreshVdsRunTimeInfo: host=lago-he-basic-suite-master-host-0.lago.local(fba14f41-7ece-46ca-af65-7476e9fd2018): PreparedStatementCallback; uncategorized SQLException for SQL [select * from getvdsdynamicbyvdsid(?)]; SQL state [null]; error code [0]; IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7d88842; nested exception is java.sql.SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7d88842 2020-11-25 09:49:24,761+01 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] Timer update runtime info failed. Exception: SQLException: IJ031013: Interrupted attempting lock: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7d88842 Any idea? -- Didi

1 1

VDSM CI patches stuck in queue for several hours
by Ales Musil 25 Nov '20

25 Nov '20

Hi, most of my patches lately are getting stuck in CI. For example [0] is hanging there for 17 hours waiting for a container that would be able to run nmstate functional tests. Can someone please take a look at that? Thanks, Ales [0] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25066/ -- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

3 2

"No module named pathlib" errors in OST
by Marcin Sobczyk 24 Nov '20

24 Nov '20

Hi, there was a new version of 'importlib_metadata' library pushed to pypi. Even though it's compatible only with py3, it's picked up by py2 pip. I posted a patch to fix the version to the previous one: https://gerrit.ovirt.org/#/c/112325/ If you see this failure in other suites than basic please notify me and I will align the patch. Regards, Marcin

2 1

Branching out 4.3 in ovirt-system-tests
by Marcin Sobczyk 23 Nov '20

23 Nov '20

Hi all, after minimizing the usage of lago in basic suite, and some minor adjustments in the network suite, we are finally able to remove lago OST plugin as a dependency [1]. This however comes with a price of keeping lots of ugly ifology, i.e. [2][3]. There's big disparity between OST runs we have on el7 and el8. There's also tons of symlink-based code sharing between suites - be it 4.3 suites and master suites or simply different types of suites. The basic suite has its own 'test_utils', which is copied/symlinked in multiple places. There's also 'ost_utils', which is really messy ATM. It's very hard to keep track and maintain all of this... At this moment, we are able to run basic suite and network suite on el8, with prebuilt ost-images and without lago plugin. HE suites should be the next step. We have patches that make them py3-compatible that probably still need some attention [4][5]. We don't have any prebuilt HE ost-images, but this will be handled in the nearest future. I think it's good time to detach ourselves from the legacy stuff and start with a clean slate. My proposition would be to branch out 4.3 in ovirt-system-tests and not use py2/el7 in the master branch at all. This would allow us to focus on py3, el8 and ost-images efforts while keeping the legacy stuff intact. WDYT? Regards, Marcin [1] https://gerrit.ovirt.org/#/c/111643/ [2] https://gerrit.ovirt.org/#/c/111643/6/basic-suite-master/control.sh [3] https://gerrit.ovirt.org/#/c/111643/6/basic-suite-master/test-scenarios/con… [4] https://gerrit.ovirt.org/108809 [5] https://gerrit.ovirt.org/110097

3 8

Can't find storage server connection
by francesco＠shellrent.com 23 Nov '20

23 Nov '20

Hi all, I'm using oVirt SDK python for retrieving info about storage domain, in several hosts (centos7/ovirt4.3 and centos8/ovirt4.4), but the script exits with the following error in some of them: Traceback (most recent call last): File "get_uuid.py", line 70, in <module> storage_domain = sds_service.list(search='name=data-foo')[0] File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/services.py", line 26296, in list return self._internal_get(headers, query, wait) File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/root/.local/lib/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "Can't find storage server connection for id '92444a95-0be7-4589-ac46-1ed6dfe7ed4c'.". HTTP response code is 500. The portion of the script that search for the storage domain is the following: sds_service = connection.system_service().storage_domains_service() storage_domain = sds_service.list(search='name={}'.format(storage_domain_name))[0] Now: I have no real clue on which ID "92444a95-0be7-4589-ac46-1ed6dfe7ed4c'" but digging in the engine logs it refers to StorageServerConnections ID: [root@ovirt-engine ovirt-engine]# zgrep 92444a95-0be7-4589-ac46-1ed6dfe7ed4c *.gz engine.log-20201108.gz:2020-11-07 06:05:54,352+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [3ffb810c] START, ConnectStorageServerVDSCommand(HostName = another-server.foo.com, StorageServerConnectionManagementVDSParameters:{hostId='7d202bc7-002b-4426-8446-99b6b346874e', storagePoolId='82d0b3de-0334-451c-8321-c3533de9a894', storageType='LOCALFS', connectionList='[StorageServerConnections:{id='92444a95-0be7-4589-ac46-1ed6dfe7ed4c', connection='/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 1509fe17 As I said, I tried to execute the script from several hosts: some of them with oVirt 4.3 and other with oVirt 4.4 but it can run, or not, on both of versions. I also did the following test: I tried execute the script on different host passing different data storage: - let's say that on host foo-host1, executing the script querying for its own data storage named foo-data1 works correctly; - and let's say that on host foo-host2, the same script querying for its own data storage foo-data2 return the mentioned error; If I execute the script on foo-host2 querying the foo-data1 domain data, I got the error. If I execute on foo-host1 querying foo-data2, I still got the error. When I try to manage the storage domain via ovirt-engine GUI on the hosts that the script exit with the mentioned error, I recieve the following error: Uncaught exception occurred. Please try reloading the page. Details: (TypeError) : Cannot read property 'a' of null Please have your administrator check the UI logs The logs: 2020-11-23 09:59:31,934+01 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-239) [] Permutation name: 68A62BEB12822F65FE66B14A9E16480A 2020-11-23 09:59:31,934+01 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-239) [] Uncaught exception: com.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read property 'a' of null at org.ovirt.engine.ui.uicommonweb.models.storage.FileStorageModel.$lambda$0(FileStorageModel.java:34) at org.ovirt.engine.ui.uicommonweb.models.storage.FileStorageModel$lambda$0$Type.onSuccess(FileStorageModel.java:34) at org.ovirt.engine.ui.frontend.Frontend$1.$onSuccess(Frontend.java:227) at org.ovirt.engine.ui.frontend.Frontend$1.onSuccess(Frontend.java:227) at org.ovirt.engine.ui.frontend.communication.OperationProcessor$1.$onSuccess(OperationProcessor.java:133) at org.ovirt.engine.ui.frontend.communication.OperationProcessor$1.onSuccess(OperationProcessor.java:133) at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.$onSuccess(GWTRPCCommunicationProvider.java:270) at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.onSuccess(GWTRPCCommunicationProvider.java:270) at com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:198) at com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:233) at com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409) at Unknown.eval(webadmin-0.js) at com.google.gwt.core.client.impl.Impl.apply(Impl.java:306) at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:345) at Unknown.eval(webadmin-0.js) Any slightest idea on what is going on? Thank you for your time and help, Francesco

1 0

[OST] Testing master suite on using python 2.7?!
by Nir Soffer 19 Nov '20

19 Nov '20

I'm trying to add a test module for image transfer: https://gerrit.ovirt.org/c/112274/ The test use concurrent.futures module form the standard library. This module is not available in python 2.7 but we don't support 2.7 in master and it was EOL since Jan 2020. The test fail when starting the suite: [2020-11-18T16:17:10.526Z] ============================= test session starts ============================== [2020-11-18T16:17:10.526Z] platform linux2 -- Python 2.7.5, pytest-4.6.9, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python2 ... [2020-11-18T16:17:11.105Z] ==================================== ERRORS ==================================== [2020-11-18T16:17:11.105Z] ___ ERROR collecting basic-suite-master/test-scenarios/008_image_transfer.py ___ [2020-11-18T16:17:11.105Z] ImportError while importing test module '/home/jenkins/agent/workspace/ovirt-system-tests_standard-check-patch/ovirt-system-tests/basic-suite-master/test-scenarios/008_image_transfer.py' [2020-11-18T16:17:11.105Z] Hint: make sure your test modules/packages have valid Python names. [2020-11-18T16:17:11.105Z] Traceback: [2020-11-18T16:17:11.105Z] ../basic-suite-master/test-scenarios/008_image_transfer.py:22: in <module> [2020-11-18T16:17:11.105Z] import concurrent.futures [2020-11-18T16:17:11.105Z] E ImportError: No module named concurrent.futures Should we use pytest.skip() to skip this test when running on python 2? Or just remove the python 2 build, I don't have any idea why we run master code python 2. Nir

2 3

vdsm CI is failing
by Marcin Sobczyk 18 Nov '20

18 Nov '20

Hi, vdsm CI is failing with: [2020-11-16T10:51:40.132Z] No match for argument: ovirt-imageio-client-2.1.1 [2020-11-16T10:51:40.132Z] Error: Unable to find a match: ovirt-imageio-client-2.1.1 example run: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… could you please take a look at it? Regards, Marcin

3 3