- Devel - oVirt List Archives

reboot after installation during hosted-engine deploy (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1915 - Failure!)
by Yedidyah Bar David 10 Feb '21

10 Feb '21

Hi all, On Wed, Feb 10, 2021 at 4:34 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1915/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1915… : 2021-02-10 03:21:31,081+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:111 TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] 2021-02-10 03:23:57,041+0100 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': SIG1 Most likely this is because by default we now reboot the host after installation. We need to not do this for the host we deploy on, something like: https://github.com/oVirt/ovirt-ansible-collection/pull/218 Do we have a bug for this? Best regards, > Build Number: 1915 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1915 > [Yedidyah Bar David] test_utils/host_status_utils: Make dc_name parametric > > [Galit Rosenthal] Add node upgrade suite > > [Ehud Yonasi] run_code: add gpg keys only for el8. > > [Ehud Yonasi] run_code: fixed check for el8. > > [Galit Rosenthal] Move ansible nightly to run on el8 > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

1 1

"Too many open files" in vdsm.log after 380 migrations
by Yedidyah Bar David 09 Feb '21

09 Feb '21

Hi all, I ran a loop of [1] (from [2]). The loop succeeded for ~ 380 iterations, then failed with 'Too many open files'. First failure was: 2021-02-08 02:21:15,702+0100 ERROR (jsonrpc/4) [storage.HSM] Could not connect to storageServer (hsm:2446) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2443, in connectStorageServer conObj.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 449, in connect return self._mountCon.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 171, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in mount cgroup=cgroup) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File "<string>", line 2, in mount File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) OSError: [Errno 24] Too many open files But obviously, once it did, it continued failing for this reason on many later operations. Is this considered a bug? Do we actively try to prevent such cases? So should I open one and attach logs? Or it can be considered a "corner case"? Using vdsm-4.40.50.3-37.git7883b3b43.el8.x86_64 from ost-images-el8-he-installed-1-202102021144.x86_64 . I can also let access to the machine(s) if needed, for now. Thanks and best regards, [1] https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=he-basic-… [2] https://gerrit.ovirt.org/c/ovirt-system-tests/+/113300 -- Didi

3 8

Network error during communication with the Host (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 857 - Still Failing!)
by Yedidyah Bar David 09 Feb '21

09 Feb '21

On Tue, Feb 9, 2021 at 7:24 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > Build Number: 857 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #855 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > Changes for Build #856 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > Changes for Build #857 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_add_nfs_master_storage_domain > > Error Message: > AssertionError: False != True after 600 seconds > > Stack Trace: > engine = <ovirtsdk4.services.SystemService object at 0x7fda58b32828> > event_id = [956] > > @contextlib.contextmanager > def wait_for_event(engine, event_id): > ''' > event_id could either be an int - a single > event ID or a list - multiple event IDs > that all will be checked > ''' > events = engine.events_service() > last_event = int(events.list(max=2)[0].id) > try: > > yield > > ../ost_utils/ost_utils/engine_utils.py:38: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > api = <ovirtsdk4.Connection object at 0x7fda58b32748> > domain = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8> > dc_name = 'test-dc' > > def add(api, domain, dc_name): > system_service = api.system_service() > sds_service = system_service.storage_domains_service() > with engine_utils.wait_for_event(system_service, 956): # USER_ADD_STORAGE_DOMAIN(956) > > sd = sds_service.add(domain) > > ../ost_utils/ost_utils/storage_utils/domain.py:33: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > storage_domain = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8> > headers = None, query = {}, wait = True, kwargs = {} > > def add( > self, > storage_domain, > headers=None, > query=None, > wait=True, > **kwargs > ): > """ > Adds a new storage domain. > Creation of a new <<types/storage_domain,StorageDomain>> requires the `name`, `type`, `host`, and `storage` > attributes. Identify the `host` attribute with the `id` or `name` attributes. In {product-name} 3.6 and > later you can enable the wipe after delete option by default on the storage domain. To configure this, specify > `wipe_after_delete` in the POST request. This option can be edited after the domain is created, but doing so will > not change the wipe after delete property of disks that already exist. > To add a new storage domain with specified `name`, `type`, `storage.type`, `storage.address`, and `storage.path`, > and using a host with an id `123`, send a request like this: > [source] > ---- > POST /ovirt-engine/api/storageDomains > ---- > With a request body like this: > [source,xml] > ---- > <storage_domain> > <name>mydata</name> > <type>data</type> > <storage> > <type>nfs</type> > <address>mynfs.example.com</address> > <path>/exports/mydata</path> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > To create a new NFS ISO storage domain send a request like this: > [source,xml] > ---- > <storage_domain> > <name>myisos</name> > <type>iso</type> > <storage> > <type>nfs</type> > <address>mynfs.example.com</address> > <path>/export/myisos</path> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > To create a new iSCSI storage domain send a request like this: > [source,xml] > ---- > <storage_domain> > <name>myiscsi</name> > <type>data</type> > <storage> > <type>iscsi</type> > <logical_units> > <logical_unit id="3600144f09dbd050000004eedbd340001"/> > <logical_unit id="3600144f09dbd050000004eedbd340002"/> > </logical_units> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > > > This method supports the following parameters: > > `storage_domain`:: The storage domain to add. > > `headers`:: Additional HTTP headers. > > `query`:: Additional URL query parameters. > > `wait`:: If `True` wait for the response. > """ > # Check the types of the parameters: > Service._check_types([ > ('storage_domain', storage_domain, types.StorageDomain), > ]) > > # Build the URL: > query = query or {} > > # Send the request and wait for the response: > > return self._internal_add(storage_domain, headers, query, wait) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:26182: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > object = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8>, headers = {} > query = {}, wait = True > > def _internal_add(self, object, headers=None, query=None, wait=None): > """ > Executes an `add` method. > """ > # Populate the headers: > headers = headers or {} > > # Send the request and wait for the response: > request = http.Request(method='POST', path=self._path, query=query, headers=headers) > request.body = writer.Writer.write(object, indent=True) > context = self._connection.send(request) > > def callback(response): > if response.code in [200, 201, 202]: > return self._internal_read_body(response) > else: > self._check_fault(response) > > future = Future(self._connection, context, callback) > > return future.wait() if wait else future > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.service.Future object at 0x7fda57ca49b0> > > def wait(self): > """ > Waits till the result of the operation that created this future is > available. > """ > response = self._connection.wait(self._context) > > return self._code(response) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > > def callback(response): > if response.code in [200, 201, 202]: > return self._internal_read_body(response) > else: > > self._check_fault(response) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > > def _check_fault(self, response): > """ > Reads the response body assuming that it contains a fault message, > converts it to an exception and raises it. > > This method is intended for internal use by other > components of the SDK. Refrain from using it directly, > as backwards compatibility isn't guaranteed. > """ > > body = self._internal_read_body(response) > if isinstance(body, types.Fault): > > self._raise_error(response, body) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > detail = <ovirtsdk4.types.Fault object at 0x7fda57ca44e0> > > @staticmethod > def _raise_error(response, detail=None): > """ > Creates and raises an error containing the details of the given HTTP > response and fault. > > This method is intended for internal use by other components of the > SDK. Refrain from using it directly, as backwards compatibility isn't > guaranteed. > """ > fault = detail if isinstance(detail, types.Fault) else None > > msg = '' > if fault: > if fault.reason: > if msg: > msg += ' ' > msg = msg + 'Fault reason is "%s".' % fault.reason > if fault.detail: > if msg: > msg += ' ' > msg = msg + 'Fault detail is "%s".' % fault.detail > if response: > if response.code: > if msg: > msg += ' ' > msg = msg + 'HTTP response code is %s.' % response.code > if response.message: > if msg: > msg += ' ' > msg = msg + 'HTTP response message is "%s".' % response.message > > if isinstance(detail, six.string_types): > if msg: > msg += ' ' > msg = msg + detail + '.' > > class_ = Error > if response is not None: > if response.code in [401, 403]: > class_ = AuthError > elif response.code == 404: > class_ = NotFoundError > > error = class_(msg) > error.code = response.code if response else None > error.fault = fault > > raise error > E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Did anyone check this? It started failing yesterday. Also for me, e.g.: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15458/ Thanks and best regards, > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error > > During handling of the above exception, another exception occurred: > > engine_api = <ovirtsdk4.Connection object at 0x7fda58b32748> > sd_nfs_host_storage_ip = '192.168.202.2' > > @order_by(_TEST_LIST) > @pytest.mark.skipif(MASTER_SD_TYPE != 'nfs', reason='not using nfs') > def test_add_nfs_master_storage_domain(engine_api, sd_nfs_host_storage_ip): > > add_nfs_storage_domain(engine_api, sd_nfs_host_storage_ip) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:545: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:554: in add_nfs_storage_domain > nfs_version='v4_2') > ../ost_utils/ost_utils/storage_utils/nfs.py:62: in add_domain > domain.add(engine_api, p, dc_name) > ../ost_utils/ost_utils/storage_utils/domain.py:37: in add > lambda: sd_service.get().status == sdk4.types.StorageDomainStatus.UNATTACHED > /usr/lib64/python3.6/contextlib.py:99: in __exit__ > self.gen.throw(type, value, traceback) > ../ost_utils/ost_utils/engine_utils.py:44: in wait_for_event > lambda: > ../ost_utils/ost_utils/assertions.py:98: in assert_true_within_long > assert_equals_within_long(func, True, allowed_exceptions) > ../ost_utils/ost_utils/assertions.py:83: in assert_equals_within_long > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > func = <function wait_for_event.<locals>.<lambda> at 0x7fda57c9c2f0> > value = True, timeout = 600, allowed_exceptions = [], initial_wait = 0 > error_message = 'False != True after 600 seconds' > > def assert_equals_within( > func, value, timeout, allowed_exceptions=None, initial_wait=10, > error_message=None > ): > allowed_exceptions = allowed_exceptions or [] > res = '<no-result-obtained>' > with _EggTimer(timeout) as timer: > while not timer.elapsed(): > try: > res = func() > if res == value: > return > except Exception as exc: > if _instance_of_any(exc, allowed_exceptions): > time.sleep(3) > continue > > LOGGER.exception("Unhandled exception in %s", func) > raise > > if initial_wait == 0: > time.sleep(3) > else: > time.sleep(initial_wait) > initial_wait = 0 > try: > if error_message is None: > error_message = '%s != %s after %s seconds' % (res, value, timeout) > > raise AssertionError(error_message) > E AssertionError: False != True after 600 seconds > > ../ost_utils/ost_utils/assertions.py:61: AssertionError_______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ND5KFKIA7BNG7… -- Didi

1 0

Connection issues during send request
by Yedidyah Bar David 08 Feb '21

08 Feb '21

Hi all, Please see [1]. engine.log has many errors like: 2021-02-08 14:01:57,529+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [c14cce8] Error: org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection issues during send request vdsm.log seems ok. Didn't try to investigate much, just tried 'ci test' again on the patch, for now. Any idea? [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15443/ -- Didi

2 1

Issue running OST basic suite master
by Galit Rosenthal 04 Feb '21

04 Feb '21

Hi, Basic suite master is failing [1] on basic-suite-master.test-scenarios.test_004_basic_sanity.test_import_vm1 Can someone have a look? Regards, Galit [1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_el8nigh… -- Galit Rosenthal SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)redhat.com T: 972-9-7692230 <https://www.redhat.com/>

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.5.4)
by Tal Nisan 03 Feb '21

03 Feb '21

1 0

Switch CentOS to Oracle Linux
by ntoxicator＠gmail.com 02 Feb '21

02 Feb '21

Hello, Oracle Linux is offering CentOS users/dev/production servers to switch to Oracle Linux free of cost. They've produced a script to do this. Oracle also has their own Repo's for oVirt and use it for their own Oracle Linux Virtualization Manager [Re-branded oVirt]. I know Red Hat has announced moving away from RHV and focusing on OpenShift virtualization. They'll happen in 2022-forward. Are we already to assume oVirt will follow suite and migrate to OpenShift?? Questions: Since announcement of CentOS going EOL and move to CentOS Stream... [Not doing it]. - Will the oVirt team continue to develop past 4.4 release as well as support Oracle Linux? - Does Oracle really develop to their own ovirt repo and virtualization manager product - What about supporting OpenSuSE or SLES? - oVirt packages on RHEL? [RH announced they will allow RHEL installed on up to 16-hosts for FREE] I am considering to test moving CentOS 7 development work-loads to Oracle Linux and use their oVirt repo. Really like to know if oVirt developers plan to continue to build on what is there moving forward. Or will the project die? I use oVirt on production work-loads [same as others] with GREAT success. This is an absolute VMWare replacement.

3 3

[CI] Wrong version of qemu-kvm in the CI on ppc64le
by Nir Soffer 02 Feb '21

02 Feb '21

We have tests that used to pass 2-3 weeks ago, but started to fail today on ppc64le with this error: qemu-kvm: Requested safe cache capability level not supported by kvm, try appending -machine cap-cfpc=broken Full log: https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/3599//arti… I can add -machine cap-cfpc=broken, but it looks like we are running the wrong qemu version in CI: 17:31:08,513 DEBUG (MainThread) [qmp] Talking with qemu 4.2.0 We should use 5.1.0, available on Centos 8.3.2011: $ dnf info qemu-kvm ... Available Packages Name : qemu-kvm Epoch : 15 Version : 5.1.0 Release : 14.el8.1 Architecture : x86_64 Size : 126 k Source : qemu-kvm-5.1.0-14.el8.1.src.rpm Repository : ovirt-master-advanced-virtualization-testing Is it possible that we don't have a build for ppc6le, or something is broken in this build? Since the tests used to pass, I don't think this is a repo issue. Nir

1 2

CentOS Stream support
by Michal Skrivanek 01 Feb '21

01 Feb '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

6 12

ovirt-engine_standard-check-patch failed to fetch remote repo 'origin'
by Eyal Shenitzky 31 Jan '21

31 Jan '21

Hi, The ovirt-engine_standard-check-patch job in jenkins failed due to the following issue - [2021-01-31T13:15:01.371Z] The recommended git tool is: NONE [2021-01-31T13:15:05.283Z] No credentials specified [2021-01-31T13:15:05.283Z] No credentials specified [2021-01-31T13:15:05.393Z] Fetching changes from 2 remote Git repositories [2021-01-31T13:15:05.420Z] Cleaning workspace [2021-01-31T13:15:05.517Z] ERROR: Error fetching remote repo 'origin' [2021-01-31T13:15:05.517Z] hudson.plugins.git.GitException: Failed to fetch from https://gerrit.ovirt.org/jenkins [2021-01-31T13:15:05.518Z] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1001) [2021-01-31T13:15:05.518Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1242) [2021-01-31T13:15:05.518Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302) [2021-01-31T13:15:05.518Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) [2021-01-31T13:15:05.518Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) [2021-01-31T13:15:05.518Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) [2021-01-31T13:15:05.518Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) [2021-01-31T13:15:05.518Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2021-01-31T13:15:05.518Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [2021-01-31T13:15:05.518Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [2021-01-31T13:15:05.518Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [2021-01-31T13:15:05.518Z] at java.lang.Thread.run(Thread.java:748) [2021-01-31T13:15:05.518Z] Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1: Taken from - https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/10151/ which run as part of https://gerrit.ovirt.org/#/c/ovirt-engine/+/113113/ Seen also in other patches that were pushed today. Can someone please have a look? Thanks, -- Regards, Eyal Shenitzky

1 0