On Tue, Mar 16, 2021 at 10:09 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > On Tue, Mar 16, 2021 at 7:06 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... > > Build Number: 962 > > Build Status: Still Failing > > Triggered By: Started by timer > > > > ------------------------------------- > > Changes Since Last Success: > > ------------------------------------- > > Changes for Build #953 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #954 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #955 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #956 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #957 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #958 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #959 > > [Michal Skrivanek] randomize /dev/shm logcollector tmp directory > > > > > > Changes for Build #960 > > [Andrej Cernek] pylint: Upgrade to 2.7 > > > > > > Changes for Build #961 > > [Andrej Cernek] pylint: Upgrade to 2.7 > > > > > > Changes for Build #962 > > [Andrej Cernek] pylint: Upgrade to 2.7 > > > > > > > > > > ----------------- > > Failed Tests: > > ----------------- > > 1 tests failed. > > FAILED: basic-suite-master.test-scenarios.test_001_initialize_engine.test_set_hostnames > > > > Error Message: > > failed on setup with "TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo'" > > > > Stack Trace: > > ansible_by_hostname = <function module_mapper_for at 0x7ffbad0acc80> > > > > @pytest.fixture(scope="session", autouse=True) > > def check_installed_packages(ansible_by_hostname): > > vms_pckgs_dict_list = [] > > for hostname in backend.default_backend().hostnames(): > > vm_pckgs_dict = _get_custom_repos_packages( > > > ansible_by_hostname(hostname)) > > > > ost_utils/ost_utils/pytest/fixtures/check_repos.py:39: > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > ost_utils/ost_utils/pytest/fixtures/check_repos.py:55: in _get_custom_repos_packages > > repo_name) > > ost_utils/ost_utils/pytest/fixtures/check_repos.py:69: in _get_installed_packages > > Package(*line) for line in result > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > > > .0 = <list_iterator object at 0x7ffba6e97860> > > > > > Package(*line) for line in result > > ] > > E TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo' > > This failed, because 'dnf repo-pkgs' has split the output to two > lines, so the first > didn't include a version [1]: > > lago-basic-suite-master-host-1 | CHANGED | rc=0 >> > Installed Packages > ovirt-ansible-collection.noarch 1.3.2-0.1.master.20210315141358.el8 @extra-src-1 > python3-ovirt-engine-sdk4.x86_64 > 4.4.10-1.20210315.gitf8b9f2a.el8 @extra-src-1 > > We should either give up on this, or rewrite the call 'dnf repo-pkgs' > in some other > language that does not require parsing of human-targeted output > (perhaps python or > ansible), or amend a bit the current code and hope it will survive longer... > > Trying last one: > > https://gerrit.ovirt.org/c/ovirt-system-tests/+/113895 Merged, but we still fail in nightly (which I ran manually): https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... 16:06:44 > raise RuntimeError('None of user custom repos has been used') 16:06:44 E RuntimeError: None of user custom repos has been used I think this is "by design" - this job runs with a "custom repo" pointing at master-snapshot, and apparently at least in this run it didn't see updates, so failed. I wonder if this is simply a design issue, or we should change the nightly run to not use a custom repo, or something else. In any case, perhaps we should consider completely reverting check_repos.py for now, until we decide what we want. It was a good idea, but we can't let basic-suite remain red for so long. And then, we can get back to the issue of ovirt-log-collector...

On 16. 3. 2021, at 15:53, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: On Tue, Mar 16, 2021 at 10:09 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: On Tue, Mar 16, 2021 at 7:06 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... Build Number: 962 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #953 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #954 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #955 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #956 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #957 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #958 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #959 [Michal Skrivanek] randomize /dev/shm logcollector tmp directory Changes for Build #960 [Andrej Cernek] pylint: Upgrade to 2.7 Changes for Build #961 [Andrej Cernek] pylint: Upgrade to 2.7 Changes for Build #962 [Andrej Cernek] pylint: Upgrade to 2.7 ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_001_initialize_engine.test_set_hostnames Error Message: failed on setup with "TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo'" Stack Trace: ansible_by_hostname = <function module_mapper_for at 0x7ffbad0acc80> @pytest.fixture(scope="session", autouse=True) def check_installed_packages(ansible_by_hostname): vms_pckgs_dict_list = [] for hostname in backend.default_backend().hostnames(): vm_pckgs_dict = _get_custom_repos_packages( ansible_by_hostname(hostname)) ost_utils/ost_utils/pytest/fixtures/check_repos.py:39: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ost_utils/ost_utils/pytest/fixtures/check_repos.py:55: in _get_custom_repos_packages repo_name) ost_utils/ost_utils/pytest/fixtures/check_repos.py:69: in _get_installed_packages Package(*line) for line in result _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .0 = <list_iterator object at 0x7ffba6e97860> Package(*line) for line in result ] E TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo' This failed, because 'dnf repo-pkgs' has split the output to two lines, so the first didn't include a version [1]: lago-basic-suite-master-host-1 | CHANGED | rc=0 >> Installed Packages ovirt-ansible-collection.noarch 1.3.2-0.1.master.20210315141358.el8 @extra-src-1 python3-ovirt-engine-sdk4.x86_64 4.4.10-1.20210315.gitf8b9f2a.el8 @extra-src-1 We should either give up on this, or rewrite the call 'dnf repo-pkgs' in some other language that does not require parsing of human-targeted output (perhaps python or ansible), or amend a bit the current code and hope it will survive longer... Trying last one: https://gerrit.ovirt.org/c/ovirt-system-tests/+/113895 Merged, but we still fail in nightly (which I ran manually): https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... 16:06:44 > raise RuntimeError('None of user custom repos has been used') 16:06:44 E RuntimeError: None of user custom repos has been used I think this is "by design" - this job runs with a "custom repo" pointing at master-snapshot, and apparently at least in this run it didn't see updates, so failed. wouldn’t it fail exactly during the night when we build a fresh ost-image and run with custom repo that has nothing newer, obviously, since we just built the image?

I wonder if this is simply a design issue, or we should change the nightly run to not use a custom repo, or something else. In any case, perhaps we should consider completely reverting check_repos.py for now, until we decide what we want. It was a good idea, but we can't let basic-suite remain red for so long. And then, we can get back to the issue of ovirt-log-collector...

we don’t need so radical changes/investment just yet, just dropping it from nightly config should be enough I think it is rather a jenkins config issue. For nightly we do not need custom repo