On 16. 3. 2021, at 15:53, Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Mar 16, 2021 at 10:09 AM Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Mar 16, 2021 at 7:06 AM <jenkins@jenkins.phx.ovirt.org> wrote:

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/962/
Build Number: 962
Build Status: Still Failing
Triggered By: Started by timer

-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #953
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #954
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #955
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #956
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #957
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #958
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #959
[Michal Skrivanek] randomize /dev/shm logcollector tmp directory

Changes for Build #960
[Andrej Cernek] pylint: Upgrade to 2.7

Changes for Build #961
[Andrej Cernek] pylint: Upgrade to 2.7

Changes for Build #962
[Andrej Cernek] pylint: Upgrade to 2.7

-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: basic-suite-master.test-scenarios.test_001_initialize_engine.test_set_hostnames

Error Message:
failed on setup with "TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo'"

Stack Trace:
ansible_by_hostname = <function module_mapper_for at 0x7ffbad0acc80>

   @pytest.fixture(scope="session", autouse=True)
   def check_installed_packages(ansible_by_hostname):
       vms_pckgs_dict_list = []
       for hostname in backend.default_backend().hostnames():
           vm_pckgs_dict = _get_custom_repos_packages(
             ansible_by_hostname(hostname))

ost_utils/ost_utils/pytest/fixtures/check_repos.py:39:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ost_utils/ost_utils/pytest/fixtures/check_repos.py:55: in _get_custom_repos_packages
   repo_name)
ost_utils/ost_utils/pytest/fixtures/check_repos.py:69: in _get_installed_packages
   Package(*line) for line in result
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <list_iterator object at 0x7ffba6e97860>

     Package(*line) for line in result
   ]
E   TypeError: __new__() missing 2 required positional arguments: 'version' and 'repo'

This failed, because 'dnf repo-pkgs' has split the output to two
lines, so the first
didn't include a version [1]:

lago-basic-suite-master-host-1 | CHANGED | rc=0 >>
Installed Packages
ovirt-ansible-collection.noarch 1.3.2-0.1.master.20210315141358.el8 @extra-src-1
python3-ovirt-engine-sdk4.x86_64
                               4.4.10-1.20210315.gitf8b9f2a.el8    @extra-src-1

We should either give up on this, or rewrite the call 'dnf repo-pkgs'
in some other
language that does not require parsing of human-targeted output
(perhaps python or
ansible), or amend a bit the current code and hope it will survive longer...

Trying last one:

https://gerrit.ovirt.org/c/ovirt-system-tests/+/113895

Merged, but we still fail in nightly (which I ran manually):

https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/963/console

16:06:44 >           raise RuntimeError('None of user custom repos has
been used')
16:06:44 E           RuntimeError: None of user custom repos has been used

I think this is "by design" - this job runs with a "custom repo"
pointing at master-snapshot, and apparently at least in this run it
didn't see updates, so failed.

wouldn’t it fail exactly during the night when we build a fresh ost-image and run with custom repo that has nothing newer, obviously, since we just built the image?