I think I know what the problem is (not 100% sure).
I see the experimental job [1] is taking mom rpms from an unmerged patch with -1 [2], so if this patch has a bug then it deployed broken mom to the experimental repo and failing tests.
I'm trying to exclude the created jobs for mom out of this workflow.

[1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/1283/
[2] https://gerrit.ovirt.org/#/c/63130/

On Thu, Sep 1, 2016 at 3:32 PM, Barak Korren <bkorren@redhat.com> wrote:
Hi there,

The oVirt "experimental" system tests form "master and 4.0 started
failing today at approximately 9:20 ([1], [2]) at the AddHost stage.

I suspect the failure is due to a mom build that was performed today
at 9:05 ([3], [4]). This is because the following can be found in the
VDSM logs ([5], [6]):

MainThread::ERROR::2016-09-01 06:18:41,076::vdsm::151::vds::(run)
Exception raised
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 149, in run
    serve_clients(log)
  File "/usr/share/vdsm/vdsm", line 104, in serve_clients
    cif = clientIF.getInstance(irs, log, scheduler)
  File "/usr/share/vdsm/clientIF.py", line 204, in getInstance
    cls._instance = clientIF(irs, log, scheduler)
  File "/usr/share/vdsm/clientIF.py", line 105, in __init__
    self._prepareMOM()
  File "/usr/share/vdsm/clientIF.py", line 273, in _prepareMOM
    self.mom = MomClient(momconf)
  File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 46, in __init__
    raise MomNotAvailableError()
MomNotAvailableError
MainThread::INFO::2016-09-01 06:18:41,077::vdsm::153::vds::(run)
Stopping threads
MainThread::INFO::2016-09-01 06:18:41,077::vdsm::164::vds::(run) Exiting


Please have a look.
Thanks,
Barak.


[1]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/1283/
[2]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/1077/
[3]: http://jenkins.ovirt.org/job/mom_master_create-rpms-el7-x86_64_created/15/
[4]: http://jenkins.ovirt.org/job/mom_4.0_create-rpms-el7-x86_64_created/11/
[5]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/1283/artifact/exported-artifacts/basic_suite_master.sh-fc24/exported-artifacts/test_logs/basic_suite_master/post-002_bootstrap.py/lago_basic_suite_master_host0/_var_log_vdsm/vdsm.log
[6]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/1077/artifact/exported-artifacts/basic_suite_4.0.sh-fc23/exported-artifacts/test_logs/basic_suite_4.0/post-002_bootstrap.py/lago_basic_suite_4_0_host0/_var_log_vdsm/vdsm.log

--
Barak Korren
bkorren@redhat.com
RHEV-CI Team



--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)