I think I know what the problem is (not 100% sure). I see the experimental job [1] is taking mom rpms from an unmerged patch with -1 [2], so if this patch has a bug then it deployed broken mom to the experimental repo and failing tests. I'm trying to exclude the created jobs for mom out of this workflow. [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/1283/ [2] https://gerrit.ovirt.org/#/c/63130/ On Thu, Sep 1, 2016 at 3:32 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > Hi there, > > The oVirt "experimental" system tests form "master and 4.0 started > failing today at approximately 9:20 ([1], [2]) at the AddHost stage. > > I suspect the failure is due to a mom build that was performed today > at 9:05 ([3], [4]). This is because the following can be found in the > VDSM logs ([5], [6]): > > MainThread::ERROR::2016-09-01 06:18:41,076::vdsm::151::vds::(run) > Exception raised > Traceback (most recent call last): > File "/usr/share/vdsm/vdsm", line 149, in run > serve_clients(log) > File "/usr/share/vdsm/vdsm", line 104, in serve_clients > cif = clientIF.getInstance(irs, log, scheduler) > File "/usr/share/vdsm/clientIF.py", line 204, in getInstance > cls._instance = clientIF(irs, log, scheduler) > File "/usr/share/vdsm/clientIF.py", line 105, in __init__ > self._prepareMOM() > File "/usr/share/vdsm/clientIF.py", line 273, in _prepareMOM > self.mom = MomClient(momconf) > File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 46, in > __init__ > raise MomNotAvailableError() > MomNotAvailableError > MainThread::INFO::2016-09-01 06:18:41,077::vdsm::153::vds::(run) > Stopping threads > MainThread::INFO::2016-09-01 06:18:41,077::vdsm::164::vds::(run) Exiting > > > Please have a look. > Thanks, > Barak. > > > [1]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma > ster/1283/ > [2]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/1077/ > [3]: http://jenkins.ovirt.org/job/mom_master_create-rpms-el7-x86_ > 64_created/15/ > [4]: http://jenkins.ovirt.org/job/mom_4.0_create-rpms-el7-x86_64_ > created/11/ > [5]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma > ster/1283/artifact/exported-artifacts/basic_suite_master.sh- > fc24/exported-artifacts/test_logs/basic_suite_master/post- > 002_bootstrap.py/lago_basic_suite_master_host0/_var_log_vdsm/vdsm.log > [6]: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4. > 0/1077/artifact/exported-artifacts/basic_suite_4.0.sh-fc23/ > exported-artifacts/test_logs/basic_suite_4.0/post-002_boots > trap.py/lago_basic_suite_4_0_host0/_var_log_vdsm/vdsm.log > > -- > Barak Korren > bkorren(a)redhat.com > RHEV-CI Team > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)