both check-patch and check-merge are using the following rpmbuild params:

rpmbuild \
    -D "_rpmdir $PWD/output" \
    -D "_topmdir $PWD/rpmbuild" \
    -D "release_suffix ${SUFFIX}" \
    -D "ovirt_build_ut $BUILD_UT" \
    -D "ovirt_build_extra_flags $EXTRA_BUILD_FLAGS" \
    -D "ovirt_build_quick 1" \
    --rebuild output/*.src.rpm


I think the     -D "ovirt_build_quick 1"  skips GWT compilation or at least some of the permutations, so if the errors was related to GWT it explains the problem.
I suggest to change check-merge to include the basic GWT compilation with a single permutation so we'll catch these at least after merge (if its doesn't take too long, worth considering adding to 
check-patch.sh).

This requires a patch to ovirt-engine automation/check-merge.sh file on master, ovirt-engine-4.0 and ovirt-engine-3.6.








On Mon, May 30, 2016 at 10:25 AM, Martin Perina <mperina@redhat.com> wrote:


On Mon, May 30, 2016 at 9:05 AM, Barak Korren <bkorren@redhat.com> wrote:
On 30 May 2016 at 09:51, Martin Perina <mperina@redhat.com> wrote:
> Hi,
>
> I just verified that local build is is broken by the same error in
> HostedEngineDeployConfiguration.
> I will post a fix for that, but upgrade-job build is correct (raised build
> failure) and check-merged jobs
> are broken (producing success although build is broken), could you please
> fix them asap?.
>

Since those are std-CI jobs, I've no clue what they do and how to fix
them, and also no permissions to do so (I'm not an engine maintainer).
Everything they do is in the automation dir.

You can try to run them locally as well, see [1]

[1]: http://ovirt-infra-docs.readthedocs.io/en/latest/CI/Build_and_test_standards.html#testing-the-scripts-locally

It is an interesting problem though, what could make the build succeed
inside mock but fail otherwise?

​There's no way how those builds could be successful regardless of using mock, GWT build had always to fail without patch [2]​, so the issue is somewhere inside CI/Jenkins/mock magic.

I'm sorry, but I need today to focus in finishing patch for 3.6.7 bug, so I don't have time to investigate the issue, but IMO it needs to be resolved ASAP, otherwise
more broken patches can be merged.

@Eyal, is there somebody in CI team who can investigate this (assuming CI team members have much better knowledge of CI/Jenkins/mock magic
 and standard engine build command raised error correctly)?

Thanks

Martin

[2] https://gerrit.ovirt.org/58242


The loges here seem to do the same things:
http://jenkins.ovirt.org/job/ovirt-engine_master_check-merged-el7-x86_64/419/artifact/exported-artifacts/logs.tgz
(See the check_merged.sh log)

--
Barak Korren
bkorren@redhat.com
RHEV-CI Team




--
Eyal Edri
Associate Manager
RHEV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)