Test failed: [ add_master_storage_domain ] Link to suspected patches: - https://gerrit.ovirt.org/#/c/83849/4

This seems to be a fairly consistent regression as all patches that follow the patch above exhibit the same issue when tested, this list of similarly failing patches includes: - https://gerrit.ovirt.org/#/c/83940/2 - https://gerrit.ovirt.org/#/c/83608/7 - https://gerrit.ovirt.org/#/c/83815/3 The following patch is an exception because it simply failed to build: - https://gerrit.ovirt.org/#/c/83370/9 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/artifa... Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "[Failed to attach Storage due to an error on the Data Center master Storage Domain. -Please activate the master Storage Domain first.]". HTTP response code is 409. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On 13 November 2017 at 10:08, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > On Mon, Nov 13, 2017 at 9:56 AM Barak Korren <bkorren(a)redhat.com&gt; wrote: >> >> Test failed: [ add_master_storage_domain ] >> >> Link to suspected patches: >> - https://gerrit.ovirt.org/#/c/83849/4 > > > This fixes grammar in few error messages, I don't see how this can cause > failures unless someone is depending on the error text... > Previous test with engine patches before it was successful: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3769 This one included the following patches which were the two commits leading up to the failing one: - https://gerrit.ovirt.org/#/c/83920/4 - https://gerrit.ovirt.org/#/c/83918/2 Also following tests with no engine patches were also successful: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3780/ so this eliminates suspicion for unrelated infra issues. I think that is sufficient grounds for suspicion. >> >> This seems to be a fairly consistent regression as all patches that >> follow the patch above exhibit the same issue when tested, this list >> of similarly failing patches includes: >> - https://gerrit.ovirt.org/#/c/83940/2 >> - https://gerrit.ovirt.org/#/c/83608/7 >> - https://gerrit.ovirt.org/#/c/83815/3 >> The following patch is an exception because it simply failed to build: >> - https://gerrit.ovirt.org/#/c/83370/9 >> >> Link to Job: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/ >> >> Link to all logs: >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/3770/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/ >> >> Error snippet from log: >> >> <error> >> >> Fault reason is "Operation Failed". Fault detail is "[Failed to attach >> Storage due to an error on the Data Center master Storage Domain. >> -Please activate the master Storage Domain first.]". HTTP response code is >> 409. >> >> </error> >> >> >> >> -- >> Barak Korren >> RHV DevOps team , RHCE, RHCi >> Red Hat EMEA >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

Commit 86bf82e746404145e8b97df46f514086e4f82e69 is probably the offending commit, taking a deeper look now, should have a fix within the hour

I merged 83920 with dependencies. Perhaps gerrit doesn't handle such chains properly (or perhaps our CI is misusing the API?). WRT current issues, posted two patches upstream: - https://gerrit.ovirt.org/#/c/83986/ - fixes the GWT build that was broken this morning, preventing a proper build altogether - https://gerrit.ovirt.org/#/c/83987/ - should fix the OST regression you noted. I'm waiting for the CI to finish running on both and will merge. On Mon, Nov 13, 2017 at 11:26 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > On 13 November 2017 at 11:10, Allon Mureinik <amureini(a)redhat.com&gt; wrote: > > Commit 86bf82e746404145e8b97df46f514086e4f82e69 is probably the > offending > > commit, taking a deeper look now, should have a fix within the hour > > > > Hmm, I see what happened there, that commit is > https://gerrit.ovirt.org/c/83920, so it was tested at: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3769 > and passed. > > BUT, it was tested along with https://gerrit.ovirt.org/#/c/83918/2 > that was merged earlier as far as git ordering goes, but somehow was > assumed to be a later patch. When we test multiple patches from the > same project we essentially use packages built from the one we deem to > be the latest. So in this case only the packages from 83918 were > actually tested. > > We went to great lengths to preserve correct patch ordering by > preserving the order of event notifications reaching us from gerrit. > But here it seems we ended out with out-of-order patches. Do you have > any idea how did we end up with this? Do you remember what was the > sequence in which you clicked the 'merge' button on the different > patches? > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >