Looks like recreating the database fails: =20 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/huds=

08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_creat=

08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_creat=

08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-eng=

08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-eng=

08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-eng=

08:54:56 dropdb: database removal failed: ERROR: database "engine" does =

08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-eng=

08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_un=

=20

=20

=20

=20

=20

=20

=20

=20

=20

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUcyGVAAoJEEBxx+HSYmnD9TYH/390JM0zMkcUuPwjLNxrOiKh GG6lygZQ2bSIrbDteEAInWnoDxYPZk9SThgeq7rPDHBmkSCnPmg4R8OWu/1ZVTUF cbAKUtWKHeKG91+0L53X/Rzr9C9P4GMNGu+80MgA4pYYv6JDd3nIQeMcEYTrcdLD Qo5+saptj2gIJKj9QIpV4oKiUCdzF3NAh0ecDM8Ed0lhmkgFtOrmH/w3ITABlHxd Ths+O3U37phCrVPgfnuVvYjP5lcUcXawpzTqzpGVrwRYqRsWNCwLi5sHEXy7EveE +o+hRAO5Ad4OA37Nn9H/0EwL1mRllia/7trgv79tMJYA7y6EGUS84BPfakGyMmM= =hPr/ -----END PGP SIGNATURE-----

=20 =20 On 24/11/14 14:16, David Caro wrote: > On 11/24, Allon Mureinik wrote: >> Looks like recreating the database fails: >> >> 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/h=

>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cr=

>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cr=

>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-=

>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-=

>> 08:54:55 ALTER ROLE >> 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-=

>> 08:54:56 dropdb: database removal failed: ERROR: database "engine" do=

>> 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-=

>> 08:54:58 CREATE DATABASE >> 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao=

>> >=20 > Those are not errors, are just unfiltered messages when it tries to > make sure the database is not there and the user has enough rights at > the startup of the job. >=20 > The first issue is: >=20 > 00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal=

> 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal=

>=20 > What to me looks like a real issue. I'll do a couple more checks, but > I don't know how the tests work or what the code does, that's your > domain. >=20 =20 I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.

=20 >> >> >> ----- Original Message ----- >>> From: "Lior Vernia" <lvernia(a)redhat.com&gt; >>> To: infra(a)ovirt.org >>> Sent: Monday, November 24, 2014 1:37:25 PM >>> Subject: DAO tests failing >>> >>> Hi, >>> >>> I've noticed recurrent failures of DAO tests on one of my patches, th=

>>> looks like something systematic as there are hundreds of failures in >>> unrelated files. >>> >>> http://gerrit.ovirt.org/#/c/34121/ >>> >>> Talked to dcaro about it on Thursday, but have been rebasing and >>> re-running the tests and they keep failing. >>> >>> Thanks, Lior. >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >>> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >=20

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUcyOxAAoJEEBxx+HSYmnDjZwH/2Iq0q9Tee3+V46e0K32ktuP 2c9Dgr+mlLrUXaiVNrb2MXjbtlaQKsJgX+6ABj54M0sufKlZ6XDvsjM1KbJSQKmR U2kq9Qe5LXb9/UVaZjGPhgt/wq7nNR9LvB8p9GClCMLlWOP/46UdEZ4rDIKPzQ+V 2TrT9L3xUdMPOvG/LLoO4KfbgdnVaapOHJEmqcd7uVSxwfONcqVUlo/Pm2F97LW1 Rvku2h1kXC77+8INGDtdS/S6OHhnPwQOqsdNtlutvBPpMEt5IpMMUm/V7ITtYBb1 Q7et4EC112gFRNTu+WaHPFMOLshIQvXYtyWDqnALdj1f/8V55eyU2i7iPgpe3+U= =2bG3 -----END PGP SIGNATURE-----

=20 =20 On 24/11/14 14:25, David Caro wrote: > On 11/24, Lior Vernia wrote: >> >> >> On 24/11/14 14:16, David Caro wrote: >>> On 11/24, Allon Mureinik wrote: >>>> Looks like recreating the database fails: >>>> >>>> 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp=

>>>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_=

>>>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_=

>>>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovir=

>>>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovir=

>>>> 08:54:55 ALTER ROLE >>>> 08:54:56 could not change directory to "/home/jenkins/workspace/ovir=

>>>> 08:54:56 dropdb: database removal failed: ERROR: database "engine" =

>>>> 08:54:56 could not change directory to "/home/jenkins/workspace/ovir=

>>>> 08:54:58 CREATE DATABASE >>>> 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_d=

>>>> >>> >>> Those are not errors, are just unfiltered messages when it tries to >>> make sure the database is not there and the user has enough rights at >>> the startup of the job. >>> >>> The first issue is: >>> >>> 00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.d=

>>> 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.d=

>>> >>> What to me looks like a real issue. I'll do a couple more checks, but >>> I don't know how the tests work or what the code does, that's your >>> domain. >>> >> >> I don't know if that specifically is a real issue or not (as it isn't >> related to my patch and I haven't researched it), I do see however that >> there are ultimately 433 failures and 371 errors, and that is spread >> across many independent tests. >=20 > You should be aware that it's not just your patch that runs, but all > the patches yours depends on, that I see are quite a lot, are you sure > that none of them introduces those failures? >=20 =20 It had failed the same way when it didn't depend on anything unmerged - I recently rebased it on two other patches so I could merge them first (and neither of them touches the dal project). =20 What do you mean by "quite a lot"? Do you see more than two?

Could these be environmental issues?

It's possible that my patch causes all of this, but I don't see how. Could it be the Allon's got it right, and the DB isn't being constructed properly?

=20 >> >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Lior Vernia" <lvernia(a)redhat.com&gt; >>>>> To: infra(a)ovirt.org >>>>> Sent: Monday, November 24, 2014 1:37:25 PM >>>>> Subject: DAO tests failing >>>>> >>>>> Hi, >>>>> >>>>> I've noticed recurrent failures of DAO tests on one of my patches, =

>>>>> looks like something systematic as there are hundreds of failures in >>>>> unrelated files. >>>>> >>>>> http://gerrit.ovirt.org/#/c/34121/ >>>>> >>>>> Talked to dcaro about it on Thursday, but have been rebasing and >>>>> re-running the tests and they keep failing. >>>>> >>>>> Thanks, Lior. >>>>> _______________________________________________ >>>>> Infra mailing list >>>>> Infra(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>> >>>> _______________________________________________ >>>> Infra mailing list >>>> Infra(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/infra >>> >=20

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUc0A0AAoJEEBxx+HSYmnDICUH/1coBQz1cmCwETUrTsMji5ri 30GL8c97nvDYXJV9Zb3fK9RBDu14PN/IM0a11uoM29Vn76BrAcrKB8Gv0VFT7pu5 8tRTMKj5QYb5SR7VSUQGqrWZ8S4FRwxam2lHLXRnng5B+C6QYLFu25yCOio7FBhf s6RAivXz1TFENCvTihGwe6ny3hwSZiMPsAHsn+YeY2tOh11psoCHw8ueZ5bYWP4g LkW91yV7SlOzN43UM+7x1KpCmcpujYE8VqB7GOQ7/y2Ym97LL/pla51cTxrplTOt 8TgAkESEDoMeouNwqBRxf5ZAuITYqMRAbmBGuq0/W1rB3r65W/a/y8cvqpEBkns= =5WW9 -----END PGP SIGNATURE-----

Lior, just a note - regardless of if this error is a false positive or not, the infra team is very much understaffed (single man operation almost), at least when it comes to jenkins stability, so it's not un-common if there are infra issues and it takes some time to resolve them. So as david proposed (on a different thread i think)- if you think a certain job shouldn't fail, please don't block on it. Talk to your manager or request the right permissions to remove it from gerrit if you're a maintainer/TL, but please make sure you do validate it locally before merging. We're planning a full day infra hackathon soon to address many issues on the oVirt infra, and everyone will be able to assist if they can/like - an official email with details will be sent in the following days. Eyal. ----- Original Message ----- > From: "David Caro" <dcaroest(a)redhat.com&gt; > To: "Lior Vernia" <lvernia(a)redhat.com&gt; > Cc: infra(a)ovirt.org > Sent: Monday, November 24, 2014 4:27:00 PM > Subject: Re: DAO tests failing > > On 11/24, Lior Vernia wrote: >> >> >> On 24/11/14 14:25, David Caro wrote: >>> On 11/24, Lior Vernia wrote: >>>> >>>> >>>> On 24/11/14 14:16, David Caro wrote: >>>>> On 11/24, Allon Mureinik wrote: >>>>>> Looks like recreating the database fails: >>>>>> >>>>>> 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh >>>>>> /tmp/hudson8811709354471707251.sh >>>>>> 08:54:54 >>>>>> /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created >>>>>> 08:54:54 >>>>>> /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created >>>>>> 08:54:55 could not change directory to >>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" >>>>>> 08:54:55 ERROR: role "engine" already exists >>>>>> 08:54:55 could not change directory to >>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" >>>>>> 08:54:55 ALTER ROLE >>>>>> 08:54:56 could not change directory to >>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" >>>>>> 08:54:56 dropdb: database removal failed: ERROR: database "engine" >>>>>> does not exist >>>>>> 08:54:56 could not change directory to >>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" >>>>>> 08:54:58 CREATE DATABASE >>>>>> 08:54:59 Creating schema >>>>>> engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082 >>>>>> >>>>> >>>>> Those are not errors, are just unfiltered messages when it tries to >>>>> make sure the database is not there and the user has enough rights at >>>>> the startup of the job. >>>>> >>>>> The first issue is: >>>>> >>>>> 00:09:49.369 2014-11-24 09:03:47,301 SEVERE >>>>> [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback >>>>> doInConnection] Can't execute batch: Batch entry 0 select * from >>>>> public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS >>>>> uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' >>>>> AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST >>>>> (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL >>>>> AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS >>>>> text)) as result was aborted. Call getNextException to see the cause. >>>>> 00:09:49.441 2014-11-24 09:03:47,303 SEVERE >>>>> [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback >>>>> doInConnection] Can't execute batch. Next exception is: ERROR: insert >>>>> or update on table "numa_node" violates foreign key constraint >>>>> "fk_numa_node_vm" >>>>> >>>>> What to me looks like a real issue. I'll do a couple more checks, but >>>>> I don't know how the tests work or what the code does, that's your >>>>> domain. >>>>> >>>> >>>> I don't know if that specifically is a real issue or not (as it isn't >>>> related to my patch and I haven't researched it), I do see however that >>>> there are ultimately 433 failures and 371 errors, and that is spread >>>> across many independent tests. >>> >>> You should be aware that it's not just your patch that runs, but all >>> the patches yours depends on, that I see are quite a lot, are you sure >>> that none of them introduces those failures? >>> >> >> It had failed the same way when it didn't depend on anything unmerged - >> I recently rebased it on two other patches so I could merge them first >> (and neither of them touches the dal project). >> >> What do you mean by "quite a lot"? Do you see more than two? > > From the gerrit page I see that the patch depended on other 5 patches, > 3 of those are already merged. So no, right now I only see two. > > >> Could these be environmental issues? > > There's always a possibility, but I think that it's quite improbable > in this case. > >> It's possible that my patch causes all of this, >> but I don't see how. Could it be the Allon's got it right, and the DB >> isn't being constructed properly? > > The DB is being constructed as part of the test. The messages Allon > pointed out are not errors but just unfiltered messages (it just drops > the db directly instead of checking if it exists first for example). > I don't see any issues on the db creation. > > ERROR: insert or update on table "network" violates foreign key > constraint "fk_network_qos_id" Detail: Key > (qos_id)=(de956031-6be2-43d6-bb90-5191c9253318) is not present in > table "qos". > > > This failure is quite a specific one, you can try looking for the > point where that key should be added. It looks like you are trying to > add an entry to the network table with an id from the qos that does > not exist. Most of the other failures also seem related to foreign > keys not being consistent. > > That does not seem like an environmental issue. The db did not exist > at the start of the test qnd it works well when run from branch HEAD, > so it's an issue that's introduced by your patch or any it depends > on. Keep in mind that it does a checkout of the change and not a > rebase. > > > I see that the first failure (if I'm not mistaken) seems to be: > > 00:02:38.691 Running > org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDaoTest > 00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time > elapsed: 0.325 sec <<< FAILURE! > > What might lead to an entry not being created on the db and the other > failures. > > I don't know the code or the tests code, so I don't think I can help > you more than that. > > >> >>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Lior Vernia" <lvernia(a)redhat.com&gt; >>>>>>> To: infra(a)ovirt.org >>>>>>> Sent: Monday, November 24, 2014 1:37:25 PM >>>>>>> Subject: DAO tests failing >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I've noticed recurrent failures of DAO tests on one of my patches, >>>>>>> this >>>>>>> looks like something systematic as there are hundreds of failures in >>>>>>> unrelated files. >>>>>>> >>>>>>> http://gerrit.ovirt.org/#/c/34121/ >>>>>>> >>>>>>> Talked to dcaro about it on Thursday, but have been rebasing and >>>>>>> re-running the tests and they keep failing. >>>>>>> >>>>>>> Thanks, Lior. >>>>>>> _______________________________________________ >>>>>>> Infra mailing list >>>>>>> Infra(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>>>> >>>>>> _______________________________________________ >>>>>> Infra mailing list >>>>>> Infra(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>> >>> > > -- > David Caro > > Red Hat S.L. > Continuous Integration Engineer - EMEA ENG Virtualization R&D > > Tel.: +420 532 294 605 > Email: dcaro(a)redhat.com > Web: www.redhat.com > RHT Global #: 82-62605 > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra >