Hi, I've noticed recurrent failures of DAO tests on one of my patches, this looks like something systematic as there are hundreds of failures in unrelated files. http://gerrit.ovirt.org/#/c/34121/ Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing. Thanks, Lior.
Looks like recreating the database fails: 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hudson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" does not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082 ----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing
Hi,
I've noticed recurrent failures of DAO tests on one of my patches, this looks like something systematic as there are hundreds of failures in unrelated files.
http://gerrit.ovirt.org/#/c/34121/
Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing.
Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--7uYPyRQQ5N0D02nI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails: =20 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/huds= on8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_creat= ed 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_creat= ed 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-eng= ine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-eng= ine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-eng= ine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" does = not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-eng= ine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_un= it_tests_created_6082 =20
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job. The first issue is: 00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.dbb= roker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execu= te batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0= -9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-= 0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8)= ,CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS n= umeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),C= AST (NULL AS text)) as result was aborted. Call getNextException to see th= e cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.dbb= roker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execu= te batch. Next exception is: ERROR: insert or update on table "numa_node" v= iolates foreign key constraint "fk_numa_node_vm" What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
=20 =20 ----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing =20 Hi, =20 I've noticed recurrent failures of DAO tests on one of my patches, this looks like something systematic as there are hundreds of failures in unrelated files. =20 http://gerrit.ovirt.org/#/c/34121/ =20 Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing. =20 Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --7uYPyRQQ5N0D02nI Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUcyGVAAoJEEBxx+HSYmnD9TYH/390JM0zMkcUuPwjLNxrOiKh GG6lygZQ2bSIrbDteEAInWnoDxYPZk9SThgeq7rPDHBmkSCnPmg4R8OWu/1ZVTUF cbAKUtWKHeKG91+0L53X/Rzr9C9P4GMNGu+80MgA4pYYv6JDd3nIQeMcEYTrcdLD Qo5+saptj2gIJKj9QIpV4oKiUCdzF3NAh0ecDM8Ed0lhmkgFtOrmH/w3ITABlHxd Ths+O3U37phCrVPgfnuVvYjP5lcUcXawpzTqzpGVrwRYqRsWNCwLi5sHEXy7EveE +o+hRAO5Ad4OA37Nn9H/0EwL1mRllia/7trgv79tMJYA7y6EGUS84BPfakGyMmM= =hPr/ -----END PGP SIGNATURE----- --7uYPyRQQ5N0D02nI--
On 24/11/14 14:16, David Caro wrote:
On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails:
08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hudson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" does not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job.
The first issue is:
00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS text)) as result was aborted. Call getNextException to see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch. Next exception is: ERROR: insert or update on table "numa_node" violates foreign key constraint "fk_numa_node_vm"
What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.
----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing
Hi,
I've noticed recurrent failures of DAO tests on one of my patches, this looks like something systematic as there are hundreds of failures in unrelated files.
http://gerrit.ovirt.org/#/c/34121/
Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing.
Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--nEsDIrWrg+hrB7l1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 11/24, Lior Vernia wrote:
On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails:
08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/h= udson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cr= eated 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cr= eated 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-= engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-= engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-= engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" do= es not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-= engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao= _unit_tests_created_6082
=20 Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job. =20 The first issue is: =20 00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal= =2Edbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't= execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('83= 0fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('772= 96e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS= int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NUL= L AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS i= nt4),CAST (NULL AS text)) as result was aborted. Call getNextException to = see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal= =2Edbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't= execute batch. Next exception is: ERROR: insert or update on table "numa_n= ode" violates foreign key constraint "fk_numa_node_vm" =20 What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain. =20 =20 I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that
=20 =20 On 24/11/14 14:16, David Caro wrote: there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.
You should be aware that it's not just your patch that runs, but all the patches yours depends on, that I see are quite a lot, are you sure that none of them introduces those failures?
=20
----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing
Hi,
I've noticed recurrent failures of DAO tests on one of my patches, th=
is
looks like something systematic as there are hundreds of failures in unrelated files.
http://gerrit.ovirt.org/#/c/34121/
Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing.
Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --nEsDIrWrg+hrB7l1 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUcyOxAAoJEEBxx+HSYmnDjZwH/2Iq0q9Tee3+V46e0K32ktuP 2c9Dgr+mlLrUXaiVNrb2MXjbtlaQKsJgX+6ABj54M0sufKlZ6XDvsjM1KbJSQKmR U2kq9Qe5LXb9/UVaZjGPhgt/wq7nNR9LvB8p9GClCMLlWOP/46UdEZ4rDIKPzQ+V 2TrT9L3xUdMPOvG/LLoO4KfbgdnVaapOHJEmqcd7uVSxwfONcqVUlo/Pm2F97LW1 Rvku2h1kXC77+8INGDtdS/S6OHhnPwQOqsdNtlutvBPpMEt5IpMMUm/V7ITtYBb1 Q7et4EC112gFRNTu+WaHPFMOLshIQvXYtyWDqnALdj1f/8V55eyU2i7iPgpe3+U= =2bG3 -----END PGP SIGNATURE----- --nEsDIrWrg+hrB7l1--
On 24/11/14 14:25, David Caro wrote:
On 11/24, Lior Vernia wrote:
On 24/11/14 14:16, David Caro wrote:
On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails:
08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hudson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" does not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job.
The first issue is:
00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS text)) as result was aborted. Call getNextException to see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch. Next exception is: ERROR: insert or update on table "numa_node" violates foreign key constraint "fk_numa_node_vm"
What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.
You should be aware that it's not just your patch that runs, but all the patches yours depends on, that I see are quite a lot, are you sure that none of them introduces those failures?
It had failed the same way when it didn't depend on anything unmerged - I recently rebased it on two other patches so I could merge them first (and neither of them touches the dal project). What do you mean by "quite a lot"? Do you see more than two? Could these be environmental issues? It's possible that my patch causes all of this, but I don't see how. Could it be the Allon's got it right, and the DB isn't being constructed properly?
----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing
Hi,
I've noticed recurrent failures of DAO tests on one of my patches, this looks like something systematic as there are hundreds of failures in unrelated files.
http://gerrit.ovirt.org/#/c/34121/
Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing.
Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--eWbcAUUbgrfSEG1c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 11/24, Lior Vernia wrote:
=20 =20 On 24/11/14 14:25, David Caro wrote:
On 11/24, Lior Vernia wrote:
On 24/11/14 14:16, David Caro wrote:
On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails:
08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp=
08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_= created 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_= created 08:54:55 could not change directory to "/home/jenkins/workspace/ovir= t-engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovir= t-engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovir= t-engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" = does not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovir= t-engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_d= ao_unit_tests_created_6082
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job.
The first issue is:
00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.d= al.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't= execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('83= 0fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('772= 96e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS= int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NUL= L AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS i= nt4),CAST (NULL AS text)) as result was aborted. Call getNextException to = see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.d= al.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't= execute batch. Next exception is: ERROR: insert or update on table "numa_n= ode" violates foreign key constraint "fk_numa_node_vm"
What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests. =20 You should be aware that it's not just your patch that runs, but all
/hudson8811709354471707251.sh the patches yours depends on, that I see are quite a lot, are you sure that none of them introduces those failures? =20 =20 It had failed the same way when it didn't depend on anything unmerged - I recently rebased it on two other patches so I could merge them first (and neither of them touches the dal project). =20 What do you mean by "quite a lot"? Do you see more than two?
=46rom the gerrit page I see that the patch depended on other 5 patches, 3 of those are already merged. So no, right now I only see two.
Could these be environmental issues?
There's always a possibility, but I think that it's quite improbable in this case.
It's possible that my patch causes all of this, but I don't see how. Could it be the Allon's got it right, and the DB isn't being constructed properly?
=20
----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing
Hi,
I've noticed recurrent failures of DAO tests on one of my patches, =
The DB is being constructed as part of the test. The messages Allon pointed out are not errors but just unfiltered messages (it just drops the db directly instead of checking if it exists first for example). I don't see any issues on the db creation. ERROR: insert or update on table "network" violates foreign key constraint "fk_network_qos_id" Detail: Key (qos_id)=3D(de956031-6be2-43d6-bb90-5191c9253318) is not present in table "qos". This failure is quite a specific one, you can try looking for the point where that key should be added. It looks like you are trying to add an entry to the network table with an id from the qos that does not exist. Most of the other failures also seem related to foreign keys not being consistent. That does not seem like an environmental issue. The db did not exist at the start of the test qnd it works well when run from branch HEAD, so it's an issue that's introduced by your patch or any it depends on. Keep in mind that it does a checkout of the change and not a rebase. I see that the first failure (if I'm not mistaken) seems to be: 00:02:38.691 Running org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDao= Test 00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time elaps= ed: 0.325 sec <<< FAILURE! What might lead to an entry not being created on the db and the other failures. I don't know the code or the tests code, so I don't think I can help you more than that. this
looks like something systematic as there are hundreds of failures in unrelated files.
http://gerrit.ovirt.org/#/c/34121/
Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing.
Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
=20
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --eWbcAUUbgrfSEG1c Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUc0A0AAoJEEBxx+HSYmnDICUH/1coBQz1cmCwETUrTsMji5ri 30GL8c97nvDYXJV9Zb3fK9RBDu14PN/IM0a11uoM29Vn76BrAcrKB8Gv0VFT7pu5 8tRTMKj5QYb5SR7VSUQGqrWZ8S4FRwxam2lHLXRnng5B+C6QYLFu25yCOio7FBhf s6RAivXz1TFENCvTihGwe6ny3hwSZiMPsAHsn+YeY2tOh11psoCHw8ueZ5bYWP4g LkW91yV7SlOzN43UM+7x1KpCmcpujYE8VqB7GOQ7/y2Ym97LL/pla51cTxrplTOt 8TgAkESEDoMeouNwqBRxf5ZAuITYqMRAbmBGuq0/W1rB3r65W/a/y8cvqpEBkns= =5WW9 -----END PGP SIGNATURE----- --eWbcAUUbgrfSEG1c--
Lior, just a note - regardless of if this error is a false positive or not, the infra team is very much understaffed (single man operation almost), at least when it comes to jenkins stability, so it's not un-common if there are infra issues and it takes some time to resolve them. So as david proposed (on a different thread i think)- if you think a certain job shouldn't fail, please don't block on it. Talk to your manager or request the right permissions to remove it from gerrit if you're a maintainer/TL, but please make sure you do validate it locally before merging. We're planning a full day infra hackathon soon to address many issues on the oVirt infra, and everyone will be able to assist if they can/like - an official email with details will be sent in the following days. Eyal. ----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Lior Vernia" <lvernia@redhat.com> Cc: infra@ovirt.org Sent: Monday, November 24, 2014 4:27:00 PM Subject: Re: DAO tests failing
On 11/24, Lior Vernia wrote:
On 24/11/14 14:25, David Caro wrote:
On 11/24, Lior Vernia wrote:
On 24/11/14 14:16, David Caro wrote:
On 11/24, Allon Mureinik wrote:
Looks like recreating the database fails:
08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hudson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" does not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job.
The first issue is:
00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS text)) as result was aborted. Call getNextException to see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch. Next exception is: ERROR: insert or update on table "numa_node" violates foreign key constraint "fk_numa_node_vm"
What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.
You should be aware that it's not just your patch that runs, but all the patches yours depends on, that I see are quite a lot, are you sure that none of them introduces those failures?
It had failed the same way when it didn't depend on anything unmerged - I recently rebased it on two other patches so I could merge them first (and neither of them touches the dal project).
What do you mean by "quite a lot"? Do you see more than two?
From the gerrit page I see that the patch depended on other 5 patches, 3 of those are already merged. So no, right now I only see two.
Could these be environmental issues?
There's always a possibility, but I think that it's quite improbable in this case.
It's possible that my patch causes all of this, but I don't see how. Could it be the Allon's got it right, and the DB isn't being constructed properly?
The DB is being constructed as part of the test. The messages Allon pointed out are not errors but just unfiltered messages (it just drops the db directly instead of checking if it exists first for example). I don't see any issues on the db creation.
ERROR: insert or update on table "network" violates foreign key constraint "fk_network_qos_id" Detail: Key (qos_id)=(de956031-6be2-43d6-bb90-5191c9253318) is not present in table "qos".
This failure is quite a specific one, you can try looking for the point where that key should be added. It looks like you are trying to add an entry to the network table with an id from the qos that does not exist. Most of the other failures also seem related to foreign keys not being consistent.
That does not seem like an environmental issue. The db did not exist at the start of the test qnd it works well when run from branch HEAD, so it's an issue that's introduced by your patch or any it depends on. Keep in mind that it does a checkout of the change and not a rebase.
I see that the first failure (if I'm not mistaken) seems to be:
00:02:38.691 Running org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDaoTest 00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE!
What might lead to an entry not being created on the db and the other failures.
I don't know the code or the tests code, so I don't think I can help you more than that.
----- Original Message ----- > From: "Lior Vernia" <lvernia@redhat.com> > To: infra@ovirt.org > Sent: Monday, November 24, 2014 1:37:25 PM > Subject: DAO tests failing > > Hi, > > I've noticed recurrent failures of DAO tests on one of my patches, > this > looks like something systematic as there are hundreds of failures in > unrelated files. > > http://gerrit.ovirt.org/#/c/34121/ > > Talked to dcaro about it on Thursday, but have been rebasing and > re-running the tests and they keep failing. > > Thanks, Lior. > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Hi Eyal, No problem, I understand the situation - did not mean to sound harsh, sorry if it came off like that! Anyway, all is good now :) (there was a likely a problem both with my patch and with Jenkins - when I ran the tests locally I got different errors and it was easier to troubleshoot) Yours, Lior. On 24/11/14 22:46, Eyal Edri wrote:
Lior,
just a note - regardless of if this error is a false positive or not, the infra team is very much understaffed (single man operation almost), at least when it comes to jenkins stability, so it's not un-common if there are infra issues and it takes some time to resolve them. So as david proposed (on a different thread i think)- if you think a certain job shouldn't fail, please don't block on it. Talk to your manager or request the right permissions to remove it from gerrit if you're a maintainer/TL, but please make sure you do validate it locally before merging.
We're planning a full day infra hackathon soon to address many issues on the oVirt infra, and everyone will be able to assist if they can/like - an official email with details will be sent in the following days.
Eyal.
----- Original Message -----
From: "David Caro" <dcaroest@redhat.com> To: "Lior Vernia" <lvernia@redhat.com> Cc: infra@ovirt.org Sent: Monday, November 24, 2014 4:27:00 PM Subject: Re: DAO tests failing
On 11/24, Lior Vernia wrote:
On 24/11/14 14:25, David Caro wrote:
On 11/24, Lior Vernia wrote:
On 24/11/14 14:16, David Caro wrote:
On 11/24, Allon Mureinik wrote: > Looks like recreating the database fails: > > 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh > /tmp/hudson8811709354471707251.sh > 08:54:54 > /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created > 08:54:54 > /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created > 08:54:55 could not change directory to > "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" > 08:54:55 ERROR: role "engine" already exists > 08:54:55 could not change directory to > "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" > 08:54:55 ALTER ROLE > 08:54:56 could not change directory to > "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" > 08:54:56 dropdb: database removal failed: ERROR: database "engine" > does not exist > 08:54:56 could not change directory to > "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created" > 08:54:58 CREATE DATABASE > 08:54:59 Creating schema > engine@localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082 >
Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at the startup of the job.
The first issue is:
00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS text)) as result was aborted. Call getNextException to see the cause. 00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch. Next exception is: ERROR: insert or update on table "numa_node" violates foreign key constraint "fk_numa_node_vm"
What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain.
I don't know if that specifically is a real issue or not (as it isn't related to my patch and I haven't researched it), I do see however that there are ultimately 433 failures and 371 errors, and that is spread across many independent tests.
You should be aware that it's not just your patch that runs, but all the patches yours depends on, that I see are quite a lot, are you sure that none of them introduces those failures?
It had failed the same way when it didn't depend on anything unmerged - I recently rebased it on two other patches so I could merge them first (and neither of them touches the dal project).
What do you mean by "quite a lot"? Do you see more than two?
From the gerrit page I see that the patch depended on other 5 patches, 3 of those are already merged. So no, right now I only see two.
Could these be environmental issues?
There's always a possibility, but I think that it's quite improbable in this case.
It's possible that my patch causes all of this, but I don't see how. Could it be the Allon's got it right, and the DB isn't being constructed properly?
The DB is being constructed as part of the test. The messages Allon pointed out are not errors but just unfiltered messages (it just drops the db directly instead of checking if it exists first for example). I don't see any issues on the db creation.
ERROR: insert or update on table "network" violates foreign key constraint "fk_network_qos_id" Detail: Key (qos_id)=(de956031-6be2-43d6-bb90-5191c9253318) is not present in table "qos".
This failure is quite a specific one, you can try looking for the point where that key should be added. It looks like you are trying to add an entry to the network table with an id from the qos that does not exist. Most of the other failures also seem related to foreign keys not being consistent.
That does not seem like an environmental issue. The db did not exist at the start of the test qnd it works well when run from branch HEAD, so it's an issue that's introduced by your patch or any it depends on. Keep in mind that it does a checkout of the change and not a rebase.
I see that the first failure (if I'm not mistaken) seems to be:
00:02:38.691 Running org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDaoTest 00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE!
What might lead to an entry not being created on the db and the other failures.
I don't know the code or the tests code, so I don't think I can help you more than that.
> > > ----- Original Message ----- >> From: "Lior Vernia" <lvernia@redhat.com> >> To: infra@ovirt.org >> Sent: Monday, November 24, 2014 1:37:25 PM >> Subject: DAO tests failing >> >> Hi, >> >> I've noticed recurrent failures of DAO tests on one of my patches, >> this >> looks like something systematic as there are hundreds of failures in >> unrelated files. >> >> http://gerrit.ovirt.org/#/c/34121/ >> >> Talked to dcaro about it on Thursday, but have been rebasing and >> re-running the tests and they keep failing. >> >> Thanks, Lior. >> _______________________________________________ >> Infra mailing list >> Infra@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> > _______________________________________________ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra
-- David Caro
Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Looks like recreating the database fails: =20 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hu= dson8811709354471707251.sh 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cre= ated 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_cre= ated 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-e= ngine_master_dao-unit-tests_created" 08:54:55 ERROR: role "engine" already exists 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-e= ngine_master_dao-unit-tests_created" 08:54:55 ALTER ROLE 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-e= ngine_master_dao-unit-tests_created" 08:54:56 dropdb: database removal failed: ERROR: database "engine" doe= s not exist 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-e= ngine_master_dao-unit-tests_created" 08:54:58 CREATE DATABASE 08:54:59 Creating schema engine@localhost:5432/ovirt_engine_master_dao_= unit_tests_created_6082 =20 =20 Those are not errors, are just unfiltered messages when it tries to make sure the database is not there and the user has enough rights at
On 11/24, Allon Mureinik wrote: the startup of the job. =20 The first issue is: =20 00:09:49.369 2014-11-24 09:03:47,301 SEVERE [org.ovirt.engine.core.dal.d= bbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't exe= cute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8= d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e0= 0-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int= 8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS= numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4)= ,CAST (NULL AS text)) as result was aborted. Call getNextException to see =
--dMyqICaxQaaUjrCL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Runnig on latest merged works properly: http://jenkins.ovirt.org/job/ovirt-engine_master_dao-unit-tests_created/= 6088/ On 11/24, David Caro wrote: the cause.
00:09:49.441 2014-11-24 09:03:47,303 SEVERE [org.ovirt.engine.core.dal.d= bbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't exe= cute batch. Next exception is: ERROR: insert or update on table "numa_node"= violates foreign key constraint "fk_numa_node_vm" =20 What to me looks like a real issue. I'll do a couple more checks, but I don't know how the tests work or what the code does, that's your domain. =20
=20 =20 ----- Original Message -----
From: "Lior Vernia" <lvernia@redhat.com> To: infra@ovirt.org Sent: Monday, November 24, 2014 1:37:25 PM Subject: DAO tests failing =20 Hi, =20 I've noticed recurrent failures of DAO tests on one of my patches, th= is looks like something systematic as there are hundreds of failures in unrelated files. =20 http://gerrit.ovirt.org/#/c/34121/ =20 Talked to dcaro about it on Thursday, but have been rebasing and re-running the tests and they keep failing. =20 Thanks, Lior. _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20
Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra =20 --=20 David Caro =20 Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D =20 Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --dMyqICaxQaaUjrCL Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUcyNeAAoJEEBxx+HSYmnDVVcH/RiX4ee3+qOKreGkbCRtWy66 ubaC1zeWLvYIGk24bDVnI5+N36hANVPtALIukL+1g7vCm/IwhhatGdbwnV8m/YFX bvX6cjMUyMBGSoNkLXc6Q+2pNreVTyiKO8oB3KZpdRClm6wkGunG2AGVKMYdy95u 1Rp09CLCCKEVPd0fbOp9TbFrkDFh1u8ozcKWryTVAy3YvJAAgMXfDeTj6l0nE6bZ jT9agVl7MsZP69iDrg6WvzUUkbeeBaDhsPT1g/pYBdI6Bt/HbvOA6LZxUEziOFfs ngirr8pO4Uq4OjCf21AqqV7No2tQzJ17Itpf1AVuJScqAMRFmkbAtsD/ht0BITU= =Rg2+ -----END PGP SIGNATURE----- --dMyqICaxQaaUjrCL--
participants (4)
-
Allon Mureinik -
David Caro -
Eyal Edri -
Lior Vernia