DAO tests failing

Mon Nov 24 14:27:00 UTC 2014

On 11/24, Lior Vernia wrote:
> 
> 
> On 24/11/14 14:25, David Caro wrote:
> > On 11/24, Lior Vernia wrote:
> >>
> >>
> >> On 24/11/14 14:16, David Caro wrote:
> >>> On 11/24, Allon Mureinik wrote:
> >>>> Looks like recreating the database fails:
> >>>>
> >>>> 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh /tmp/hudson8811709354471707251.sh
> >>>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created
> >>>> 08:54:54 /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created
> >>>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
> >>>> 08:54:55 ERROR:  role "engine" already exists
> >>>> 08:54:55 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
> >>>> 08:54:55 ALTER ROLE
> >>>> 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
> >>>> 08:54:56 dropdb: database removal failed: ERROR:  database "engine" does not exist
> >>>> 08:54:56 could not change directory to "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
> >>>> 08:54:58 CREATE DATABASE
> >>>> 08:54:59 Creating schema engine at localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082
> >>>>
> >>>
> >>> Those are not errors, are just unfiltered messages when it tries to
> >>> make sure the database is not there and the user has enough rights at
> >>> the startup of the job.
> >>>
> >>> The first issue is:
> >>>
> >>> 00:09:49.369 2014-11-24 09:03:47,301 SEVERE  [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch: Batch entry 0 select * from public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355' AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS text)) as result was aborted.  Call getNextException to see the cause.
> >>> 00:09:49.441 2014-11-24 09:03:47,303 SEVERE  [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback doInConnection] Can't execute batch. Next exception is: ERROR: insert or update on table "numa_node" violates foreign key constraint "fk_numa_node_vm"
> >>>
> >>> What to me looks like a real issue. I'll do a couple more checks, but
> >>> I don't know how the tests work or what the code does, that's your
> >>> domain.
> >>>
> >>
> >> I don't know if that specifically is a real issue or not (as it isn't
> >> related to my patch and I haven't researched it), I do see however that
> >> there are ultimately 433 failures and 371 errors, and that is spread
> >> across many independent tests.
> > 
> > You should be aware that it's not just your patch that runs, but all
> > the patches yours depends on, that I see are quite a lot, are you sure
> > that none of them introduces those failures?
> > 
> 
> It had failed the same way when it didn't depend on anything unmerged -
> I recently rebased it on two other patches so I could merge them first
> (and neither of them touches the dal project).
> 
> What do you mean by "quite a lot"? Do you see more than two?

From the gerrit page I see that the patch depended on other 5 patches,
3 of those are already merged. So no, right now I only see two.

> Could these be environmental issues?

There's always a possibility, but I think that it's quite improbable
in this case.

> It's possible that my patch causes all of this,
> but I don't see how. Could it be the Allon's got it right, and the DB
> isn't being constructed properly?

The DB is being constructed as part of the test. The messages Allon
pointed out are not errors but just unfiltered messages (it just drops
the db directly instead of checking if it exists first for example).
I don't see any issues on the db creation.

ERROR: insert or update on table "network" violates foreign key
constraint "fk_network_qos_id"   Detail: Key
(qos_id)=(de956031-6be2-43d6-bb90-5191c9253318) is not present in
table "qos".

This failure is quite a specific one, you can try looking for the
point where that key should be added. It looks like you are trying to
add an entry to the network table with an id from the qos that does
not exist. Most of the other failures also seem related to foreign
keys not being consistent.

That does not seem like an environmental issue. The db did not exist
at the start of the test qnd it works well when run from branch HEAD,
so it's an issue that's introduced by your patch or any it depends
on. Keep in mind that it does a checkout of the change and not a
rebase.

I see that the first failure (if I'm not mistaken) seems to be:

00:02:38.691 Running org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDaoTest
00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE!

What might lead to an entry not being created on the db and the other
failures.

I don't know the code or the tests code, so I don't think I can help
you more than that.

> 
> >>
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Lior Vernia" <lvernia at redhat.com>
> >>>>> To: infra at ovirt.org
> >>>>> Sent: Monday, November 24, 2014 1:37:25 PM
> >>>>> Subject: DAO tests failing
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I've noticed recurrent failures of DAO tests on one of my patches, this
> >>>>> looks like something systematic as there are hundreds of failures in
> >>>>> unrelated files.
> >>>>>
> >>>>> http://gerrit.ovirt.org/#/c/34121/
> >>>>>
> >>>>> Talked to dcaro about it on Thursday, but have been rebasing and
> >>>>> re-running the tests and they keep failing.
> >>>>>
> >>>>> Thanks, Lior.
> >>>>> _______________________________________________
> >>>>> Infra mailing list
> >>>>> Infra at ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/infra
> >>>>>
> >>>> _______________________________________________
> >>>> Infra mailing list
> >>>> Infra at ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/infra
> >>>
> > 

-- 
David Caro

Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D

Tel.: +420 532 294 605
Email: dcaro at redhat.com
Web: www.redhat.com
RHT Global #: 82-62605
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20141124/cae22031/attachment.sig>