DAO tests failing

Tue Nov 25 06:47:51 UTC 2014

Hi Eyal,

No problem, I understand the situation - did not mean to sound harsh,
sorry if it came off like that! Anyway, all is good now :)

(there was a likely a problem both with my patch and with Jenkins - when
I ran the tests locally I got different errors and it was easier to
troubleshoot)

Yours, Lior.

On 24/11/14 22:46, Eyal Edri wrote:
> Lior,
> 
> just a note - regardless of if this error is a false positive or not, 
> the infra team is very much understaffed (single man operation almost), at least when it comes
> to jenkins stability, so it's not un-common if there are infra issues and it takes some time to 
> resolve them.
> So as david proposed (on a different thread i think)- if you think a certain job shouldn't fail, please don't block on it.
> Talk to your manager or request the right permissions to remove it from gerrit if you're a maintainer/TL,
> but please make sure you do validate it locally before merging.
> 
> We're planning a full day infra hackathon soon to address many issues on the oVirt infra,
> and everyone will be able to assist if they can/like - an official email with details will be
> sent in the following days.
> 
> Eyal.
> 
> 
> 
> ----- Original Message -----
>> From: "David Caro" <dcaroest at redhat.com>
>> To: "Lior Vernia" <lvernia at redhat.com>
>> Cc: infra at ovirt.org
>> Sent: Monday, November 24, 2014 4:27:00 PM
>> Subject: Re: DAO tests failing
>>
>> On 11/24, Lior Vernia wrote:
>>>
>>>
>>> On 24/11/14 14:25, David Caro wrote:
>>>> On 11/24, Lior Vernia wrote:
>>>>>
>>>>>
>>>>> On 24/11/14 14:16, David Caro wrote:
>>>>>> On 11/24, Allon Mureinik wrote:
>>>>>>> Looks like recreating the database fails:
>>>>>>>
>>>>>>> 08:54:54 [ovirt-engine_master_dao-unit-tests_created] $ /bin/sh
>>>>>>> /tmp/hudson8811709354471707251.sh
>>>>>>> 08:54:54
>>>>>>> /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created
>>>>>>> 08:54:54
>>>>>>> /home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created
>>>>>>> 08:54:55 could not change directory to
>>>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
>>>>>>> 08:54:55 ERROR:  role "engine" already exists
>>>>>>> 08:54:55 could not change directory to
>>>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
>>>>>>> 08:54:55 ALTER ROLE
>>>>>>> 08:54:56 could not change directory to
>>>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
>>>>>>> 08:54:56 dropdb: database removal failed: ERROR:  database "engine"
>>>>>>> does not exist
>>>>>>> 08:54:56 could not change directory to
>>>>>>> "/home/jenkins/workspace/ovirt-engine_master_dao-unit-tests_created"
>>>>>>> 08:54:58 CREATE DATABASE
>>>>>>> 08:54:59 Creating schema
>>>>>>> engine at localhost:5432/ovirt_engine_master_dao_unit_tests_created_6082
>>>>>>>
>>>>>>
>>>>>> Those are not errors, are just unfiltered messages when it tries to
>>>>>> make sure the database is not there and the user has enough rights at
>>>>>> the startup of the job.
>>>>>>
>>>>>> The first issue is:
>>>>>>
>>>>>> 00:09:49.369 2014-11-24 09:03:47,301 SEVERE
>>>>>> [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback
>>>>>> doInConnection] Can't execute batch: Batch entry 0 select * from
>>>>>> public.insertnumanode(CAST ('830fd8d0-9332-4d81-bb80-54beee7d5b59' AS
>>>>>> uuid),CAST (NULL AS uuid),CAST ('77296e00-0cad-4e5a-9299-008a7b6f4355'
>>>>>> AS uuid),CAST ('0' AS int2),CAST ('0' AS int8),CAST ('4' AS int2),CAST
>>>>>> (NULL AS int8),CAST (NULL AS int4),CAST (NULL AS numeric),CAST (NULL
>>>>>> AS numeric),CAST (NULL AS numeric),CAST (NULL AS int4),CAST (NULL AS
>>>>>> text)) as result was aborted.  Call getNextException to see the cause.
>>>>>> 00:09:49.441 2014-11-24 09:03:47,303 SEVERE
>>>>>> [org.ovirt.engine.core.dal.dbbroker.BatchProcedureExecutionConnectionCallback
>>>>>> doInConnection] Can't execute batch. Next exception is: ERROR: insert
>>>>>> or update on table "numa_node" violates foreign key constraint
>>>>>> "fk_numa_node_vm"
>>>>>>
>>>>>> What to me looks like a real issue. I'll do a couple more checks, but
>>>>>> I don't know how the tests work or what the code does, that's your
>>>>>> domain.
>>>>>>
>>>>>
>>>>> I don't know if that specifically is a real issue or not (as it isn't
>>>>> related to my patch and I haven't researched it), I do see however that
>>>>> there are ultimately 433 failures and 371 errors, and that is spread
>>>>> across many independent tests.
>>>>
>>>> You should be aware that it's not just your patch that runs, but all
>>>> the patches yours depends on, that I see are quite a lot, are you sure
>>>> that none of them introduces those failures?
>>>>
>>>
>>> It had failed the same way when it didn't depend on anything unmerged -
>>> I recently rebased it on two other patches so I could merge them first
>>> (and neither of them touches the dal project).
>>>
>>> What do you mean by "quite a lot"? Do you see more than two?
>>
>> From the gerrit page I see that the patch depended on other 5 patches,
>> 3 of those are already merged. So no, right now I only see two.
>>
>>
>>> Could these be environmental issues?
>>
>> There's always a possibility, but I think that it's quite improbable
>> in this case.
>>
>>> It's possible that my patch causes all of this,
>>> but I don't see how. Could it be the Allon's got it right, and the DB
>>> isn't being constructed properly?
>>
>> The DB is being constructed as part of the test. The messages Allon
>> pointed out are not errors but just unfiltered messages (it just drops
>> the db directly instead of checking if it exists first for example).
>> I don't see any issues on the db creation.
>>
>> ERROR: insert or update on table "network" violates foreign key
>> constraint "fk_network_qos_id"   Detail: Key
>> (qos_id)=(de956031-6be2-43d6-bb90-5191c9253318) is not present in
>> table "qos".
>>
>>
>> This failure is quite a specific one, you can try looking for the
>> point where that key should be added. It looks like you are trying to
>> add an entry to the network table with an id from the qos that does
>> not exist. Most of the other failures also seem related to foreign
>> keys not being consistent.
>>
>> That does not seem like an environmental issue. The db did not exist
>> at the start of the test qnd it works well when run from branch HEAD,
>> so it's an issue that's introduced by your patch or any it depends
>> on. Keep in mind that it does a checkout of the change and not a
>> rebase.
>>
>>
>> I see that the first failure (if I'm not mistaken) seems to be:
>>
>> 00:02:38.691 Running
>> org.ovirt.engine.core.dao.VmAndTemplatesGenerationsDaoTest
>> 00:02:39.007 Tests run: 18, Failures: 13, Errors: 1, Skipped: 0, Time
>> elapsed: 0.325 sec <<< FAILURE!
>>
>> What might lead to an entry not being created on the db and the other
>> failures.
>>
>> I don't know the code or the tests code, so I don't think I can help
>> you more than that.
>>
>>
>>>
>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Lior Vernia" <lvernia at redhat.com>
>>>>>>>> To: infra at ovirt.org
>>>>>>>> Sent: Monday, November 24, 2014 1:37:25 PM
>>>>>>>> Subject: DAO tests failing
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I've noticed recurrent failures of DAO tests on one of my patches,
>>>>>>>> this
>>>>>>>> looks like something systematic as there are hundreds of failures in
>>>>>>>> unrelated files.
>>>>>>>>
>>>>>>>> http://gerrit.ovirt.org/#/c/34121/
>>>>>>>>
>>>>>>>> Talked to dcaro about it on Thursday, but have been rebasing and
>>>>>>>> re-running the tests and they keep failing.
>>>>>>>>
>>>>>>>> Thanks, Lior.
>>>>>>>> _______________________________________________
>>>>>>>> Infra mailing list
>>>>>>>> Infra at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Infra mailing list
>>>>>>> Infra at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>>
>>>>
>>
>> --
>> David Caro
>>
>> Red Hat S.L.
>> Continuous Integration Engineer - EMEA ENG Virtualization R&D
>>
>> Tel.: +420 532 294 605
>> Email: dcaro at redhat.com
>> Web: www.redhat.com
>> RHT Global #: 82-62605
>>
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>