February 2019 - Infra - oVirt List Archives

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 14 Feb '19

14 Feb '19

Failed to run project_setup.sh for: #3578 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-m06pr.

1 0

[JIRA] (OVIRT-2617) Allow rerunning only failed test lanes
by Roman Mohr (oVirt JIRA) 14 Feb '19

14 Feb '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2617?page=com.atlassian.jira.… ] Roman Mohr commented on OVIRT-2617: ----------------------------------- So think about sprint ends. Many people have PRs, they produce load. Things are already a little bit more unstable than usual, which will increase the amount of failed lanes. So people will hit "retest this please" more often ... > Allow rerunning only failed test lanes > -------------------------------------- > > Key: OVIRT-2617 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2617 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roman Mohr > Assignee: infra > > Hi, > In KubeVirt we run a lot of differnent test lanes. Even if our tests are > written in a good way and if the underlying CI infra is stable and fast, > there will always be cases where for-whatever reason one or two tests can > fail out of hundreds in a lane. > It would be very helpful if we could have a /retest command or something > similar where only failed lanes are re-run. That would be similar to how it > is handled in Kubernetes. > Right now it is an all-or-nothing thing. > What I would expect, is that it is still possible to see the test history > somewhere then (e.g. the first two attempts where it failed and then the > third one where it succeeds). > Best Regards, > Roman -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100098)

1 0

[JIRA] (OVIRT-2617) Allow rerunning only failed test lanes
by Roman Mohr (oVirt JIRA) 14 Feb '19

14 Feb '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2617?page=com.atlassian.jira.… ] Roman Mohr commented on OVIRT-2617: ----------------------------------- Well if one suspicious lane fails, we write "retest this please". This runs everything again. > Allow rerunning only failed test lanes > -------------------------------------- > > Key: OVIRT-2617 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2617 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Reporter: Roman Mohr > Assignee: infra > > Hi, > In KubeVirt we run a lot of differnent test lanes. Even if our tests are > written in a good way and if the underlying CI infra is stable and fast, > there will always be cases where for-whatever reason one or two tests can > fail out of hundreds in a lane. > It would be very helpful if we could have a /retest command or something > similar where only failed lanes are re-run. That would be similar to how it > is handled in Kubernetes. > Right now it is an all-or-nothing thing. > What I would expect, is that it is still possible to see the test history > somewhere then (e.g. the first two attempts where it failed and then the > third one where it succeeds). > Best Regards, > Roman -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100098)

1 0

[JIRA] (OVIRT-2590) Cache Docker images in the datacenter
by Roman Mohr (oVirt JIRA) 14 Feb '19

14 Feb '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2590?page=com.atlassian.jira.… ] Roman Mohr commented on OVIRT-2590: ----------------------------------- Thinking about our CI images it can avoid peaks on the network consumption. Right now you need all of our lanes to have run on each bare-metal machine until all nodes have a cache for each image. They have up to 8gb. It is another measure to reduce the amount of failing tests because of networking issues. Setting it up should be pretty simple. The mirror needs then just to be injected before starting docker. > Cache Docker images in the datacenter > ------------------------------------- > > Key: OVIRT-2590 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2590 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Roman Mohr > Assignee: infra > > What? > As a user, I expect that I don't have to care about caching to speed up > builds for the good of the CI system itself. > Right now there exists a whitelist for docker images, which will not be > remove from the build slot after the build. Instead of that I expect a > clean build environment and that in general all images which I regularly > use are cached in the cluster via e.g. a pull-through-cache [1]. > Why? > 1) Caching in a build slot is not very effective. CI runs do really-a-lot > of almost identical things in a small time-window (e.g. days). If caching > happens in the build-slot and many slots are present, then the cache > utilization will be very low. > 2) Whitelisting docker images extra for a slot where the registry runs in, > is very error prone and since it is not cached across the cluster it is > also very intransparent what the clear benefit for the user is. Especially > when thinking about scaling a CI system, that seems to leak internal > optimizations to the user. Fast builds are twice as important for the CI > system than they are for the users (by default faster and lower utilization > is always better than asking people to optimize on their side). > [1] https://docs.docker.com/registry/recipes/mirror/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100098)

1 0

[CQ]: 97578, 9 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 14 Feb '19

14 Feb '19

Change 97578,9 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/97578/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12879/

1 0

[CQ]: 97694, 4 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Feb '19

14 Feb '19

A system test invoked by the "ovirt-master" change queue including change 97694,4 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 97726,5 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 97726,5 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/97694/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/97726/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12876/

3 2

[CQ]: 97733, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 14 Feb '19

14 Feb '19

A system test invoked by the "ovirt-master" change queue including change 97733,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 97726,5 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 97726,5 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/97733/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/97726/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12877/

2 1

[CQ]: 97726,5 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 14 Feb '19

14 Feb '19

Change 97726,5 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/97726/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12875/

1 0

oVirt infra daily report - unstable production jobs - 735
by jenkins＠jenkins.phx.ovirt.org 14 Feb '19

14 Feb '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/735//artifact/exported-a… Cheers, Jenkins

1 0

[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 13-02-2019 ] [ 001_initialize_engine.initialize_engine ]
by Dafna Ron 13 Feb '19

13 Feb '19

Hi, we are failing on project ovirt-engine in master branch on initialize engine Change suspected: https://gerrit.ovirt.org/#/c/95743/8 - core: Remove unused field VdsStatic.vdsStrength setup log is here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12864/artifa… full logs are here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12864/artifa… Can you please check the issue? Thanks, Dafna <error> INSERT INTO gluster_config_master(config_key, config_description, minimum_supported_cluster, config_possible_values, config_feature) values('use_meta_volume', 'Meta volume for the geo-replication session', '3.5', 'false;true', 'geo_replication'); ************************** INSERT 0 1 2019-02-13 09:03:04,406-0500 Saving custom users permissions on database objects... ********* QUERY ********** copy ( select count(*) from pg_available_extensions where name = 'uuid-ossp' and installed_version IS NOT NULL ) to stdout with delimiter as '|'; ************************** 2019-02-13 09:03:04,906-0500 dbfunc_psql_die --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_00_0000_set_version.sql ********* QUERY ********** select 4000000; ************************** 4000000 2019-02-13 09:03:05,534-0500 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.executeRaw:863 execute-result: ['/usr/share/ovirt-engine/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'engine', '-d', 'engine', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20190213090242-msviji.log', '-c', 'apply'], rc=1 2019-02-13 09:03:05,534-0500 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.execute:921 execute-output: ['/usr/share/ovirt-engine/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'engine', '-d', 'engine', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20190213090242-msviji.log', '-c', 'apply'] stdout: 2019-02-13 09:03:05,535-0500 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.execute:926 execute-output: ['/usr/share/ovirt-engine/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'engine', '-d', 'engine', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20190213090242-msviji.log', '-c', 'apply'] stderr: FATAL: Operation aborted, found duplicate version: 04030790 2019-02-13 09:03:05,535-0500 ERROR otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema schema._misc:435 schema.sh: FATAL: Operation aborted, found duplicate version: 04030790 2019-02-13 09:03:05,535-0500 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 437, in _misc raise RuntimeError(_('Engine schema refresh failed')) RuntimeError: Engine schema refresh failed 2019-02-13 09:03:05,536-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Misc configuration': Engine schema refresh failed 2019-02-13 09:03:05,562-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe <error>

2 3