December 2018 - Infra - oVirt List Archives

[CQ]: 96301, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 21 Dec '18

21 Dec '18

A system test invoked by the "ovirt-master" change queue including change 96301,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 96040,9 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96040,9 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96301/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96040/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12065/

1 0

[CQ]: 96363, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 21 Dec '18

21 Dec '18

A system test invoked by the "ovirt-master" change queue including change 96363,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 96040,9 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96040,9 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96363/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96040/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12059/

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Nir Soffer (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2630: ----------------------------------- Third time today: Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-15072b7e/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-15072b7e/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0013.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… On Thu, Dec 20, 2018 at 10:01 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > Two incidents on a different slaves today: > > Error: > sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp: > No such file or directory > mv: cannot stat > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp': > No such file or directory > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0088.workers-phx.ovirt.org > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > > > Error: > sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp: > No such file or directory > mv: cannot stat > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp': > No such file or directory > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0013.workers-phx.ovirt.org > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > > On Wed, Dec 19, 2018 at 4:36 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > > > I have seen this before, very strange failure. > > > > The job run normally and then abort when we start to run the tests. The error > > does not come from our tests. > > > > Error: > > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > > api_response_test.py bridge_test.py caps_test.py client_test.py > > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > > fuser_test.py glob_test.py gluster_cli_test.py > > gluster_exception_test.py glusterTestData.py > > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > > permutation_test.py protocoldetector_test.py response_test.py > > rngsources_test.py schedule_test.py schemavalidation_test.py > > sigutils_test.py sparsify_test.py stompadapter_test.py > > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > > network/*_test.py devices/parsing/complex_vm_test.py > > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > > common/contextlib_test.py common/fileutils_test.py > > common/function_test.py common/hostutils_test.py > > common/libvirtconnection_test.py common/logutils_test.py > > common/network_test.py common/osutils_test.py common/proc_test.py > > common/properties_test.py common/pthread_test.py common/time_test.py > > common/validate_test.py > > > > missing workspace > > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > > vm0038.workers-phx.ovirt.org > > > > > > Build: > > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

oVirt infra daily report - unstable production jobs - 680
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/680//artifact/exported-a… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

Failed to run project_setup.sh for: #970 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0090.workers-phx.ovirt.org.

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Nir Soffer (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2630: ----------------------------------- Two incidents on a different slaves today: Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0088.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0013.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… On Wed, Dec 19, 2018 at 4:36 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > I have seen this before, very strange failure. > > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

planned datacenter maintenance 21.12.2018 20:30 UTC
by Evgheni Dereveanchin 20 Dec '18

20 Dec '18

Hi everyone, The network switch stack used by the oVirt PHX datacenter needs a reboot which is scheduled for tomorrow. It is expected to be a fast task yet it may cut all networking including shared storage access for all of our hypervisors for a couple of minutes. For this reason I'll shut down non-critical services beforehand and pause CI to minimize I/O activity and protect against potential VM disk corruption. Maintenance window: 21.12.2018 20:30 UTC - 21:30 UTC Services that may be unreachable for short periods of time during this outage are: * Package repositories * Glance image repository * Jenkins CI Other services such as the website, gerrit and mailing lists are not affected and will be operating as usual. -- Regards, Evgheni Dereveanchin

1 0

[CQ]: 96323, 2 (ovirt-node-ng-image) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 20 Dec '18

20 Dec '18

A system test invoked by the "ovirt-master" change queue including change 96323,2 (ovirt-node-ng-image) failed. However, this change seems not to be the root cause for this failure. Change 96265,2 (ovirt-node-ng-image) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96265,2 (ovirt-node-ng-image) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96323/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96265/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12046/

1 0

Fwd: ** PROBLEM Service Alert: Resources file server/Memory usage is WARNING **
by Eyal Edri 20 Dec '18

20 Dec '18

FYI ---------- Forwarded message --------- From: <nagios(a)monitoring.ovirt.org> Date: Thu, Dec 20, 2018 at 4:09 PM Subject: ** PROBLEM Service Alert: Resources file server/Memory usage is WARNING ** To: <eedri(a)redhat.com> ***** Nagios ***** Notification Type: PROBLEM Service: Memory usage Host: Resources file server Address: resources.ovirt.org State: WARNING Date/Time: Thu Dec 20 14:09:50 UTC 2018 Additional Info: [MEMORY] Total: 3813 MB - Used: 3133 MB - 82% [SWAP] Total: 2047 MB - Used: 388 MB - 18% -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[JIRA] (OVIRT-2632) No space left on device
by Eyal Edri (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2632?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2632: ---------------------------------- Evgheni, How much space do we have on the VMs? are we doing any cleanups on the disks? On Thu, Dec 20, 2018 at 2:53 PM sbonazzo (oVirt JIRA) < -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) > No space left on device > ----------------------- > > Key: OVIRT-2632 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2632 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Hi, > looks like on some slave there's not enough space for handling the jobs: > https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x… > Can you please check if we need larger disks or if it's a case of cleanup > failing? > In both cases, please take action on fixing the issue. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0