- Infra - oVirt List Archives

[CQ]: 96363, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 21 Dec '18

21 Dec '18

A system test invoked by the "ovirt-master" change queue including change 96363,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 96040,9 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96040,9 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96363/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96040/9 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12059/

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Nir Soffer (oVirt JIRA) 21 Dec '18

21 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2630: ----------------------------------- Third time today: Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-15072b7e/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-15072b7e/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0013.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… On Thu, Dec 20, 2018 at 10:01 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > Two incidents on a different slaves today: > > Error: > sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp: > No such file or directory > mv: cannot stat > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp': > No such file or directory > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0088.workers-phx.ovirt.org > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > > > Error: > sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp: > No such file or directory > mv: cannot stat > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp': > No such file or directory > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0013.workers-phx.ovirt.org > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > > On Wed, Dec 19, 2018 at 4:36 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > > > I have seen this before, very strange failure. > > > > The job run normally and then abort when we start to run the tests. The error > > does not come from our tests. > > > > Error: > > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > > api_response_test.py bridge_test.py caps_test.py client_test.py > > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > > fuser_test.py glob_test.py gluster_cli_test.py > > gluster_exception_test.py glusterTestData.py > > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > > permutation_test.py protocoldetector_test.py response_test.py > > rngsources_test.py schedule_test.py schemavalidation_test.py > > sigutils_test.py sparsify_test.py stompadapter_test.py > > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > > network/*_test.py devices/parsing/complex_vm_test.py > > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > > common/contextlib_test.py common/fileutils_test.py > > common/function_test.py common/hostutils_test.py > > common/libvirtconnection_test.py common/logutils_test.py > > common/network_test.py common/osutils_test.py common/proc_test.py > > common/properties_test.py common/pthread_test.py common/time_test.py > > common/validate_test.py > > > > missing workspace > > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > > vm0038.workers-phx.ovirt.org > > > > > > Build: > > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

oVirt infra daily report - unstable production jobs - 680
by jenkins＠jenkins.phx.ovirt.org 21 Dec '18

21 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/680//artifact/exported-a… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

Failed to run project_setup.sh for: #970 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0090.workers-phx.ovirt.org.

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Nir Soffer (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2630: ----------------------------------- Two incidents on a different slaves today: Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-d993b7a2/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0088.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… Error: sh: /home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp: No such file or directory mv: cannot stat '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm(a)tmp/durable-edec94a0/jenkins-result.txt.tmp': No such file or directory missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0013.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… On Wed, Dec 19, 2018 at 4:36 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > I have seen this before, very strange failure. > > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > > > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

planned datacenter maintenance 21.12.2018 20:30 UTC
by Evgheni Dereveanchin 20 Dec '18

20 Dec '18

Hi everyone, The network switch stack used by the oVirt PHX datacenter needs a reboot which is scheduled for tomorrow. It is expected to be a fast task yet it may cut all networking including shared storage access for all of our hypervisors for a couple of minutes. For this reason I'll shut down non-critical services beforehand and pause CI to minimize I/O activity and protect against potential VM disk corruption. Maintenance window: 21.12.2018 20:30 UTC - 21:30 UTC Services that may be unreachable for short periods of time during this outage are: * Package repositories * Glance image repository * Jenkins CI Other services such as the website, gerrit and mailing lists are not affected and will be operating as usual. -- Regards, Evgheni Dereveanchin

1 0

[CQ]: 96323, 2 (ovirt-node-ng-image) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 20 Dec '18

20 Dec '18

A system test invoked by the "ovirt-master" change queue including change 96323,2 (ovirt-node-ng-image) failed. However, this change seems not to be the root cause for this failure. Change 96265,2 (ovirt-node-ng-image) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96265,2 (ovirt-node-ng-image) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96323/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96265/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12046/

1 0

Fwd: ** PROBLEM Service Alert: Resources file server/Memory usage is WARNING **
by Eyal Edri 20 Dec '18

20 Dec '18

FYI ---------- Forwarded message --------- From: <nagios(a)monitoring.ovirt.org> Date: Thu, Dec 20, 2018 at 4:09 PM Subject: ** PROBLEM Service Alert: Resources file server/Memory usage is WARNING ** To: <eedri(a)redhat.com> ***** Nagios ***** Notification Type: PROBLEM Service: Memory usage Host: Resources file server Address: resources.ovirt.org State: WARNING Date/Time: Thu Dec 20 14:09:50 UTC 2018 Additional Info: [MEMORY] Total: 3813 MB - Used: 3133 MB - 82% [SWAP] Total: 2047 MB - Used: 388 MB - 18% -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

1 0

[JIRA] (OVIRT-2632) No space left on device
by Eyal Edri (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2632?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2632: ---------------------------------- Evgheni, How much space do we have on the VMs? are we doing any cleanups on the disks? On Thu, Dec 20, 2018 at 2:53 PM sbonazzo (oVirt JIRA) < -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) > No space left on device > ----------------------- > > Key: OVIRT-2632 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2632 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Hi, > looks like on some slave there's not enough space for handling the jobs: > https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x… > Can you please check if we need larger disks or if it's a case of cleanup > failing? > In both cases, please take action on fixing the issue. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2632) No space left on device
by sbonazzo (oVirt JIRA) 20 Dec '18

20 Dec '18

sbonazzo created OVIRT-2632: ------------------------------- Summary: No space left on device Key: OVIRT-2632 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2632 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: sbonazzo Assignee: infra Hi, looks like on some slave there's not enough space for handling the jobs: https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x… Can you please check if we need larger disks or if it's a case of cleanup failing? In both cases, please take action on fixing the issue. -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

2 1

[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
by Eyal Edri (oVirt JIRA) 20 Dec '18

20 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2631?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2631: ---------------------------------- Evgheni, Do we have an issue with our proxy or its a problem in Fedora mirrors? On Thu, Dec 20, 2018 at 2:25 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > Looks like creating mock env fail because of slow network. > > Can we have local mirror to avoid such issues? > > Error: > > # /usr/bin/dnf --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/ > --releasever 28 --disableplugin=local --setopt=deltarpm=False install > @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git > iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw > make mom openvswitch ovirt-imageio-common policycoreutils-python > python2-augeas python2-blivet1 python2-coverage python2-dateutil > python2-dbus python2-decorator python2-devel python2-dmidecode > python2-enum34 python2-inotify python2-ioprocess python2-ipaddress > python2-magic python2-mock python2-netaddr python2-pthreading > python2-pyudev python2-pyyaml python2-requests python2-setuptools > python2-six python2-subprocess32 python3-PyYAML python3-augeas > python3-coverage python3-dateutil python3-dbus python3-decorator > python3-devel python3-magic python3-netaddr python3-nose > python3-pyudev python3-six python3-yaml rpm-build > ... > Transaction Summary > > ================================================================================ > Install 675 Packages > > Total download size: 424 M > Installed size: 1.5 G > Downloading Packages: > ... > (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00 > (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00 > (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00 > [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for > http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for > http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > Error: Error downloading packages: > Curl error (28): Timeout was reached for > > http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > > > Build: > > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/EMCHJTOHQYDYN… > -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) > [VDSM] Build fail with "Curl error (28): Timeout was reached for..." > -------------------------------------------------------------------- > > Key: OVIRT-2631 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > Looks like creating mock env fail because of slow network. > Can we have local mirror to avoid such issues? > Error: > # /usr/bin/dnf --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/ > --releasever 28 --disableplugin=local --setopt=deltarpm=False install > @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git > iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw > make mom openvswitch ovirt-imageio-common policycoreutils-python > python2-augeas python2-blivet1 python2-coverage python2-dateutil > python2-dbus python2-decorator python2-devel python2-dmidecode > python2-enum34 python2-inotify python2-ioprocess python2-ipaddress > python2-magic python2-mock python2-netaddr python2-pthreading > python2-pyudev python2-pyyaml python2-requests python2-setuptools > python2-six python2-subprocess32 python3-PyYAML python3-augeas > python3-coverage python3-dateutil python3-dbus python3-decorator > python3-devel python3-magic python3-netaddr python3-nose > python3-pyudev python3-six python3-yaml rpm-build > ... > Transaction Summary > ================================================================================ > Install 675 Packages > Total download size: 424 M > Installed size: 1.5 G > Downloading Packages: > ... > (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00 > (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00 > (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00 > [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was > reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > Error: Error downloading packages: > Curl error (28): Timeout was reached for > http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… > [Operation too slow. Less than 1000 bytes/sec transferred the last 30 > seconds] > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2631) [VDSM] Build fail with "Curl error (28): Timeout was reached for..."
by Nir Soffer (oVirt JIRA) 20 Dec '18

20 Dec '18

Nir Soffer created OVIRT-2631: --------------------------------- Summary: [VDSM] Build fail with "Curl error (28): Timeout was reached for..." Key: OVIRT-2631 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2631 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra Looks like creating mock env fail because of slow network. Can we have local mirror to avoid such issues? Error: # /usr/bin/dnf --installroot /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-15798/root/ --releasever 28 --disableplugin=local --setopt=deltarpm=False install @buildsys-build autoconf automake createrepo dnf dnf-utils gdb git iproute-tc libguestfs-tools-c libselinux-python3 libvirt-python3 lshw make mom openvswitch ovirt-imageio-common policycoreutils-python python2-augeas python2-blivet1 python2-coverage python2-dateutil python2-dbus python2-decorator python2-devel python2-dmidecode python2-enum34 python2-inotify python2-ioprocess python2-ipaddress python2-magic python2-mock python2-netaddr python2-pthreading python2-pyudev python2-pyyaml python2-requests python2-setuptools python2-six python2-subprocess32 python3-PyYAML python3-augeas python3-coverage python3-dateutil python3-dbus python3-decorator python3-devel python3-magic python3-netaddr python3-nose python3-pyudev python3-six python3-yaml rpm-build ... Transaction Summary ================================================================================ Install 675 Packages Total download size: 424 M Installed size: 1.5 G Downloading Packages: ... (530/675): rest-0.8.1-2.fc28.x86_64.rpm 211 kB/s | 69 kB 00:00 (531/675): libXau-1.0.8-11.fc28.x86_64.rpm 193 kB/s | 34 kB 00:00 (532/675): libogg-1.3.2-10.fc28.x86_64.rpm 147 kB/s | 30 kB 00:00 [MIRROR] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] [FAILED] gtk3-3.22.30-1.fc28.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] Error: Error downloading packages: Curl error (28): Timeout was reached for http://mirrors.phx.ovirt.org/repos/yum/fedora-base-fc28/base/Packages/g/gtk… [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #2022
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune > git rev-parse origin/test^{commit} # timeout=10 > git rev-parse test^{commit} # timeout=10 Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test) > git config core.sparsecheckout # timeout=10 > git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 Commit message: "mirror-reposync: remove gluster-3.10 mirror" > git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2015484664767842987.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile result = self.grab.urlgrab(misc.to_utf8(relative), local, File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda> grab = property(lambda self: self._getgrab()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab self._setupGrab() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab urls = self.urls File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda> urls = property(fget=lambda self: self._geturls(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls self._baseurlSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup mirrorurls.extend(list(self.metalink_data.urls())) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 914, in _getMetalink self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename) File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 189, in __init__ raise MetaLinkRepoErrorParseFail, "File %s is not XML" % filename yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml is not XML Build step 'Execute shell' marked build as failure

1 2

Update to latest master failed with Failed to execute stage 'Setup validation': 'changed'
by Michael Burman 20 Dec '18

20 Dec '18

Hi I'm trying to update the latest master version 4.3.0-0.4.master.20181219115514.git1d803c4.el7 and on engine-setup i get: [ ERROR ] Failed to execute stage 'Setup validation': 'changed' 2018-12-20 08:44:50,803+0200 DEBUG otopi.context context._executeMethod:128 Stage validation METHOD otopi.plugins.ovirt_engine_common.base.core.uninstall.Plugin._validatio n_changed_files 2018-12-20 08:44:50,819+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/uninstall.py", line 247, in _validation_changed_files if info['changed'] KeyError: 'changed' 2018-12-20 08:44:50,821+0200 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': 'changed' Known issue? Thanks, -- Michael Burman Senior Quality engineer - rhv network - redhat israel Red Hat <https://www.redhat.com> mburman(a)redhat.com M: 0545355725 IM: mburman <https://red.ht/sig>

2 3

[oVirt Jenkins] ovirt-appliance_ovirt-4.3_build-artifacts-fc28-x86_64 - Build # 32 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc28… Build: http://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc28… Build Number: 32 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #32 [Yuval Turgeman] build: change root fs to be a logical volume ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 679
by jenkins＠jenkins.phx.ovirt.org 20 Dec '18

20 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/679//artifact/exported-a… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct kubevirt_ovs-cni_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

Failed to run project_setup.sh for: #205 ovs-cni [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-wkw5q.

1 0

[JENKINS] Failed to setup proejct standard-manual-runner-gal
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

Failed to run project_setup.sh for: #20 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-guaranteed-qos-9p1n5.

1 0

[JENKINS] Failed to setup proejct standard-manual-runner-gal
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

Failed to run project_setup.sh for: #20 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-guaranteed-qos-hcz5f.

1 0

Build failed in Jenkins: system-sync_mirrors-fedora-updates-fc27-x86_64 #1183
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-updates-fc27-x86_64…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-updates-fc27-x86_64…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-fedora-updates-fc27-x86_64] $ /bin/bash -xe /tmp/jenkins4306815876383484277.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-updates-fc27 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile result = self.grab.urlgrab(misc.to_utf8(relative), local, File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda> grab = property(lambda self: self._getgrab()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab self._setupGrab() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab urls = self.urls File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda> urls = property(fget=lambda self: self._geturls(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls self._baseurlSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup mirrorurls.extend(list(self.metalink_data.urls())) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 897, in _getMetalink raise Errors.RepoError, msg yum.Errors.RepoError: Cannot retrieve metalink for repository: fedora-updates-fc27. Please verify its path and try again Build step 'Execute shell' marked build as failure

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #2240
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins6421262932256916641.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Dafna Ron (oVirt JIRA) 19 Dec '18

19 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Dafna Ron updated OVIRT-2630: ----------------------------- Resolution: Duplicate Status: Done (was: To Do) OVIRT-2587 > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Dafna Ron (oVirt JIRA) 19 Dec '18

19 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2630?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2630: ---------------------------------- clone of https://ovirt-jira.atlassian.net/browse/OVIRT-2587 > [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" > --------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2630 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > I have seen this before, very strange failure. > The job run normally and then abort when we start to run the tests. The error > does not come from our tests. > Error: > ../tests/run_tests_local.sh API_test.py alignmentscan_test.py > api_response_test.py bridge_test.py caps_test.py client_test.py > clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py > cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py > executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py > fuser_test.py glob_test.py gluster_cli_test.py > gluster_exception_test.py glusterTestData.py > gluster_thinstorage_test.py hooks_test.py hostdev_test.py > hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py > jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py > modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py > mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py > permutation_test.py protocoldetector_test.py response_test.py > rngsources_test.py schedule_test.py schemavalidation_test.py > sigutils_test.py sparsify_test.py stompadapter_test.py > stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py > sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py > tool_test.py throttledlog_test.py unicode_test.py utils_test.py > validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py > vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py > network/*_test.py devices/parsing/complex_vm_test.py > common/cache_test.py common/cmdutils_test.py common/concurrent_test.py > common/contextlib_test.py common/fileutils_test.py > common/function_test.py common/hostutils_test.py > common/libvirtconnection_test.py common/logutils_test.py > common/network_test.py common/osutils_test.py common/proc_test.py > common/properties_test.py common/pthread_test.py common/time_test.py > common/validate_test.py > missing workspace > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on > vm0038.workers-phx.ovirt.org > Build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

Failed to run project_setup.sh for: #2959 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-mxn3w.

1 0

[JIRA] (OVIRT-2630) [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org"
by Nir Soffer (oVirt JIRA) 19 Dec '18

19 Dec '18

Nir Soffer created OVIRT-2630: --------------------------------- Summary: [VDSM] Tests fail with "missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org" Key: OVIRT-2630 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2630 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra I have seen this before, very strange failure. The job run normally and then abort when we start to run the tests. The error does not come from our tests. Error: ../tests/run_tests_local.sh API_test.py alignmentscan_test.py api_response_test.py bridge_test.py caps_test.py client_test.py clientif_test.py cmdutils_test.py config_test.py cpu_profile_test.py cpuinfo_test.py domcaps_test.py encoding_test.py exception_test.py executor_test.py eventfd_test.py fallocate_test.py filecontrol_test.py fuser_test.py glob_test.py gluster_cli_test.py gluster_exception_test.py glusterTestData.py gluster_thinstorage_test.py hooks_test.py hostdev_test.py hoststats_test.py hugepages_test.py hwinfo_test.py jobs_test.py jsonRpcClient_test.py jsonrpc_test.py loopback_test.py mkimage_test.py modprobe.py moduleloader_test.py monkeypatch_test.py mom_test.py mompolicy_test.py osinfo_test.py osutils_test.py passwords_test.py permutation_test.py protocoldetector_test.py response_test.py rngsources_test.py schedule_test.py schemavalidation_test.py sigutils_test.py sparsify_test.py stompadapter_test.py stompasyncclient_test.py stompasyncdispatcher_test.py stomp_test.py sysprep_test.py taskset_test.py testlib_test.py tool_confmeta_test.py tool_test.py throttledlog_test.py unicode_test.py utils_test.py validate_test.py vdsmapi_test.py vdsmdumpchains_test.py verify.py vmapi_test.py vmfakelib_test.py vmTestsData.py zombiereaper_test.py network/*_test.py devices/parsing/complex_vm_test.py common/cache_test.py common/cmdutils_test.py common/concurrent_test.py common/contextlib_test.py common/fileutils_test.py common/function_test.py common/hostutils_test.py common/libvirtconnection_test.py common/logutils_test.py common/network_test.py common/osutils_test.py common/proc_test.py common/properties_test.py common/pthread_test.py common/time_test.py common/validate_test.py missing workspace /home/jenkins/workspace/vdsm_standard-check-patch/vdsm on vm0038.workers-phx.ovirt.org Build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

oVirt infra weekly meeting notes 19.12.2018
by Evgheni Dereveanchin 19 Dec '18

19 Dec '18

Hi everyone, Please find this week's infra updates below: PHX: - Jenkins - Core and plugins updated - JVM optimizations enabled - SSE gateway mem leak still not fixed - Dropping sessions from /monitoring helps with load - RAM upgraded on 4 bare metals, 6 to go - oVirt 4.2.7 upgrade started OSAS: - Network issues in community cage still being investigated AI: - Review hawkular in OpenShift - Review SSL facilities for OpenShift -- Regards, Evgheni Dereveanchin

1 0

[JIRA] (OVIRT-2629) repoman CI Fails (as of 96127 change)
by Anton Marchukov (oVirt JIRA) 19 Dec '18

19 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2629?page=com.atlassian.jira.… ] Anton Marchukov reassigned OVIRT-2629: -------------------------------------- Assignee: Anton Marchukov (was: infra) > repoman CI Fails (as of 96127 change) > ------------------------------------- > > Key: OVIRT-2629 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2629 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: repoman > Reporter: Anton Marchukov > Assignee: Anton Marchukov > > repoman CI fails again for [~eyonasi(a)redhat.com] patch [1]. This is most likely due to pip packages / distros changes. Since repoman is not changed that often we usually collect multiple issues there. We need to fix it to the point of repoman CI working again, so we can proceed with merging new changes. > [1] https://gerrit.ovirt.org/#/c/96127/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2629) repoman CI Fails (as of 96127 change)
by Anton Marchukov (oVirt JIRA) 19 Dec '18

19 Dec '18

Anton Marchukov created OVIRT-2629: -------------------------------------- Summary: repoman CI Fails (as of 96127 change) Key: OVIRT-2629 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2629 Project: oVirt - virtualization made easy Issue Type: Bug Components: repoman Reporter: Anton Marchukov Assignee: infra repoman CI fails again for [~eyonasi(a)redhat.com] patch [1]. This is most likely due to pip packages / distros changes. Since repoman is not changed that often we usually collect multiple issues there. We need to fix it to the point of repoman CI working again, so we can proceed with merging new changes. [1] https://gerrit.ovirt.org/#/c/96127/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[CQ]: 96265, 2 (ovirt-node-ng-image) failed "ovirt-master" system tests
by oVirt Jenkins 19 Dec '18

19 Dec '18

Change 96265,2 (ovirt-node-ng-image) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/96265/2 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12022/

3 2

Build failed in Jenkins: system-sync_mirrors-fedora-base-fc29-x86_64 #123
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/12…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins3631681965288692372.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/fedora-base-fc29 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/fedora-base-fc29' Build step 'Execute shell' marked build as failure

1 1

planned Jenkins restart
by Evgheni Dereveanchin 19 Dec '18

19 Dec '18

Hi everyone, I'll be performing a planned Jenkins restart within the next hour. No new jobs will be scheduled during this maintenance period. I will inform you once it is over. Regards, Evgheni Dereveanchin

1 1

oVirt infra daily report - unstable production jobs - 678
by jenkins＠jenkins.phx.ovirt.org 19 Dec '18

19 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/678//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt CI] standard-enqueue - Build #18083 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 18 Dec '18

18 Dec '18

Build: http://jenkins.ovirt.org/job/standard-enqueue/18083/ Build Name: #18083 Build Description: Gerrit: 96312 - ovirt-engine-nodejs-modules (4.1, 4.2, master) Build Status: FAILURE Gerrit change: https://gerrit.ovirt.org/96312 - title: Update to pull current deps for ovirt-web-ui - project: ovirt-engine-nodejs-modules - branch: master - author: Scott Dickerson <sdickers(a)redhat.com>

1 0

Clone job broken?
by Tal Nisan 18 Dec '18

18 Dec '18

Tried to close a bug, it failed with ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/bugzilla_clo…

2 1

[JIRA] (OVIRT-2628) Unable to use virt-install in mock
by Evgheni Dereveanchin (oVirt JIRA) 18 Dec '18

18 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2628?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2628: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Unable to use virt-install in mock > ---------------------------------- > > Key: OVIRT-2628 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2628 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Yuval Turgeman > Assignee: Evgheni Dereveanchin > > Some jenkins jobs that run virt-install (node-ng, appliance) randomly fail on "ERROR Host does not support any virtualization options". -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2628) Unable to use virt-install in mock
by Yuval Turgeman (oVirt JIRA) 18 Dec '18

18 Dec '18

Yuval Turgeman created OVIRT-2628: ------------------------------------- Summary: Unable to use virt-install in mock Key: OVIRT-2628 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2628 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Yuval Turgeman Assignee: infra Some jenkins jobs that run virt-install (node-ng, appliance) randomly fail on "ERROR Host does not support any virtualization options". -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-1318) configure backup of status.ovirt.org
by Evgheni Dereveanchin (oVirt JIRA) 18 Dec '18

18 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1318?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-1318: --------------------------------------------- It's running in the PHX OpenShift instance and no backup is configured there. Data for the status page stored in a MySQL database and there is no documented way to backup/restore data in Cachet itself. An interesting concept is using a separate container that connects to the DB and backs it up, then we sync the volume outside. Here's a sample of such a container: https://hub.docker.com/r/confirm/mysql-backup/ > configure backup of status.ovirt.org > ------------------------------------ > > Key: OVIRT-1318 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1318 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Low > > status.ovirt.org was set up - need to investigate if it needs to be backed up or we can produce documentation to set up a new stats page in case something happens to the existing OpenShift gear. > We can use OpenShift's built in backup/restore functionality: > https://developers.openshift.com/managing-your-applications/backing-up-appl… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #2017
by jenkins＠jenkins.phx.ovirt.org 18 Dec '18

18 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune > git rev-parse origin/test^{commit} # timeout=10 > git rev-parse test^{commit} # timeout=10 Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test) > git config core.sparsecheckout # timeout=10 > git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 Commit message: "mirror-reposync: remove gluster-3.10 mirror" > git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins3621589975492533033.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Eyal Edri (oVirt JIRA) 18 Dec '18

18 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Eyal Edri updated OVIRT-2586: ----------------------------- Status: In Progress (was: To Do) > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > Attachments: usedMemory_year.png > > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

Build failed in Jenkins: system-sync_mirrors-fedora-base-fc29-x86_64 #119
by jenkins＠jenkins.phx.ovirt.org 18 Dec '18

18 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/11…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins2192842989505263347.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile result = self.grab.urlgrab(misc.to_utf8(relative), local, File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda> grab = property(lambda self: self._getgrab()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab self._setupGrab() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab urls = self.urls File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda> urls = property(fget=lambda self: self._geturls(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls self._baseurlSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup mirrorurls.extend(list(self.metalink_data.urls())) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 906, in _getMetalink shutil.move(result, self.metalink_filename) File "/usr/lib64/python2.7/shutil.py", line 301, in move copy2(src, real_dst) File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml.tmp' Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 677
by jenkins＠jenkins.phx.ovirt.org 18 Dec '18

18 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/677//artifact/exported-a… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2627) upgrade jenkins.ovirt.org to 2.150.1 LTS
by Evgheni Dereveanchin (oVirt JIRA) 17 Dec '18

17 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2627?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2627: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > upgrade jenkins.ovirt.org to 2.150.1 LTS > ---------------------------------------- > > Key: OVIRT-2627 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2627 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > New LTS is out along with some plugins. Updates applied to Staging, this is a tracker for updating Prod -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[JIRA] (OVIRT-2627) upgrade jenkins.ovirt.org to 2.150.1 LTS
by Evgheni Dereveanchin (oVirt JIRA) 17 Dec '18

17 Dec '18

Evgheni Dereveanchin created OVIRT-2627: ------------------------------------------- Summary: upgrade jenkins.ovirt.org to 2.150.1 LTS Key: OVIRT-2627 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2627 Project: oVirt - virtualization made easy Issue Type: Improvement Reporter: Evgheni Dereveanchin Assignee: infra New LTS is out along with some plugins. Updates applied to Staging, this is a tracker for updating Prod -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #2234
by jenkins＠jenkins.phx.ovirt.org 17 Dec '18

17 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins3437419995021721713.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML oxml = self._saveOldRepoXML(local) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1300, in _saveOldRepoXML shutil.copy2(local, old_local) File "/usr/lib64/python2.7/shutil.py", line 131, in copy2 copystat(src, dst) File "/usr/lib64/python2.7/shutil.py", line 100, in copystat os.chmod(dst, mode) OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/centos-base-el6/repomd.xml.old.tmp' Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2621) setlocale warnings in mock logs
by Evgheni Dereveanchin (oVirt JIRA) 17 Dec '18

17 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2621?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2621: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > setlocale warnings in mock logs > ------------------------------- > > Key: OVIRT-2621 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2621 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > There's a bunch of setlocale warnings in mock logs which didn't exist before: > https://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2-pre_build-artifacts-e… > We need to verify what happened to get rid of these messages as they are distracting attention from real failure causes -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100096)

1 0

[oVirt Jenkins] ovirt-node-ng-image_4.3_build-artifacts-el7-x86_64 - Build # 27 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Dec '18

17 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-el7-x8… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-el7-x8… Build Number: 27 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #27 [Sandro Bonazzola] build: prepare for 4.3.0 alpha release ----------------- Failed Tests: ----------------- No tests ran.

1 1

Build failed in Jenkins: system-sync_mirrors-fedora-base-fc29-x86_64 #115
by jenkins＠jenkins.phx.ovirt.org 17 Dec '18

17 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/11…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins2058003799727988653.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7/gen error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7/gen' Build step 'Execute shell' marked build as failure

1 2

Build failed in Jenkins: system-sync_mirrors-centos-updates-el7-x86_64 #2232
by jenkins＠jenkins.phx.ovirt.org 17 Dec '18

17 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el7-x86_64/…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 Commit message: "ost: Add 'COVERAGE' option" > git rev-list --no-walk 176abf92c7f6ef0fb587732da7f1a3d7dc95ac79 # timeout=10 [system-sync_mirrors-centos-updates-el7-x86_64] $ /bin/bash -xe /tmp/jenkins2368424498586860659.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-updates-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1443, in _latestRepoXML oxml = self._saveOldRepoXML(local) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1300, in _saveOldRepoXML shutil.copy2(local, old_local) File "/usr/lib64/python2.7/shutil.py", line 131, in copy2 copystat(src, dst) File "/usr/lib64/python2.7/shutil.py", line 98, in copystat os.utime(dst, (st.st_atime, st.st_mtime)) OSError: [Errno 2] No such file or directory: '/home/jenkins/mirrors_cache/centos-extras-el7/repomd.xml.old.tmp' Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 676
by jenkins＠jenkins.phx.ovirt.org 17 Dec '18

17 Dec '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/676//artifact/exported-a… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2625) lists.ovirt.org not showing threads
by Eyal Edri (oVirt JIRA) 16 Dec '18

16 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2625?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2625: ---------------------------------- Hi [~duck] can you help? > lists.ovirt.org not showing threads > ----------------------------------- > > Key: OVIRT-2625 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2625 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Greg Sheremeta > Assignee: infra > Attachments: Selection_258.png > > > Hi, > lists.ovirt.org hyperkitty is once again not showing threads. I tried to > find my old Jira ticket, but Jira is down. > For example, this doesn't show the thread, but there are 8 replies: > https://lists.ovirt.org/archives/list/users@ovirt.org/thread/SDDH2OC5RBOVYY… > Screenshot: https://imgur.com/a/xHKDN6K > [image: Selection_258.png] > -- > GREG SHEREMETA > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX > Red Hat NA > <https://www.redhat.com/> > gshereme(a)redhat.com IRC: gshereme > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0