- Infra - oVirt List Archives

[oVirt Jenkins] ovirt-node-ng_4.2_build-artifacts-el7-x86_64 - Build # 317 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Jan '19

10 Jan '19

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_4.2_build-artifacts-el7-x86_64/ Build: http://jenkins.ovirt.org/job/ovirt-node-ng_4.2_build-artifacts-el7-x86_64/3… Build Number: 317 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #317 [Gal Ben Haim] ovirt-guest-agent: Drop 4.1 jobs [Yuval Turgeman] build: add version and release.dist to iso name ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x86_64 - Build # 535 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Jan '19

10 Jan '19

Project: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x8… Build: http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x8… Build Number: 535 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #535 [Gal Ben Haim] ovirt-guest-agent: Drop 4.1 jobs [Yuval Turgeman] automation: disable check-iso in build-artifacts ----------------- Failed Tests: ----------------- No tests ran.

1 1

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #2087
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune > git rev-parse origin/test^{commit} # timeout=10 > git rev-parse test^{commit} # timeout=10 Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test) > git config core.sparsecheckout # timeout=10 > git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 Commit message: "mirror-reposync: remove gluster-3.10 mirror" > git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins5895850176287367408.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7' Build step 'Execute shell' marked build as failure

1 2

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Failed to run project_setup.sh for: #1332 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0035.workers-phx.ovirt.org.

1 0

[oVirt Jenkins] ovirt-appliance_ovirt-4.3_build-artifacts-fc28-x86_64 - Build # 53 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Project: http://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc28… Build: http://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc28… Build Number: 53 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #53 [Gal Ben Haim] ovirt-guest-agent: Drop 4.1 jobs [Yuval Turgeman] build: change root fs to be a logical volume ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 700
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/700//artifact/exported-a… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Failed to run project_setup.sh for: #1315 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0035.workers-phx.ovirt.org.

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Failed to run project_setup.sh for: #1315 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0012.workers-phx.ovirt.org.

1 0

oVirt infra weekly meeting notes 09.01.2019
by Evgheni Dereveanchin 09 Jan '19

09 Jan '19

Hi everyone, Please find this week's infra updates below: PHX - FC27 systems replaced with FC29 - FC29 has incompatible repo parser in DNF - OVIRT-2646 - Installing yum-deprecated on FC29 fixes this behavior - OpenShift staging rebuilt - Web console SSL certificate fails to install - PVs created for Jenkins on OpenShift - New subnet configured on foreman and several VMs migrated as test - Stats.phx.ovirt.org Action items: - Review ansible jenkins patches - Move PVs to temporary project and re-assign - subnet migration: - move guest subnet to new IPs - Verify firewall settings - Forward PTR zone for mail sending -- Regards, Evgheni Dereveanchin

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

Failed to run project_setup.sh for: #3090 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-jw0xw.

1 0

[JIRA] (OVIRT-2447) Make mock_runner no use proxy for localhost connections
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2447?page=com.atlassian.jira.… ] Anton Marchukov reassigned OVIRT-2447: -------------------------------------- Assignee: Barak Korren (was: infra) > Make mock_runner no use proxy for localhost connections > ------------------------------------------------------- > > Key: OVIRT-2447 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2447 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Barak Korren > Assignee: Barak Korren > > As a developer using STDCI, I do not want the PHX proxy to intercept connections my test code is doing to localhost, as that will make such connections fail. > h3. Acceptance Criteria > # When inside a proxied mock_runner environment, the no_proxy environment variable is set to 'localhost,127.0.0.1' -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2447) Make mock_runner no use proxy for localhost connections
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2447?page=com.atlassian.jira.… ] Anton Marchukov commented on OVIRT-2447: ---------------------------------------- [~bkorren(a)redhat.com] I think this is done? > Make mock_runner no use proxy for localhost connections > ------------------------------------------------------- > > Key: OVIRT-2447 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2447 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: mock_runner > Reporter: Barak Korren > Assignee: infra > > As a developer using STDCI, I do not want the PHX proxy to intercept connections my test code is doing to localhost, as that will make such connections fail. > h3. Acceptance Criteria > # When inside a proxied mock_runner environment, the no_proxy environment variable is set to 'localhost,127.0.0.1' -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2267) POC - Try to run Lago inside a container
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2267?page=com.atlassian.jira.… ] Anton Marchukov reassigned OVIRT-2267: -------------------------------------- Assignee: Gal Ben Haim (was: infra) > POC - Try to run Lago inside a container > ---------------------------------------- > > Key: OVIRT-2267 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2267 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Lago > Reporter: Gal Ben Haim > Assignee: Gal Ben Haim > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2640) drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2640?page=com.atlassian.jira.… ] Anton Marchukov reassigned OVIRT-2640: -------------------------------------- Assignee: Eyal Edri (was: infra) > drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore > ------------------------------------------------------------- > > Key: OVIRT-2640 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2640 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Eyal Edri > Assignee: Eyal Edri > > We have old el6 mirrors, 4.0 and 4.1 repos that shouldn't be used anymore, no point in keeping them and taking up space -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2640) drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2640?page=com.atlassian.jira.… ] Anton Marchukov updated OVIRT-2640: ----------------------------------- Status: In Progress (was: To Do) > drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore > ------------------------------------------------------------- > > Key: OVIRT-2640 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2640 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Eyal Edri > Assignee: infra > > We have old el6 mirrors, 4.0 and 4.1 repos that shouldn't be used anymore, no point in keeping them and taking up space -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2640) drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2640?page=com.atlassian.jira.… ] Anton Marchukov updated OVIRT-2640: ----------------------------------- Status: To Do (was: In Progress) > drop el6,4.0,4.1 oVirt CI mirrors that are not needed anymore > ------------------------------------------------------------- > > Key: OVIRT-2640 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2640 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Eyal Edri > Assignee: infra > > We have old el6 mirrors, 4.0 and 4.1 repos that shouldn't be used anymore, no point in keeping them and taking up space -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[CQ]: 96715,3 (ovirt-host) failed "ovirt-master" system tests
by oVirt Jenkins 09 Jan '19

09 Jan '19

Change 96715,3 (ovirt-host) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/96715/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12266/

2 1

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Anton Marchukov updated OVIRT-2586: ----------------------------------- Status: Blocked (was: To Do) Waiting for jenkins ssi plugin fix from u/s > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > Attachments: usedMemory_year.png > > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Anton Marchukov (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Anton Marchukov updated OVIRT-2586: ----------------------------------- Status: To Do (was: In Progress) > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > Attachments: usedMemory_year.png > > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2645) [VDSM] Tests mounting loop devices fail randomly again
by Nir Soffer (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2645?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2645: ----------------------------------- On Wed, Jan 9, 2019 at 2:25 AM Nir Soffer (oVirt JIRA) < After discussion with Barak and Daniel, we decided to create more loop devices. If this does not fix the issue we will have to invest more time in this. https://gerrit.ovirt.org/c/96752/ > [VDSM] Tests mounting loop devices fail randomly again > ------------------------------------------------------ > > Key: OVIRT-2645 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2645 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > We had this issue few years ago, and it was solved when we started > to create loop devices to the slave in check-path.sh. > Before we run the tests, we created loop devices. Looking at the log, we > had 5 > nodes and we added 3 nodes - this is typical log. > + create_loop_devices 8 > + local last=7 > + local min > ++ seq 0 7 > + for min in '`seq 0 $last`' > + local name=/dev/loop0 > + '[' '!' -e /dev/loop0 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop1 > + '[' '!' -e /dev/loop1 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop2 > + '[' '!' -e /dev/loop2 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop3 > + '[' '!' -e /dev/loop3 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop4 > + '[' '!' -e /dev/loop4 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop5 > + '[' '!' -e /dev/loop5 ']' > + mknod --mode 0666 /dev/loop5 b 7 5 > + for min in '`seq 0 $last`' > + local name=/dev/loop6 > + '[' '!' -e /dev/loop6 ']' > + mknod --mode 0666 /dev/loop6 b 7 6 > + for min in '`seq 0 $last`' > + local name=/dev/loop7 > + '[' '!' -e /dev/loop7 ']' > + mknod --mode 0666 /dev/loop7 b 7 7 > But mounting images failed later as you can see bellow. > Failed build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > The same tests run fine in travis inside a container: > https://travis-ci.org/nirs/vdsm/jobs/471920126 > Anyone has idea why mounting fail randomly in Jenkins slaves, and how we > can fix this issue? > ====================================================================== > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,720 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,731 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > at /tmp/tmpSejvmr (mount:204) > 2019-01-08 23:46:19,731 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > /tmp/tmpSejvmr (cwd None) (commands:198) > 2019-01-08 23:46:19,743 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,750 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,759 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,761 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > at /tmp/tmpiDcstj (mount:204) > 2019-01-08 23:46:19,762 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > /tmp/tmpiDcstj (cwd None) (commands:198) > 2019-01-08 23:46:19,773 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,779 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,789 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,790 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > at /tmp/tmpX4UcrB (mount:204) > 2019-01-08 23:46:19,791 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > /tmp/tmpX4UcrB (cwd None) (commands:198) > 2019-01-08 23:46:19,803 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,809 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,821 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,822 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > at /tmp/tmpJNTkWb (mount:204) > 2019-01-08 23:46:19,823 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > /tmp/tmpJNTkWb (cwd None) (commands:198) > 2019-01-08 23:46:19,836 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << > --------------------Coverage.py warning: Module > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never > imported. (module-not-imported) > 2019-01-08 23:46:19,851 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mkisofs -R -J -o > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > -V fslabel /tmp/tmphPPnKt (cwd None) (commands:198) > 2019-01-08 23:46:19,882 DEBUG (MainThread) [root] SUCCESS: <err> = > 'Total translation table size: 0\nTotal rockridge attributes bytes: > 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk > space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 > (commands:219) > 2019-01-08 23:46:19,885 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > at /tmp/vdsm-mkimage-testsS16Km7/work (mount:204) > 2019-01-08 23:46:19,886 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -o loop > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > /tmp/vdsm-mkimage-testsS16Km7/work (cwd None) (commands:198) > 2019-01-08 23:46:19,898 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,907 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mkisofs -R -J -o > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > /tmp/tmpfk8az9 (cwd None) (commands:198) > 2019-01-08 23:46:19,931 DEBUG (MainThread) [root] SUCCESS: <err> = > 'Total translation table size: 0\nTotal rockridge attributes bytes: > 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk > space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 > (commands:219) > 2019-01-08 23:46:19,933 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > at /tmp/vdsm-mkimage-testsxv7k7G/work (mount:204) > 2019-01-08 23:46:19,934 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -o loop > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > /tmp/vdsm-mkimage-testsxv7k7G/work (cwd None) (commands:198) > 2019-01-08 23:46:19,946 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2645) [VDSM] Tests mounting loop devices fail randomly again
by Nir Soffer (oVirt JIRA) 09 Jan '19

09 Jan '19

Nir Soffer created OVIRT-2645: --------------------------------- Summary: [VDSM] Tests mounting loop devices fail randomly again Key: OVIRT-2645 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2645 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We had this issue few years ago, and it was solved when we started to create loop devices to the slave in check-path.sh. Before we run the tests, we created loop devices. Looking at the log, we had 5 nodes and we added 3 nodes - this is typical log. + create_loop_devices 8 + local last=7 + local min ++ seq 0 7 + for min in '`seq 0 $last`' + local name=/dev/loop0 + '[' '!' -e /dev/loop0 ']' + for min in '`seq 0 $last`' + local name=/dev/loop1 + '[' '!' -e /dev/loop1 ']' + for min in '`seq 0 $last`' + local name=/dev/loop2 + '[' '!' -e /dev/loop2 ']' + for min in '`seq 0 $last`' + local name=/dev/loop3 + '[' '!' -e /dev/loop3 ']' + for min in '`seq 0 $last`' + local name=/dev/loop4 + '[' '!' -e /dev/loop4 ']' + for min in '`seq 0 $last`' + local name=/dev/loop5 + '[' '!' -e /dev/loop5 ']' + mknod --mode 0666 /dev/loop5 b 7 5 + for min in '`seq 0 $last`' + local name=/dev/loop6 + '[' '!' -e /dev/loop6 ']' + mknod --mode 0666 /dev/loop6 b 7 6 + for min in '`seq 0 $last`' + local name=/dev/loop7 + '[' '!' -e /dev/loop7 ']' + mknod --mode 0666 /dev/loop7 b 7 7 But mounting images failed later as you can see bellow. Failed build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… The same tests run fine in travis inside a container: https://travis-ci.org/nirs/vdsm/jobs/471920126 Anyone has idea why mounting fail randomly in Jenkins slaves, and how we can fix this issue? ====================================================================== ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,720 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img 1440 -n FSLABEL (cwd None) (commands:198) 2019-01-08 23:46:19,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,731 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img at /tmp/tmpSejvmr (mount:204) 2019-01-08 23:46:19,731 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img /tmp/tmpSejvmr (cwd None) (commands:198) 2019-01-08 23:46:19,743 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,750 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img 1440 (cwd None) (commands:198) 2019-01-08 23:46:19,759 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,761 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img at /tmp/tmpiDcstj (mount:204) 2019-01-08 23:46:19,762 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img /tmp/tmpiDcstj (cwd None) (commands:198) 2019-01-08 23:46:19,773 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,779 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img 1440 -n FSLABEL (cwd None) (commands:198) 2019-01-08 23:46:19,789 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,790 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img at /tmp/tmpX4UcrB (mount:204) 2019-01-08 23:46:19,791 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img /tmp/tmpX4UcrB (cwd None) (commands:198) 2019-01-08 23:46:19,803 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,809 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img 1440 (cwd None) (commands:198) 2019-01-08 23:46:19,821 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,822 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img at /tmp/tmpJNTkWb (mount:204) 2019-01-08 23:46:19,823 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img /tmp/tmpJNTkWb (cwd None) (commands:198) 2019-01-08 23:46:19,836 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkIsoFs creating an image and checking its content ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs m.mount(mntOpts='loop') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << --------------------Coverage.py warning: Module /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never imported. (module-not-imported) 2019-01-08 23:46:19,851 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img -V fslabel /tmp/tmphPPnKt (cwd None) (commands:198) 2019-01-08 23:46:19,882 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) 2019-01-08 23:46:19,885 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img at /tmp/vdsm-mkimage-testsS16Km7/work (mount:204) 2019-01-08 23:46:19,886 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img /tmp/vdsm-mkimage-testsS16Km7/work (cwd None) (commands:198) 2019-01-08 23:46:19,898 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkIsoFs creating an image and checking its content ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs m.mount(mntOpts='loop') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,907 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/tmpfk8az9 (cwd None) (commands:198) 2019-01-08 23:46:19,931 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) 2019-01-08 23:46:19,933 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img at /tmp/vdsm-mkimage-testsxv7k7G/work (mount:204) 2019-01-08 23:46:19,934 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/vdsm-mkimage-testsxv7k7G/work (cwd None) (commands:198) 2019-01-08 23:46:19,946 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

2 1

[CQ]: 96621, 3 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 09 Jan '19

09 Jan '19

Change 96621,3 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/96621/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12251/

2 1

[JIRA] (OVIRT-2646) FC29 Is not compatible with older versions of dnf and yum
by Ehud Yonasi (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2646?page=com.atlassian.jira.… ] Ehud Yonasi reassigned OVIRT-2646: ---------------------------------- Assignee: Ehud Yonasi (was: infra) > FC29 Is not compatible with older versions of dnf and yum > --------------------------------------------------------- > > Key: OVIRT-2646 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2646 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: Ehud Yonasi > Assignee: Ehud Yonasi > > FC29 Is not compatible with older versions of dnf and yum, > thus we can't run mock with older versions. > [~ederevea] sent a PR and is trying to fix it on the side of dnf. > I found that what i tested with fc29 running el7 mock had yum-deprecated package that mock > used that package in order to bring mock up. > I will sent a patch to global_setup to add that package to dnf. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2646) FC29 Is not compatible with older versions of dnf and yum
by Ehud Yonasi (oVirt JIRA) 09 Jan '19

09 Jan '19

Ehud Yonasi created OVIRT-2646: ---------------------------------- Summary: FC29 Is not compatible with older versions of dnf and yum Key: OVIRT-2646 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2646 Project: oVirt - virtualization made easy Issue Type: Outage Reporter: Ehud Yonasi Assignee: infra FC29 Is not compatible with older versions of dnf and yum, thus we can't run mock with older versions. [~ederevea] sent a PR and is trying to fix it on the side of dnf. I found that what i tested with fc29 running el7 mock had yum-deprecated package that mock used that package in order to bring mock up. I will sent a patch to global_setup to add that package to dnf. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2645) [VDSM] Tests mounting loop devices fail randomly again
by Daniel Belenky (oVirt JIRA) 09 Jan '19

09 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2645?page=com.atlassian.jira.… ] Daniel Belenky commented on OVIRT-2645: --------------------------------------- Nir, I think that you have a bug in create_loop_devices() From the log you've sent, it seems that it created just devices 5, 6 and 7dsf while skipping devices 1-4 because they already existed. I'm not sure how your tests know which loop devices were created by you and which already existed and might be busy, but maybe you should create loop devices until you have enough and store them in a file? On Wed, Jan 9, 2019 at 2:24 AM Nir Soffer <nsoffer(a)redhat.com> wrote: > We had this issue few years ago, and it was solved when we started > to create loop devices to the slave in check-path.sh. > > Before we run the tests, we created loop devices. Looking at the log, we > had 5 > nodes and we added 3 nodes - this is typical log. > > + create_loop_devices 8 > + local last=7 > + local min > ++ seq 0 7 > + for min in '`seq 0 $last`' > + local name=/dev/loop0 > + '[' '!' -e /dev/loop0 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop1 > + '[' '!' -e /dev/loop1 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop2 > + '[' '!' -e /dev/loop2 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop3 > + '[' '!' -e /dev/loop3 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop4 > + '[' '!' -e /dev/loop4 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop5 > + '[' '!' -e /dev/loop5 ']' > + mknod --mode 0666 /dev/loop5 b 7 5 > + for min in '`seq 0 $last`' > + local name=/dev/loop6 > + '[' '!' -e /dev/loop6 ']' > + mknod --mode 0666 /dev/loop6 b 7 6 > + for min in '`seq 0 $last`' > + local name=/dev/loop7 > + '[' '!' -e /dev/loop7 ']' > + mknod --mode 0666 /dev/loop7 b 7 7 > > > But mounting images failed later as you can see bellow. > > Failed build: > > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > > > The same tests run fine in travis inside a container: > https://travis-ci.org/nirs/vdsm/jobs/471920126 > > Anyone has idea why mounting fail randomly in Jenkins slaves, and how we > can fix this issue? > > ====================================================================== > > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,720 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,731 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img at /tmp/tmpSejvmr (mount:204) > 2019-01-08 23:46:19,731 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img /tmp/tmpSejvmr (cwd None) (commands:198) > 2019-01-08 23:46:19,743 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > ====================================================================== > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,750 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,759 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,761 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img at /tmp/tmpiDcstj (mount:204) > 2019-01-08 23:46:19,762 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img /tmp/tmpiDcstj (cwd None) (commands:198) > 2019-01-08 23:46:19,773 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,779 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,789 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,790 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img at /tmp/tmpX4UcrB (mount:204) > 2019-01-08 23:46:19,791 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img /tmp/tmpX4UcrB (cwd None) (commands:198) > 2019-01-08 23:46:19,803 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,809 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,821 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,822 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img at /tmp/tmpJNTkWb (mount:204) > 2019-01-08 23:46:19,823 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img /tmp/tmpJNTkWb (cwd None) (commands:198) > 2019-01-08 23:46:19,836 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << --------------------Coverage.py warning: Module /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never imported. (module-not-imported) > > 2019-01-08 23:46:19,851 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img -V fslabel /tmp/tmphPPnKt (cwd None) (commands:198) > 2019-01-08 23:46:19,882 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,885 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img at /tmp/vdsm-mkimage-testsS16Km7/work (mount:204) > 2019-01-08 23:46:19,886 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img /tmp/vdsm-mkimage-testsS16Km7/work (cwd None) (commands:198) > 2019-01-08 23:46:19,898 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,907 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/tmpfk8az9 (cwd None) (commands:198) > 2019-01-08 23:46:19,931 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) > 2019-01-08 23:46:19,933 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img at /tmp/vdsm-mkimage-testsxv7k7G/work (mount:204) > 2019-01-08 23:46:19,934 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/vdsm-mkimage-testsxv7k7G/work (cwd None) (commands:198) > 2019-01-08 23:46:19,946 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) > --------------------- >> end captured logging << --------------------- > > > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/USZV6PWV5PS7J… > -- DANIEL BELENKY > [VDSM] Tests mounting loop devices fail randomly again > ------------------------------------------------------ > > Key: OVIRT-2645 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2645 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > We had this issue few years ago, and it was solved when we started > to create loop devices to the slave in check-path.sh. > Before we run the tests, we created loop devices. Looking at the log, we > had 5 > nodes and we added 3 nodes - this is typical log. > + create_loop_devices 8 > + local last=7 > + local min > ++ seq 0 7 > + for min in '`seq 0 $last`' > + local name=/dev/loop0 > + '[' '!' -e /dev/loop0 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop1 > + '[' '!' -e /dev/loop1 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop2 > + '[' '!' -e /dev/loop2 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop3 > + '[' '!' -e /dev/loop3 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop4 > + '[' '!' -e /dev/loop4 ']' > + for min in '`seq 0 $last`' > + local name=/dev/loop5 > + '[' '!' -e /dev/loop5 ']' > + mknod --mode 0666 /dev/loop5 b 7 5 > + for min in '`seq 0 $last`' > + local name=/dev/loop6 > + '[' '!' -e /dev/loop6 ']' > + mknod --mode 0666 /dev/loop6 b 7 6 > + for min in '`seq 0 $last`' > + local name=/dev/loop7 > + '[' '!' -e /dev/loop7 ']' > + mknod --mode 0666 /dev/loop7 b 7 7 > But mounting images failed later as you can see bellow. > Failed build: > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… > The same tests run fine in travis inside a container: > https://travis-ci.org/nirs/vdsm/jobs/471920126 > Anyone has idea why mounting fail randomly in Jenkins slaves, and how we > can fix this issue? > ====================================================================== > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,720 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,731 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > at /tmp/tmpSejvmr (mount:204) > 2019-01-08 23:46:19,731 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img > /tmp/tmpSejvmr (cwd None) (commands:198) > 2019-01-08 23:46:19,743 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 211, in test_mkFloppyFs > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,750 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,759 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,761 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > at /tmp/tmpiDcstj (mount:204) > 2019-01-08 23:46:19,762 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img > /tmp/tmpiDcstj (cwd None) (commands:198) > 2019-01-08 23:46:19,773 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,779 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > 1440 -n FSLABEL (cwd None) (commands:198) > 2019-01-08 23:46:19,789 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,790 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > at /tmp/tmpX4UcrB (mount:204) > 2019-01-08 23:46:19,791 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img > /tmp/tmpX4UcrB (cwd None) (commands:198) > 2019-01-08 23:46:19,803 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 230, in test_mkFloppyFs_overwrite > floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 137, in mkFloppyFs > injectFilesToFs(floppy, files, 'vfat') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", > line 113, in injectFilesToFs > m.mount(mntOpts='loop', vfstype=fstype) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,809 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/sbin/mkfs.msdos -C > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > 1440 (cwd None) (commands:198) > 2019-01-08 23:46:19,821 DEBUG (MainThread) [root] SUCCESS: <err> = ''; > <rc> = 0 (commands:219) > 2019-01-08 23:46:19,822 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > at /tmp/tmpJNTkWb (mount:204) > 2019-01-08 23:46:19,823 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -t vfat -o loop > /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img > /tmp/tmpJNTkWb (cwd None) (commands:198) > 2019-01-08 23:46:19,836 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << > --------------------Coverage.py warning: Module > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never > imported. (module-not-imported) > 2019-01-08 23:46:19,851 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mkisofs -R -J -o > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > -V fslabel /tmp/tmphPPnKt (cwd None) (commands:198) > 2019-01-08 23:46:19,882 DEBUG (MainThread) [root] SUCCESS: <err> = > 'Total translation table size: 0\nTotal rockridge attributes bytes: > 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk > space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 > (commands:219) > 2019-01-08 23:46:19,885 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > at /tmp/vdsm-mkimage-testsS16Km7/work (mount:204) > 2019-01-08 23:46:19,886 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -o loop > /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img > /tmp/vdsm-mkimage-testsS16Km7/work (cwd None) (commands:198) > 2019-01-08 23:46:19,898 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- > ====================================================================== > ERROR: Tests mkimage.mkIsoFs creating an image and checking its content > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", > line 142, in wrapper > return f(self, *args) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", > line 194, in wrapper > return f(*args, **kwargs) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", > line 252, in test_mkIsoFs > m.mount(mntOpts='loop') > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 207, in mount > cgroup=cgroup) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 277, in _mount > _runcmd(cmd) > File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", > line 305, in _runcmd > raise MountError(rc, b";".join((out, err))) > MountError: (32, ';mount: > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: > failed to setup loop device: No such file or directory\n') > -------------------- >> begin captured logging << -------------------- > 2019-01-08 23:46:19,907 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mkisofs -R -J -o > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > /tmp/tmpfk8az9 (cwd None) (commands:198) > 2019-01-08 23:46:19,931 DEBUG (MainThread) [root] SUCCESS: <err> = > 'Total translation table size: 0\nTotal rockridge attributes bytes: > 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk > space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 > (commands:219) > 2019-01-08 23:46:19,933 INFO (MainThread) [storage.Mount] mounting > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > at /tmp/vdsm-mkimage-testsxv7k7G/work (mount:204) > 2019-01-08 23:46:19,934 DEBUG (MainThread) [root] /usr/bin/taskset > --cpu-list 0-1 /usr/bin/mount -o loop > /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img > /tmp/vdsm-mkimage-testsxv7k7G/work (cwd None) (commands:198) > 2019-01-08 23:46:19,946 DEBUG (MainThread) [root] FAILED: <err> = > 'mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: > failed to setup loop device: No such file or directory\n'; <rc> = 32 > (commands:219) > --------------------- >> end captured logging << --------------------- -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[VDSM] Tests mounting loop devices fail randomly again
by Nir Soffer 09 Jan '19

09 Jan '19

We had this issue few years ago, and it was solved when we started to create loop devices to the slave in check-path.sh. Before we run the tests, we created loop devices. Looking at the log, we had 5 nodes and we added 3 nodes - this is typical log. + create_loop_devices 8 + local last=7 + local min ++ seq 0 7 + for min in '`seq 0 $last`' + local name=/dev/loop0 + '[' '!' -e /dev/loop0 ']' + for min in '`seq 0 $last`' + local name=/dev/loop1 + '[' '!' -e /dev/loop1 ']' + for min in '`seq 0 $last`' + local name=/dev/loop2 + '[' '!' -e /dev/loop2 ']' + for min in '`seq 0 $last`' + local name=/dev/loop3 + '[' '!' -e /dev/loop3 ']' + for min in '`seq 0 $last`' + local name=/dev/loop4 + '[' '!' -e /dev/loop4 ']' + for min in '`seq 0 $last`' + local name=/dev/loop5 + '[' '!' -e /dev/loop5 ']' + mknod --mode 0666 /dev/loop5 b 7 5 + for min in '`seq 0 $last`' + local name=/dev/loop6 + '[' '!' -e /dev/loop6 ']' + mknod --mode 0666 /dev/loop6 b 7 6 + for min in '`seq 0 $last`' + local name=/dev/loop7 + '[' '!' -e /dev/loop7 ']' + mknod --mode 0666 /dev/loop7 b 7 7 But mounting images failed later as you can see bellow. Failed build: https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_st… The same tests run fine in travis inside a container: https://travis-ci.org/nirs/vdsm/jobs/471920126 Anyone has idea why mounting fail randomly in Jenkins slaves, and how we can fix this issue? ====================================================================== ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,720 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img 1440 -n FSLABEL (cwd None) (commands:198) 2019-01-08 23:46:19,729 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,731 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img at /tmp/tmpSejvmr (mount:204) 2019-01-08 23:46:19,731 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img /tmp/tmpSejvmr (cwd None) (commands:198) 2019-01-08 23:46:19,743 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxtGqen/images/vmId_floppy.e83b907eacb704d3942b8cab79561d2a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkFloppyFs creating an image and checking its content. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 211, in test_mkFloppyFs floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,750 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img 1440 (cwd None) (commands:198) 2019-01-08 23:46:19,759 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,761 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img at /tmp/tmpiDcstj (mount:204) 2019-01-08 23:46:19,762 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img /tmp/tmpiDcstj (cwd None) (commands:198) 2019-01-08 23:46:19,773 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsnlTJrM/images/vmId_floppy.4f26a597fe7f8a916fd808350e62bdc0.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,779 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img 1440 -n FSLABEL (cwd None) (commands:198) 2019-01-08 23:46:19,789 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,790 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img at /tmp/tmpX4UcrB (mount:204) 2019-01-08 23:46:19,791 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img /tmp/tmpX4UcrB (cwd None) (commands:198) 2019-01-08 23:46:19,803 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testslSTYsV/images/vmId_floppy.3af4c566a929c30f605091dd7a5b7deb.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Test that mkimage.mkFloppyFs handle situation when the floppy image ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 230, in test_mkFloppyFs_overwrite floppy = mkimage.mkFloppyFs("vmId_floppy", self.files, label) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 137, in mkFloppyFs injectFilesToFs(floppy, files, 'vfat') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/mkimage.py", line 113, in injectFilesToFs m.mount(mntOpts='loop', vfstype=fstype) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,809 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/sbin/mkfs.msdos -C /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img 1440 (cwd None) (commands:198) 2019-01-08 23:46:19,821 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:219) 2019-01-08 23:46:19,822 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img at /tmp/tmpJNTkWb (mount:204) 2019-01-08 23:46:19,823 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -t vfat -o loop /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img /tmp/tmpJNTkWb (cwd None) (commands:198) 2019-01-08 23:46:19,836 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsOy036Z/images/vmId_floppy.7d36c87a8c8c6c8ad83dc972edfcb3cc.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkIsoFs creating an image and checking its content ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs m.mount(mntOpts='loop') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << --------------------Coverage.py warning: Module /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/vdsm was never imported. (module-not-imported) 2019-01-08 23:46:19,851 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img -V fslabel /tmp/tmphPPnKt (cwd None) (commands:198) 2019-01-08 23:46:19,882 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) 2019-01-08 23:46:19,885 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img at /tmp/vdsm-mkimage-testsS16Km7/work (mount:204) 2019-01-08 23:46:19,886 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img /tmp/vdsm-mkimage-testsS16Km7/work (cwd None) (commands:198) 2019-01-08 23:46:19,898 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsS16Km7/images/vmId_iso.1d6daeff12530405cbf56a511812e581.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << --------------------- ====================================================================== ERROR: Tests mkimage.mkIsoFs creating an image and checking its content ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testlib.py", line 142, in wrapper return f(self, *args) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/testValidation.py", line 194, in wrapper return f(*args, **kwargs) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/mkimage_test.py", line 252, in test_mkIsoFs m.mount(mntOpts='loop') File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 207, in mount cgroup=cgroup) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 277, in _mount _runcmd(cmd) File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/storage/mount.py", line 305, in _runcmd raise MountError(rc, b";".join((out, err))) MountError: (32, ';mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n') -------------------- >> begin captured logging << -------------------- 2019-01-08 23:46:19,907 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mkisofs -R -J -o /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/tmpfk8az9 (cwd None) (commands:198) 2019-01-08 23:46:19,931 DEBUG (MainThread) [root] SUCCESS: <err> = 'Total translation table size: 0\nTotal rockridge attributes bytes: 1796\nTotal directory bytes: 8192\nPath table size(bytes): 64\nMax brk space used 22000\n199 extents written (0 MB)\n'; <rc> = 0 (commands:219) 2019-01-08 23:46:19,933 INFO (MainThread) [storage.Mount] mounting /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img at /tmp/vdsm-mkimage-testsxv7k7G/work (mount:204) 2019-01-08 23:46:19,934 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img /tmp/vdsm-mkimage-testsxv7k7G/work (cwd None) (commands:198) 2019-01-08 23:46:19,946 DEBUG (MainThread) [root] FAILED: <err> = 'mount: /tmp/vdsm-mkimage-testsxv7k7G/images/vmId_iso.afeec73d2b486c5f33df966a0306836a.img: failed to setup loop device: No such file or directory\n'; <rc> = 32 (commands:219) --------------------- >> end captured logging << ---------------------

2 1

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #2085
by jenkins＠jenkins.phx.ovirt.org 09 Jan '19

09 Jan '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune > git rev-parse origin/test^{commit} # timeout=10 > git rev-parse test^{commit} # timeout=10 Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test) > git config core.sparsecheckout # timeout=10 > git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 Commit message: "mirror-reposync: remove gluster-3.10 mirror" > git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins557032158534410757.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7' Build step 'Execute shell' marked build as failure

1 1

Build failed in Jenkins: system-sync_mirrors-fedora-base-fc29-x86_64 #188
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/18…> Changes: [Gal Ben Haim] ovirt-guest-agent: Drop 4.1 jobs ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 3980a490238a988869b7a2e070c14777d44a1226 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 3980a490238a988869b7a2e070c14777d44a1226 Commit message: "ovirt-guest-agent: Drop 4.1 jobs" > git rev-list --no-walk 41ba10d144e43636af927b6600c9a897656644c2 # timeout=10 [system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins2213412723554411166.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=fedora-base-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs fedora-base-fc29 yum + local repo_name=fedora-base-fc29 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-base-fc29 /var/www/html/repos/yum/fedora-base-fc29/base + check_yum_sync_needed fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=fedora-base-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/fedora-base-fc29 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=fedora-base-fc29 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-base-fc29 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-base-fc29/base --norepopath --newest-only --urls --quiet Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7' + reposync_out= Build step 'Execute shell' marked build as failure

1 1

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #1292 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0035.workers-phx.ovirt.org.

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #1291 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0011.workers-phx.ovirt.org.

1 0

[JENKINS] Failed to setup proejct vdsm_standard-check-patch
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #1293 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0090.workers-phx.ovirt.org.

1 0

oVirt infra daily report - unstable production jobs - 699
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/699//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 96505, 4 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 08 Jan '19

08 Jan '19

A system test invoked by the "ovirt-master" change queue including change 96505,4 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 96409,3 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 96409,3 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/96505/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/96409/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/12258/

1 0

Build failed in Jenkins: system-sync_mirrors-centos-base-el6-x86_64 #2167
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-base-el6-x86_64/216…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-base-el6-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 41ba10d144e43636af927b6600c9a897656644c2 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 41ba10d144e43636af927b6600c9a897656644c2 Commit message: "runner-node:explicityly set !requiretty to jenkins" > git rev-list --no-walk 41ba10d144e43636af927b6600c9a897656644c2 # timeout=10 [system-sync_mirrors-centos-base-el6-x86_64] $ /bin/bash -xe /tmp/jenkins7256725620018086083.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-base-el6 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror centos-base-el6 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror centos-base-el6 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=centos-base-el6 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs centos-base-el6 yum + local repo_name=centos-base-el6 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/centos-base-el6 /var/www/html/repos/yum/centos-base-el6/base + check_yum_sync_needed centos-base-el6 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=centos-base-el6 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/centos-base-el6 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync centos-base-el6 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=centos-base-el6 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=centos-base-el6 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/centos-base-el6/base --norepopath --newest-only --urls --quiet Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1474, in _commonLoadRepoXML if self._latestRepoXML(local): File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1452, in _latestRepoXML repomd = self.metalink_data.repomd File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda> metalink_data = property(fget=lambda self: self._getMetalink(), File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 914, in _getMetalink self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename) File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 189, in __init__ raise MetaLinkRepoErrorParseFail, "File %s is not XML" % filename yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/epel-el7/metalink.xml is not XML + reposync_out='Could not get metalink https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64 error was 16: error opening file from https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=x86_64, IOError: [Errno 2] No such file or directory: '\''/home/jenkins/mirrors_cache/epel-el7/metalink.xml.tmp'\''' Build step 'Execute shell' marked build as failure

1 1

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3082 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-9h98k.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3081 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-29gn4.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3080 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-ff533.

1 0

[JENKINS] Failed to setup proejct kubevirt_containerized-data-importer_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #1029 containerized-data-importer [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-0htk6.

1 0

[JENKINS] Failed to setup proejct kubevirt_containerized-data-importer_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #1030 containerized-data-importer [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-hrs2m.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3073 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-bhkhk.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3073 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-cs4rw.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

Failed to run project_setup.sh for: #3074 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-90nw5.

1 0

[JIRA] (OVIRT-2644) [VDSM] Error in CI check-patch linter
by Dafna Ron (oVirt JIRA) 08 Jan '19

08 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2644?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2644: ---------------------------------- [~nsoffer(a)redhat.com] before I went on holidays we discussed cleaning up the automation/check-patch.repos.el7 as we had problems on other repos vdsm use which were solved here: https://gerrit.ovirt.org/#/c/96335/ if you give us a list of repos you need (as the file seems to be outdated) we can check the mirrors exist on our side and submit a patch to make sure you use internal mirrors. > [VDSM] Error in CI check-patch linter > ------------------------------------- > > Key: OVIRT-2644 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2644 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Daniel Erez > Assignee: infra > > Hi, > I'm encountering a CI failure in the following test: > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/1236/artifact/check… > ... > Start(bootstrap): yum install > ERROR: Command failed: > # /usr/bin/yum --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-bootstrap-23494/root/ > --releasever 28 install dnf dnf-plugins-core distribution-gpg-keys > --setopt=tsflags=nocontexts > Failed to set locale, defaulting to C > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > ... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2644) [VDSM] Error in CI check-patch linter
by Anton Marchukov (oVirt JIRA) 08 Jan '19

08 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2644?page=com.atlassian.jira.… ] Anton Marchukov commented on OVIRT-2644: ---------------------------------------- I see this is passed on the following run, so it seems to be temporarily connection problems to Fedora copr. I am not sure if we have the mirroring for this as it is a private copr, but we might arrange it if this happens often. > [VDSM] Error in CI check-patch linter > ------------------------------------- > > Key: OVIRT-2644 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2644 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Daniel Erez > Assignee: infra > > Hi, > I'm encountering a CI failure in the following test: > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/1236/artifact/check… > ... > Start(bootstrap): yum install > ERROR: Command failed: > # /usr/bin/yum --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-bootstrap-23494/root/ > --releasever 28 install dnf dnf-plugins-core distribution-gpg-keys > --setopt=tsflags=nocontexts > Failed to set locale, defaulting to C > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > ... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

Build failed in Jenkins: system-sync_mirrors-fedora-base-fc29-x86_64 #185
by jenkins＠jenkins.phx.ovirt.org 08 Jan '19

08 Jan '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/18…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 41ba10d144e43636af927b6600c9a897656644c2 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 41ba10d144e43636af927b6600c9a897656644c2 Commit message: "runner-node:explicityly set !requiretty to jenkins" > git rev-list --no-walk 41ba10d144e43636af927b6600c9a897656644c2 # timeout=10 [system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins2693244642056953145.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=fedora-base-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs fedora-base-fc29 yum + local repo_name=fedora-base-fc29 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-base-fc29 /var/www/html/repos/yum/fedora-base-fc29/base + check_yum_sync_needed fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=fedora-base-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/fedora-base-fc29 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=fedora-base-fc29 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-base-fc29 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-base-fc29/base --norepopath --newest-only --urls --quiet Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML local = self.cachedir + '/repomd.xml' File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> cachedir = property(lambda self: self._dirGetAttr('cachedir')) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr self.dirSetup() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup self._dirSetupMkdir_p(dir) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p raise Errors.RepoError, msg yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-kvm-common-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-kvm-common-el7' + reposync_out= Build step 'Execute shell' marked build as failure

1 1

oVirt infra daily report - unstable production jobs - 698
by jenkins＠jenkins.phx.ovirt.org 07 Jan '19

07 Jan '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/698//artifact/exported-a… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2644) [VDSM] Error in CI check-patch linter
by Nir Soffer (oVirt JIRA) 07 Jan '19

07 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2644?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-2644: ----------------------------------- On Mon, Jan 7, 2019 at 3:19 PM Dafna Ron (oVirt JIRA) < Dafna, can you suggest actual change in vdsm automation that will avoid this issue? > [VDSM] Error in CI check-patch linter > ------------------------------------- > > Key: OVIRT-2644 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2644 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Daniel Erez > Assignee: infra > > Hi, > I'm encountering a CI failure in the following test: > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/1236/artifact/check… > ... > Start(bootstrap): yum install > ERROR: Command failed: > # /usr/bin/yum --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-bootstrap-23494/root/ > --releasever 28 install dnf dnf-plugins-core distribution-gpg-keys > --setopt=tsflags=nocontexts > Failed to set locale, defaulting to C > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > ... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2644) [VDSM] Error in CI check-patch linter
by Dafna Ron (oVirt JIRA) 07 Jan '19

07 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2644?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2644: ---------------------------------- as you can see here, you are trying to access external mirrors: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pa… vdsm needs to fix their projects repos to use local mirrors. I can see that the repo is back up so try to re-run the job > [VDSM] Error in CI check-patch linter > ------------------------------------- > > Key: OVIRT-2644 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2644 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Daniel Erez > Assignee: infra > > Hi, > I'm encountering a CI failure in the following test: > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/1236/artifact/check… > ... > Start(bootstrap): yum install > ERROR: Command failed: > # /usr/bin/yum --installroot > /var/lib/mock/fedora-28-x86_64-4a1ad1766b02e7fc37cf465ffb164e2f-bootstrap-23494/root/ > --releasever 28 install dnf dnf-plugins-core distribution-gpg-keys > --setopt=tsflags=nocontexts > Failed to set locale, defaulting to C > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > [Errno 12] Timeout on > https://copr-be.cloud.fedoraproject.org/results/rhawalsh/dm-vdo/fedora-28-x…: > (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes > received') > Trying other mirror. > ... -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

2 1

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 07 Jan '19

07 Jan '19

Failed to run project_setup.sh for: #3061 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-sbcrg.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 07 Jan '19

07 Jan '19

Failed to run project_setup.sh for: #3057 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-7bhb5.

1 0