January 2019 - Infra - oVirt List Archives

[JIRA] (OVIRT-2641) mirror_mgr.sh - Build error
by Liora Milbaum (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2641?page=com.atlassian.jira.… ] Liora Milbaum reopened OVIRT-2641: ---------------------------------- > mirror_mgr.sh - Build error > --------------------------- > > Key: OVIRT-2641 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2641 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: CI Mirrors > Reporter: Liora Milbaum > Assignee: infra > > From job run: > [https://jenkins.ovirt.org/blue/organizations/jenkins/system-sync_mirrors-ep…] > {code}+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf > Checking if mirror needs a resync > Traceback (most recent call last): > File "/usr/bin/reposync", line 343, in <module> > main() > File "/usr/bin/reposync", line 175, in main > my.doRepoSetup() > File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup > return self._getRepos(thisrepo, True) > File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos > self._repos.doSetup(thisrepo) > File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup > self.retrieveAllMD() > File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD > dl = repo._async and repo._commonLoadRepoXML(repo) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML > local = self.cachedir + '/repomd.xml' > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> > cachedir = property(lambda self: self._dirGetAttr('cachedir')) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr > self.dirSetup() > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup > self._dirSetupMkdir_p(dir) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p > raise Errors.RepoError, msg > yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2641) mirror_mgr.sh - Build error
by Liora Milbaum (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2641?page=com.atlassian.jira.… ] Liora Milbaum updated OVIRT-2641: --------------------------------- Resolution: Won't Fix Status: Done (was: To Do) > mirror_mgr.sh - Build error > --------------------------- > > Key: OVIRT-2641 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2641 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: CI Mirrors > Reporter: Liora Milbaum > Assignee: infra > > From job run: > [https://jenkins.ovirt.org/blue/organizations/jenkins/system-sync_mirrors-ep…] > {code}+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror epel-el6 x86_64 jenkins/data/mirrors-reposync.conf > Checking if mirror needs a resync > Traceback (most recent call last): > File "/usr/bin/reposync", line 343, in <module> > main() > File "/usr/bin/reposync", line 175, in main > my.doRepoSetup() > File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup > return self._getRepos(thisrepo, True) > File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos > self._repos.doSetup(thisrepo) > File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup > self.retrieveAllMD() > File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD > dl = repo._async and repo._commonLoadRepoXML(repo) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1465, in _commonLoadRepoXML > local = self.cachedir + '/repomd.xml' > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 774, in <lambda> > cachedir = property(lambda self: self._dirGetAttr('cachedir')) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 757, in _dirGetAttr > self.dirSetup() > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 735, in dirSetup > self._dirSetupMkdir_p(dir) > File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 712, in _dirSetupMkdir_p > raise Errors.RepoError, msg > yum.Errors.RepoError: Error making cache directory: /home/jenkins/mirrors_cache/centos-updates-el7 error was: [Errno 17] File exists: '/home/jenkins/mirrors_cache/centos-updates-el7' > {code} -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 15 Jan '19

15 Jan '19

Failed to run project_setup.sh for: #3159 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-kubevirt-org-5f98w.

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2649: ------------------------------------- looking at the job logs linked above: {code} 12:17:48 Filesystem Size Used Avail Use% Mounted on 12:17:48 /dev/vda3 36G 24G 13G 66% / 12:17:48 devtmpfs 6.1G 0 6.1G 0% /dev 12:17:48 tmpfs 6.1G 329M 5.8G 6% /dev/shm 12:17:48 tmpfs 6.1G 828K 6.1G 1% /run 12:17:48 tmpfs 6.1G 0 6.1G 0% /sys/fs/cgroup 12:17:48 /dev/vda1 253M 152M 102M 60% /boot 12:17:48 tmpfs 1.3G 0 1.3G 0% /run/user/1012 12:17:48 tmpfs 1.3G 0 1.3G 0% /run/user/0 12:17:48 ############################################################### 12:17:48 # Slave cleanup done: SUCCESS # 12:17:48 ############################################################### {code} We had 13G available on the salve when we started - how much space do we need exactly? > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2649: ------------------------------------- We can probably make {{slave_cleanup.sh}} take more drastic steps if space on the slave goes below a certain threshold. But we need to know what that threshold is. > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren edited comment on OVIRT-2649 at 1/15/19 8:11 AM: -------------------------------------------------------------- I'm not sure bigger disk space is the right solution, we'll just run out again - we need some details before reaching such a decision: # How much space is needed to build appliance and node? # How much space we have on the slaves right now? # What else is taking up space? was (Author: bkorren(a)redhat.com) I'm not sure bigger disk space is the right solution, we'll just run out again - we need some details before reaching such a decision: # How much space is needed to buil,appliance? # How much space we have on the slaves right now? # What else is taking up space? > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren commented on OVIRT-2649: ------------------------------------- I'm not sure bigger disk space is the right solution, we'll just run out again - we need some details before reaching such a decision: # How much space is needed to buil,appliance? # How much space we have on the slaves right now? # What else is taking up space? > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Eyal Edri (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Eyal Edri commented on OVIRT-2649: ---------------------------------- the slave got offline automatically due to out of space. Since this only happens on appliance and node build jobs I suggest to create new big VMs for now with more space and assign the appliance and node build jobs to run only them ( we could also run them on Bare metal since its running nightly ) So I suggest to update the jobs to run on bare-metals as a temp WA ( since they run once a day ) and create new VMs with bigger disk space for future. [~dron] can you help with that? > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren edited comment on OVIRT-2649 at 1/15/19 8:07 AM: -------------------------------------------------------------- we usually want {{/var/cache}} to stick around since we need the caches to work... but maybe not all of them, I suspect we may have an issue with cleaning unused mock environments. was (Author: bkorren(a)redhat.com) we usually waht {{/var/cache}} to stick around since we nee the caches to work... but maybe no all of them, I suspect we may have an issue with cleaning unused mock environments. > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0

[JIRA] (OVIRT-2649) Not enough space on slaves for building
by Barak Korren (oVirt JIRA) 15 Jan '19

15 Jan '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2649?page=com.atlassian.jira.… ] Barak Korren edited comment on OVIRT-2649 at 1/15/19 8:07 AM: -------------------------------------------------------------- we usually waht {{/var/cache}} to stick around since we nee the caches to work... but maybe no all of them, I suspect we may have an issue with cleaning unused mock environments. was (Author: bkorren(a)redhat.com) we ususal;y waht {{/var/cache}} to stick around since we nee the caches to work... but maybe no all of them, I suspect we may have an issue with cleaning unused mock environments. > Not enough space on slaves for building > --------------------------------------- > > Key: OVIRT-2649 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2649 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > Some slave have not enough space for building ovirt-engine appliance > Example: > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.309* Building remotely on vm0092.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0092.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:53:22.207* pigz: abort: write error on <stdout> (No space left on > device) > https://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.3_build-artifacts-fc2… > *00:00:00.558* Building remotely on vm0095.workers-phx.ovirt.org > <https://jenkins.ovirt.org/computer/vm0095.workers-phx.ovirt.org> > (libvirt el7 phx nested) > *00:50:49.368* pigz: abort: write error on <stdout> (No space left on > device) > Please ensure slaves have enough space. > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100097)

1 0