[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=40640#comment-40640 ] Evgheni Dereveanchin commented on OVIRT-2972: --------------------------------------------- Suspect jobs that were reporting issues this weekend: [https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/|https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/] [https://jenkins.ovirt.org/job/ovirt_4.3_publish-rpms_nightly/|https://jenkins.ovirt.org/job/ovirt_4.3_publish-rpms_nightly/] [https://jenkins.ovirt.org/job/ovirt_4.2_publish-rpms_nightly/|https://jenkins.ovirt.org/job/ovirt_4.2_publish-rpms_nightly/] The errors are as follows: {noformat}13:15:32 2020-07-19 11:15:32,180::INFO::root::Saving /home/jenkins/ovirt-master-snapshot.tmp/rpm/fc29/x86_64/ovirt-engine-wildfly-18.0.0-1.fc29.x86_64.rpm 13:15:32 Traceback (most recent call last): 13:15:32 File "/bin/repoman", line 10, in <module> 13:15:32 sys.exit(main()) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/cmd.py", line 461, in main 13:15:32 exit_code = do_add(args, config, repo) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/cmd.py", line 361, in do_add 13:15:32 repo.save() 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/repo.py", line 42, in _func 13:15:32 return func(self, *args, **kwargs) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/repo.py", line 201, in save 13:15:32 store.save() 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/stores/RPM/__init__.py", line 263, in save 13:15:32 self._save(**args) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/stores/RPM/__init__.py", line 302, in _save 13:15:32 save_file(pkg.path, dst_path) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/utils.py", line 436, in save_file 13:15:32 copy(src_path, dst_path) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/utils.py", line 296, in copy 13:15:32 shutil.copy2(what, where) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 13:15:32 copyfile(src, dst) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile 13:15:32 copyfileobj(fsrc, fdst) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj 13:15:32 fdst.write(buf) 13:15:32 IOError: [Errno 28] No space left on device 13:15:32 2020-07-19 11:15:32,917::INFO::repoman.common.repo::Cleaning up temporary dir /tmp/tmpRKo543 13:15:34 Waiting for scan_for_artifacts.sh cron job to finish (attempt 1 of 50) {noformat} looks like the job ran out of tmpfs space - did we publish too many files for some reason? Not entirely sure what tmp is used for in these jobs.

resources.ovirt.org ran out of space for artifact publishing ------------------------------------------------------------

Key: OVIRT-2972 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra Attachments: image-20200720-083159.png, prom.png

This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. The partition is free again now so this is not blocking anything right now.

-- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)