system_cleanup-workspaces job bugged?
Hi, can you take a look at http://jenkins.ovirt.org/job/system_cleanup-workspaces/112/console ? Looks like the job won't end: 17:06:35 ====== 17:06:35 INFO::node: fc21-vm04.phx.ovirt.org, free space: 9GB 17:06:35 INFO::Node already offline. 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64_merged:: Wiping out 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64_merged:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tests_gerrit:: Wiping out 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tests_gerrit:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created:: Wiping out 17:06:38 WARN::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created:: It seems we can't remove as default user... sudoing! 17:06:38 $ /usr/bin/sudo rm -Rf /home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created http://jenkins.ovirt.org/computer/fc21-vm04.phx.ovirt.org/ shows: Slave fc21-vm04.phx.ovirt.org Disconnected by SYSTEM : workspace cleanup (http://jenkins.ovirt.org/job/system_cleanup-workspaces/111/) So it seems that after cleanup the node has been left disconnected -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
--azLHFNyN32YCQGCU Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 11/06, Sandro Bonazzola wrote:
Hi, can you take a look at http://jenkins.ovirt.org/job/system_cleanup-worksp= aces/112/console ? Looks like the job won't end: =20 17:06:35 =3D=3D=3D=3D=3D=3D 17:06:35 INFO::node: fc21-vm04.phx.ovirt.org, free space: 9GB 17:06:35 INFO::Node already offline. 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64= _merged:: Wiping out 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64= _merged:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tes= ts_gerrit:: Wiping out 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tes= ts_gerrit:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc2= 0_created:: Wiping out 17:06:38 WARN::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc2= 0_created:: It seems we can't remove as default user... sudoing! 17:06:38 $ /usr/bin/sudo rm -Rf /home/jenkins/workspace/vdsm_master_insta= ll-rpm-sanity-fc20_created =20 http://jenkins.ovirt.org/computer/fc21-vm04.phx.ovirt.org/ shows: Slave fc21-vm04.phx.ovirt.org =20 Disconnected by SYSTEM : workspace cleanup (http://jenkins.ovirt.org/job/= system_cleanup-workspaces/111/) =20 So it seems that after cleanup the node has been left disconnected =20
I'm looking into this, the issue is that it seems some mock jobs did not cleanup properly and left some mount inside the working space, that prevnts from cleaning them up with a simple run, I'm trying ot get a small script that uounts those stalled directories prior to it.
=20 --=20 Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
--=20 David Caro Red Hat S.L. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.redhat.com RHT Global #: 82-62605 --azLHFNyN32YCQGCU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUWzglAAoJEEBxx+HSYmnDmVoIAJcgTvlgixsTGiwbPO9TaXYl KnY3cIIpDPoYXx3Pn85/+OTbJlCV4Ob4OpPc1Ptu7o37V8ZXa2w2CW6PJsGZl68k /WF7+379HH0MO/3B7p6PsqIlfnmjGdU2a3VgynpDjMPnDn+Qm+MtgLqO3mk2jD3S 1eqypkEPQsPSvBK8zndxK/xnasee2imWXFRMF5Sr5b1MU2MRjR1f++OX1QMM8TDb e7SaesOPkesuCE6cn4o3T1rrlJLLgtHQXI/e7One1kmhqwARxkX4CpPAjhUMjpJM /rmB/FXsYPek8LxjHc3J+sPPBBvyy6OdYELGtv1jZLrqzuVQeVh8Wzmc2AK9C4k= =+KuZ -----END PGP SIGNATURE----- --azLHFNyN32YCQGCU--
Il 06/11/2014 09:58, David Caro ha scritto:
On 11/06, Sandro Bonazzola wrote:
Hi, can you take a look at http://jenkins.ovirt.org/job/system_cleanup-workspaces/112/console ? Looks like the job won't end:
17:06:35 ====== 17:06:35 INFO::node: fc21-vm04.phx.ovirt.org, free space: 9GB 17:06:35 INFO::Node already offline. 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64_merged:: Wiping out 17:06:36 INFO::/home/jenkins/workspace/vdsm_master_create-rpms-el7-x86_64_merged:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tests_gerrit:: Wiping out 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_network_functional_tests_gerrit:: Deleted 17:06:37 INFO::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created:: Wiping out 17:06:38 WARN::/home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created:: It seems we can't remove as default user... sudoing! 17:06:38 $ /usr/bin/sudo rm -Rf /home/jenkins/workspace/vdsm_master_install-rpm-sanity-fc20_created
http://jenkins.ovirt.org/computer/fc21-vm04.phx.ovirt.org/ shows: Slave fc21-vm04.phx.ovirt.org
Disconnected by SYSTEM : workspace cleanup (http://jenkins.ovirt.org/job/system_cleanup-workspaces/111/)
So it seems that after cleanup the node has been left disconnected
I'm looking into this, the issue is that it seems some mock jobs did not cleanup properly and left some mount inside the working space, that prevnts from cleaning them up with a simple run, I'm trying ot get a small script that uounts those stalled directories prior to it.
Maybe also better to understand which job left the mock chroot in a dirty state and fix it cleanup phase.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
participants (2)
-
David Caro -
Sandro Bonazzola