[JIRA] (OVIRT-2768) All loader-container nodes are offline, 99 jobs in the queue, jobs stuck for 2 hours

All loader-container nodes are offline, 99 jobs in the queue, jobs stuck for 2 hours ------------------------------------------------------------------------------------ Key: OVIRT-2768 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2768 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra We depend on jenkins to run our tests and build artifacts in a timely manner. Here is example job, stuck for 2 hours: https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9212/ This looks like the issue: [image: collapse] <https://jenkins.ovirt.org/toggleCollapse?paneId=executors>Build Executor Status <https://jenkins.ovirt.org/computer/> master <https://jenkins.ovirt.org/computer/(master)/> 1 Idle 2 Idle 3 Idle 4 Idle lfedora1.lf-dev.marist.edu <https://jenkins.ovirt.org/computer/lfedora1.lf-dev.marist.edu/> 1 Idle loader-container-28ljp <https://jenkins.ovirt.org/computer/loader-container-28ljp/> (offline) (suspended) loader-container-35tlc <https://jenkins.ovirt.org/computer/loader-container-35tlc/> (offline) (suspended) loader-container-3j0ps <https://jenkins.ovirt.org/computer/loader-container-3j0ps/> (offline) (suspended) loader-container-4391q <https://jenkins.ovirt.org/computer/loader-container-4391q/> (offline) (suspended) loader-container-58v3d <https://jenkins.ovirt.org/computer/loader-container-58v3d/> (offline) (suspended) loader-container-66r5r <https://jenkins.ovirt.org/computer/loader-container-66r5r/> (offline) (suspended) loader-container-6k57j <https://jenkins.ovirt.org/computer/loader-container-6k57j/> (offline) (suspended) loader-container-7813f <https://jenkins.ovirt.org/computer/loader-container-7813f/> (offline) (suspended) loader-container-8dkj1 <https://jenkins.ovirt.org/computer/loader-container-8dkj1/> (offline) (suspended) loader-container-8hb98 <https://jenkins.ovirt.org/computer/loader-container-8hb98/> (offline) (suspended) loader-container-8v609 <https://jenkins.ovirt.org/computer/loader-container-8v609/> (offline) (suspended) loader-container-99f36 <https://jenkins.ovirt.org/computer/loader-container-99f36/> (offline) (suspended) loader-container-c16rq <https://jenkins.ovirt.org/computer/loader-container-c16rq/> (offline) (suspended) loader-container-c1vkj <https://jenkins.ovirt.org/computer/loader-container-c1vkj/> (offline) (suspended) loader-container-c4g8c <https://jenkins.ovirt.org/computer/loader-container-c4g8c/> (offline) (suspended) loader-container-cqrx5 <https://jenkins.ovirt.org/computer/loader-container-cqrx5/> (offline) (suspended) loader-container-d1r8d <https://jenkins.ovirt.org/computer/loader-container-d1r8d/> (offline) (suspended) loader-container-dpp70 <https://jenkins.ovirt.org/computer/loader-container-dpp70/> (offline) (suspended) loader-container-gz52q <https://jenkins.ovirt.org/computer/loader-container-gz52q/> (offline) (suspended) loader-container-h9kcf <https://jenkins.ovirt.org/computer/loader-container-h9kcf/> (offline) (suspended) loader-container-hjq0g <https://jenkins.ovirt.org/computer/loader-container-hjq0g/> (offline) (suspended) loader-container-ht75p <https://jenkins.ovirt.org/computer/loader-container-ht75p/> (offline) (suspended) loader-container-htstk <https://jenkins.ovirt.org/computer/loader-container-htstk/> (offline) (suspended) loader-container-j47s3 <https://jenkins.ovirt.org/computer/loader-container-j47s3/> (offline) (suspended) loader-container-k2p10 <https://jenkins.ovirt.org/computer/loader-container-k2p10/> (offline) (suspended) loader-container-kc066 <https://jenkins.ovirt.org/computer/loader-container-kc066/> (offline) (suspended) loader-container-l1vd9 <https://jenkins.ovirt.org/computer/loader-container-l1vd9/> (offline) (suspended) loader-container-l7lqn <https://jenkins.ovirt.org/computer/loader-container-l7lqn/> (offline) (suspended) loader-container-lqqkj <https://jenkins.ovirt.org/computer/loader-container-lqqkj/> (offline) (suspended) loader-container-m8fkq <https://jenkins.ovirt.org/computer/loader-container-m8fkq/> (offline) (suspended) loader-container-n2zml <https://jenkins.ovirt.org/computer/loader-container-n2zml/> (offline) (suspended) loader-container-p7rxw <https://jenkins.ovirt.org/computer/loader-container-p7rxw/> (offline) (suspended) loader-container-pqg56 <https://jenkins.ovirt.org/computer/loader-container-pqg56/> (offline) (suspended) loader-container-qtsdr <https://jenkins.ovirt.org/computer/loader-container-qtsdr/> (offline) (suspended) loader-container-qxwnp <https://jenkins.ovirt.org/computer/loader-container-qxwnp/> (offline) (suspended) loader-container-rdlww <https://jenkins.ovirt.org/computer/loader-container-rdlww/> (offline) (suspended) loader-container-rp1rc <https://jenkins.ovirt.org/computer/loader-container-rp1rc/> (offline) (suspended) loader-container-s6ndk <https://jenkins.ovirt.org/computer/loader-container-s6ndk/> (offline) (suspended) loader-container-sv0pt <https://jenkins.ovirt.org/computer/loader-container-sv0pt/> (offline) (suspended) loader-container-t6qd9 <https://jenkins.ovirt.org/computer/loader-container-t6qd9/> (offline) (suspended) loader-container-vb5dj <https://jenkins.ovirt.org/computer/loader-container-vb5dj/> (offline) (suspended) loader-container-vhdck <https://jenkins.ovirt.org/computer/loader-container-vhdck/> (offline) (suspended) loader-container-vmww3 <https://jenkins.ovirt.org/computer/loader-container-vmww3/> (offline) (suspended) loader-container-vv202 <https://jenkins.ovirt.org/computer/loader-container-vv202/> (offline) (suspended) loader-container-w7n89 <https://jenkins.ovirt.org/computer/loader-container-w7n89/> (offline) (suspended) loader-container-x5gwl <https://jenkins.ovirt.org/computer/loader-container-x5gwl/> (offline) (suspended) loader-container-xmthb <https://jenkins.ovirt.org/computer/loader-container-xmthb/> (offline) (suspended) loader-container-xxcl1 <https://jenkins.ovirt.org/computer/loader-container-xxcl1/> (offline) (suspended) loader-container-zp4w9 <https://jenkins.ovirt.org/computer/loader-container-zp4w9/> (offline) (suspended)