The s390x VM seems to operate normally again. I've also cleaned up the workspace of the VM to make more space available.
It's back online and running s390x builds for VDSM successfully so we can skip disabling the arch for now.

On Wed, Aug 14, 2019 at 11:48 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Aug 14, 2019 at 9:14 PM Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Jenkins is back in operation after restoring configs and restarting it.
New jobs should be processed normally. Older ones that partially ran when the restart happened may fail - please re-trigger those.

There is still an issue with the s390x VM that we use for CI. It is unrelated to the Jenkins outage: looks like the mainframe it's hosted on is overloaded.
Until the load drops it won't be able to run CI jobs so some jobs that build for this platform (mostly VDSM) may be stuck in the queue.
The VM belongs to the Fedora project and we share access to it with several other projects.
I'll try to reach out to Fedora about this issue but if it persists we may need to temporarily disable s390x jobs.

Indeed s390x are stuck, here is example build:

I posted this to disable s390x builds until the vm is functional again:
 

On Wed, Aug 14, 2019 at 7:11 PM Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi Nir,

There has been an issue today caused by our configuration management system (puppet) that accidentally downgraded Jenkins which caused some plugin configs to be wiped.
This caused the issue you see as new workers can't start and process the queue. I've recovered the old config from bacup and am now working on restoring Jenkins operation.

Will update you once the problem is resolved. The tracker ticket is here:

Sorry for the inconvenience caused.

On Wed, Aug 14, 2019 at 6:58 PM Nir Soffer <nsoffer@redhat.com> wrote:
After waiting for build-artifacts job for 36 minutes (typically finish in 15 minutes)
I found that jeknins seems to be broken.

We depend on CI and OST for merging patches.

Can someone check this?

---

collapseBuild Queue (51)
vdsm_standard-on-mergecancel this build
vdsm_standard-on-mergecancel this build
vdsm_standard-on-mergecancel this build
vdsm_standard-on-mergecancel this build
part of standard-manual-runner #622   cancel this build
part of jenkins_standard-check-patch #3199  cancel this build
part of standard-manual-runner #621   cancel this build
part of jenkins_standard-check-patch #3197  cancel this build
part of vdsm_standard-check-patch #10405  cancel this build
part of vdsm_standard-check-patch #10404  cancel this build
part of vdsm_standard-check-patch #10403  cancel this build
part of ovirt-system-tests_standard-check-patch #5206   cancel this build
part of ovirt-engine-ui-extensions_standard-on-merge #54   cancel this build
part of nmstate_nmstate_standard-check-pr #2095   cancel this build
part of standard-manual-runner #620   cancel this build
part of standard-manual-runner #619   cancel this build
part of vdsm_standard-check-patch #10402  cancel this build
part of vdsm_standard-check-patch #10401  cancel this build
part of vdsm_standard-check-patch #10398  cancel this build
part of vdsm_standard-check-patch #10395  cancel this build
part of vdsm_standard-check-patch #10400  cancel this build
part of vdsm_standard-check-patch #10394  cancel this build
part of vdsm_standard-check-patch #10396  cancel this build
part of vdsm_standard-check-patch #10399  cancel this build
part of vdsm_standard-check-patch #10397  cancel this build
part of vdsm_standard-check-patch #10386  cancel this build
part of vdsm_standard-check-patch #10391  cancel this build
part of vdsm_standard-check-patch #10392  cancel this build
part of vdsm_standard-check-patch #10389  cancel this build
part of vdsm_standard-check-patch #10388  cancel this build
part of vdsm_standard-check-patch #10385  cancel this build
part of vdsm_standard-check-patch #10387  cancel this build
part of vdsm_standard-check-patch #10390  cancel this build
part of vdsm_standard-check-patch #10393  cancel this build
part of standard-manual-runner #618   cancel this build
part of vdsm_standard-check-patch #10384  cancel this build
part of vdsm_standard-check-patch #10381  cancel this build
part of vdsm_standard-check-patch #10383  cancel this build
part of vdsm_standard-check-patch #10382  cancel this build
part of vdsm_standard-check-patch #10380  cancel this build
part of vdsm_standard-check-patch #10379  cancel this build
part of vdsm_standard-check-patch #10378  cancel this build
part of nmstate_nmstate_standard-check-pr #2094   cancel this build
part of ovirt-system-tests_standard-check-patch #5204   cancel this build
ovirt-system-tests_manual   cancel this build
part of vdsm_standard-check-patch #10376  cancel this build
part of vdsm_standard-check-patch #10377  cancel this build
part of repoman_standard-check-patch #205  cancel this build
part of standard-manual-runner #617   cancel this build
part of vdsm_standard-on-merge #1391   cancel this build
part of vdsm_standard-on-merge #1390 vdsm [check-merged]   cancel this build
 master
1Idle
2Idle
3Idle
4Idle
 lfedora1.lf-dev.marist.edu  (offline)
 mirrors.phx.ovirt.org
1Idle
2Idle
3Idle
4Idle
5Idle
 ovirt-srv18.phx.ovirt.org
1Idle
 ovirt-srv19.phx.ovirt.org
1Idle
 ovirt-srv21.phx.ovirt.org
1Idle
 ovirt-srv22.phx.ovirt.org
1Idle
 ovirt-srv23.phx.ovirt.org
1Idle
 vm0001.workers-phx.ovirt.org
1Idle
 vm0003.workers-phx.ovirt.org
1Idle
 vm0004.workers-phx.ovirt.org
1Idle
 vm0005.workers-phx.ovirt.org
1Idle
 vm0006.workers-phx.ovirt.org
1Idle
 vm0007.workers-phx.ovirt.org  (offline)
 vm0008.workers-phx.ovirt.org
1Idle
 vm0009.workers-phx.ovirt.org
1Idle
 vm0010.workers-phx.ovirt.org
1Idle
 vm0011.workers-phx.ovirt.org
1Idle
 vm0012.workers-phx.ovirt.org
1Idle
 vm0013.workers-phx.ovirt.org  (offline)
 vm0014.workers-phx.ovirt.org
1Idle
 vm0015.workers-phx.ovirt.org
1Idle
 vm0016.workers-phx.ovirt.org
1Idle
 vm0034.workers-phx.ovirt.org
1Idle
 vm0035.workers-phx.ovirt.org
1Idle
 vm0036.workers-phx.ovirt.org
1Idle
 vm0037.workers-phx.ovirt.org
1Idle
 vm0038.workers-phx.ovirt.org
1Idle
 vm0039.workers-phx.ovirt.org
1Idle
 vm0040.workers-phx.ovirt.org
1Idle
 vm0041.workers-phx.ovirt.org
1Idle
 vm0042.workers-phx.ovirt.org
1Idle
 vm0043.workers-phx.ovirt.org
1Idle
 vm0044.workers-phx.ovirt.org
1Idle
 vm0045.workers-phx.ovirt.org
1Idle
 vm0046.workers-phx.ovirt.org  (offline)
 vm0047.workers-phx.ovirt.org
1Idle
 vm0048.workers-phx.ovirt.org
1Idle
 vm0049.workers-phx.ovirt.org
1Idle
 vm0051.workers-phx.ovirt.org
1Idle
 vm0052.workers-phx.ovirt.org
1Idle
 vm0053.workers-phx.ovirt.org
1Idle
 vm0054.workers-phx.ovirt.org
1Idle
 vm0055.workers-phx.ovirt.org
1Idle
 vm0056.workers-phx.ovirt.org
1Idle
 vm0057.workers-phx.ovirt.org
1Idle
 vm0058.workers-phx.ovirt.org
1Idle
 vm0059.workers-phx.ovirt.org
1Idle
 vm0060.workers-phx.ovirt.org
1Idle
 vm0061.workers-phx.ovirt.org
1Idle
 vm0062.workers-phx.ovirt.org
1Idle
 vm0063.workers-phx.ovirt.org
1Idle
 vm0064.workers-phx.ovirt.org
1Idle
 vm0065.workers-phx.ovirt.org
1Idle
 vm0084.workers-phx.ovirt.org
1Idle
 vm0085.workers-phx.ovirt.org
1Idle
 vm0086.workers-phx.ovirt.org
1Idle
 vm0087.workers-phx.ovirt.org
1Idle
 vm0088.workers-phx.ovirt.org  (offline)
 vm0089.workers-phx.ovirt.org
1Idle
 vm0090.workers-phx.ovirt.org
1Idle
 vm0091.workers-phx.ovirt.org
1Idle
 vm0092.workers-phx.ovirt.org
1Idle
 vm0093.workers-phx.ovirt.org
1Idle
 vm0094.workers-phx.ovirt.org
1Idle
 vm0095.workers-phx.ovirt.org
1Idle
 vm0096.workers-phx.ovirt.org
1Idle
 vm0097.workers-phx.ovirt.org
1Idle
 vm0098.workers-phx.ovirt.org
1Idle
 vm0099.workers-phx.ovirt.org
1Idle
 vm0134.workers-phx.ovirt.org  (offline)
 vm0135.workers-phx.ovirt.org
1Idle
 vm0136.workers-phx.ovirt.org
1Idle
 vm0137.workers-phx.ovirt.org
1Idle
 vm0138.workers-phx.ovirt.org
1Idle
 vm0139.workers-phx.ovirt.org
1Idle
 vm0140.workers-phx.ovirt.org
1Idle
 vm0141.workers-phx.ovirt.org
1Idle
 vm0142.workers-phx.ovirt.org
1Idle
 vm0143.workers-phx.ovirt.org
1Idle
 vm0144.workers-phx.ovirt.org
1Idle
 vm0145.workers-phx.ovirt.org
1Idle
 vm0146.workers-phx.ovirt.org
1Idle
 vm0147.workers-phx.ovirt.org
1Idle
 vm0148.workers-phx.ovirt.org
1Idle
 vm0149.workers-phx.ovirt.org
1Idle
 vm0151.workers-phx.ovirt.org
1Idle
 vm0152.workers-phx.ovirt.org
1Idle
 vm0153.workers-phx.ovirt.org
1Idle
 vm0154.workers-phx.ovirt.org
1Idle
 vm0155.workers-phx.ovirt.org
1Idle
 vm0156.workers-phx.ovirt.org
1Idle
 vm0157.workers-phx.ovirt.org
1Idle
 vm0158.workers-phx.ovirt.org
1Idle
 vm0159.workers-phx.ovirt.org
1Idle
 vm0160.workers-phx.ovirt.org
1Idle
 vm0161.workers-phx.ovirt.org
1Idle
 vm0162.workers-phx.ovirt.org
1Idle
 vm0163.workers-phx.ovirt.org
1Idle
 vm0164.workers-phx.ovirt.org
1Idle
 vm0165.workers-phx.ovirt.org
1Idle
 vm0200.workers-phx.ovirt.org
1Idle
 vm0201.workers-phx.ovirt.org
1Idle
 vm0203.workers-phx.ovirt.org
1Idle
 vm0204.workers-phx.ovirt.org
1Idle
 vm0205.workers-phx.ovirt.org
1Idle
 vm0206.workers-phx.ovirt.org
1Idle
 vm0207.workers-phx.ovirt.org
1Idle
 vm0208.workers-phx.ovirt.org
1Idle
 vm0215.workers-phx.ovirt.org
1Idle
 vm0216.workers-phx.ovirt.org
1Idle
 vm0217.workers-phx.ovirt.org
1Idle
 vm0218.workers-phx.ovirt.org
1Idle
 vm0219.workers-phx.ovirt.org
1Idle
 vm0220.workers-phx.ovirt.org
1Idle
 vm0221.workers-phx.ovirt.org
1Idle
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7TXF7RYGFYGWB4JXQDCKQMMN2EEW3X2I/


--
Regards,
Evgheni Dereveanchin


--
Regards,
Evgheni Dereveanchin


--
Regards,
Evgheni Dereveanchin