[JIRA] (OVIRT-1161) Nightly publisher is broken

[Barak Korren (oVirt JIRA)]

     [ https://ovirt-jira.atlassian.net/browse/OVIRT-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Barak Korren updated OVIRT-1161:
--------------------------------
    Description: 
Nightly publisher is currently broken, this has caused some of the oVirt snapshot repos to become empty, which in turn caused OST failures, among others.

The error output from the nightly publisher job is as follows:
{code}
[/usr/local/libexec/releng-tools/scan_for_artifacts.sh] ##### Starting artifacts publishing Sun Feb 19 09:00:01 UTC 2017
FATAL: We still have files under /home/jenkins/artifacts/ovirt-master-snapshot.ready. Please recheck
FATAL: Failed inside the loop
{code}

This is probably caused by YUM metadata that has been placed in the {{/home/jenkins/artifacts/ovirt-master-snapshot.ready}} by repoman that had been added in OVIRT-1152.

This as also surfaced another issue - if the {{\*-snapshot.ready}} directory exists while the nightly publisher job runs (which means the cron job did not finish yet), it will happily move the {{\*-snaphot.tmp}} directory into it. What it should do instead is wait a while for it to vanish, and then fail if it does not.

  was:
Nightly publisher is currently broken, this has caused some of the oVirt snapshot repos to become empty, which in turn caused OST failures, among others.

The error output from the nightly publisher job is as follows:
{code}
[/usr/local/libexec/releng-tools/scan_for_artifacts.sh] ##### Starting artifacts publishing Sun Feb 19 09:00:01 UTC 2017
FATAL: We still have files under /home/jenkins/artifacts/ovirt-master-snapshot.ready. Please recheck
FATAL: Failed inside the loop
{code}

This is probably caused by YUM metadata that has been placed in the {{/home/jenkins/artifacts/ovirt-master-snapshot.ready}} by repoman that had been added in OVIRT-1152.

This as also surfaced another issue - if the {{*-snapshot.ready}} directory exists while the nightly publisher job runs (which means the cron job did not finish yet), it will happily move the {{*-snaphot.tmp}} directory into it. What it should do instead is wait a while for it to vanish, and then fail if it does not.


> Nightly publisher is broken
> ---------------------------
>
>                 Key: OVIRT-1161
>                 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1161
>             Project: oVirt - virtualization made easy
>          Issue Type: Bug
>          Components: Repositories Mgmt
>            Reporter: Barak Korren
>            Assignee: infra
>            Priority: Highest
>
> Nightly publisher is currently broken, this has caused some of the oVirt snapshot repos to become empty, which in turn caused OST failures, among others.
> The error output from the nightly publisher job is as follows:
> {code}
> [/usr/local/libexec/releng-tools/scan_for_artifacts.sh] ##### Starting artifacts publishing Sun Feb 19 09:00:01 UTC 2017
> FATAL: We still have files under /home/jenkins/artifacts/ovirt-master-snapshot.ready. Please recheck
> FATAL: Failed inside the loop
> {code}
> This is probably caused by YUM metadata that has been placed in the {{/home/jenkins/artifacts/ovirt-master-snapshot.ready}} by repoman that had been added in OVIRT-1152.
> This as also surfaced another issue - if the {{\*-snapshot.ready}} directory exists while the nightly publisher job runs (which means the cron job did not finish yet), it will happily move the {{\*-snaphot.tmp}} directory into it. What it should do instead is wait a while for it to vanish, and then fail if it does not.



--
This message was sent by Atlassian JIRA
(v1000.769.1#100032)