Indeed, these were in the build queue when I was restarting Jenkins and the jobs tried to re-run before it was fully up. I also see the following error when publishing el7:

Temporary repodata directory /srv/resources/repos/ovirt/tested/./master/rpm/el7/.repodata/ already exists! (Another createrepo process is running?)
https://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/7790/console

Looks like it's also caused by me cancelling a job that left behind a .repodata directory. Will clean it up now to unblock el7 publishing.


On Wed, Jun 5, 2019 at 8:07 AM Emil Natan <ena@redhat.com> wrote:
A bunch of deploy to tested jobs failed as result of the maintenance restarts earlier today. I'm re-triggering those now.

On Wed, Jun 5, 2019 at 1:38 AM <jenkins@jenkins.phx.ovirt.org> wrote:
See <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/7776/display/redirect>

------------------------------------------
Started by upstream project "ovirt-master_change-queue-tester" build number 14434
originally caused by:
 Started by upstream project "ovirt-master_change-queue" build number 42136
 originally caused by:
  Started by upstream project "vdsm_standard-on-merge" build number 795
  originally caused by:
   Triggered by Gerrit: https://gerrit.ovirt.org/100463
[EnvInject] - Loading node environment variables.
Building remotely on vm0010.workers-phx.ovirt.org (el7 phx nested) in workspace <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/>
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-epHRQo8YDSQE/agent.19927
SSH_AGENT_PID=19929
[ssh-agent] Started.
Running ssh-add (command line suppressed)
Identity added: <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/@tmp/private_key_5326099156985161464.key> (<http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/@tmp/private_key_5326099156985161464.key)>
[ssh-agent] Using credentials deploy-ovirt-experimental (SSH key for deploying to the tested repo)
[deploy-to_ovirt-master_tested] $ /bin/bash -xe /tmp/jenkins7284571172790321491.sh
+ [[ jenkins:http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795/ == '' ]]
+ queue_name=ovirt-master
+ ssh -o StrictHostKeyChecking=no deploy-ovirt-experimental@resources.ovirt.org
+ echo repo-extra-dir:master
+ echo jenkins:http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795/
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ BASE_DIR=/srv/resources/repos/ovirt/tested
+ PUBLISH_MD_COPIES=50
+ main
+ local tmp_dir
+ mkdir -p /srv/resources/repos/ovirt/tested
++ mktemp -d /srv/resources/repos/ovirt/tested/.deploy.XXXXXXXXXX
Collecting packages
+ tmp_dir=/srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v
+ trap 'rm -rf '\''/srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v'\''' EXIT HUP
+ echo 'Collecting packages'
+ collect_packages /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v
+ local repoman_dst=/srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v
+ repoman --temp-dir generate-in-repo --option main.allowed_repo_paths=/srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v --option main.on_empty_source=warn /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v add conf:stdin
2019-06-04 22:37:56,171::INFO::repoman.cmd::
2019-06-04 22:37:56,171::INFO::repoman.cmd::Adding artifacts to the repo /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v
2019-06-04 22:37:56,171::INFO::repoman.common.repo::Adding repo extra dir master
2019-06-04 22:37:56,173::INFO::repoman.common.stores.RPM::Loading repo /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/master
2019-06-04 22:37:56,173::INFO::repoman.common.stores.RPM::Repo /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/master loaded
2019-06-04 22:37:56,174::INFO::repoman.common.stores.iso::Loading repo /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/master
2019-06-04 22:37:56,175::INFO::repoman.common.stores.iso::Repo /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/master loaded
2019-06-04 22:37:56,183::INFO::repoman.common.repo::Resolving artifact source jenkins:http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795/
2019-06-04 22:37:56,308::ERROR::repoman.common.sources.jenkins::URL: http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795//api/json?depth=3
Status: 503
Reason: Service Unavailable
Headers: {'cache-control': 'no-cache,no-store,must-revalidate',
 'connection': 'close',
 'content-encoding': 'gzip',
 'content-length': '456',
 'content-type': 'text/html;charset=utf-8',
 'date': 'Tue, 04 Jun 2019 22:37:54 GMT',
 'expires': '0',
 'server': 'Jetty(9.4.z-SNAPSHOT)',
 'via': '1.1 jenkins.phx.ovirt.org',
 'x-content-type-options': 'nosniff'}
Body:


    <!DOCTYPE html><html lang="en-US"><head resURL="/static/539a36f3" data-rooturl="" data-resurl="/static/539a36f3"><title>Starting Jenkins</title><meta name="ROBOTS" content="NOFOLLOW" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.theme.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/loading.css" type="text/css" /></head><body><div class="simple-page" role="main"><div class="modal signup"><div class="signupIntroDefault"><div class="logo"></div><h1 class="loading">
                            Please wait while Jenkins is getting ready to work
                            <span>.</span><span>.</span><span>.</span></h1><p>Your browser will reload automatically when Jenkins is ready.</p></div></div></div><script src="/static/539a36f3/scripts/loading.js" type="text/javascript"></script><script>safeRedirector(window.location.href);</script></body></html>

2019-06-04 22:37:58,360::ERROR::repoman.common.sources.jenkins::URL: http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795//api/json?depth=3
Status: 503
Reason: Service Unavailable
Headers: {'cache-control': 'no-cache,no-store,must-revalidate',
 'connection': 'close',
 'content-encoding': 'gzip',
 'content-length': '456',
 'content-type': 'text/html;charset=utf-8',
 'date': 'Tue, 04 Jun 2019 22:37:56 GMT',
 'expires': '0',
 'server': 'Jetty(9.4.z-SNAPSHOT)',
 'via': '1.1 jenkins.phx.ovirt.org',
 'x-content-type-options': 'nosniff'}
Body:


    <!DOCTYPE html><html lang="en-US"><head resURL="/static/539a36f3" data-rooturl="" data-resurl="/static/539a36f3"><title>Starting Jenkins</title><meta name="ROBOTS" content="NOFOLLOW" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.theme.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/loading.css" type="text/css" /></head><body><div class="simple-page" role="main"><div class="modal signup"><div class="signupIntroDefault"><div class="logo"></div><h1 class="loading">
                            Please wait while Jenkins is getting ready to work
                            <span>.</span><span>.</span><span>.</span></h1><p>Your browser will reload automatically when Jenkins is ready.</p></div></div></div><script src="/static/539a36f3/scripts/loading.js" type="text/javascript"></script><script>safeRedirector(window.location.href);</script></body></html>

2019-06-04 22:38:00,436::ERROR::repoman.common.sources.jenkins::URL: http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795//api/json?depth=3
Status: 503
Reason: Service Unavailable
Headers: {'cache-control': 'no-cache,no-store,must-revalidate',
 'connection': 'close',
 'content-encoding': 'gzip',
 'content-length': '456',
 'content-type': 'text/html;charset=utf-8',
 'date': 'Tue, 04 Jun 2019 22:37:58 GMT',
 'expires': '0',
 'server': 'Jetty(9.4.z-SNAPSHOT)',
 'via': '1.1 jenkins.phx.ovirt.org',
 'x-content-type-options': 'nosniff'}
Body:


    <!DOCTYPE html><html lang="en-US"><head resURL="/static/539a36f3" data-rooturl="" data-resurl="/static/539a36f3"><title>Starting Jenkins</title><meta name="ROBOTS" content="NOFOLLOW" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.theme.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/loading.css" type="text/css" /></head><body><div class="simple-page" role="main"><div class="modal signup"><div class="signupIntroDefault"><div class="logo"></div><h1 class="loading">
                            Please wait while Jenkins is getting ready to work
                            <span>.</span><span>.</span><span>.</span></h1><p>Your browser will reload automatically when Jenkins is ready.</p></div></div></div><script src="/static/539a36f3/scripts/loading.js" type="text/javascript"></script><script>safeRedirector(window.location.href);</script></body></html>

2019-06-04 22:38:02,514::ERROR::repoman.common.sources.jenkins::URL: http://jenkins.ovirt.org/job/vdsm_standard-on-merge/795//api/json?depth=3
Status: 503
Reason: Service Unavailable
Headers: {'cache-control': 'no-cache,no-store,must-revalidate',
 'connection': 'close',
 'content-encoding': 'gzip',
 'content-length': '456',
 'content-type': 'text/html;charset=utf-8',
 'date': 'Tue, 04 Jun 2019 22:38:01 GMT',
 'expires': '0',
 'server': 'Jetty(9.4.z-SNAPSHOT)',
 'via': '1.1 jenkins.phx.ovirt.org',
 'x-content-type-options': 'nosniff'}
Body:


    <!DOCTYPE html><html lang="en-US"><head resURL="/static/539a36f3" data-rooturl="" data-resurl="/static/539a36f3"><title>Starting Jenkins</title><meta name="ROBOTS" content="NOFOLLOW" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/simple-page.theme.css" type="text/css" /><link rel="stylesheet" href="/static/539a36f3/css/loading.css" type="text/css" /></head><body><div class="simple-page" role="main"><div class="modal signup"><div class="signupIntroDefault"><div class="logo"></div><h1 class="loading">
                            Please wait while Jenkins is getting ready to work
                            <span>.</span><span>.</span><span>.</span></h1><p>Your browser will reload automatically when Jenkins is ready.</p></div></div></div><script src="/static/539a36f3/scripts/loading.js" type="text/javascript"></script><script>safeRedirector(window.location.href);</script></body></html>

2019-06-04 22:38:04,516::ERROR::repoman.cmd::
2019-06-04 22:38:04,516::ERROR::repoman.cmd::Error while adding conf:stdin
2019-06-04 22:38:04,518::INFO::repoman.common.repo::Cleaning up temporary dir /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/.lago_tmp/tmpQ4NzXM/tmpgqw_Ee
2019-06-04 22:38:04,519::INFO::repoman.common.repo::Cleaning up temporary dir /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v/.lago_tmp/tmpQ4NzXM
+ rm -rf /srv/resources/repos/ovirt/tested/.deploy.5Xqeujnn6v
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 19929 killed;
[ssh-agent] Stopped.
_______________________________________________
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ETOIPTHC5I4FBQLVX5HWCIUYFRW4CTL7/


--
Emil Natan
RHV/CNV DevOps
_______________________________________________
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B5TI3BQQFN5C7VECQTRAMEUL5J3A7MLP/


--
Regards,
Evgheni Dereveanchin