Build failed in Jenkins: deploy-to_ovirt-master_tested #4259
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/4259/display/r...>
------------------------------------------
Started by upstream project "ovirt-release_master_build-artifacts-fc28-x86_64" build number 37
originally caused by:
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on vm0061.workers-phx.ovirt.org (libvirt phx fc28 nested) in workspace <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/>
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-yqOi4OyuEUe2/agent.24000
SSH_AGENT_PID=24002
[ssh-agent] Started.
$ ssh-add <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/@tmp/privat...>
Identity added: <http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/@tmp/privat...> (<http://jenkins.ovirt.org/job/deploy-to_ovirt-master_tested/ws/@tmp/privat...)>
[ssh-agent] Using credentials deploy-ovirt-experimental (SSH key for deploying to the tested repo)
[deploy-to_ovirt-master_tested] $ /bin/bash -xe /tmp/jenkins5529326099273699447.sh
+ [[ http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8... == '' ]]
+ queue_name=ovirt-master
+ ssh -o StrictHostKeyChecking=no deploy-ovirt-experimental(a)resources.ovirt.org
+ echo repo-extra-dir:master
+ echo http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ BASE_DIR=/srv/resources/repos/ovirt/tested
+ PUBLISH_MD_COPIES=50
+ main
+ local tmp_dir
+ mkdir -p /srv/resources/repos/ovirt/tested
++ mktemp -d /srv/resources/repos/ovirt/tested/.deploy.XXXXXXXXXX
Collecting packages
+ tmp_dir=/srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
+ trap 'rm -rf '\''/srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG'\''' EXIT HUP
+ echo 'Collecting packages'
+ collect_packages /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
+ local repoman_dst=/srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
+ repoman --temp-dir generate-in-repo --option main.allowed_repo_paths=/srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG --option main.on_empty_source=warn /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG add conf:stdin
2018-07-12 00:55:20,804::INFO::repoman.cmd::
2018-07-12 00:55:20,805::INFO::repoman.cmd::Adding artifacts to the repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
2018-07-12 00:55:20,805::INFO::repoman.common.repo::Adding repo extra dir master
2018-07-12 00:55:20,812::INFO::repoman.common.stores.RPM::Loading repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:20,813::INFO::repoman.common.stores.RPM::Repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master loaded
2018-07-12 00:55:20,818::INFO::repoman.common.stores.iso::Loading repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:20,818::INFO::repoman.common.stores.iso::Repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master loaded
2018-07-12 00:55:20,838::INFO::repoman.common.repo::Resolving artifact source http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,069::INFO::repoman.common.sources.jenkins::Parsing jenkins URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,083::INFO::repoman.common.sources.jenkins:: Got URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,083::INFO::repoman.common.sources.jenkins:: Got URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,084::INFO::repoman.common.sources.jenkins:: Got URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,084::INFO::repoman.common.sources.jenkins:: Got URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,085::INFO::repoman.common.sources.jenkins:: Got URL: http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8...
2018-07-12 00:55:21,087::INFO::root:: Done
2018-07-12 00:55:21,110::INFO::root::Downloading http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8..., length 7K ...
%[-----------------------25------------------------50-----------------------75------------------------] %[====================================================================================================]
2018-07-12 00:55:21,289::INFO::repoman.common.stores.RPM::Adding package /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj/ovirt-node-ng-image-update-placeholder-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm to repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,319::INFO::root::Downloading http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8..., length 327K ...
%[-----------------------25------------------------50-----------------------75------------------------] %[====================================================================================================]
2018-07-12 00:55:21,448::INFO::repoman.common.stores.RPM::Adding package /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj/ovirt-release-host-node-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm to repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,473::INFO::root::Downloading http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8..., length 11K ...
%[-----------------------25------------------------50-----------------------75------------------------] %[====================================================================================================]
2018-07-12 00:55:21,590::INFO::repoman.common.stores.RPM::Adding package /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj/ovirt-release-master-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm to repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,625::INFO::root::Downloading http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8..., length 396K ...
%[-----------------------25------------------------50-----------------------75------------------------] %[====================================================================================================]
2018-07-12 00:55:21,756::INFO::repoman.common.stores.RPM::Adding package /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj/ovirt-release-master-4.3.0-0.1.master.20180712005424.git356809a.fc28.src.rpm to repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,781::INFO::root::Downloading http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-fc28-x8..., length 11K ...
%[-----------------------25------------------------50-----------------------75------------------------] %[====================================================================================================]
2018-07-12 00:55:21,902::INFO::repoman.common.stores.RPM::Adding package /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj/ovirt-release-master-tested-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm to repo /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,903::INFO::repoman.cmd::
2018-07-12 00:55:21,904::INFO::repoman.common.stores.RPM::Saving new added rpms into /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:21,904::INFO::root::Saving /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master/rpm/fc28/noarch/ovirt-node-ng-image-update-placeholder-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm
2018-07-12 00:55:21,906::INFO::root::Saving /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master/rpm/fc28/noarch/ovirt-release-host-node-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm
2018-07-12 00:55:21,907::INFO::root::Saving /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master/rpm/fc28/noarch/ovirt-release-master-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm
2018-07-12 00:55:21,907::INFO::root::Saving /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master/rpm/fc28/SRPMS/ovirt-release-master-4.3.0-0.1.master.20180712005424.git356809a.fc28.src.rpm
2018-07-12 00:55:21,908::INFO::root::Saving /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master/rpm/fc28/noarch/ovirt-release-master-tested-4.3.0-0.1.master.20180712005424.git356809a.fc28.noarch.rpm
2018-07-12 00:55:21,909::INFO::repoman.common.stores.RPM::
2018-07-12 00:55:21,909::INFO::repoman.common.stores.RPM::Updating metadata
2018-07-12 00:55:21,909::INFO::repoman.common.stores.RPM:: Creating metadata for fc28
2018-07-12 00:55:22,916::INFO::repoman.common.stores.RPM::
2018-07-12 00:55:22,917::INFO::repoman.common.stores.RPM::Creating symlinks
2018-07-12 00:55:22,918::INFO::repoman.common.stores.RPM::
2018-07-12 00:55:22,918::INFO::repoman.common.stores.RPM::Saved /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:22,918::INFO::repoman.common.stores.iso::Saving new added isos into /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:22,919::INFO::repoman.common.stores.iso::
2018-07-12 00:55:22,919::INFO::repoman.common.stores.iso::Saved /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/master
2018-07-12 00:55:22,924::INFO::repoman.common.repo::Cleaning up temporary dir /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw/tmp9vcmLj
2018-07-12 00:55:22,926::INFO::repoman.common.repo::Cleaning up temporary dir /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG/.lago_tmp/tmpeXEynw
Publishing to repo
+ echo 'Publishing to repo'
+ push_to_tested /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG /srv/resources/repos/ovirt/tested
+ local pkg_src=/srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
+ local pkg_dst=/srv/resources/repos/ovirt/tested
+ cd /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
+ find . -type d '!' -name repodata
+ tac
+ read dir
+ install -o deploy-ovirt-experimental -m 755 -d /srv/resources/repos/ovirt/tested/./.lago_tmp
+ find ./.lago_tmp -maxdepth 1 '!' -type d -print0
+ xargs -0 -r cp -RPplf -t /srv/resources/repos/ovirt/tested/./.lago_tmp
+ [[ -d ./.lago_tmp/repodata ]]
+ read dir
+ install -o deploy-ovirt-experimental -m 755 -d /srv/resources/repos/ovirt/tested/./master/rpm/fc28/noarch
+ find ./master/rpm/fc28/noarch -maxdepth 1 '!' -type d -print0
+ xargs -0 -r cp -RPplf -t /srv/resources/repos/ovirt/tested/./master/rpm/fc28/noarch
+ [[ -d ./master/rpm/fc28/noarch/repodata ]]
+ read dir
+ install -o deploy-ovirt-experimental -m 755 -d /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS
+ find ./master/rpm/fc28/SRPMS -maxdepth 1 '!' -type d -print0
+ xargs -0 -r cp -RPplf -t /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS
+ [[ -d ./master/rpm/fc28/SRPMS/repodata ]]
+ xargs -P 8 -d ' '
+ comm -23 /dev/fd/63 /dev/fd/62
++ date +%Y-%m-%d-%H:%M
++ sort
++ repomanage -k1 --new -c /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS
++ sort
++ find /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS -name '*.rpm' -type f -mtime +14
++ echo /srv/resources/repos/ovirt/tested
++ tr / _
+ createrepo_c --update --retain-old-md 50 --workers 8 /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS
Directory walk started
Directory walk done - 365 packages
Loaded information about 364 packages
Temporary output repo path: /srv/resources/repos/ovirt/tested/./master/rpm/fc28/SRPMS/.repodata/
Preparing sqlite DBs
Pool started (with 8 workers)
Pool finished
+ read dir
+ install -o deploy-ovirt-experimental -m 755 -d /srv/resources/repos/ovirt/tested/./master/rpm/fc28
+ find ./master/rpm/fc28 -maxdepth 1 '!' -type d -print0
+ xargs -0 -r cp -RPplf -t /srv/resources/repos/ovirt/tested/./master/rpm/fc28
+ [[ -d ./master/rpm/fc28/repodata ]]
+ xargs -P 8 -d ' '
+ comm -23 /dev/fd/63 /dev/fd/62
++ date +%Y-%m-%d-%H:%M
++ sort
++ repomanage -k1 --new -c /srv/resources/repos/ovirt/tested/./master/rpm/fc28
++ sort
++ find /srv/resources/repos/ovirt/tested/./master/rpm/fc28 -name '*.rpm' -type f -mtime +14
++ echo /srv/resources/repos/ovirt/tested
++ tr / _
xargs: argument line too long
+ rm -rf /srv/resources/repos/ovirt/tested/.deploy.6zhXK8URFG
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 24002 killed;
[ssh-agent] Stopped.
6 years, 2 months
[JIRA] (OVIRT-2311) ovirt-master vdsm failure - extend disk test
times out
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2311?page=com.atlassian.jir... ]
Dafna Ron commented on OVIRT-2311:
----------------------------------
I see a lot of fluentd errors but its hard to tell if its the same time since the logs are not in the same time zone as the console.
but at the end if messages log in engine we can see errors from fluentd:
https://pastebin.com/mprJ0VMD
> ovirt-master vdsm failure - extend disk test times out
> -------------------------------------------------------
>
> Key: OVIRT-2311
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2311
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: infra
> Labels: ost_failures
>
> There is no failure of a test but we can see a gap of an hour from start of extend_disk1 until unhanded exception is thrown.
> Although the error is reported on 004_basic_sanity the logs stop on post-003_basic_networking.py.
> looking at the available logs it may be a task that is not cleared.
> I do not see a connection between the patch and the timeout unless something in the tests is looking ERROR: InterpreterNotFound: python3.6 errors in order to exit.
> patch that failed: https://gerrit.ovirt.org/#/c/92847/
> job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8648/
> Error I can see in the logs is:
> 2018-07-12 01:23:14,863-04 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] An error occurred while fetching unregistered disks from Storage Domain id '597bfe1f-d88c-4961-b1dc-f6c771af9
> fff'
> but I can't see an error in vdsm that would suggest we had a failure.
> you can see the gap in the job:
> 05:31:45 [basic-suit] # extend_disk1:
> 06:31:56 [basic-suit] * Unhandled exception in <function <lambda> at 0x7f5dd403c5f0>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100088)
6 years, 2 months
[JIRA] (OVIRT-2311) ovirt-master vdsm failure - extend disk test
times out
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2311?page=com.atlassian.jir... ]
Dafna Ron commented on OVIRT-2311:
----------------------------------
[~gbenhaim(a)redhat.com] [~nsoffer(a)redhat.com] [~edwardh]
Can you please review the logs/job?
It seems to me a timeout from ost and perhaps a task that was not cleaned?
> ovirt-master vdsm failure - extend disk test times out
> -------------------------------------------------------
>
> Key: OVIRT-2311
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2311
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: infra
> Labels: ost_failures
>
> There is no failure of a test but we can see a gap of an hour from start of extend_disk1 until unhanded exception is thrown.
> Although the error is reported on 004_basic_sanity the logs stop on post-003_basic_networking.py.
> looking at the available logs it may be a task that is not cleared.
> I do not see a connection between the patch and the timeout unless something in the tests is looking ERROR: InterpreterNotFound: python3.6 errors in order to exit.
> patch that failed: https://gerrit.ovirt.org/#/c/92847/
> job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8648/
> Error I can see in the logs is:
> 2018-07-12 01:23:14,863-04 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] An error occurred while fetching unregistered disks from Storage Domain id '597bfe1f-d88c-4961-b1dc-f6c771af9
> fff'
> but I can't see an error in vdsm that would suggest we had a failure.
> you can see the gap in the job:
> 05:31:45 [basic-suit] # extend_disk1:
> 06:31:56 [basic-suit] * Unhandled exception in <function <lambda> at 0x7f5dd403c5f0>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100088)
6 years, 2 months
[JIRA] (OVIRT-2311) ovirt-master vdsm failure - extend disk test
times out
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2311:
--------------------------------
Summary: ovirt-master vdsm failure - extend disk test times out
Key: OVIRT-2311
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2311
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
There is no failure of a test but we can see a gap of an hour from start of extend_disk1 until unhanded exception is thrown.
Although the error is reported on 004_basic_sanity the logs stop on post-003_basic_networking.py.
looking at the available logs it may be a task that is not cleared.
I do not see a connection between the patch and the timeout unless something in the tests is looking ERROR: InterpreterNotFound: python3.6 errors in order to exit.
patch that failed: https://gerrit.ovirt.org/#/c/92847/
job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8648/
Error I can see in the logs is:
2018-07-12 01:23:14,863-04 ERROR [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [] An error occurred while fetching unregistered disks from Storage Domain id '597bfe1f-d88c-4961-b1dc-f6c771af9
fff'
but I can't see an error in vdsm that would suggest we had a failure.
you can see the gap in the job:
05:31:45 [basic-suit] # extend_disk1:
06:31:56 [basic-suit] * Unhandled exception in <function <lambda> at 0x7f5dd403c5f0>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100088)
6 years, 2 months
[JIRA] (OVIRT-2310) Make STDCI agnistic to wither and where you say
'please'
by Barak Korren (oVirt JIRA)
Barak Korren created OVIRT-2310:
-----------------------------------
Summary: Make STDCI agnistic to wither and where you say 'please'
Key: OVIRT-2310
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2310
Project: oVirt - virtualization made easy
Issue Type: Bug
Components: Standard CI (Pipelines)
Reporter: Barak Korren
Assignee: infra
In STDCI V1 we supported:
{code}
ci please build
{code}
In STDCI V2 we now support:
{code}
ci build please
{code}
We should make STDCI not care so it easier for people to use it.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100088)
6 years, 2 months
