Build failed in Jenkins: ovirt_master_system-tests #854
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_master_system-tests/854/changes>
Changes:
[Yaniv Kaul] CHAP and IPv6 for iSCSI
[Anton Marchukov] Updated Fedora distro list for repoman (-fc23 +fc25)
------------------------------------------
[...truncated 749 lines...]
## Sat Dec 3 02:46:05 UTC 2016 Finished env: el7:epel-7-x86_64
## took 892 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.basic_suite_master.sh/basic_suite_master.sh.log
##!
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 746 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=master
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson6244207805446786015.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=master
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
+ OVIRT_SUITE=master
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_master_system-tests/854/artifact/expor...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_master_system-tests/854/artifact/expor...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/854/artifact/expor...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
for log in "${logs[@]}"
do
echo "Copying ${log} to exported-artifacts"
mv $log exported-artifacts/
done
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
# Drop all left over libvirt domains
for UUID in $(virsh list --all --uuid); do
virsh destroy $UUID || :
sleep 2
virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || :
done
[ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson7539346291946703264.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ for log in '"${logs[@]}"'
+ echo 'Copying ./ovirt-system-tests/logs to exported-artifacts'
Copying ./ovirt-system-tests/logs to exported-artifacts
+ mv ./ovirt-system-tests/logs exported-artifacts/
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-epel-7-x86_64.el7.cfg
+ mock_root=mocker-epel-7-x86_64.el7
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-epel-7-x86_64.el7'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te....> Using default...
INFO: mock.py version 1.2.21 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...>
+ mock_root=epel-7-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ [[ -n epel-7-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep epel-7-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/epel-7-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
++ virsh list --all --uuid
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
Archiving artifacts
7 years, 11 months
oVirt infra daily report - unstable production jobs - 158
by jenkins＠jenkins.phx.ovirt.org
------=_Part_271_237532515.1480892409104
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/158//artifact/exported...
Cheers,
Jenkins
------=_Part_271_237532515.1480892409104
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 04/12/2016
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Jobs (Production)</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/cockpit-ovirt_ovirt-4.1_build-artifacts-fc23...">cockpit-ovirt_ovirt-4.1_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/fabric-ovirt_master_check-merged-el7-x86_64/">fabric-ovirt_master_check-merged-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-engine-api-explorer_master_build-artif...">ovirt-engine-api-explorer_master_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-fc2...">ovirt-guest-agent_master_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifact...">ovirt-hosted-engine-ha_master_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifact...">ovirt-hosted-engine-ha_master_build-artifacts-fc24-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-live_master_create-iso-el7-x86_64/">ovirt-live_master_create-iso-el7-x86_64</a>
</td><td>
<h3>This job generates a nightly iso of ovirt-live </h3>
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-live_master_experimental_create-iso-el...">ovirt-live_master_experimental_create-iso-el7-x86_64</a>
</td><td>
<h3>This job generates a nightly iso of ovirt-live using experimental repos</h3>
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-el7-x86...">ovirt-node-ng_master_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-3.6_build-artifacts-el7-...">ovirt-node-ng_ovirt-3.6_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.0_build-artifacts-el7-...">ovirt-node-ng_ovirt-4.0_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-pre_build-artifacts-...">ovirt-node-ng_ovirt-4.1-pre_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_buil...">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/">ovirt-node_ovirt-3.6_create-iso-el7_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-optimizer_4.0_build-artifacts-el6-x86_64/">ovirt-optimizer_4.0_build-artifacts-el6-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/">ovirt_3.6_he-system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/">ovirt_master_system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests_debug/">ovirt_master_system-tests_debug</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/system-backup_jenkins_ovirt_org/">system-backup_jenkins_ovirt_org</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/vdsm_4.0_check-merged-fc23-x86_64/">vdsm_4.0_check-merged-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/">vdsm_master_check-merged-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_271_237532515.1480892409104--
7 years, 11 months
Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM
by Yaniv Kaul
On Dec 4, 2016 10:57 PM, "Eyal Edri" <eedri(a)redhat.com> wrote:
tests are back to stable, but with a cost of not testing 4.1 CL.
DC level.
But we are not explicitly using any of the 4.1 features for the time being.
(I'd like to believe implictly we did, qcow2v3 for example).
Iets hope we get centos 7.3 soon.
Indeed - but we also need virt-builder image for it.
Y.
On Dec 4, 2016 22:41, "Yaniv Kaul" <ykaul(a)redhat.com> wrote:
>
>
> On Dec 4, 2016 6:42 PM, "Arik Hadas" <ahadas(a)redhat.com> wrote:
>
> Yaniv will try to lower the cluster level used in the system-tests to 4.0
> - this is supposed to solve the issue.
>
>
> Done.
> Y.
>
> If it won't help (we will know it in about an hour), we'll add a db-script
> that changes the rng device of the blank template only.
>
> On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <eedri(a)redhat.com> wrote:
>
>> FYI,
>>
>> I opened a bug [1] to track this issue since I don't see any attempts to
>> resolve the issue on the thread, hopefully a bug will get more attention.
>> Opened on VDSM since we see the libvirt error there, feel free to move
>> product/team.
>>
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303
>>
>> On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri(a)redhat.com> wrote:
>>
>>> Not sure if relevant, but Juan posted a fix for SDK4 last time it
>>> happened ( but different failure on log-collector ):
>>>
>>> https://gerrit.ovirt.org/#/c/67213/
>>>
>>> * Added `urandom` to the `RngSource` enumerated type.
>>>
>>> On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri(a)redhat.com> wrote:
>>>
>>>> And its still failing from Friday,
>>>> Since we don't have official Centos 7.3 repos yet ( hopefully we'll
>>>> have it this week, but as of this moment its not published yet ) , we have
>>>> to either revert the offending patch
>>>> or send a quick fix.
>>>>
>>>> Right now all experimental flows for master are not working and nightly
>>>> rpms are not refreshed with new RPMs.
>>>>
>>>>
>>>>
>>>> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com> wrote:
>>>>>
>>>>> Hello Martin.
>>>>>
>>>>> Do by outdated you mean the old libvirt? If so that is that livirt
>>>>> available in CentOS 7.2? There is no 7.3 yet.
>>>>>
>>>>>
>>>>> Right, this is the issue.
>>>>> Y.
>>>>>
>>>>>
>>>>> Anton.
>>>>>
>>>>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote:
>>>>>>
>>>>>>> Hello All.
>>>>>>>
>>>>>>> Engine log can be viewed here:
>>>>>>>
>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>>>>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi
>>>>>>> rt-engine/engine.log
>>>>>>>
>>>>>>> I see the following exception there:
>>>>>>>
>>>>>>> 2016-12-02 04:29:24,030-05 DEBUG
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
>>>>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id":
>>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true}
>>>>>>> 2016-12-02 04:29:24,030-05 ERROR
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
>>>>>>> [83b6b5d] Not able to update response for
>>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c"
>>>>>>> 2016-12-02 04:29:24,041-05 DEBUG
>>>>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener]
>>>>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling
>>>>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref
>>>>>>> reshLightWeightData#-9223372036854775775
>>>>>>> as there is no unfired trigger.
>>>>>>> 2016-12-02 04:29:24,024-05 DEBUG
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception:
>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>>>>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
>>>>>>> at org.ovirt.engine.core.vdsbroke
>>>>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73)
>>>>>>> [vdsbroker.jar:]
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> 2016-12-02 04:29:24,042-05 ERROR
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for
>>>>>>> VDSM response: Internal timeout occured
>>>>>>> 2016-12-02 04:29:24,044-05 DEBUG
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD
>>>>>>> SCommand]
>>>>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START,
>>>>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0,
>>>>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
>>>>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5',
>>>>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9
>>>>>>> 188-685b6c64a2f5]'}),
>>>>>>> log id: 58f448b8
>>>>>>> 2016-12-02 04:29:24,044-05 DEBUG
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND
>>>>>>> destination:jms.topic.vdsm_requests
>>>>>>> reply-to:jms.topic.vdsm_responses
>>>>>>> content-length:105
>>>>>>>
>>>>>>>
>>>>>>> Please note that this runs on localhost with local bridge. So it is
>>>>>>> not
>>>>>>> likely to be network itself.
>>>>>>>
>>>>>>
>>>>>> The main issue I see is that the VM run command has actually failed
>>>>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was
>>>>>> done as engine patch and according to git log, posted around Mon Nov
>>>>>> 28. Also adding Jakub - this should either not happen from engine's
>>>>>> point of view or the lago host is outdated.
>>>>>>
>>>>>> [1]
>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>>>>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log
>>>>>>
>>>>>>
>>>>>> Anton.
>>>>>>>
>>>>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <
>>>>>>> amarchuk(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> FYI. Experimental flow for master currently fails to run a VM. The
>>>>>>>> tests
>>>>>>>> times out while waiting for 180 seconds:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>>>>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/
>>>>>>>>
>>>>>>>> This is reproducible over 23 runs of this happened tonight, sounds
>>>>>>>> like a
>>>>>>>> regression to me:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/
>>>>>>>>
>>>>>>>> I will update here with additional information once I find it.
>>>>>>>>
>>>>>>>> Last successful run was with this patch:
>>>>>>>>
>>>>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters
>>>>>>>> fixup
>>>>>>>> in a method)
>>>>>>>>
>>>>>>>> Known to start failing around this patch:
>>>>>>>>
>>>>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string
>>>>>>>> formatting)
>>>>>>>>
>>>>>>>> Please notes that we do not have gating implemented yet, so
>>>>>>>> everything
>>>>>>>> that was merged in between those patches might have caused this (not
>>>>>>>> necessary in vdsm project).
>>>>>>>>
>>>>>>>> Anton.
>>>>>>>> --
>>>>>>>> Anton Marchukov
>>>>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anton Marchukov
>>>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel(a)ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anton Marchukov
>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Eyal Edri
>>>> Associate Manager
>>>> RHV DevOps
>>>> EMEA ENG Virtualization R&D
>>>> Red Hat Israel
>>>>
>>>> phone: +972-9-7692018
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>
>>>
>>>
>>>
>>> --
>>> Eyal Edri
>>> Associate Manager
>>> RHV DevOps
>>> EMEA ENG Virtualization R&D
>>> Red Hat Israel
>>>
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>
>>
>>
>>
>> --
>> Eyal Edri
>> Associate Manager
>> RHV DevOps
>> EMEA ENG Virtualization R&D
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
7 years, 11 months
Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM
by Eyal Edri
And its still failing from Friday,
Since we don't have official Centos 7.3 repos yet ( hopefully we'll have it
this week, but as of this moment its not published yet ) , we have to
either revert the offending patch
or send a quick fix.
Right now all experimental flows for master are not working and nightly
rpms are not refreshed with new RPMs.
On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>
> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com> wrote:
>
> Hello Martin.
>
> Do by outdated you mean the old libvirt? If so that is that livirt
> available in CentOS 7.2? There is no 7.3 yet.
>
>
> Right, this is the issue.
> Y.
>
>
> Anton.
>
> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com>
> wrote:
>
>> On 02/12/16 10:55 +0100, Anton Marchukov wrote:
>>
>>> Hello All.
>>>
>>> Engine log can be viewed here:
>>>
>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi
>>> rt-engine/engine.log
>>>
>>> I see the following exception there:
>>>
>>> 2016-12-02 04:29:24,030-05 DEBUG
>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id":
>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true}
>>> 2016-12-02 04:29:24,030-05 ERROR
>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
>>> [83b6b5d] Not able to update response for
>>> "ec254aad-441b-47e7-a644-aebddcc1d62c"
>>> 2016-12-02 04:29:24,041-05 DEBUG
>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener]
>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling
>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref
>>> reshLightWeightData#-9223372036854775775
>>> as there is no unfired trigger.
>>> 2016-12-02 04:29:24,024-05 DEBUG
>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception:
>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
>>> at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.g
>>> et(FutureVDSCommand.java:73)
>>> [vdsbroker.jar:]
>>>
>>> ...
>>>
>>> 2016-12-02 04:29:24,042-05 ERROR
>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for
>>> VDSM response: Internal timeout occured
>>> 2016-12-02 04:29:24,044-05 DEBUG
>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START,
>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0,
>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5',
>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9
>>> 188-685b6c64a2f5]'}),
>>> log id: 58f448b8
>>> 2016-12-02 04:29:24,044-05 DEBUG
>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default
>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND
>>> destination:jms.topic.vdsm_requests
>>> reply-to:jms.topic.vdsm_responses
>>> content-length:105
>>>
>>>
>>> Please note that this runs on localhost with local bridge. So it is not
>>> likely to be network itself.
>>>
>>
>> The main issue I see is that the VM run command has actually failed
>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was
>> done as engine patch and according to git log, posted around Mon Nov
>> 28. Also adding Jakub - this should either not happen from engine's
>> point of view or the lago host is outdated.
>>
>> [1]
>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log
>>
>>
>> Anton.
>>>
>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <amarchuk(a)redhat.com>
>>> wrote:
>>>
>>> FYI. Experimental flow for master currently fails to run a VM. The tests
>>>> times out while waiting for 180 seconds:
>>>>
>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/
>>>>
>>>> This is reproducible over 23 runs of this happened tonight, sounds like
>>>> a
>>>> regression to me:
>>>>
>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/
>>>>
>>>> I will update here with additional information once I find it.
>>>>
>>>> Last successful run was with this patch:
>>>>
>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters
>>>> fixup
>>>> in a method)
>>>>
>>>> Known to start failing around this patch:
>>>>
>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string
>>>> formatting)
>>>>
>>>> Please notes that we do not have gating implemented yet, so everything
>>>> that was merged in between those patches might have caused this (not
>>>> necessary in vdsm project).
>>>>
>>>> Anton.
>>>> --
>>>> Anton Marchukov
>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>
>>>>
>>>>
>>>
>>> --
>>> Anton Marchukov
>>> Senior Software Engineer - RHEV CI - Red Hat
>>>
>>
>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>
>
> --
> Anton Marchukov
> Senior Software Engineer - RHEV CI - Red Hat
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
7 years, 11 months
