[JIRA] (OVIRT-620) Optimize how Jenkins fetchs ovirt-engine from gerrit
by Nadav Goldin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-620?page=com.atlassian.jira... ]
Nadav Goldin commented on OVIRT-620:
------------------------------------
from man git gc:
{quote}
--aggressive
Usually git gc runs very quickly while providing good disk space utilization and performance. This option will cause git gc to more aggressively optimize the repository at the expense of taking much more time. The effects of this optimization are persistent, so this option only needs to be used occasionally; every few hundred changesets or so.
{quote}
I am not sure if we have 100s of patchsets/daily, but maybe weekly/monthly. Other than that I never used it, so not sure if it could be a source of other problems.
> Optimize how Jenkins fetchs ovirt-engine from gerrit
> ----------------------------------------------------
>
> Key: OVIRT-620
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-620
> Project: oVirt - virtualization made easy
> Issue Type: Sub-task
> Reporter: Nadav Goldin
> Assignee: infra
>
> ovirt-engine git repository is at the time of writing is of size 244MB, where ovirt-engine/.git alone takes 160MB.
> There are two things which might be done:
> 1. Find a way to optimize ovirt-engine.git/.git size, this needs to be explored deeply.
> 2. Check if it is possible to configure git plugin to not download the entire history, although Jenkins attempts to optimize that, it still looks like the git plugin is downloading the entire history(this is what a random check on the slaves using du ../jenkins/workspace/...../.git -h -s showed)
> for instance the following command takes only 91Mb:
> {quote}git clone --depth 1 --branch master {quote}
> Although this will not solve real network/storage issues, it could help reduce those problems and in the overall decrease jobs runtime. As Gerrit is still on AWS, this is a considerable amount of traffic for each job we might be able to reduce.
--
This message was sent by Atlassian JIRA
(v1000.126.1#100004)
7 years, 9 months
Build failed in Jenkins: ovirt_3.6_system-tests #198
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/198/changes>
Changes:
[Eyal Edri] add hystrix deps to yum repos include list
------------------------------------------
[...truncated 554 lines...]
## took 2192 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
##!
@ Collect artifacts:
# ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host
# [Thread-1] lago_basic_suite_3_6_engine:
# [Thread-2] lago_basic_suite_3_6_host1:
# [Thread-3] lago_basic_suite_3_6_host0:
# [Thread-4] lago_basic_suite_3_6_storage:
# [Thread-2] lago_basic_suite_3_6_host1: Success (in 0:00:15)
# [Thread-3] lago_basic_suite_3_6_host0: Success (in 0:00:15)
# [Thread-4] lago_basic_suite_3_6_storage: Success (in 0:00:15)
# [Thread-1] lago_basic_suite_3_6_engine: Success (in 0:00:15)
@ Collect artifacts: Success (in 0:00:15)
<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests>
@@@@ ERROR: Failed running <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...>
#########################
======== Cleaning up
----------- Cleaning with lago
----------- Cleaning with lago done
======== Cleanup done
Took 2033 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=3.6
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson7736730615706189616.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=3.6
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
+ OVIRT_SUITE=3.6
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/198/artifact/exported...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/198/artifact/exported...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/198/artifact/exported...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
rm -rf "${logs[@]}"
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
[ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson7140842544403926302.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
./ovirt-system-tests/logs/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
+ rm -rf ./ovirt-system-tests/logs
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-fedora-23-x86_64.fc23.cfg
+ mock_root=mocker-fedora-23-x86_64.fc23
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests....> Using default...
INFO: mock.py version 1.2.17 starting (python version = 3.4.3)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...>
+ mock_root=fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
+ [[ -n fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/fedora-23-x86_64-ecdfd9c866cbf1a2d41342199fdb011f
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
Archiving artifacts
7 years, 9 months
Error in vdsm functional tests
by Yaniv Bronheim
Hi,
Recently we're getting the following error, even-though we require
python-mock in check-merged.packages.f23 (https://gerrit.ovirt.org/59797)
*09:52:43* # Deploy environment: Success (in 0:09:06)*09:52:44* @
Deploy oVirt environment: Success (in 0:09:07)*09:52:44* + lago shell
vdsm_functional_tests_host-fc23 -c 'mount -t tmpfs tmpfs
/sys/kernel/mm/ksm'*09:52:44* + tee
/home/jenkins/workspace/vdsm_master_check-merged-fc23-x86_64/vdsm/exported-artifacts/functional_tests_stdout.fc23.log*09:52:44*
current session does not belong to lago group.*09:52:46* + lago shell
vdsm_functional_tests_host-fc23 -c ' cd
/usr/share/vdsm/tests*09:52:46* ./run_tests.sh
--with-xunit
--xunit-file=/tmp/nosetests-fc23.xml -s
functional/supervdsmFuncTests.py '*09:52:47*
current session does not belong to lago group.*09:52:49* Traceback
(most recent call last):*09:52:49* File "../tests/testrunner.py",
line 42, in <module>*09:52:49* import testlib*09:52:49* File
"/usr/share/vdsm/tests/testlib.py", line 44, in <module>*09:52:49*
import mock*09:52:49* ImportError: No module named mock*09:52:49* +
failed=1*09:52:49* + lago copy-from-vm vdsm_functional_tests_host-fc23
/tmp/nosetests-fc23.xml
/home/jenkins/workspace/vdsm_master_check-merged-fc23-x86_64/vdsm/exported-artifacts/nosetests-fc23.xml*09:52:50*
current session does not belong to lago group.*09:52:51* Error
occured, aborting*09:52:51* Traceback (most recent call
last):*09:52:51* File
"/usr/lib/python2.7/site-packages/lago/cmd.py", line 691, in
main*09:52:51* cli_plugins[args.verb].do_run(args)*09:52:51*
File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180,
in do_run*09:52:51* self._do_run(**vars(args))*09:52:51* File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in
wrapper*09:52:51* return func(*args, **kwargs)*09:52:51* File
"/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in
wrapper*09:52:51* return func(*args, prefix=prefix,
**kwargs)*09:52:51* File
"/usr/lib/python2.7/site-packages/lago/cmd.py", line 514, in
do_copy_from_vm*09:52:51* host.copy_from(remote_path,
local_path)*09:52:51* File
"/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 319, in
copy_from*09:52:51* local_path=local_path,*09:52:51* File
"/usr/lib/python2.7/site-packages/scp.py", line 125, in get*09:52:51*
self._recv_all()*09:52:51* File
"/usr/lib/python2.7/site-packages/scp.py", line 257, in
_recv_all*09:52:51* raise SCPException(str(msg).strip())*09:52:51*
SCPException: scp: /tmp/nosetests-fc23.xml: No such file or
directory*09:52:51* + :*09:52:51* + lago stop
vdsm_functional_tests_host-fc23
--
*Yaniv Bronhaim.*
7 years, 9 months
[JIRA] (OVIRT-620) Optimize how Jenkins fetchs ovirt-engine from gerrit
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-620?page=com.atlassian.jira... ]
Barak Korren commented on OVIRT-620:
------------------------------------
Besides potential Gerrit slownes while this runs, any reason not to do this?
> Optimize how Jenkins fetchs ovirt-engine from gerrit
> ----------------------------------------------------
>
> Key: OVIRT-620
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-620
> Project: oVirt - virtualization made easy
> Issue Type: Sub-task
> Reporter: Nadav Goldin
> Assignee: infra
>
> ovirt-engine git repository is at the time of writing is of size 244MB, where ovirt-engine/.git alone takes 160MB.
> There are two things which might be done:
> 1. Find a way to optimize ovirt-engine.git/.git size, this needs to be explored deeply.
> 2. Check if it is possible to configure git plugin to not download the entire history, although Jenkins attempts to optimize that, it still looks like the git plugin is downloading the entire history(this is what a random check on the slaves using du ../jenkins/workspace/...../.git -h -s showed)
> for instance the following command takes only 91Mb:
> {quote}git clone --depth 1 --branch master {quote}
> Although this will not solve real network/storage issues, it could help reduce those problems and in the overall decrease jobs runtime. As Gerrit is still on AWS, this is a considerable amount of traffic for each job we might be able to reduce.
--
This message was sent by Atlassian JIRA
(v1000.126.1#100004)
7 years, 9 months
[JIRA] (OVIRT-620) Optimize how Jenkins fetchs ovirt-engine from gerrit
by Nadav Goldin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-620?page=com.atlassian.jira... ]
Nadav Goldin commented on OVIRT-620:
------------------------------------
[gerrit garbage collector job|http://jenkins-old.ovirt.org/job/system_gerrit-garbage-collect/] is running without --aggressive, when executing:
{quote}git clone git://gerrit.ovirt.org/ovirt-engine.git
cd ovirt-engine
git gc --aggressive
{quote}
this reduces the repository size by 62MB.
maybe we should add this option(from the docs it only says it will run slower, this is nightly anyways) or add it only to ovirt-engine.
[~eedri], [~bkorren(a)redhat.com] - what do you think?
> Optimize how Jenkins fetchs ovirt-engine from gerrit
> ----------------------------------------------------
>
> Key: OVIRT-620
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-620
> Project: oVirt - virtualization made easy
> Issue Type: Sub-task
> Reporter: Nadav Goldin
> Assignee: infra
>
> ovirt-engine git repository is at the time of writing is of size 244MB, where ovirt-engine/.git alone takes 160MB.
> There are two things which might be done:
> 1. Find a way to optimize ovirt-engine.git/.git size, this needs to be explored deeply.
> 2. Check if it is possible to configure git plugin to not download the entire history, although Jenkins attempts to optimize that, it still looks like the git plugin is downloading the entire history(this is what a random check on the slaves using du ../jenkins/workspace/...../.git -h -s showed)
> for instance the following command takes only 91Mb:
> {quote}git clone --depth 1 --branch master {quote}
> Although this will not solve real network/storage issues, it could help reduce those problems and in the overall decrease jobs runtime. As Gerrit is still on AWS, this is a considerable amount of traffic for each job we might be able to reduce.
--
This message was sent by Atlassian JIRA
(v1000.126.1#100004)
7 years, 9 months
[JIRA] (OVIRT-620) Optimize how Jenkins fetchs ovirt-engine from gerrit
by Nadav Goldin (oVirt JIRA)
Nadav Goldin created OVIRT-620:
----------------------------------
Summary: Optimize how Jenkins fetchs ovirt-engine from gerrit
Key: OVIRT-620
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-620
Project: oVirt - virtualization made easy
Issue Type: Sub-task
Reporter: Nadav Goldin
Assignee: infra
ovirt-engine git repository is at the time of writing is of size 244MB, where ovirt-engine/.git alone takes 160MB.
There are two things which might be done:
1. Find a way to optimize ovirt-engine.git/.git size, this needs to be explored deeply.
2. Check if it is possible to configure git plugin to not download the entire history, although Jenkins attempts to optimize that, it still looks like the git plugin is downloading the entire history(this is what a random check on the slaves using du ../jenkins/workspace/...../.git -h -s showed)
for instance the following command takes only 91Mb:
{quote}git clone --depth 1 --branch master {quote}
Although this will not solve real network/storage issues, it could help reduce those problems and in the overall decrease jobs runtime. As Gerrit is still on AWS, this is a considerable amount of traffic for each job we might be able to reduce.
--
This message was sent by Atlassian JIRA
(v1000.126.1#100004)
7 years, 9 months