- Infra - oVirt List Archives

[JIRA] (OVIRT-926) check why 3.6 experimental job failed even if not error in shown on job
by Barak Korren (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-926?page=com.atlassian.jira.p… ] Barak Korren commented on OVIRT-926: ------------------------------------ I'm seeing this in the end of the cleanup script: {quote} 18:35:39 + false {quote} So maybe its returning a false value that gets returned up the groovy stack (the groovy code seems to imply that 'sh' will throw an exception on script failure, but maybe that is not the case) Will need to write some simple Groovy jobs to understand this. > check why 3.6 experimental job failed even if not error in shown on job > ----------------------------------------------------------------------- > > Key: OVIRT-926 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-926 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_3.6/4322/ > [~gbenhaim@redhat.com][~ngoldin@redhat.com][~ederevea][~bkorren@redhat.com] any ideas? -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fix repoproxy service and logging mess
by Evgheni Dereveanchin (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] Evgheni Dereveanchin reassigned OVIRT-933: ------------------------------------------ Assignee: Evgheni Dereveanchin (was: infra) > Fix repoproxy service and logging mess > -------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: Evgheni Dereveanchin > Priority: High > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fix repoproxy service and logging mess
by eyal edri [Administrator] (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-933: ------------------------------------------------- [~ederevea] didn't we disable or stopped using the repo proxy? I remember another ticket on it. > Fix repoproxy service and logging mess > -------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > Priority: High > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fix repoproxy service and logging mess
by Yaniv Kaul (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] Yaniv Kaul updated OVIRT-933: ----------------------------- Priority: High (was: Medium) > Fix repoproxy service and logging mess > -------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > Priority: High > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fix repoproxy service and logging mess
by Yaniv Kaul (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] Yaniv Kaul commented on OVIRT-933: ---------------------------------- Moving priority to High, based on 'will cause it to crash' statement above. > Fix repoproxy service and logging mess > -------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > Priority: High > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fix repoproxy service and logging mess
by Barak Korren (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] Barak Korren updated OVIRT-933: ------------------------------- Summary: Fix repoproxy service and logging mess (was: Fixe repoproxy service and logging mess) > Fix repoproxy service and logging mess > -------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-931) profile why ovirt-engine check-patch takes between 10-50 min
by Evgheni Dereveanchin (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-931?page=com.atlassian.jira.p… ] Evgheni Dereveanchin reassigned OVIRT-931: ------------------------------------------ Assignee: Evgheni Dereveanchin (was: infra) > profile why ovirt-engine check-patch takes between 10-50 min > ------------------------------------------------------------- > > Key: OVIRT-931 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-931 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: eyal edri [Administrator] > Assignee: Evgheni Dereveanchin > Priority: High > > Same job sometimes takes little as 10 min but sometimes high as 50 min. > We need to understand why. > http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-el7-x86_64/ > http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-el7-x86_64/bui… -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fixe repoproxy service and logging mess
by Barak Korren (oVirt JIRA) 12 Dec '16

12 Dec '16

Barak Korren created OVIRT-933: ---------------------------------- Summary: Fixe repoproxy service and logging mess Key: OVIRT-933 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 Project: oVirt - virtualization made easy Issue Type: Improvement Reporter: Barak Korren Assignee: infra There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

[JIRA] (OVIRT-933) Fixe repoproxy service and logging mess
by Barak Korren (oVirt JIRA) 12 Dec '16

12 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-933?page=com.atlassian.jira.p… ] Barak Korren updated OVIRT-933: ------------------------------- Epic Link: OVIRT-403 > Fixe repoproxy service and logging mess > --------------------------------------- > > Key: OVIRT-933 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-933 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: Barak Korren > Assignee: infra > > There are a couple of issues with the way repoproxy runs right now, these issues probably cause it to be less stable then it can: > # repoproxy is can currently be invoked only from puppet, because a lot of the data on how to run it is stored right now in puppet and nowhere else. We should make puppet create a systemd unit file to run repoproxy instead of running it itself. This way we can get rid of the cron job that runs puppet every 2 minutes on the proxy. > # repoproxy is configured to run as the '{{squid}}' user and log to '{{/var/log/repoproxy.log}}'. That file is pre-created with the right permissions bu Puppet so repoproxy can write to it. However, the Python logger in repoproxy is configured to try and rotate the log when it becomes too big. That, in turn, will cause it to crash because the '{{squid}}' user cannot rename file in '{{/var/log}}'. We should move repoproxy logs somewhere else. -- This message was sent by Atlassian JIRA (v1000.620.0#100023)

1 0

Build failed in Jenkins: ovirt_4.0_system-tests #733
by jenkins＠jenkins.phx.ovirt.org 12 Dec '16

12 Dec '16

See <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/733/changes> Changes: [Your Name] basic networking: modify ip config to dhcp [Barak Korren] Add http_proxy env var to proxied mock envs ------------------------------------------ [...truncated 632 lines...] ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.basic_suite_4.0.sh/basic_suite_4.0.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/d…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/d…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 230 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/hudson7138590860841628533.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/e…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/733/artifact/exported-a…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/733/artifact/exported-a…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/e…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/733/artifact/exported-a…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then for log in "${logs[@]}"; do [[ "$log" = ./exported-artifacts/* ]] && continue echo "Copying ${log} to exported-artifacts" mv $log exported-artifacts/ done fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_4.0_system-tests] $ /bin/bash -x /tmp/hudson6396308667636953382.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + for log in '"${logs[@]}"' + [[ ./ovirt-system-tests/logs = ./exported-artifacts/* ]] + echo 'Copying ./ovirt-system-tests/logs to exported-artifacts' Copying ./ovirt-system-tests/logs to exported-artifacts + mv ./ovirt-system-tests/logs exported-artifacts/ + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/m…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/l….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests/m…> + mock_root=epel-7-x86_64-ed452d1ea0f6dadf50c66d3b3f6b3a6e + [[ -n epel-7-x86_64-ed452d1ea0f6dadf50c66d3b3f6b3a6e ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-ed452d1ea0f6dadf50c66d3b3f6b3a6e + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

2 5