[ https://ovirt-jira.atlassian.net/browse/OVIRT-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=39422#comment-39422 ] Barak Korren edited comment on OVIRT-2741 at 6/24/19 7:39 AM: -------------------------------------------------------------- This is the real cause of failure: {code} [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html {code} It means we have root-owned files win the VDSM workspace on the slave - this is caused by people manually aborting jobs before the cleanup code can run. I see the slave is clean now, probably got cleaned by a job for a different project that did not stumble on the vdsm files. Going to close this ticket, since this is not really an ongoing infra issue. was (Author: bkorren@redhat.com): This is the real cause of failure: {code} [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html [2019-06-20T22:20:54.330Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html {code} It means we have root-owned files win the VDSM workspace on the slave - this is caused by people manually aborting jobs before the cleanup code can run. I see the slave is clean now, probally got cleaned by a job for a different project that did not stumble on the vdsm files. Goind to close this ticket, since this is not really an ongoing infra issue.

Most el7 vdsm build fail to fetch source - bad slave? -----------------------------------------------------

Key: OVIRT-2741 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2741 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra

I have see many such failures today, here is 2 bulids - both on same slave (slave issue?) 1. [2019-06-20T22:23:12.401Z] Running on node: vm0015.workers-phx.ovirt.org (el7 phx nested local_disk) https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... 2. [2019-06-20T22:20:50.288Z] Running on node: vm0015.workers-phx.ovirt.org (el7 phx nested local_disk) https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... Looks like there was no build on this slave in the last 2 days: https://jenkins.ovirt.org/computer/vm0015.workers-phx.ovirt.org/builds The failure looks like this: [2019-06-20T22:23:16.416Z] No credentials specified [2019-06-20T22:23:16.440Z] Fetching changes from the remote Git repository [2019-06-20T22:23:16.456Z] Cleaning workspace [2019-06-20T22:23:16.566Z] ERROR: Error fetching remote repo 'origin' [2019-06-20T22:23:16.566Z] hudson.plugins.git.GitException: Failed to fetch from https://gerrit.ovirt.org/vdsm [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894) [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) [2019-06-20T22:23:16.566Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2019-06-20T22:23:16.566Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [2019-06-20T22:23:16.566Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [2019-06-20T22:23:16.566Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [2019-06-20T22:23:16.566Z] at java.lang.Thread.run(Thread.java:748) [2019-06-20T22:23:16.566Z] Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1: [2019-06-20T22:23:16.566Z] stdout: [2019-06-20T22:23:16.566Z] stderr: warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage___init___py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockVolume_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockdev_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_check_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_clusterlock_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_compat_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_constants_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_curlImgWrap_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_devicemapper_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_directio_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_dispatcher_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_exception_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fallocate_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileSD_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileUtils_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileVolume_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_formatconverter_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fsutils_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fuser_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_glusterSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_glusterVolume_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_guarded_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_hba_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_hsm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_image_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_imageSharing_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_imagetickets_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_iscsi_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_iscsiadm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_localFsSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvmconf_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvmfilter_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mailbox_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_managedvolume_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/index.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_managedvolumedb_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_merge_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_misc_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_monitor_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mount_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mpathhealth_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_multipath_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nbd_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nfsSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nos_brick_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_operation_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_outOfProcess_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_persistent_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_qemuimg_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_resourceFactories_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/status.json [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/style.css [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.ba-throttle-debounce.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.hotkeys.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.isonscreen.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.tablesorter.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/coverage_html.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/keybd_closed.png [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/keybd_open.png [2019-06-20T22:23:16.567Z] [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:828) [2019-06-20T22:23:16.567Z] at hudson.plugins.git.GitAPI.clean(GitAPI.java:311) [2019-06-20T22:23:16.567Z] at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) [2019-06-20T22:23:16.567Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [2019-06-20T22:23:16.567Z] at java.lang.reflect.Method.invoke(Method.java:498) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) [2019-06-20T22:23:16.567Z] at hudson.remoting.UserRequest.perform(UserRequest.java:212) [2019-06-20T22:23:16.567Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54) [2019-06-20T22:23:16.567Z] at hudson.remoting.Request$2.run(Request.java:369) [2019-06-20T22:23:16.567Z] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) [2019-06-20T22:23:16.567Z] ... 4 more [2019-06-20T22:23:16.568Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vm0015.workers-phx.ovirt.org [2019-06-20T22:23:16.568Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) [2019-06-20T22:23:16.568Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) [2019-06-20T22:23:16.568Z] at hudson.remoting.Channel.call(Channel.java:957) [2019-06-20T22:23:16.568Z] at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) [2019-06-20T22:23:16.568Z] at com.sun.proxy.$Proxy83.clean(Unknown Source) [2019-06-20T22:23:16.568Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:450) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:30) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:890) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) [2019-06-20T22:23:16.568Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) [2019-06-20T22:23:16.569Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2019-06-20T22:23:16.569Z] ... 4 more [2019-06-20T22:23:16.421Z] > git rev-parse --is-inside-work-tree # timeout=10 [2019-06-20T22:23:16.444Z] > git config remote.origin.url https://gerrit.ovirt.org/vdsm # timeout=10 [2019-06-20T22:23:16.459Z] > git rev-parse --verify HEAD # timeout=10 [2019-06-20T22:23:16.466Z] Resetting working tree [2019-06-20T22:23:16.466Z] > git reset --hard # timeout=10 [2019-06-20T22:23:16.514Z] > git clean -fdx # timeout=10 Error fetching remote repo 'origin'

-- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100105)