[JIRA] (OVIRT-2741) Most el7 vdsm build fail to fetch source - bad slave?

Most el7 vdsm build fail to fetch source - bad slave? ----------------------------------------------------- Key: OVIRT-2741 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2741 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra I have see many such failures today, here is 2 bulids - both on same slave (slave issue?) 1. [2019-06-20T22:23:12.401Z] Running on node: vm0015.workers-phx.ovirt.org (el7 phx nested local_disk) https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-... 2. [2019-06-20T22:20:50.288Z] Running on node: vm0015.workers-phx.ovirt.org (el7 phx nested local_disk) https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-... Looks like there was no build on this slave in the last 2 days: https://jenkins.ovirt.org/computer/vm0015.workers-phx.ovirt.org/builds The failure looks like this: [2019-06-20T22:23:16.416Z] No credentials specified [2019-06-20T22:23:16.440Z] Fetching changes from the remote Git repository [2019-06-20T22:23:16.456Z] Cleaning workspace [2019-06-20T22:23:16.566Z] ERROR: Error fetching remote repo 'origin' [2019-06-20T22:23:16.566Z] hudson.plugins.git.GitException: Failed to fetch from https://gerrit.ovirt.org/vdsm [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894) [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) [2019-06-20T22:23:16.566Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) [2019-06-20T22:23:16.566Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) [2019-06-20T22:23:16.566Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2019-06-20T22:23:16.566Z] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [2019-06-20T22:23:16.566Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [2019-06-20T22:23:16.566Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [2019-06-20T22:23:16.566Z] at java.lang.Thread.run(Thread.java:748) [2019-06-20T22:23:16.566Z] Caused by: hudson.plugins.git.GitException: Command "git clean -fdx" returned status code 1: [2019-06-20T22:23:16.566Z] stdout: [2019-06-20T22:23:16.566Z] stderr: warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage___init___py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockVolume_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockdev_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_check_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_clusterlock_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_compat_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_constants_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_curlImgWrap_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_devicemapper_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_directio_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_dispatcher_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_exception_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fallocate_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileSD_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileUtils_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fileVolume_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_formatconverter_py.html [2019-06-20T22:23:16.566Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fsutils_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_fuser_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_glusterSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_glusterVolume_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_guarded_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_hba_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_hsm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_image_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_imageSharing_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_imagetickets_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_iscsi_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_iscsiadm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_localFsSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvm_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvmconf_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_lvmfilter_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mailbox_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_managedvolume_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/index.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_managedvolumedb_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_merge_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_misc_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_monitor_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mount_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_mpathhealth_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_multipath_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nbd_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nfsSD_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_nos_brick_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_operation_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_outOfProcess_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_persistent_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_qemuimg_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_resourceFactories_py.html [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/status.json [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/style.css [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.ba-throttle-debounce.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.hotkeys.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.isonscreen.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/jquery.tablesorter.min.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/coverage_html.js [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/keybd_closed.png [2019-06-20T22:23:16.567Z] warning: failed to remove tests/htmlcov-storage-py27/keybd_open.png [2019-06-20T22:23:16.567Z] [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650) [2019-06-20T22:23:16.567Z] at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:828) [2019-06-20T22:23:16.567Z] at hudson.plugins.git.GitAPI.clean(GitAPI.java:311) [2019-06-20T22:23:16.567Z] at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) [2019-06-20T22:23:16.567Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [2019-06-20T22:23:16.567Z] at java.lang.reflect.Method.invoke(Method.java:498) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) [2019-06-20T22:23:16.567Z] at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) [2019-06-20T22:23:16.567Z] at hudson.remoting.UserRequest.perform(UserRequest.java:212) [2019-06-20T22:23:16.567Z] at hudson.remoting.UserRequest.perform(UserRequest.java:54) [2019-06-20T22:23:16.567Z] at hudson.remoting.Request$2.run(Request.java:369) [2019-06-20T22:23:16.567Z] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) [2019-06-20T22:23:16.567Z] ... 4 more [2019-06-20T22:23:16.568Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vm0015.workers-phx.ovirt.org [2019-06-20T22:23:16.568Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) [2019-06-20T22:23:16.568Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) [2019-06-20T22:23:16.568Z] at hudson.remoting.Channel.call(Channel.java:957) [2019-06-20T22:23:16.568Z] at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) [2019-06-20T22:23:16.568Z] at com.sun.proxy.$Proxy83.clean(Unknown Source) [2019-06-20T22:23:16.568Z] at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:450) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:30) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:890) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) [2019-06-20T22:23:16.568Z] at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) [2019-06-20T22:23:16.568Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77) [2019-06-20T22:23:16.569Z] at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) [2019-06-20T22:23:16.569Z] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [2019-06-20T22:23:16.569Z] ... 4 more [2019-06-20T22:23:16.421Z] > git rev-parse --is-inside-work-tree # timeout=10 [2019-06-20T22:23:16.444Z] > git config remote.origin.url https://gerrit.ovirt.org/vdsm # timeout=10 [2019-06-20T22:23:16.459Z] > git rev-parse --verify HEAD # timeout=10 [2019-06-20T22:23:16.466Z] Resetting working tree [2019-06-20T22:23:16.466Z] > git reset --hard # timeout=10 [2019-06-20T22:23:16.514Z] > git clean -fdx # timeout=10 Error fetching remote repo 'origin'