[JIRA] (OVIRT-1870) kubevirt_kubevirt_standard-check-pr jobs often get stuck

kubevirt_kubevirt_standard-check-pr jobs often get stuck -------------------------------------------------------- Key: OVIRT-1870 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1870 Project: oVirt - virtualization made easy Issue Type: Outage Components: CI client projects, Standard CI (Pipelines) Reporter: Evgheni Dereveanchin Assignee: Barak Korren Priority: Highest kubevirt_kubevirt_standard-check-pr jobs often get stuck waiting forever for a connection to get established: http://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/376/console http://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/377/console http://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/378/console http://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/380/console The following message keeps repeating all the time: 17:03:32 [check-patch.el7.x86_64] ++ awk '/virt-controller/ && /true/' 17:03:32 [check-patch.el7.x86_64] ++ kubectl get pods -n kube-system '-ocustom-columns=status:status.containerStatuses[*].ready,metadata:metadata.name' --no-headers 17:03:32 [check-patch.el7.x86_64] ++ wc -l 17:03:32 [check-patch.el7.x86_64] ++ cluster/kubectl.sh get pods -n kube-system '-ocustom-columns=status:status.containerStatuses[*].ready,metadata:metadata.name' --no-headers 17:03:44 [check-patch.el7.x86_64] Unable to connect to the server: dial tcp 192.168.121.111:6443: getsockopt: no route to host 17:03:44 [check-patch.el7.x86_64] + '[' 0 -lt 1 ']' 17:03:44 [check-patch.el7.x86_64] + echo 'Waiting for KubeVirt virt-controller container to become ready ...' 17:03:44 [check-patch.el7.x86_64] Waiting for KubeVirt virt-controller container to become ready ... 17:03:44 [check-patch.el7.x86_64] + kubectl get pods -n kube-system '-ocustom-columns=status:status.containerStatuses[*].ready,metadata:metadata.name' --no-headers 17:03:44 [check-patch.el7.x86_64] + awk '/virt-controller/ && /true/' 17:03:44 [check-patch.el7.x86_64] + cluster/kubectl.sh get pods -n kube-system '-ocustom-columns=status:status.containerStatuses[*].ready,metadata:metadata.name' --no-headers 17:03:44 [check-patch.el7.x86_64] + wc -l 17:03:56 [check-patch.el7.x86_64] Unable to connect to the server: dial tcp 192.168.121.111:6443: getsockopt: no route to host 17:03:56 [check-patch.el7.x86_64] 0 17:03:56 [check-patch.el7.x86_64] + sleep 10 Need to implement timeouts as this takes up bare metal systems for days and weeks until someone manually kills the job