Re: [ovirt-infra] jenkins is misbehaving

Opened a ticket for tracking[1], still didn't get to do more tests. Also, this never happens in VDSM, probably because ovirt-engine.git is 244MB while vdsm.git is only 52MB. another failure: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-db-from-3.6_el6_cre... [1] https://ovirt-jira.atlassian.net/browse/OVIRT-403 On Thu, Jun 30, 2016 at 4:11 AM, Nadav Goldin <ngoldin(a)redhat.com&gt; wrote: > I was more or less able to reproduce the problem. I ran git clone on > ovirt-engine.git from one of the VMs in the Jenkins_CentOS cluster for > 200 times, with time out set to 90 seconds, and 15 seconds between > each clone. It had 13/200 failures, which is exactly 6.5%. This > explains why we don't see it often, it might be more severe as this > testing was done during the night when Jenkins/Gerrit aren't busy. > During that time there were few, but not 13, exceptions in gerrit's > error_log: > [2016-06-29 19:18:26,933] [NioProcessor-1] WARN > com.google.gerrit.sshd.GerritServerSession : Exception caught > org.apache.sshd.common.SshException: Received 97 on unknown channel 0 > at org.apache.sshd.common.session.AbstractConnectionService.getChannel(AbstractConnectionService.java:301) > ..... > Since it doesn't have the client IP log, its hard to tell if it is > correlated, even if it is, not all attempts reached to the exception > log. So there is a problem, independent of Jenkins itself. Will need > to dig deeper to find out what is causing it.. > > > > > On Wed, Jun 29, 2016 at 8:22 PM, Nadav Goldin <ngoldin(a)redhat.com&gt; wrote: >> 1. Its the second time it happens this week[1] >> 2. Around a month a go, I did a log analyse of how often this happens, >> and it was more than 10 times a week. >> 3. After Shlomi resolved few issues on Gerrit, it seem to have gone away. >> My guess is that this is network related or overload on Gerrit - it >> either fails when trying to connect to Gerrit, or while cloning(like >> in [1]). I didn't find any consistency in the error, which makes it >> hard to reproduce. The current re-trigger Anton did was on a BM metal >> slave, so I doubt its related to overload on the Jenkins slave itself. >> >> [1] http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/... >> >> On Wed, Jun 29, 2016 at 7:47 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>> Shlomi arw you running anything on gerrit now? >>> If you're copying the content please stop as it might affect gerrit >>> performance. >>> >>> On Jun 29, 2016 7:37 PM, "Anton Marchukov" <amarchuk(a)redhat.com&gt; wrote: >>>> >>>> Hello All. >>>> >>>> I tried to clone manually and this works: >>>> >>>> [amarchuk@ovirt-srv22 ~]$ git clone >>>> git://gerrit.ovirt.org/ovirt-engine.git >>>> Cloning into 'ovirt-engine'... >>>> remote: Counting objects: 784726, done. >>>> remote: Compressing objects: 100% (204209/204209), done. >>>> remote: Total 784726 (delta 360293), reused 777805 (delta 358840) >>>> Receiving objects: 100% (784726/784726), 136.26 MiB | 28.66 MiB/s, done. >>>> Resolving deltas: 100% (360293/360293), done. >>>> >>>> >>>> But failed in the job >>>> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/... >>>> >>>> So it is either not 100% reproducible or some Jenkins issue. >>>> >>>> Anybody did anything on Jenkins recently that can be correlated with this? >>>> >>>> Also gerrit plugin started to loose events... >>>> >>>> Anton. >>>> >>>> On Wed, Jun 29, 2016 at 6:14 PM, Piotr Kliczewski >>>> <piotr.kliczewski(a)gmail.com&gt; wrote: >>>>> >>>>> Some time ago jenkins did not update the patches with the score. Now I >>>>> see that builds are not triggered. One of the builds that I triggered >>>>> manually [1] failed with: >>>>> >>>>> 16:04:47 ERROR: Timeout after 10 minutes >>>>> 16:04:47 ERROR: Error cloning remote repo 'origin' >>>>> 16:04:47 hudson.plugins.git.GitException: Command "git fetch --tags >>>>> --progress git://gerrit.ovirt.org/ovirt-engine.git >>>>> +refs/heads/*:refs/remotes/origin/*" returned status code 143: >>>>> 16:04:47 stdout: >>>>> 16:04:47 stderr: >>>>> 16:04:47 at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640) >>>>> >>>>> Thanks, >>>>> Piotr >>>>> >>>>> [1] >>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-el7-x86_64/3... >>>>> _______________________________________________ >>>>> Infra mailing list >>>>> Infra(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> >>>> >>>> >>>> >>>> -- >>>> Anton Marchukov >>>> Senior Software Engineer - RHEV CI - Red Hat >>>> >>>> >>>> _______________________________________________ >>>> Infra mailing list >>>> Infra(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> >>> >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >>>