On Thu, Sep 8, 2016 at 5:45 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > Adding devel. > > On Thu, Sep 8, 2016 at 5:43 PM, Shlomo Ben David <sbendavi(a)redhat.com&gt; > wrote: >> >> Hi, >> >> Job [1] is failing with the following error: >> >> lago.ssh: DEBUG: Command 8de75538 on lago_basic_suite_master_engine >> errors: >> ERROR: Failed to collect logs from: 192.168.200.2; /bin/ls: >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d- 9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >> No such file or directory >> /bin/ls: cannot open directory >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d- 9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >> No such file or directory This looks like a lago issue - it should never read anything inside /rhev This is a private directory for vdsm, no other process should ever depend on the content inside this directory, or even on the fact that it exists. In particular, /rhev/data-center/mnt/blockSD/*/master/tasks/*.temp Is not a log file, and lago should not collect it. Nir >> lago.utils: ERROR: Error while running thread >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in >> _ret_via_queue >> queue.put({'return': func()}) >> File >> "/home/jenkins/workspace/test-repo_ovirt_experimental_ master/ovirt-system-tests/basic_suite_master/test- scenarios/002_bootstrap.py", >> line 493, in log_collector >> result.code, 0, 'log collector failed. Exit code is %s' % result.code >> File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, >> in eq_ >> raise AssertionError(msg or "%r != %r" % (a, b)) >> AssertionError: log collector failed. Exit code is 2 >> >> >> * The previous issue already fixed (SDK) and now we have a new issue on >> the same area. >> >> >> [1] - >> http://jenkins.ovirt.org/view/experimental%20jobs/job/test- repo_ovirt_experimental_master/1462/testReport/(root)/ 002_bootstrap/add_secondary_storage_domains/ >> >> >> Best Regards, >> >> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL >> RHCSA | RHCE >> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) >> >> OPEN SOURCE - 1 4 011 && 011 4 1 > > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

Indeed, this is the log collector. I wonder if we collect its logs... Y.

On Thu, Sep 8, 2016 at 6:54 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > I'm pretty sure lago or ovirt system tests aren't doing it but its the > log collector which is running during that test, I'm not near a computer so > can't verify it yet. > > On Sep 8, 2016 6:05 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: > >> On Thu, Sep 8, 2016 at 5:45 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >> > Adding devel. >> > >> > On Thu, Sep 8, 2016 at 5:43 PM, Shlomo Ben David <sbendavi(a)redhat.com&gt; >> > wrote: >> >> >> >> Hi, >> >> >> >> Job [1] is failing with the following error: >> >> >> >> lago.ssh: DEBUG: Command 8de75538 on lago_basic_suite_master_engine >> >> errors: >> >> ERROR: Failed to collect logs from: 192.168.200.2; /bin/ls: >> >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d-945189 >> 0fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >> >> No such file or directory >> >> /bin/ls: cannot open directory >> >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d-945189 >> 0fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >> >> No such file or directory >> >> This looks like a lago issue - it should never read anything inside /rhev >> >> This is a private directory for vdsm, no other process should ever depend >> on the content inside this directory, or even on the fact that it exists. >> >> In particular, /rhev/data-center/mnt/blockSD/*/master/tasks/*.temp >> Is not a log file, and lago should not collect it. >> >> Nir >> >> >> lago.utils: ERROR: Error while running thread >> >> Traceback (most recent call last): >> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in >> >> _ret_via_queue >> >> queue.put({'return': func()}) >> >> File >> >> "/home/jenkins/workspace/test-repo_ovirt_experimental_master >> /ovirt-system-tests/basic_suite_master/test-scenarios/002_bootstrap.py", >> >> line 493, in log_collector >> >> result.code, 0, 'log collector failed. Exit code is %s' % >> result.code >> >> File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", >> line 29, >> >> in eq_ >> >> raise AssertionError(msg or "%r != %r" % (a, b)) >> >> AssertionError: log collector failed. Exit code is 2 >> >> >> >> >> >> * The previous issue already fixed (SDK) and now we have a new issue >> on >> >> the same area. >> >> >> >> >> >> [1] - >> >> http://jenkins.ovirt.org/view/experimental%20jobs/job/test-r >> epo_ovirt_experimental_master/1462/testReport/(root)/002_boo >> tstrap/add_secondary_storage_domains/ >> >> >> >> >> >> Best Regards, >> >> >> >> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL >> >> RHCSA | RHCE >> >> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) >> >> >> >> OPEN SOURCE - 1 4 011 && 011 4 1 >> > >> > >> > >> > >> > -- >> > Eyal Edri >> > Associate Manager >> > RHV DevOps >> > EMEA ENG Virtualization R&D >> > Red Hat Israel >> > >> > phone: +972-9-7692018 >> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > >> > _______________________________________________ >> > Devel mailing list >> > Devel(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/devel >> >

Hi all, bumping this thread due to an almost identical failure[1]: ovirt-log-collector/ovirt-log-collector-20161030053238.log:2016-10-30 05:33:09::ERROR::__main__::791::root:: Failed to collect logs from: 192.168.200.4; /bin/ls: /rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5- 5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7- 52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.job.0: No such file or directory ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls: cannot access /rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5- 5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7- 52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.recover.1: No such file or directory ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls: cannot access /rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5- 5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7- 52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.task: No such file or directory ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls: cannot access /rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5- 5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7- 52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.recover.0: No such file or directory To ensure I've checked lago/OST, and couldn't find any stage where there is a reference to '/rhv' nor any manipulation to ovirt-log-collector, only customizations made is a 'ovirt-log-collector.conf' with user/password. The code that pulls the logs in OST[2] runs the following command on the engine VM(and there it fails): ovirt-log-collector --conf /rot/ovirt-log-collector.conf The failure comes right after 'add_secondary_storage_domains'[3] test, which all of its steps ran successfully.

Can anyone look into this?

Thanks, Nadav. [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_ check-patch-fc24-x86_64/141/console [2] https://github.com/oVirt/ovirt-system-tests/blob/ master/basic_suite_master/test-scenarios/002_bootstrap.py#L490 [3] https://github.com/oVirt/ovirt-system-tests/blob/ master/basic_suite_master/test-scenarios/002_bootstrap.py#L243 On Tue, Sep 20, 2016 at 9:45 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > > On Fri, Sep 9, 2016 at 1:19 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >> >> Indeed, this is the log collector. I wonder if we collect its logs... >> Y. > > > This can't be log-collector, it can be sos vdsm plugin. > That said, if we run log-collector within lago we should collect the results as job artifacts. > > >> >> >> >> On Thu, Sep 8, 2016 at 6:54 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>> >>> I'm pretty sure lago or ovirt system tests aren't doing it but its the log collector which is running during that test, I'm not near a computer so can't verify it yet. >>> >>> >>> On Sep 8, 2016 6:05 PM, "Nir Soffer" <nsoffer(a)redhat.com&gt; wrote: >>>> >>>> On Thu, Sep 8, 2016 at 5:45 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>> > Adding devel. >>>> > >>>> > On Thu, Sep 8, 2016 at 5:43 PM, Shlomo Ben David < sbendavi(a)redhat.com&gt; >>>> > wrote: >>>> >> >>>> >> Hi, >>>> >> >>>> >> Job [1] is failing with the following error: >>>> >> >>>> >> lago.ssh: DEBUG: Command 8de75538 on lago_basic_suite_master_engine >>>> >> errors: >>>> >> ERROR: Failed to collect logs from: 192.168.200.2; /bin/ls: >>>> >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d- 9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >>>> >> No such file or directory >>>> >> /bin/ls: cannot open directory >>>> >> /rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d- 9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp: >>>> >> No such file or directory >>>> >>>> This looks like a lago issue - it should never read anything inside /rhev >>>> >>>> This is a private directory for vdsm, no other process should ever depend >>>> on the content inside this directory, or even on the fact that it exists. >>>> >>>> In particular, /rhev/data-center/mnt/blockSD/*/master/tasks/*.temp >>>> Is not a log file, and lago should not collect it. >>>> >>>> Nir >>>> >>>> >> lago.utils: ERROR: Error while running thread >>>> >> Traceback (most recent call last): >>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in >>>> >> _ret_via_queue >>>> >> queue.put({'return': func()}) >>>> >> File >>>> >> "/home/jenkins/workspace/test-repo_ovirt_experimental_ master/ovirt-system-tests/basic_suite_master/test- scenarios/002_bootstrap.py", >>>> >> line 493, in log_collector >>>> >> result.code, 0, 'log collector failed. Exit code is %s' % result.code >>>> >> File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, >>>> >> in eq_ >>>> >> raise AssertionError(msg or "%r != %r" % (a, b)) >>>> >> AssertionError: log collector failed. Exit code is 2 >>>> >> >>>> >> >>>> >> * The previous issue already fixed (SDK) and now we have a new issue on >>>> >> the same area. >>>> >> >>>> >> >>>> >> [1] - >>>> >> http://jenkins.ovirt.org/view/experimental%20jobs/job/test- repo_ovirt_experimental_master/1462/testReport/(root)/ 002_bootstrap/add_secondary_storage_domains/ >>>> >> >>>> >> >>>> >> Best Regards, >>>> >> >>>> >> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL >>>> >> RHCSA | RHCE >>>> >> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci) >>>> >> >>>> >> OPEN SOURCE - 1 4 011 && 011 4 1 >>>> > >>>> > >>>> > >>>> > >>>> > -- >>>> > Eyal Edri >>>> > Associate Manager >>>> > RHV DevOps >>>> > EMEA ENG Virtualization R&D >>>> > Red Hat Israel >>>> > >>>> > phone: +972-9-7692018 >>>> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> > >>>> > _______________________________________________ >>>> > Devel mailing list >>>> > Devel(a)ovirt.org >>>> > http://lists.ovirt.org/mailman/listinfo/devel >> >> > > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

On Sun, Oct 30, 2016 at 12:40 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > Not exactly. My bad, missed that the tests run in parallel, though what this means is that 'ovirt-log-collector' can fail when there are ongoing tasks(such as adding the storage domains), I assume that is not the expected behaviour. I'll send a patch separating the test for now.