Hi all, bumping this thread due to an almost identical failure[1]:
ovirt-log-collector/ovirt-log-collector-20161030053238.log:2016-10-30
05:33:09::ERROR::__main__::791::root:: Failed to collect logs from:
192.168.200.4; /bin/ls:
/rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5-5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7-52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.job.0:
No such file or directory
ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls:
cannot access
/rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5-5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7-52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.recover.1:
No such file or directory
ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls:
cannot access
/rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5-5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7-52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.task:
No such file or directory
ovirt-log-collector/ovirt-log-collector-20161030053238.log-/bin/ls:
cannot access
/rhev/data-center/mnt/blockSD/63c4fdd3-5d0f-4d16-b1e5-5f43caa4cf82/master/tasks/6b3b6aa1-808c-42df-9db7-52349f8533f2/6b3b6aa1-808c-42df-9db7-52349f8533f2.recover.0:
No such file or directory
To ensure I've checked lago/OST, and couldn't find any stage where
there is a reference to '/rhv' nor any manipulation to
ovirt-log-collector, only customizations made is a
'ovirt-log-collector.conf' with user/password. The code that pulls the
logs in OST[2] runs the following command on the engine VM(and there
it fails):
ovirt-log-collector --conf /rot/ovirt-log-collector.conf
The failure comes right after 'add_secondary_storage_domains'[3] test,
which all of its steps ran successfully.
Can anyone look into this?
Thanks,
Nadav.
[1]
On Tue, Sep 20, 2016 at 9:45 AM, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
On Fri, Sep 9, 2016 at 1:19 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
> Indeed, this is the log collector. I wonder if we collect its logs...
> Y.
This can't be log-collector, it can be sos vdsm plugin.
That said, if we run log-collector within lago we should collect the results as job
artifacts.
>
>
>
> On Thu, Sep 8, 2016 at 6:54 PM, Eyal Edri <eedri(a)redhat.com> wrote:
>>
>> I'm pretty sure lago or ovirt system tests aren't doing it but its the
log collector which is running during that test, I'm not near a computer so can't
verify it yet.
>>
>>
>> On Sep 8, 2016 6:05 PM, "Nir Soffer" <nsoffer(a)redhat.com> wrote:
>>>
>>> On Thu, Sep 8, 2016 at 5:45 PM, Eyal Edri <eedri(a)redhat.com> wrote:
>>> > Adding devel.
>>> >
>>> > On Thu, Sep 8, 2016 at 5:43 PM, Shlomo Ben David
<sbendavi(a)redhat.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Job [1] is failing with the following error:
>>> >>
>>> >> lago.ssh: DEBUG: Command 8de75538 on lago_basic_suite_master_engine
>>> >> errors:
>>> >> ERROR: Failed to collect logs from: 192.168.200.2; /bin/ls:
>>> >>
/rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d-9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp:
>>> >> No such file or directory
>>> >> /bin/ls: cannot open directory
>>> >>
/rhev/data-center/mnt/blockSD/eb8c9f48-5f23-48dc-ab7d-9451890fd422/master/tasks/1350bed7-443e-4ae6-ae1f-9b24d18c70a8.temp:
>>> >> No such file or directory
>>>
>>> This looks like a lago issue - it should never read anything inside /rhev
>>>
>>> This is a private directory for vdsm, no other process should ever depend
>>> on the content inside this directory, or even on the fact that it exists.
>>>
>>> In particular, /rhev/data-center/mnt/blockSD/*/master/tasks/*.temp
>>> Is not a log file, and lago should not collect it.
>>>
>>> Nir
>>>
>>> >> lago.utils: ERROR: Error while running thread
>>> >> Traceback (most recent call last):
>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py",
line 53, in
>>> >> _ret_via_queue
>>> >> queue.put({'return': func()})
>>> >> File
>>> >>
"/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic_suite_master/test-scenarios/002_bootstrap.py",
>>> >> line 493, in log_collector
>>> >> result.code, 0, 'log collector failed. Exit code is %s'
% result.code
>>> >> File
"/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29,
>>> >> in eq_
>>> >> raise AssertionError(msg or "%r != %r" % (a, b))
>>> >> AssertionError: log collector failed. Exit code is 2
>>> >>
>>> >>
>>> >> * The previous issue already fixed (SDK) and now we have a new issue
on
>>> >> the same area.
>>> >>
>>> >>
>>> >> [1] -
>>> >>
http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_exp...
>>> >>
>>> >>
>>> >> Best Regards,
>>> >>
>>> >> Shlomi Ben-David | DevOps Engineer | Red Hat ISRAEL
>>> >> RHCSA | RHCE
>>> >> IRC: shlomibendavid (on #rhev-integ, #rhev-dev, #rhev-ci)
>>> >>
>>> >> OPEN SOURCE - 1 4 011 && 011 4 1
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Eyal Edri
>>> > Associate Manager
>>> > RHV DevOps
>>> > EMEA ENG Virtualization R&D
>>> > Red Hat Israel
>>> >
>>> > phone: +972-9-7692018
>>> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>> >
>>> > _______________________________________________
>>> > Devel mailing list
>>> > Devel(a)ovirt.org
>>> >
http://lists.ovirt.org/mailman/listinfo/devel
>
>
--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at
redhat.com
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel