[ovirt-devel] [ OST Failure Report ] [ ovirt master ] [ 14.02.17 ] [storage]

[Nir Soffer]

On Tue, Feb 14, 2017 at 9:16 PM, Dusan Fodor <dfodor at redhat.com> wrote:
> Hi,
>
> storage task manager returns following error (vdsm.log):
>
> 2017-02-14 06:58:27,975-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task]
> (Task='1bc5b2ca-a44e-4794-81f9-d2415f27ad48') Unexpected error (task:871)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in
> _run
>     return fn(*args, **kargs)
>   File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in
> wrapper
>     res = f(*args, **kwargs)
>   File "/usr/share/vdsm/storage/hsm.py", line 2215, in getAllTasksInfo
>     allTasksInfo = self._pool.getAllTasksInfo()
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line
> 77, in wrapper
>     raise SecureError("Secured object is not in safe state")
> SecureError: Secured object is not in safe state
> 2017-02-14 06:58:27,977-0500 INFO  (jsonrpc/1) [storage.TaskManager.Task]
> (Task='1bc5b2ca-a44e-4794-81f9-d2415f27ad48') aborting: Task is aborted:
> u'Secured object is not in safe state' - code 100 (task:1176)
> 2017-02-14 06:58:27,978-0500 ERROR (jsonrpc/1) [storage.Dispatcher] Secured
> object is not in safe state (dispatcher:80)
> Traceback (most recent call last):
>   File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
>     result = ctask.prepare(func, *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 106, in
> wrapper
>     return m(self, *a, **kw)
>   File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1184,
> in prepare
>     raise self.error
> SecureError: Secured object is not in safe state
>
> It doesn't make OST fail, but i believe it is not desired.
> Sadly, I'm not able to find patch that might be related, since all builds in
> history have this error.
> Link to build:
> http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_master/5301
> Link to vdsm.log:
> http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_master/5301/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host0/_var_log/vdsm/vdsm.log

Thanks for reporting this,

This log is too noisy, but it may reveal real error in the caller, calling
spm methods on a host which is not the spm.

However, I'm not sure it is possible to avoid this situation. This should
probably be a one line warning instead of this noisy exception.

But the real issue is the caller of getAllTasksInfo - why is it calling
this on a host which is not the SPM? this is probably engine bug, or
maybe code using engine REST api incorrectly - maybe the test
framework?

We plan to clean the logs during 4.1.z.

Nir