<div dir="ltr">It happened again in [1]<div><ol style="color:rgb(51,51,51);font-family:monospace;font-size:11px"><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">2016-11-20 10:48:12,106 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='6c1ec6e7-fb37-465b-8e30-1613317683b2') Unexpected error (task:870)</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">Traceback (most recent call last):</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/share/vdsm/storage/task.py", line 877, in _run</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> return fn(*args, **kargs)</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> res = f(*args, **kwargs)</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/share/vdsm/storage/hsm.py", line 2205, in getAllTasksInfo</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> allTasksInfo = sp.getAllTasksInfo()</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> raise SecureError("Secured object is not in safe state")</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">SecureError: Secured object is not in safe state</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">2016-11-20 10:48:12,109 INFO (jsonrpc/2) [storage.TaskManager.Task] (Task='6c1ec6e7-fb37-465b-8e30-1613317683b2') aborting: Task is aborted: u'Secured object is not in safe state' - code 100 (task:1175)</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">2016-11-20 10:48:12,110 ERROR (jsonrpc/2) [storage.Dispatcher] Secured object is not in safe state (dispatcher:80)</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">Traceback (most recent call last):</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> result = ctask.prepare(func, *args, **kwargs)</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/share/vdsm/storage/task.py", line 105, in wrapper</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> return m(self, *a, **kw)</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> File "/usr/share/vdsm/storage/task.py", line 1183, in prepare</div></li><li class="gmail-li2" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:rgb(248,248,248)"><div class="gmail-de2" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top"> raise self.error</div></li><li class="gmail-li1" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div class="gmail-de1" style="font-stretch:normal;font-size:1em;line-height:1.2em;margin:0px;padding:0px;background-image:none;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;background-color:initial;vertical-align:top">SecureError: Secured object is not in safe state</div></li></ol><div><font color="#333333" face="monospace"><span style="font-size:11px"><br></span></font></div></div><div><font color="#333333" face="monospace"><span style="font-size:11px"><a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-006_network_by_label.py/lago-basic-suite-master-host1/_var_log_vdsm/vdsm.log">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-006_network_by_label.py/lago-basic-suite-master-host1/_var_log_vdsm/vdsm.log</a></span><br></font></div><div><font color="#333333" face="monospace"><span style="font-size:11px"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:11px">The storage VM is running on the same VM as engine ( to save memory ) and its serving both NFS & ISCSI. </span></font></div><div><font color="#333333" face="monospace"><span style="font-size:11px">Do you think running it on the same VM as engine might cause such issues? </span></font></div><div><font color="#333333" face="monospace"><span style="font-size:11px"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:11px"><br></span></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 17, 2016 at 11:45 PM, Adam Litke <span dir="ltr"><<a href="mailto:alitke@redhat.com" target="_blank">alitke@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 17/10/16 11:51 +0200, Piotr Kliczewski wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Adam,<br>
<br>
I see constant failures due to this and found:<br>
<br>
2016-10-17 03:55:21,045 ERROR (jsonrpc/3) [storage.TaskManager.Task]<br>
Task=`8989d694-7099-449b-bd66-<wbr>4d63786be089`::Unexpected error<br>
(task:870)<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/storage/task.<wbr>py", line 877, in _run<br>
return fn(*args, **kargs)<br>
File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/logUtils.py", line 50, in wrapper<br>
res = f(*args, **kwargs)<br>
File "/usr/share/vdsm/storage/hsm.p<wbr>y", line 2212, in getAllTasksInfo<br>
allTasksInfo = sp.getAllTasksInfo()<br>
File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/securable.py"<wbr>,<br>
line 77, in wrapper<br>
raise SecureError("Secured object is not in safe state")<br>
SecureError: Secured object is not in safe state<br>
</blockquote>
<br></span>
This usually indicates that the SPM role has been lost which happens<br>
most likely due to connection issues with the storage. What is the<br>
storage environment being used for the system tests?<span class="im HOEnZb"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Please take a look not sure whether it is related. You can find latest<br>
build here [1]<br>
<br>
Thanks,<br>
Piotr<br>
<br>
[1] <a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/668/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt_master_system-tests/668/</a><br>
<br>
On Fri, Oct 14, 2016 at 11:22 AM, Evgheni Dereveanchin<br>
<<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
We've got several cases today where system tests failed<br>
when attempting to export templates:<br>
<br>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/testReport/junit/(root)/004_basic_sanity/template_export/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt_master_system-tests/655/t<wbr>estReport/junit/(root)/004_bas<wbr>ic_sanity/template_export/</a><br>
<br>
Related engine.log looks something like this:<br>
<a href="https://paste.fedoraproject.org/449936/47643643/raw/" rel="noreferrer" target="_blank">https://paste.fedoraproject.or<wbr>g/449936/47643643/raw/</a><br>
<br>
I could not find any obvious issues in SPM logs, could someone<br>
please take a look to confirm what may be causing this issue?<br>
<br>
Full logs from the test are available here:<br>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/artifact/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt_master_system-tests/655/a<wbr>rtifact/</a><br>
<br>
Regards,<br>
Evgheni Dereveanchin<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
</blockquote></blockquote>
<br></span><span class="HOEnZb"><font color="#888888">
-- <br>
Adam Litke</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>