<p dir="ltr"></p>
<p dir="ltr">On Nov 20, 2016 6:33 PM, "Nir Soffer" <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
><br>
> On Sun, Nov 20, 2016 at 6:25 PM, Eyal Edri <<a href="mailto:eedri@redhat.com">eedri@redhat.com</a>> wrote:<br>
> > It happened again in [1]<br>
> ><br>
> > 2016-11-20 10:48:12,106 ERROR (jsonrpc/2) [storage.TaskManager.Task]<br>
> > (Task='6c1ec6e7-fb37-465b-8e30-1613317683b2') Unexpected error (task:870)<br>
> > Traceback (most recent call last):<br>
> > File "/usr/share/vdsm/storage/task.py", line 877, in _run<br>
> > return fn(*args, **kargs)<br>
> > File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in<br>
> > wrapper<br>
> > res = f(*args, **kwargs)<br>
> > File "/usr/share/vdsm/storage/hsm.py", line 2205, in getAllTasksInfo<br>
> > allTasksInfo = sp.getAllTasksInfo()<br>
> > File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line<br>
> > 77, in wrapper<br>
> > raise SecureError("Secured object is not in safe state")<br>
> > SecureError: Secured object is not in safe state<br>
> > 2016-11-20 10:48:12,109 INFO (jsonrpc/2) [storage.TaskManager.Task]<br>
> > (Task='6c1ec6e7-fb37-465b-8e30-1613317683b2') aborting: Task is aborted:<br>
> > u'Secured object is not in safe state' - code 100 (task:1175)<br>
> > 2016-11-20 10:48:12,110 ERROR (jsonrpc/2) [storage.Dispatcher] Secured<br>
> > object is not in safe state (dispatcher:80)<br>
> > Traceback (most recent call last):<br>
> > File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper<br>
> > result = ctask.prepare(func, *args, **kwargs)<br>
> > File "/usr/share/vdsm/storage/task.py", line 105, in wrapper<br>
> > return m(self, *a, **kw)<br>
> > File "/usr/share/vdsm/storage/task.py", line 1183, in prepare<br>
> > raise self.error<br>
> > SecureError: Secured object is not in safe state<br>
><br>
> This can also mean that the SPM is not started yet. Maybe you are not<br>
> waiting until the SPM is ready before you try to perform an operation?<br>
><br>
> Who is the owner of this test? This person should debug this test.</p>
<p dir="ltr">The relevant team for the feature. </p>
<p dir="ltr">><br>
> > <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-006_network_by_label.py/lago-basic-suite-master-host1/_var_log_vdsm/vdsm.log">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-006_network_by_label.py/lago-basic-suite-master-host1/_var_log_vdsm/vdsm.log</a><br>
> ><br>
> > The storage VM is running on the same VM as engine ( to save memory ) and<br>
> > its serving both NFS & ISCSI.<br>
> > Do you think running it on the same VM as engine might cause such issues?<br>
><br>
> I don't think so, but this prevents testing lot of interesting negative flows.</p>
<p dir="ltr">Which don't belong to CI. </p>
<p dir="ltr">><br>
> For example, when one storage server is down, the system should be<br>
> able to use the other storage domain. Having each storage server in<br>
> its own vm makes this possible.</p>
<p dir="ltr">You have both NFS and ISCSI there. It's trival to set multiple of each if needed, of course. <br>
I do wish to add more IPs and test iSCSI bonding as well as both NFSv3 and NFSv4. </p>
<p dir="ltr">><br>
> Also, we may like to test multiple storage servers of same type.<br>
> the storage servers should be decoupled so we can start any number<br>
> of them as needed for the current test.</p>
<p dir="ltr">Right, but not on this suite.<br>
Again, it's trivial to do so. The main motivation was to conserve resources so everyone could run the tests. </p>
<p dir="ltr">Y.</p>
<p dir="ltr">><br>
> > On Mon, Oct 17, 2016 at 11:45 PM, Adam Litke <<a href="mailto:alitke@redhat.com">alitke@redhat.com</a>> wrote:<br>
> >><br>
> >> On 17/10/16 11:51 +0200, Piotr Kliczewski wrote:<br>
> >>><br>
> >>> Adam,<br>
> >>><br>
> >>> I see constant failures due to this and found:<br>
> >>><br>
> >>> 2016-10-17 03:55:21,045 ERROR (jsonrpc/3) [storage.TaskManager.Task]<br>
> >>> Task=`8989d694-7099-449b-bd66-4d63786be089`::Unexpected error<br>
> >>> (task:870)<br>
> >>> Traceback (most recent call last):<br>
> >>> File "/usr/share/vdsm/storage/task.py", line 877, in _run<br>
> >>> return fn(*args, **kargs)<br>
> >>> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in<br>
> >>> wrapper<br>
> >>> res = f(*args, **kwargs)<br>
> >>> File "/usr/share/vdsm/storage/hsm.py", line 2212, in getAllTasksInfo<br>
> >>> allTasksInfo = sp.getAllTasksInfo()<br>
> >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",<br>
> >>> line 77, in wrapper<br>
> >>> raise SecureError("Secured object is not in safe state")<br>
> >>> SecureError: Secured object is not in safe state<br>
> >><br>
> >><br>
> >> This usually indicates that the SPM role has been lost which happens<br>
> >> most likely due to connection issues with the storage. What is the<br>
> >> storage environment being used for the system tests?<br>
> >><br>
> >>><br>
> >>> Please take a look not sure whether it is related. You can find latest<br>
> >>> build here [1]<br>
> >>><br>
> >>> Thanks,<br>
> >>> Piotr<br>
> >>><br>
> >>> [1] <a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/668/">http://jenkins.ovirt.org/job/ovirt_master_system-tests/668/</a><br>
> >>><br>
> >>> On Fri, Oct 14, 2016 at 11:22 AM, Evgheni Dereveanchin<br>
> >>> <<a href="mailto:ederevea@redhat.com">ederevea@redhat.com</a>> wrote:<br>
> >>>><br>
> >>>> Hello,<br>
> >>>><br>
> >>>> We've got several cases today where system tests failed<br>
> >>>> when attempting to export templates:<br>
> >>>><br>
> >>>><br>
> >>>> <a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/testReport/junit/(root)/004_basic_sanity/template_export/">http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/testReport/junit/(root)/004_basic_sanity/template_export/</a><br>
> >>>><br>
> >>>> Related engine.log looks something like this:<br>
> >>>> <a href="https://paste.fedoraproject.org/449936/47643643/raw/">https://paste.fedoraproject.org/449936/47643643/raw/</a><br>
> >>>><br>
> >>>> I could not find any obvious issues in SPM logs, could someone<br>
> >>>> please take a look to confirm what may be causing this issue?<br>
> >>>><br>
> >>>> Full logs from the test are available here:<br>
> >>>> <a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/artifact/">http://jenkins.ovirt.org/job/ovirt_master_system-tests/655/artifact/</a><br>
> >>>><br>
> >>>> Regards,<br>
> >>>> Evgheni Dereveanchin<br>
> >>>> _______________________________________________<br>
> >>>> Devel mailing list<br>
> >>>> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> >>>> <a href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a><br>
> >><br>
> >><br>
> >> --<br>
> >> Adam Litke<br>
> >><br>
> >> _______________________________________________<br>
> >> Devel mailing list<br>
> >> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> >> <a href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a><br>
> >><br>
> >><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Eyal Edri<br>
> > Associate Manager<br>
> > RHV DevOps<br>
> > EMEA ENG Virtualization R&D<br>
> > Red Hat Israel<br>
> ><br>
> > phone: +972-9-7692018<br>
> > irc: eedri (on #tlv #rhev-dev #rhev-integ)<br>
> ><br>
> > _______________________________________________<br>
> > Devel mailing list<br>
> > <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> > <a href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a><br>
> _______________________________________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a><br></p>