On Thu, Feb 23, 2017 at 9:37 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > Test failed: [ add_secondary_storage_domains ] > > Note: > - This may or may not be related to > https://bugzilla.redhat.com/show_bug.cgi?id=1421945 > The BZ talks about sporadic failures, while this seems to be > happening consistently (for 6 runs so far) > > Link to suspected patches: > - https://gerrit.ovirt.org/70415 > - https://gerrit.ovirt.org/69157 Why do you suspect these patches?

Did you try to run the tests with the latest patches before these patches?

On Thu, Feb 23, 2017 at 4:38 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > On 23 February 2017 at 16:35, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> On Thu, Feb 23, 2017 at 9:37 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: >>> Test failed: [ add_secondary_storage_domains ] >>> >>> Note: >>> - This may or may not be related to >>> https://bugzilla.redhat.com/show_bug.cgi?id=1421945 >>> The BZ talks about sporadic failures, while this seems to be >>> happening consistently (for 6 runs so far) >>> >>> Link to suspected patches: >>> - https://gerrit.ovirt.org/70415 >>> - https://gerrit.ovirt.org/69157 >> >> Why do you suspect these patches? > > Because the test right before them passed. > These are all the changes that caused the failing OST job to run. > >> Did you try to run the tests with the latest patches before these patches? > > Yes, the test before them pass. Your are correct, these patches are broken: 2017-02-22 16:13:00,745-0500 ERROR (jsonrpc/1) [storage.TaskManager.Task] (Task='4f670db2-70c2-4c21-96ff-114f57de70c0') Unexpected error (task:871) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 878, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 989, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1051, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 672, in connect self.__createMailboxMonitor() File "/usr/share/vdsm/storage/sp.py", line 485, in __createMailboxMonitor outbox = self._master_volume_path("inbox") File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 77, in wrapper raise SecureError("Secured object is not in safe state") SecureError: Secured object is not in safe state I'm sending a fix. Nir > > > -- > Barak Korren > bkorren(a)redhat.com > RHCE, RHCi, RHV-DevOps Team > https://ifireball.wordpress.com/ _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

It's very confusing that this error is sometimes harmless and sometimes isn't - how did you identify it as problematic? Y.

On Thu, Feb 23, 2017 at 4:51 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > On Thu, Feb 23, 2017 at 4:43 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Thu, Feb 23, 2017 at 4:38 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: >> > On 23 February 2017 at 16:35, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Thu, Feb 23, 2017 at 9:37 AM, Barak Korren <bkorren(a)redhat.com&gt; >> >> wrote: >> >>> Test failed: [ add_secondary_storage_domains ] >> >>> >> >>> Note: >> >>> - This may or may not be related to >> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1421945 >> >>> The BZ talks about sporadic failures, while this seems to be >> >>> happening consistently (for 6 runs so far) >> >>> >> >>> Link to suspected patches: >> >>> - https://gerrit.ovirt.org/70415 >> >>> - https://gerrit.ovirt.org/69157 >> >> >> >> Why do you suspect these patches? >> > >> > Because the test right before them passed. >> > These are all the changes that caused the failing OST job to run. >> > >> >> Did you try to run the tests with the latest patches before these >> >> patches? >> > >> > Yes, the test before them pass. >> >> Your are correct, these patches are broken: >> >> 2017-02-22 16:13:00,745-0500 ERROR (jsonrpc/1) >> [storage.TaskManager.Task] >> (Task='4f670db2-70c2-4c21-96ff-114f57de70c0') Unexpected error >> (task:871) >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >> 878, in _run >> return fn(*args, **kargs) >> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in >> wrapper >> res = f(*args, **kwargs) >> File "/usr/share/vdsm/storage/hsm.py", line 989, in connectStoragePool >> spUUID, hostID, msdUUID, masterVersion, domainsMap) >> File "/usr/share/vdsm/storage/hsm.py", line 1051, in _connectStoragePool >> res = pool.connect(hostID, msdUUID, masterVersion) >> File "/usr/share/vdsm/storage/sp.py", line 672, in connect >> self.__createMailboxMonitor() >> File "/usr/share/vdsm/storage/sp.py", line 485, in >> __createMailboxMonitor >> outbox = self._master_volume_path("inbox") >> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", >> line 77, in wrapper >> raise SecureError("Secured object is not in safe state") >> SecureError: Secured object is not in safe state > > > It's very confusing that this error is sometimes harmless and sometimes > isn't - how did you identify it as problematic? It depends on the context. Here we called __createMailboxMontior, which is something we call when creating an instance, and is marked as @unsecured. This call is calling now a new helper introduced in 7cf19dafd7cd, but the helper was not marked as @unsecured. This will raise UnsecureError, which will fail the current flow. We have another instance of this during upgrade domain flow - I think we have the same issue there, but this needs investigation. Other errors means that a real secured method is called when a host is not hte spm. This may be bad client code, or unavoidable, since there is no race-free way to check that a host is the spm before calling a method on the spm. > Y. > >> >> >> I'm sending a fix. >> >> Nir >> >> > >> > >> > -- >> > Barak Korren >> > bkorren(a)redhat.com >> > RHCE, RHCi, RHV-DevOps Team >> > https://ifireball.wordpress.com/ >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel

Great! This OST experimental run is verifying that: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5503/ Hope it doesn't fail on something else... On 23 February 2017 at 17:38, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > Fixed in > > commit 726e946257174926ea2591a1c4a3be2dae4297ea > Author: Nir Soffer <nsoffer(a)redhat.com&gt; > Date: Thu Feb 23 16:45:45 2017 +0200 > > sp: Mark helper method as @unsecured > > In commit 7cf19dafd7cd (storage_mailbox: make inbox/outbox mailbox > args), we added a helper that is used before the spm is started, but the > helper was not marked as @unsecure. This cause the call to fail with: > > File "/usr/share/vdsm/storage/sp.py", line 485, in > __createMailboxMonitor > outbox = self._master_volume_path("inbox") > File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", > line 77, in wrapper > raise SecureError("Secured object is not in safe state") > SecureError: Secured object is not in safe state > > As this helper doesn't change the state of the storage pool, there is no > reason to treat it as a secured method, which is the default for this > class. > > Change-Id: Icf92b9474c9000840a5c15e3b91f2ced4d02aca2 > Signed-off-by: Nir Soffer <nsoffer(a)redhat.com&gt; > > Verified with http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tes... > > Thanks for reporting this. > > Nir > > > On Thu, Feb 23, 2017 at 5:03 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> On Thu, Feb 23, 2017 at 4:51 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>> >>> >>> On Thu, Feb 23, 2017 at 4:43 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>> >>>> On Thu, Feb 23, 2017 at 4:38 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: >>>> > On 23 February 2017 at 16:35, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>> >> On Thu, Feb 23, 2017 at 9:37 AM, Barak Korren <bkorren(a)redhat.com&gt; >>>> >> wrote: >>>> >>> Test failed: [ add_secondary_storage_domains ] >>>> >>> >>>> >>> Note: >>>> >>> - This may or may not be related to >>>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1421945 >>>> >>> The BZ talks about sporadic failures, while this seems to be >>>> >>> happening consistently (for 6 runs so far) >>>> >>> >>>> >>> Link to suspected patches: >>>> >>> - https://gerrit.ovirt.org/70415 >>>> >>> - https://gerrit.ovirt.org/69157 >>>> >> >>>> >> Why do you suspect these patches? >>>> > >>>> > Because the test right before them passed. >>>> > These are all the changes that caused the failing OST job to run. >>>> > >>>> >> Did you try to run the tests with the latest patches before these >>>> >> patches? >>>> > >>>> > Yes, the test before them pass. >>>> >>>> Your are correct, these patches are broken: >>>> >>>> 2017-02-22 16:13:00,745-0500 ERROR (jsonrpc/1) >>>> [storage.TaskManager.Task] >>>> (Task='4f670db2-70c2-4c21-96ff-114f57de70c0') Unexpected error >>>> (task:871) >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >>>> 878, in _run >>>> return fn(*args, **kargs) >>>> File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 52, in >>>> wrapper >>>> res = f(*args, **kwargs) >>>> File "/usr/share/vdsm/storage/hsm.py", line 989, in connectStoragePool >>>> spUUID, hostID, msdUUID, masterVersion, domainsMap) >>>> File "/usr/share/vdsm/storage/hsm.py", line 1051, in _connectStoragePool >>>> res = pool.connect(hostID, msdUUID, masterVersion) >>>> File "/usr/share/vdsm/storage/sp.py", line 672, in connect >>>> self.__createMailboxMonitor() >>>> File "/usr/share/vdsm/storage/sp.py", line 485, in >>>> __createMailboxMonitor >>>> outbox = self._master_volume_path("inbox") >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", >>>> line 77, in wrapper >>>> raise SecureError("Secured object is not in safe state") >>>> SecureError: Secured object is not in safe state >>> >>> >>> It's very confusing that this error is sometimes harmless and sometimes >>> isn't - how did you identify it as problematic? >> >> It depends on the context. >> >> Here we called __createMailboxMontior, which is something we call >> when creating an instance, and is marked as @unsecured. >> >> This call is calling now a new helper introduced in 7cf19dafd7cd, >> but the helper was not marked as @unsecured. This will raise >> UnsecureError, which will fail the current flow. >> >> We have another instance of this during upgrade domain flow - I think >> we have the same issue there, but this needs investigation. >> >> Other errors means that a real secured method is called when a host >> is not hte spm. This may be bad client code, or unavoidable, since >> there is no race-free way to check that a host is the spm before >> calling a method on the spm. >> >>> Y. >>> >>>> >>>> >>>> I'm sending a fix. >>>> >>>> Nir >>>> >>>> > >>>> > >>>> > -- >>>> > Barak Korren >>>> > bkorren(a)redhat.com >>>> > RHCE, RHCi, RHV-DevOps Team >>>> > https://ifireball.wordpress.com/ >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel -- Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/

On 23 February 2017 at 18:11, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > I see this error there now: > > 00:02:59.460 [upgrade-from-release_suit_el7] + yum install > --nogpgcheck -y --downloaddir=/dev/shm ntp ovirt-engine > ovirt-log-collector 'ovirt-engine-extension-aaa-ldap*' > 00:02:59.460 [upgrade-from-release_suit_el7] > 00:02:59.461 [upgrade-from-release_suit_el7] > 00:02:59.461 [upgrade-from-release_suit_el7] One of the configured Yeah looks like a temp issue with CentOS repos. We should probably switch back to blocking outside repo access from Lago VMs. Next run passed finally: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5504/