<div dir="ltr">Thank you for the fast reply and help. <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 8:21 PM, Dominik Holler <span dir="ltr">&lt;<a href="mailto:dholler@redhat.com" target="_blank">dholler@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, 15 Mar 2018 16:24:10 +0000<br>
Dafna Ron &lt;<a href="mailto:dron@redhat.com">dron@redhat.com</a>&gt; wrote:<br>
<br>
&gt; Hi,<br>
&gt;<br>
&gt; We have a failure on master for test<br>
&gt; 098_ovirt_provider_ovn.use_<wbr>ovn_provider in project cockpit-ovirt.<br>
&gt; This seems to be a race because object is locked. also, the actual<br>
&gt; failure is logged as WARN and not ERROR.<br>
&gt;<br>
&gt; I don&#39;t think the patch is actually related to the failure but I<br>
&gt; think the test should be fixed.<br>
&gt; can you please review to make sure we do not have an actual<br>
&gt; regression and let me know if we need to open a bz to fix the test?<br>
&gt;<br>
&gt;<br>
</span>&gt; *Link and headline of suspected patches: *<br>
&gt; *<a href="https://gerrit.ovirt.org/#/c/89020/2" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>89020/2</a><br>
&gt; &lt;<a href="https://gerrit.ovirt.org/#/c/89020/2" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>89020/2</a>&gt; - *<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; *wizard: Enable scroll on start page for low-res screensLink to<br>
<span class="">&gt; Job:<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/<wbr>job/ovirt-master_change-queue-<wbr>tester/6374</a><br>
&gt; &lt;<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/6374</a>&gt;Link<br>
&gt; to all<br>
&gt; logs:<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374/artifacts" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/<wbr>job/ovirt-master_change-queue-<wbr>tester/6374/artifacts</a><br>
&gt; &lt;<a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6374/artifacts" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-master_change-queue-<wbr>tester/6374/artifacts</a>&gt;(<wbr>Relevant)<br>
&gt; error snippet from the log: &lt;error&gt;2018-03-15 10:05:00,160-04 DEBUG<br>
&gt; [org.ovirt.engine.core.sso.<wbr>servlets.<wbr>OAuthTokenInfoServlet] (default<br>
</span>&gt; task-10) [] Sending json response2018-03-15 10:05:00,160-04 DEBUG<br>
&gt; [org.ovirt.engine.core.sso.<wbr>utils.TokenCleanupUtility] (default<br>
&gt; task-10) [] Not cleaning up expired tokens2018-03-15 10:05:00,169-04<br>
<span class="">&gt; INFO<br>
&gt; [org.ovirt.engine.core.bll.<wbr>provider.network.<wbr>SyncNetworkProviderCommand]<br>
&gt; (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-90) [789edb23] Lock<br>
&gt; Acquired to object<br>
</span>&gt; &#39;EngineLock:{exclusiveLocks=&#39;[<wbr>c38a67ec-0b48-4e6f-be85-<wbr>70c700df5483=PROVIDER]&#39;,<br>
&gt; sharedLocks=&#39;&#39;}&#39;2018-03-15 10:05:00,184-04 INFO<br>
<span class="">&gt; [org.ovirt.engine.core.bll.<wbr>provider.network.<wbr>SyncNetworkProviderCommand]<br>
&gt; (EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-90) [789edb23]<br>
&gt; Running command: SyncNetworkProviderCommand internal: true.2018-03-15<br>
&gt; 10:05:00,228-04 DEBUG<br>
&gt; [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall]<br>
&gt; (default task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] Compiled<br>
&gt; stored procedure. Call string is [{call<br>
</span>&gt; getdcidbyexternalnetworkid(?)}<wbr>]2018-03-15 10:05:00,228-04 DEBUG<br>
<span class="">&gt; [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall]<br>
&gt; (default task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] SqlCall for<br>
&gt; procedure [GetDcIdByExternalNetworkId] compiled2018-03-15<br>
&gt; 10:05:00,229-04 DEBUG<br>
&gt; [org.ovirt.engine.core.common.<wbr>di.interceptor.<wbr>DebugLoggingInterceptor]<br>
&gt; (default task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] method:<br>
&gt; runQuery, params: [<wbr>GetAllExternalNetworksOnProvid<wbr>er,<br>
&gt; IdQueryParameters:{refresh=&#39;<wbr>false&#39;, filtered=&#39;false&#39;}], timeElapsed:<br>
</span>&gt; 353ms2018-03-15 10:05:00,239-04 INFO<br>
<span class="">&gt; [org.ovirt.engine.core.bll.<wbr>network.dc.AddNetworkCommand] (default<br>
&gt; task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] Failed to Acquire<br>
&gt; Lock to object &#39;EngineLock:{exclusiveLocks=&#39;[<wbr>network_1=NETWORK,<br>
&gt; c38a67ec-0b48-4e6f-be85-<wbr>70c700df5483=PROVIDER]&#39;,<br>
&gt; sharedLocks=&#39;&#39;}&#39;2018-03-15 10:05:00,239-04 WARN<br>
&gt; [org.ovirt.engine.core.bll.<wbr>network.dc.AddNetworkCommand] (default<br>
&gt; task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] Validation of action<br>
&gt; &#39;AddNetwork&#39; failed for user admin@internal-authz. Reasons:<br>
&gt; VAR__TYPE__NETWORK,VAR__<wbr>ACTION__ADD,ACTION_TYPE_<wbr>FAILED_PROVIDER_LOCKED,$<wbr>providerId<br>
</span>&gt; c38a67ec-0b48-4e6f-be85-<wbr>70c700df54832018-03-15 10:05:00,240-04 DEBUG<br>
<span class="">&gt; [org.ovirt.engine.core.common.<wbr>di.interceptor.<wbr>DebugLoggingInterceptor]<br>
&gt; (default task-13) [e1328379-17b7-49f8-beb2-<wbr>cf8331784828] method:<br>
&gt; runAction, params: [AddNetwork,<br>
&gt; AddNetworkStoragePoolParameter<wbr>s:{commandId=&#39;61b365ec-27c1-<wbr>49af-ad72-f907df8befcd&#39;,<br>
&gt; user=&#39;null&#39;, commandType=&#39;Unknown&#39;}], timeElapsed: 10ms2018-03-15<br>
&gt; 10:05:00,250-04 ERROR<br>
&gt; [org.ovirt.engine.api.restapi.<wbr>resource.<wbr>AbstractBackendResource]<br>
&gt; (default task-13) [] Operation Failed: [Cannot add Network. Related<br>
&gt; operation on provider with the id<br>
&gt; c38a67ec-0b48-4e6f-be85-<wbr>70c700df5483 is currently in progress. Please<br>
&gt; try again later.]2018-03-15 10:05:00,254-04 DEBUG<br>
&gt; [org.ovirt.engine.core.utils.<wbr>servlet.LocaleFilter] (default task-14)<br>
&gt; [] Incoming locale &#39;en-US&#39;. Filter determined locale to be<br>
&gt; &#39;en-US&#39;2018-03-15 10:05:00,254-04 DEBUG<br>
&gt; [org.ovirt.engine.core.sso.<wbr>servlets.OAuthTokenServlet] (default<br>
&gt; task-14) [] Entered OAuthTokenServlet Query String: null,<br>
&gt; Parameters : password = ***, grant_type = password, scope =<br>
&gt; ovirt-app-api ovirt-ext=token-info:validate, username =<br>
</span>&gt; admin@internal, &lt;/error&gt;*<br>
<br>
I will care about this.<br>
The problem is that SyncNetworkProviderCommand is running in the<br>
background and locking the provider, which blocks the lock for the<br>
tested AddNetworkCommand.<br>
The related changes are<br>
core: Add locking for Add and RemoveNetworkCommand<br>
<a href="https://gerrit.ovirt.org/#/c/85480/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>85480/</a><br>
and<br>
core: Add SyncNetworkProviderCommand<br>
<a href="https://gerrit.ovirt.org/#/c/85134/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>85134/</a><br>
<br>
<br>
</blockquote></div><br></div>