November 2017 - Devel - oVirt List Archives

Default MAC address range
by Yedidyah Bar David 22 Nov '17

22 Nov '17

Hi all, The default range went through several changes during the lifetime of the project - from being small and constant until 3.2, to small and random in 3.3 until 3.5, back to fixed and small in 3.6, slightly enlarged (to 1024 addresses) in current master. In all cases, the address range was taken from the IANA-assigned range 001A4A of Qumranet (now Red Hat) [1]. I suggest to: 1. Default to a random range again - So that two or (a few) more setups using the same network segment are less likely to use the same range 2. One in the locally-administered address range [2] - So that we have a larger address range 3. Make it a bit larger, say 65000 addresses - So that admins do not have to handle this manually, even for large setups, unless they have specific needs If there is objection to (2.), we can still have the others - among the 24 bits in the 001A4A prefix range, choose among the first 8 a random value during engine-setup, and have the pool of size 16 bits. Comments/ideas/etc. are welcome. Best regards, [1] https://regauth.standards.ieee.org/standards-ra-web/pub/view.html [2] https://en.wikipedia.org/wiki/MAC_address#Universal_vs._local -- Didi

2 2

[ OST Failure Report ] [ oVirt 4.2 ] [ 22-11-2017 ] [ 002_bootstrap.add_dc ]
by Dafna Ron 22 Nov '17

22 Nov '17

This is a multi-part message in MIME format. --------------4E0BDA3C10A37D37E4CBA152 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I am not sure we should be alerting on 4.2 versions yet but since we are getting several cq failures for the same reasons and test, I decided this should be communicated. The failed test is 002_bootstrap.add_dc in OST 4.2. and all are pointing to one patch. Remove mandatory option for cluster in register operation - https://gerrit.ovirt.org/#/c/84359/* * can you please have a look? I was unable to determine from what log the error was coming from as well but looking at the failed test and the patch reported there seem to be a connection. ** *Link and headline of suspected patches: * ** Patch that is marked as root cause: Remove mandatory option for cluster in register operation - https://gerrit.ovirt.org/#/c/84359/* * Patches reported as failed: 1. Add discoveredTargets property to HostService - https://gerrit.ovirt.org/#/c/84489/ *2. Document the fact that sparse/raw isn't supported on block - https://gerrit.ovirt.org/#/c/84490/* 3. Update CHANGES.adoc for release 4.2.25 - https://gerrit.ovirt.org/#/c/84499/* * 4. [maven-release-plugin] prepare release 4.2.26 - https://gerrit.ovirt.org/#/c/84500/* * ** * Link to Job: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/* * * *Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/* * * *(Relevant) error snippet from the log: * * <error> * Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/002_bootstrap.py", line 109, in add_dc api = prefix.virt_env.engine_vm().get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 329, in get_api return self.get_api_v4() File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 338, in get_api_v4 self._api_v4 = self._get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 316, in _get_api while not testapi.test(): File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 744, in test six.reraise(*sys.exc_info()) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 741, in test self.system_service().get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 21865, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get context = self._connection.send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 370, in send return self.__send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 388, in __send self.authenticate() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 381, in authenticate self._sso_token = self._get_access_token() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 617, in _get_access_token sso_response = self._get_sso_response(self._sso_url, post_data) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 702, in _get_sso_response self._check_content_type(self.__JSON_CONTENT_TYPE_RE, 'JSON', header_lines) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 887, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=UTF-8' isn't the expected JSON ** *</error>* ** --------------4E0BDA3C10A37D37E4CBA152 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, I am not sure we should be alerting on 4.2 versions yet but since we are getting several cq failures for the same reasons and test, I decided this should be communicated. The failed test is 002_bootstrap.add_dc in OST 4.2. and all are pointing to one patch. Remove mandatory option for cluster in register operation - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84359/">https://gerrit.ovirt.org/#/c/84359/</a> can you please have a look? I was unable to determine from what log the error was coming from as well but looking at the failed test and the patch reported there seem to be a connection. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link and headline of suspected patches: Patch that is marked as root cause: Remove mandatory option for cluster in register operation - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84359/">https://gerrit.ovirt.org/#/c/84359/</a> Patches reported as failed: 1. Add discoveredTargets property to HostService - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84489/">https://gerrit.ovirt.org/#/c/84489/</a> 2. Document the fact that sparse/raw isn't supported on block - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84490/">https://gerrit.ovirt.org/#/c/84490/</a> 3. Update CHANGES.adoc for release 4.2.25 - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84499/">https://gerrit.ovirt.org/#/c/84499/</a> 4. [maven-release-plugin] prepare release 4.2.26 - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84500/">https://gerrit.ovirt.org/#/c/84500/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/002_bootstrap.py", line 109, in add_dc api = prefix.virt_env.engine_vm().get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 329, in get_api return self.get_api_v4() File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 338, in get_api_v4 self._api_v4 = self._get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 316, in _get_api while not testapi.test(): File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 744, in test six.reraise(*sys.exc_info()) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 741, in test self.system_service().get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 21865, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get context = self._connection.send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 370, in send return self.__send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 388, in __send self.authenticate() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 381, in authenticate self._sso_token = self._get_access_token() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 617, in _get_access_token sso_response = self._get_sso_response(self._sso_url, post_data) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 702, in _get_sso_response self._check_content_type(self.__JSON_CONTENT_TYPE_RE, 'JSON', header_lines) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 887, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=UTF-8' isn't the expected JSON</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------4E0BDA3C10A37D37E4CBA152--

3 2

Re: [ovirt-devel] Question about availability of image-io in ovirt master repo
by Nir Soffer 21 Nov '17

21 Nov '17

On Tue, Nov 21, 2017 at 9:38 PM Idan Shaby <ishaby(a)redhat.com> wrote: > So that what it gives me: > - el6 - nothing. > - el7: > ovirt-imageio-common-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > - fc27 - nothing. > - fc26: > ovirt-imageio-common-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > - fc25: > ovirt-imageio-common-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > fc24 - nothing. > > So I guess it means we don't deliver imageio1.2.0 on el6, fc27 and fc24. > fc27 should be a problem, right? > I hope that tomorrow we will have also fc27 and fcraw, but these are not supported platforms yet. Looks like we can require 1.2 in vdsm. > > Regards, > Idan > > ---------- Forwarded message ---------- > From: Barak Korren <bkorren(a)redhat.com> > Date: Tue, Nov 21, 2017 at 9:27 PM > Subject: Re: Question about availability of image-io in ovirt master repo > To: Idan Shaby <ishaby(a)redhat.com> > Cc: infra <infra(a)ovirt.org> > > > On 21 November 2017 at 21:04, Idan Shaby <ishaby(a)redhat.com> wrote: > > Hi, > > > > How do I check if our ovirt master repos contain version 1.2.0 of > > ovirt-imageio-common and ovirt-imageio-daemon? > > I want to add these requirements to vdsm and I am not sure how to verify > it. > > This searches for a certain package in a certain repo: > > repoquery --repofrompath=r, > http://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7 > --repoid=r > <http://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7%0D--repoid=r> > list '*imageio*' > Adding devel, may be useful for people adding dependencies. > > You can change the url slightly to check for other distros and oVirt > versions. > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > >

1 0

[ OST Failure Report ] [ oVirt master ] [ 20-11-2017 ] [004_basic_sanity.vm_run ]
by Dafna Ron 20 Nov '17

20 Nov '17

This is a multi-part message in MIME format. --------------2EEBD942435F80A2268AD3F9 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in OST on test 004_basic_sanity.vm_run. it seems to be an error in vm type which is related to the patch reported. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/84343/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact (Relevant) error snippet from the log: <error> vdsm log: * *2017-11-20 07:40:12,779-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1341, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 508, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1664, in getStats stats.update(self._getConfigVmStats()) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1703, in _getConfigVmStats 'vmType': self.conf['vmType'], KeyError: 'vmType' * * * engine log: ***2017-11-20 07:43:07,675-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "5bf12e5a-4a09-4999-a6ce-a7dd639d3833", "error": {"message": "Internal JSON-RPC error: {'reason': \"'vmType'\"}", "code": -32603}} 2017-11-20 07:43:07,676-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'r eason': "'vmType'"}] 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Failed in 'GetAllVmStatsVDS' method 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdV DSCommandParametersBase:{hostId='1af28f2c-79db-4069-aa53-5bb46528c5e9'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 2017-11-20 07:43:07,676-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGeneric Exception: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42) [vdsbroker.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) * *</error>* ** --------------2EEBD942435F80A2268AD3F9 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a failure in OST on test 004_basic_sanity.vm_run. it seems to be an error in vm type which is related to the patch reported. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84343/">https://gerrit.ovirt.org/#/c/84343/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> vdsm log: 2017-11-20 07:40:12,779-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1341, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 508, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1664, in getStats stats.update(self._getConfigVmStats()) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1703, in _getConfigVmStats 'vmType': self.conf['vmType'], KeyError: 'vmType' engine log: 2017-11-20 07:43:07,675-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "5bf12e5a-4a09-4999-a6ce-a7dd639d3833", "error": {"message": "Internal JSON-RPC error: {'reason': \"'vmType'\"}", "code": -32603}} 2017-11-20 07:43:07,676-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'r eason': "'vmType'"}] 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Failed in 'GetAllVmStatsVDS' method 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdV DSCommandParametersBase:{hostId='1af28f2c-79db-4069-aa53-5bb46528c5e9'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 2017-11-20 07:43:07,676-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGeneric Exception: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42) [vdsbroker.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------2EEBD942435F80A2268AD3F9--

2 1

[ OST Failure Report ] [ oVirt master ] [ 20-11-1017 ] [ 002_bootstrap.verify_add_all_hosts ]
by Dafna Ron 20 Nov '17

20 Nov '17

This is a multi-part message in MIME format. --------------FBD48B98231B5BEAA6AAA899 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in OST for test 002_bootstrap.verify_add_all_hosts. >From the logs I can see that vdsm on host0 was reporting that it cannot find the physical volume but eventually the storage was created and is reported as responsive. However, Host1 is reported to became non-operational with storage domain does not exist error and I think that there is a race. I think that we create the storage domain while host1 is being installed and if the domain is not created and reported as activated in time, host1 will become nonOperational. are we starting installation of host1 before host0 and storage are active? ** *Link to suspected patches: I do not think that the patch reported is related to the error* * ** ** https://gerrit.ovirt.org/#/c/84133/** Link to Job: ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/ Link to all logs: * *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/ * * * *(Relevant) error snippet from the log: * * <error> Lago log: * 2017-11-18 11:15:25,472::log_utils.py::end_log_task::670::nose::INFO:: # add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09) 2017-11-18 11:15:25,472::log_utils.py::start_log_task::655::nose::INFO:: # add_secondary_storage_domains: ESC[0mESC[0m 2017-11-18 11:16:47,455::log_utils.py::end_log_task::670::nose::INFO:: # add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21) 2017-11-18 11:16:47,456::log_utils.py::start_log_task::655::nose::INFO:: # import_templates: ESC[0mESC[0m 2017-11-18 11:16:47,513::testlib.py::stopTest::198::nose::INFO:: * SKIPPED: Exported domain generation not supported yet 2017-11-18 11:16:47,514::log_utils.py::end_log_task::670::nose::INFO:: # import_templates: ESC[32mSuccessESC[0m (in 0:00:00) 2017-11-18 11:16:47,514::log_utils.py::start_log_task::655::nose::INFO:: # verify_add_all_hosts: ESC[0mESC[0m 2017-11-18 11:16:47,719::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function <lambda> at 0x2909230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 430, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 129, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 149, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-basic-suite-master-host-1: non_operational 2017-11-18 11:16:47,722::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir 2017-11-18 11:16:47,722::workdir.py::resolve_workdir_path::361::lago.workdir::DEBUG::Checking if /dev/shm/ost/deployment-basic-suite-master is a workdir 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m vdsm host0: 2017-11-18 06:14:23,980-0500 INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3, guids=[u'360014059618895272774e97a2aaf5dd6'], checkStatus=False, options={}) from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff 364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:46) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014059618895272774e97a2aaf5dd6".'] (lvm:322) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM] getPV failed for guid: 360014059618895272774e97a2aaf5dd6 (hsm:1973) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1970, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014059618895272774e97a2aaf5dd6',)" 2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getDeviceList return={'devList': [{'status': 'unknown', 'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev': '4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '', 'pathlist': [{'initiatorname': u'default', 'connection': u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}, {'initiatorname': u'default', 'connection': u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}], 'logicalblocksize': '512', 'discard_max_bytes': 1073741824, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': 'SLIO-ORG_lun0_bdev_96188952-7277-4e97-a2aa-f5dd6aad6fc2', 'GUID': '360014059618895272774e97a2aaf5dd6', 'productID': 'lun0_bdev'}]} from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:52) 2017-11-18 06:14:31,788-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-66079a39f123', 'vgMetadataDevice': '360014059618895272774e97a2aaf5dd6', 'vguuid': '7ifbmt-0elj-uWZZ-zS LG-plA8-8hd3-JG298b', 'metadataDevice': '360014059618895272774e97a2aaf5dd6', 'state': 'OK', 'version': '4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data', 'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,45636, flow_id=2c1876 99, task_id=c2080b61-d4a5-4bdb-9d75-f81580a8257a (api: vdsm host1: 2017-11-18 06:16:34,315-0500 ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for c65437ce-339f-4b01-aeb5-45c1d486bf49 failed (monitor:329) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 177, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c65437ce-339f-4b01-aeb5-45c1d486bf49',) 2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.201.4,58722 (api:46) 2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:46) 2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'c65437ce-339f-4b01-aeb5-45c1d486bf49': {'code': 358, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.1', 'valid': False}, u'cc61e074-a3b6-4371-9185-66079a39f123': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00103987', 'lastCheck': '6.5', 'valid': True}} from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:52) engine log: 2017-11-18 06:15:54,040-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-45c1d486bf49:nfs' was reported with error code '358' 2017-11-18 06:15:54,041-05 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is in problem in host 'lago-basic-suite-master-host-1' 2017-11-18 06:15:54,045-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host lago-basic-suite-master-host-1 reports about one of the Active Storage Domains as Problematic.** ** *** * *</error>* ** --------------FBD48B98231B5BEAA6AAA899 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in OST for test 002_bootstrap.verify_add_all_hosts. From the logs I can see that vdsm on host0 was reporting that it cannot find the physical volume but eventually the storage was created and is reported as responsive. However, Host1 is reported to became non-operational with storage domain does not exist error and I think that there is a race. I think that we create the storage domain while host1 is being installed and if the domain is not created and reported as activated in time, host1 will become nonOperational. are we starting installation of host1 before host0 and storage are active? <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: I do not think that the patch reported is related to the error <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/">https://gerrit.ovirt.org/#/c/84133/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Lago log: 2017-11-18 11:15:25,472::log_utils.py::end_log_task::670::nose::INFO:: # add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09) 2017-11-18 11:15:25,472::log_utils.py::start_log_task::655::nose::INFO:: # add_secondary_storage_domains: ESC[0mESC[0m 2017-11-18 11:16:47,455::log_utils.py::end_log_task::670::nose::INFO:: # add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21) 2017-11-18 11:16:47,456::log_utils.py::start_log_task::655::nose::INFO:: # import_templates: ESC[0mESC[0m 2017-11-18 11:16:47,513::testlib.py::stopTest::198::nose::INFO:: * SKIPPED: Exported domain generation not supported yet 2017-11-18 11:16:47,514::log_utils.py::end_log_task::670::nose::INFO:: # import_templates: ESC[32mSuccessESC[0m (in 0:00:00) 2017-11-18 11:16:47,514::log_utils.py::start_log_task::655::nose::INFO:: # verify_add_all_hosts: ESC[0mESC[0m 2017-11-18 11:16:47,719::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function <lambda> at 0x2909230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 430, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 129, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 149, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-basic-suite-master-host-1: non_operational 2017-11-18 11:16:47,722::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir 2017-11-18 11:16:47,722::workdir.py::resolve_workdir_path::361::lago.workdir::DEBUG::Checking if /dev/shm/ost/deployment-basic-suite-master is a workdir 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m vdsm host0: 2017-11-18 06:14:23,980-0500 INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3, guids=[u'360014059618895272774e97a2aaf5dd6'], checkStatus=False, options={}) from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff 364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:46) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014059618895272774e97a2aaf5dd6".'] (lvm:322) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM] getPV failed for guid: 360014059618895272774e97a2aaf5dd6 (hsm:1973) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1970, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014059618895272774e97a2aaf5dd6',)" 2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getDeviceList return={'devList': [{'status': 'unknown', 'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev': '4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '', 'pathlist': [{'initiatorname': u'default', 'connection': u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}, {'initiatorname': u'default', 'connection': u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}], 'logicalblocksize': '512', 'discard_max_bytes': 1073741824, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': 'SLIO-ORG_lun0_bdev_96188952-7277-4e97-a2aa-f5dd6aad6fc2', 'GUID': '360014059618895272774e97a2aaf5dd6', 'productID': 'lun0_bdev'}]} from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:52) 2017-11-18 06:14:31,788-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-66079a39f123', 'vgMetadataDevice': '360014059618895272774e97a2aaf5dd6', 'vguuid': '7ifbmt-0elj-uWZZ-zS LG-plA8-8hd3-JG298b', 'metadataDevice': '360014059618895272774e97a2aaf5dd6', 'state': 'OK', 'version': '4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data', 'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,45636, flow_id=2c1876 99, task_id=c2080b61-d4a5-4bdb-9d75-f81580a8257a (api: vdsm host1: 2017-11-18 06:16:34,315-0500 ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for c65437ce-339f-4b01-aeb5-45c1d486bf49 failed (monitor:329) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 177, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c65437ce-339f-4b01-aeb5-45c1d486bf49',) 2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.201.4,58722 (api:46) 2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:46) 2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'c65437ce-339f-4b01-aeb5-45c1d486bf49': {'code': 358, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.1', 'valid': False}, u'cc61e074-a3b6-4371-9185-66079a39f123': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00103987', 'lastCheck': '6.5', 'valid': True}} from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:52) engine log: 2017-11-18 06:15:54,040-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-45c1d486bf49:nfs' was reported with error code '358' 2017-11-18 06:15:54,041-05 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is in problem in host 'lago-basic-suite-master-host-1' 2017-11-18 06:15:54,045-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host lago-basic-suite-master-host-1 reports about one of the Active Storage Domains as Problematic. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------FBD48B98231B5BEAA6AAA899--

2 2

Where is subprocess32 for EL7?
by Nir Soffer 17 Nov '17

17 Nov '17

I'm trying to update ioprocess to use subprocess32 instead of cpopen. The build fail on el7-* http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-x86_64/262/co… http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-ppc64le/32/ I think I'm missing a secret repository. Who can tell me where subprocess32 is hiding? Nir

3 4

sos 3.5 has been released
by Sandro Bonazzola 17 Nov '17

17 Nov '17

Hi, sos 3.5 has been released upstream. I packaged it for Fedora >= 26 (and waiting karma[fc26][fc27]) being 25 almost dead. Is there interest in packaging it for CentOS Virt SIG considering CentOS 7.4 is still shipping sos 3.4? Among the improvements in sos 3.5 there's support for PostgreSQL from SCL, the same we are using in oVirt 4.2. Or are we fine waiting until sos 3.5 will be packaged and shipped within RHEL? [fc26] https://bodhi.fedoraproject.org/updates/FEDORA-2017-bef483c3a0 [fc27] https://bodhi.fedoraproject.org/updates/FEDORA-2017-4bd9528c50 Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Adding new features to ovirt.org
by John Marks 16 Nov '17

16 Nov '17

Hi, Thanks to everyone who helped with the feature categorization effort on the ovirt feature page <https://ovirt.org/develop/release-management/features/>. We've made significant progress, and thanks also to Eldan's UX work, the page is looking much better organized. There are a *few simple things* you can do to help keep the page organized, and to improve it. 1. Adding a new feature to ovirt.org: Please add your feature to a suitable category: - Gluster (The location in the repository is: /develop/release-management/features/gluster) - Infra (Same pattern as above) - Integration - Metrics - Network - Node - SLA - Storage - UX - VDSM - Virt 2. ‎To help us all find which release each feature is associated with, please add a link to your feature under *Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release>. * The folder location in the source repository is under: source/develop/release-management/releases/x.x/feature.html.md x.x = release version 3. We want to continue to list all *existing features* under the correct category and under Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release> . You can help by visiting this spreadsheet and completing the missing feature category and release version info: https://docs.google.com/spreadsheets/d/1c7Di6xz0N0j0rPFRzogBhErJj7rnTCatAZe… *Legend* Green*: *The feature is listed under the correct category and release *Brown**: *The feature needs to be moved to a different category / link needs to be added to the feature under the relevant release. Let me know if you have any questions. Thanks! John -- John Marks Community Manager & Technical Writer, oVirt redhat Israel Cell: +972 52 8644 491

4 6

[ OST Failure Report ] [ oVirt master ] [ 16-11-2017 ] [ 002_bootstrap.add_dc ]
by Dafna Ron 16 Nov '17

16 Nov '17

This is a multi-part message in MIME format. --------------D85A420919595D580C076AD7 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. ** *Link to suspected patches: * ** These are the reported patches: https://gerrit.ovirt.org/#/c/83790/ https://gerrit.ovirt.org/#/c/83779/ * ** Here is the original patch for the sso support: https://gerrit.ovirt.org/#/c/82997/ Link to Job: *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/ ** *Link to all logs:* *** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/ (Relevant) error snippet from the log: <error> * 2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException ** *</error>* ** --------------D85A420919595D580C076AD7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">These are the reported patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83790/">https://gerrit.ovirt.org/#/c/83790/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83779/">https://gerrit.ovirt.org/#/c/83779/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> Here is the original patch for the sso support: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82997/">https://gerrit.ovirt.org/#/c/82997/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/</a><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/"> </a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------D85A420919595D580C076AD7--

4 4

[ OST Failure Report ] [ oVirt master ] [ 14-11-2017 ] [ 002_bootstrap.add_hosts ]
by Dafna Ron 15 Nov '17

15 Nov '17

This is a multi-part message in MIME format. --------------5185435FD38913B821CED967 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. ** *Link to suspected patches: As I said,**I do not think its related, but this is the patch that was reported. * * https://gerrit.ovirt.org/#/c/83854/ Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/* * * *Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/* * * *(Relevant) error snippet from the log: * * <error> ** _*Test error: *_** ** * Error Message False != True after 900 seconds Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds ** ** ** **_lago log: _* * *2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::ssh::81::lago.ssh::DEBUG::Command bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::INFO::ESC[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') * * * **_from host: _* * ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb)** * * *</error>* ** --------------5185435FD38913B821CED967 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: As I said, I do not think its related, but this is the patch that was reported. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83854/">https://gerrit.ovirt.org/#/c/83854/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Test error: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Error Message</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">False != True after 900 seconds</pre> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Stacktrace</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> lago log: 2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::<a class="moz-txt-link-freetext" href="ssh::81::lago.ssh::DEBUG::Command">ssh::81::lago.ssh::DEBUG::Command</a> bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::153::lago.ssh::DEBUG::Wait">ssh::153::lago.ssh::DEBUG::Wait</a> succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') from host: ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb) <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------5185435FD38913B821CED967--

3 4

more aggressive static asset caching in ovirt-engine
by Greg Sheremeta 15 Nov '17

15 Nov '17

Hi, I made some changes [1][2] to ovirt-engine that make it now more aggressively (and correctly) cache all static assets. We were previously caching some things, but not all. (Thanks to ykaul for noticing.) The upside is reduced server hits and faster page loads. The downside is you'll have to clear your browser cache more often as you are developing. (You can use a browser plugin, incognito mode, F12 > disable cache, etc. if you don't want to keep opening the dialog.) [1] https://gerrit.ovirt.org/#/c/83879/ [2] https://gerrit.ovirt.org/#/c/83871/ Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

2 2

[ OST Failure Report ] [ oVirt master ] [ 15-11-2017 ] [ 001_upgrade_engine.test_initialize_engine ]
by Dafna Ron 15 Nov '17

15 Nov '17

This is a multi-part message in MIME format. --------------1EB3285B75C820E3A1990192 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a few alerts from cq all pointing to the same patch. Adding Sandro since he's working on fixing the packages issue and resolve this. ** *Link to suspected patches: **https://gerrit.ovirt.org/#/c/83852/* ** ** *Link to Job:* *** * *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824* http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo… http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/* * * * *Link to all logs:* * ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/ (Relevant) error snippet from the log: <error> * Removing obsolete ovirt-engine-setup-plugin-websocket-proxy-4.1.7.7-0.0.master.20171114102253.git9267e4c.el7.centos.noarch.rpm cockpit-ovirt-dashboard-0.10.9 FAILED (1/145): cockpit-ovirt-das 0% [ ] 0.0 B/s | 0 B --:-- ETA cockpit-ovirt-uiplugin-0.10.9- FAILED (1/145): cockpit-ovirt-uip 0% [ ] 0.0 B/s | 0 B --:-- ETA Public key for cockpit-ovirt-uiplugin-0.10.9-1.el7.centos.noarch.rpm is not installed ovirt-ansible-roles-1.0.4-1.el FAILED ovirt-engine-setup-plugin-live FAILED ovirt-engine-yarn-0.19.1-2.el7 FAILED ** *</error>* ** --------------1EB3285B75C820E3A1990192 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a few alerts from cq all pointing to the same patch. Adding Sandro since he's working on fixing the packages issue and resolve this. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83852/">https://gerrit.ovirt.org/#/c/83852/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/testRepo…</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3824/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3829/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3833/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3839/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3844/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3850/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Removing obsolete ovirt-engine-setup-plugin-websocket-proxy-4.1.7.7-0.0.master.20171114102253.git9267e4c.el7.centos.noarch.rpm <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> cockpit-ovirt-dashboard-0.10.9 FAILED (1/145): cockpit-ovirt-das 0% [ ] 0.0 B/s | 0 B --:-- ETA cockpit-ovirt-uiplugin-0.10.9- FAILED (1/145): cockpit-ovirt-uip 0% [ ] 0.0 B/s | 0 B --:-- ETA Public key for cockpit-ovirt-uiplugin-0.10.9-1.el7.centos.noarch.rpm is not installed ovirt-ansible-roles-1.0.4-1.el FAILED ovirt-engine-setup-plugin-live FAILED ovirt-engine-yarn-0.19.1-2.el7 FAILED </pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------1EB3285B75C820E3A1990192--

2 1

speed up engine development with SKIP_CHECKS
by Greg Sheremeta 15 Nov '17

15 Nov '17

Hi, I've added a SKIP_CHECKS flag to the ovirt-engine build here [1]. Thanks to Roy for the idea [2]. TL;DR: you'll save roughly 2 minutes 10 seconds per full compile with SKIP_CHECKS. You'll save more (3 min 10 sec) if you don't need a 'clean'. You'll save the most (about 3 min 20 sec) if you can skip rebuilding GWT. Here are some statistics (3 runs per command, for averages sake) My test machine is a Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz 256GB Samsung SSD (old, using SATA) 32GB DDR3 @ 1333 MT/s --- fresh master checkout no clean, no gwt, skip checks: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 1:04.38 total 1:06.04 total 1:04.41 total same as above, but now change one non-GWT class in the utils project make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 1:05.49 total Ok, pretend we recompiled and our utils class change had an error. Same as above, change the one non-GWT class in the utils project again, and use the resume (-pl) functionality: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 EXTRA_BUILD_FLAGS="-pl org.ovirt.engine.core:utils" 17.958 total (yes, 18 seconds, obviously a world record for any engine build) [The resume piece is EXTRA_BUILD_FLAGS="-pl org.ovirt.engine.ui:webadmin" -- I personally need to use this more often!] --- no clean, no gwt, run checks (but skip unit tests): make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 BUILD_UT=0 3:07.66 total 3:16.69 total 3:14.28 total clean, no gwt, skip checks: make clean install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=0 SKIP_CHECKS=1 4:17.74 total 4:15.76 total 4:13.90 total no clean, gwt (1 permutation for chrome), skip checks: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=1 SKIP_CHECKS=1 DEV_EXTRA_BUILD_FLAGS_GWT_DEFAULTS="-Dgwt.userAgent=safari" 4:25.63 total 4:26.71 total 4:30.69 total no clean, gwt (1 permutation for chrome), run checks but skip unit tests: make install-dev PREFIX=/home/greg/ovirt-engine BUILD_GWT=1 BUILD_UT=0 DEV_EXTRA_BUILD_FLAGS_GWT_DEFAULTS="-Dgwt.userAgent=safari" 6:44.23 total And the whole enchilada [3] ... clean, gwt (1 permutation for chrome), run checks, run unit tests: 11:52.25 total ... wow :) Hope this helps you have a faster change-compile-check cycle :) Best wishes, Greg [1] https://gerrit.ovirt.org/#/c/83961/ [2] https://www.mail-archive.com/devel@ovirt.org/msg10377.html [3] https://idioms.thefreedictionary.com/whole+enchilada -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 2

[ OST Failure Report ] [ oVirt master ] [ 2017-11-13 ] [add_master_storage_domain]
by Barak Korren 13 Nov '17

13 Nov '17

Test failed: [ add_master_storage_domain ] Link to suspected patches: - https://gerrit.ovirt.org/#/c/83849/4 This seems to be a fairly consistent regression as all patches that follow the patch above exhibit the same issue when tested, this list of similarly failing patches includes: - https://gerrit.ovirt.org/#/c/83940/2 - https://gerrit.ovirt.org/#/c/83608/7 - https://gerrit.ovirt.org/#/c/83815/3 The following patch is an exception because it simply failed to build: - https://gerrit.ovirt.org/#/c/83370/9 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/artifact… Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "[Failed to attach Storage due to an error on the Data Center master Storage Domain. -Please activate the master Storage Domain first.]". HTTP response code is 409. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

3 8

failing check-patch on ovirt-engine
by Greg Sheremeta 13 Nov '17

13 Nov '17

Hi, I'm seeing a weird issue in ovirt-engine check-patch. There's no way this failure is related to my patch [ https://gerrit.ovirt.org/#/c/83950/] which is a 1 character typo fix in the constants file. Anyone have any ideas or also seeing this? It seems random, too -- sometimes 1 or 2 of the jobs will succeed [el7, fc25] but the other will fail. Etc. *14:48:08* + automation/packaging-setup-tests.sh*14:48:08* + trap popd 0*14:48:08* +++ readlink -f automation/packaging-setup-tests.sh*14:48:08* ++ dirname /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/packaging-setup-tests.sh*14:48:08* + pushd /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/..*14:48:08* ~ ~*14:48:08* + export PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + python -m pytest packaging/setup*14:48:08* ============================= test session starts ==============================*14:48:08* platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.4.34, pluggy-0.4.0*14:48:08* rootdir: /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine, inifile:*14:48:08* collected 0 items / 1 errors*14:48:08* *14:48:08* ==================================== ERRORS ====================================*14:48:08* ERROR collecting packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py *14:48:08* ImportError while importing test module '/home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py'.*14:48:08* Hint: make sure your test modules/packages have valid Python names.*14:48:08* Traceback:*14:48:08* packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py:19: in <module>*14:48:08* import ovirt_engine_setup.engine_common.database as under_test # isort:skip # noqa: E402*14:48:08* packaging/setup/ovirt_engine_setup/engine_common/database.py:30: in <module>*14:48:08* from otopi import base*14:48:08* E ImportError: No module named otopi*14:48:08* !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!*14:48:08* =========================== 1 error in 0.33 seconds ============================*14:48:08* + popd*14:48:08* ~ ---------- Forwarded message ---------- From: Code Review <gerrit(a)ovirt.org> Date: Sun, Nov 12, 2017 at 10:31 AM Subject: Change in ovirt-engine[master]: engine: typo fix snaphot -> snapshot To: Greg Sheremeta <gshereme(a)redhat.com> Jenkins CI *posted comments* on this change. View Change <https://gerrit.ovirt.org/83950> Patch set 1:Continuous-Integration -1 Build Failed http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc25-x86_64/17231/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc26-x86_64/157/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-el7-x86_64/33193/ : SUCCESS To view, visit change 83950 <https://gerrit.ovirt.org/83950>. To unsubscribe, visit settings <https://gerrit.ovirt.org/settings>. Gerrit-Project: ovirt-engine Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b4d75855c0abe2f0eddd8b282c172205db2e42 Gerrit-Change-Number: 83950 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Alexander Wels <awels(a)redhat.com> Gerrit-Reviewer: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Tal Nisan <tnisan(a)redhat.com> Gerrit-Reviewer: Vojtech Szocs <vszocs(a)redhat.com> Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org> Gerrit-Comment-Date: Sun, 12 Nov 2017 15:31:27 +0000 Gerrit-HasComments: No -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 2

oVirt 4.1.9 will be the last 4.1 release
by Sandro Bonazzola 13 Nov '17

13 Nov '17

Hi, we are now targeting oVirt 4.2.0 GA around December 12th and oVirt 4.1.8 around December 5th. Against tradition that would have set 4.1.8 as last 4.1 release, this round we'll release also oVIrt 4.1.9 being it already in development just before 4.2.0 GA. We suggest to be prepared to upgrade to 4.2 since 4.1 won't receive any further update after 4.1.9 release. -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Forking 'ansible' into oVirt org on GitHub
by Barak Korren 13 Nov '17

13 Nov '17

Guys, I'm not sure who did that, but somebody forked the core 'ansible' repo in to the ovirt organisation on GitHub. IMO this does not look good. Why was this done? -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

[ FIXED ] Missing release RPMs in resources.ovirt.org/pub/yum-repo/
by Daniel Belenky 12 Nov '17

12 Nov '17

Hi all, Some of you might have encountered an error while trying to download the latest ovirt release (4.1/4.1/master) from resources.ovirt.org/yum-repo/ During the weekend we had an issue with our nightly publisher's job. Those jobs are responsible to move artifacts from Jenkins to the matching snapshot repo in resources.ovirt.org. Since the publishers failed, ovirt-release-master.rpm and ovirt-release-4.x-snapshot.rpms (which are symlinks to ovirt-${version}-snapshot) were unavailable. The nightly publisher is fixed now and, the rpms were properly deployed to the repo. Sorry for any inconvenience, -- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>

3 2

[ OST Failure Report ] [ oVirt 4.1 ] [ 09-11-2017 ] [ 001_upgrade_engine.test_initialize_engine ]
by Dafna Ron 09 Nov '17

09 Nov '17

This is a multi-part message in MIME format. --------------1E382631B1CFE8CFC947FDB5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in installing engine with a dependency error between ovirt-engine-backend and vdsm-jsonrpc-java Please note that we have other issues which are not related to this patch so it might be hard to find this issue. I also added the lago log where you can find the package dependency error. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/82995/* * Link to Job:http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291 * http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex… * * *Link to all logs: http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact* * (Relevant) error snippet from the log: <error> * [ INFO ] Checking for an update for Setup... [ ERROR ] Yum [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Yum Performing yum transaction rollback [ ERROR ] Failed to execute stage 'Environment customization': [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20171109042111-tpncix.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20171109042115-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed ** *</error>* ** --------------1E382631B1CFE8CFC947FDB5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in installing engine with a dependency error between ovirt-engine-backend and vdsm-jsonrpc-java Please note that we have other issues which are not related to this patch so it might be hard to find this issue. I also added the lago log where you can find the package dependency error. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82995/">https://gerrit.ovirt.org/#/c/82995/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job:<a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex…">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">[ INFO ] Checking for an update for Setup... [ ERROR ] Yum [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Yum Performing yum transaction rollback [ ERROR ] Failed to execute stage 'Environment customization': [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20171109042111-tpncix.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20171109042115-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed </pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------1E382631B1CFE8CFC947FDB5--

3 3

VM start fails due to missing disk
by Viktor Mihajlovski 08 Nov '17

08 Nov '17

Hi, I ran into a problem after building ovirt-engine from current master to verify my last outstanding patch: but now I can't start my VM anymore, because the <disk> element is omitted from the final libvirt XML created by vdsm. The xml argument sent by the engine for the VM.create call contains a valid disk definition, but it disappears from the final libvirt XML. The preceding prepareImage call returns a seemingly valid path info... I am currently at loss and would appreciate any hints. -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski

1 1

Infra related events message - review needed
by Dafna Ron 08 Nov '17

08 Nov '17

This is a multi-part message in MIME format. --------------C2C1B6C457A9CCD4A0E14F7B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had some confusion last week when I sent a message about an infrastructure related issue we have found and people thought I was reporting an error on OST so I decided that perhaps it would be a good idea to create a general message that would alert people on an on-going infra event and not just OST related issues. iBelow is the format I was thinking of using: ############################################## Subject: Ovirt CI - [Infrastructure Incident] [$Date] [$TIME] Hello, We encountered an incident that can suggest there is an ongoing infrastructure incident. Our team is currently investigating the issue and we will send a mail once we have more information or the issue is resolved. Issue summary: ################################################ Please let me know what you're ideas on this are and if the message is at all lacking in some way. Thanks. Dafna --------------C2C1B6C457A9CCD4A0E14F7B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had some confusion last week when I sent a message about an infrastructure related issue we have found and people thought I was reporting an error on OST so I decided that perhaps it would be a good idea to create a general message that would alert people on an on-going infra event and not just OST related issues. iBelow is the format I was thinking of using: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">############################################## <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Subject: Ovirt CI - [Infrastructure Incident] [$Date] [$TIME] <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Hello, <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">We encountered an incident that can suggest there is an ongoing infrastructure incident. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Our team is currently investigating the issue and we will send a mail once we have more information or the issue is resolved. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Issue <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">summary<span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">################################################ Please let me know what you're ideas on this are and if the message is at all lacking in some way. Thanks. Dafna </body> </html> --------------C2C1B6C457A9CCD4A0E14F7B--

2 1

[ OST Failure Report ] [ oVirt master ] [ 07-11-2017 ] [ 004_basic_sanity.disk_operations ]
by Dafna Ron 08 Nov '17

08 Nov '17

This is a multi-part message in MIME format. --------------106EB129C1BC1AEAACE8C19F Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in test 004_basic_sanity.disk_operations. >From the log I can see that vdsm has a KeyError: 'diskReplicate' and than reports the drive not found. Can you please check the issue? ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/83641/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/ engine log: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact… vdsm log: ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact… (Relevant) error snippet from the log: <error> >From engine log: * 2017-11-07 09:20:42,396-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Failed in 'VmReplicateDiskFinishVDS' method 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-basic-suite-mas ter-host-0 command VmReplicateDiskFinishVDS failed: Drive image file could not be found 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDisk FinishVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]' 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] HostName = lago-basic-suite-master-host-0 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'VmReplicateDiskFinishVDSCommand(HostName = lago-basic-sui te-master-host-0, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c9 1d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'})' execution failed: VDSGenericException: VDSErrorException: Fai led to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 2017-11-07 09:20:42,399-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorExcep tion: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand.executeVdsBrokerCommand(VmReplicateDiskFinishVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor244.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskFinish(LiveMigrateDiskCommand.java:374) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.handleDestDisk(LiveMigrateDiskCommand.java:681) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.endWithFailure(LiveMigrateDiskCommand.java:337) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:783) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:712) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1989) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:574) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:125) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:156) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:65) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:141) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:106) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2017-11-07 09:20:42,400-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] FINISH, VmReplicateDiskFinishVDSCommand, log id: 32c5a56e 2017-11-07 09:20:42,400-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: runVdsCommand, params: [VmReplicateDiskFinish, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'}], timeElapsed: 12ms 2017-11-07 09:20:42,400-05 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Replication end of disk '36837a9b-6396-4d65-a354-df97ce808f01' in vm '61d328bc-c920-4691-997f-09bbecb5de10' back to the source failed, skipping deletion of the target disk 2017-11-07 09:20:42,408-05 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Trying to release exclusive lock which does not exist, lock key: '61d328bc-c920-4691-997f-09bbecb5de10LIVE_STORAGE_MIGRATION' 2017-11-07 09:20:42,408-05 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Lock freed to object 'EngineLock:{exclusiveLocks='[61d328bc-c920-4691-997f-09bbecb5de10=LIVE_STORAGE_MIGRATION]', sharedLocks=''}' 2017-11-07 09:20:42,428-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: get, params: [038b0c9a-f64c-4aa7-8e36-72e435771dab], timeElapsed: 2ms 2017-11-07 09:20:42,430-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk vm0_disk0 to domain iscsi.** ** * * * * vdsm log: 2017-11-07 09:20:41,178-0500 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeSize return={'truesize': '2415919104', 'apparentsize': '2415919104'} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e, task_id=042481eb-74ac-42a8-9d1e-8fb1c84db234 (api:52) 2017-11-07 09:20:41,193-0500 ERROR (jsonrpc/6) [api] FINISH diskReplicateFinish error='diskReplicate' (api:127) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 558, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4477, in diskReplicateFinish self._delDiskReplica(drive) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4528, in _delDiskReplica del dev['diskReplicate'] KeyError: 'diskReplicate' 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'General Exception: ("\'diskReplicate\'",)', 'code': 100}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 100) in 0.79 seconds (__init__:630) 2017-11-07 09:20:41,253-0500 INFO (jsonrpc/2) [vdsm.api] START finalizeMerge(spUUID=u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', subchainInfo={u'img_id': u'8401cd0f-419c-417c-8da3-52da285fa47c', u'sd_id': u'4a088fb2-5497-4b4d-adec-4c91d998c5 6e', u'top_id': u'7a295270-fdf5-4561-b688-033c5d312d1a', u'base_id': u'e77f8714-9e0f-4f39-8437-e75a3fa44c74'}) from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (ap i:46) 2017-11-07 09:20:41,267-0500 INFO (jsonrpc/2) [vdsm.api] FINISH finalizeMerge return=None from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (api:52) 2017-11-07 09:20:41,285-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.finalizeMerge succeeded in 0.03 seconds (__init__:630) 2017-11-07 09:20:41,285-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] START task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x2f0a560>>, args=None) (threadPool:208) 2017-11-07 09:20:41,295-0500 INFO (tasks/8) [storage.merge] Finalizing subchain after merge: <SubchainInfo sd_id=4a088fb2-5497-4b4d-adec-4c91d998c56e, img_id=8401cd0f-419c-417c-8da3-52da285fa47c, top_id=7a295270-fdf5-4561-b688-033c5d312d 1a, base_id=e77f8714-9e0f-4f39-8437-e75a3fa44c74 base_generation=None at 0x30a7990> (merge:254) 2017-11-07 09:20:41,332-0500 INFO (tasks/8) [storage.SANLock] Acquiring Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:377) 2017-11-07 09:20:41,345-0500 INFO (tasks/8) [storage.SANLock] Successfully acquired Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:415) 2017-11-07 09:20:41,372-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/e77f8714-9e0f-4f39-8437-e75a3fa44c74 (volume:558) 2017-11-07 09:20:41,377-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/7a295270-fdf5-4561-b688-033c5d312d1a (volume:558) 2017-11-07 09:20:41,395-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (volume:558) 2017-11-07 09:20:41,399-0500 INFO (tasks/8) [storage.VolumeManifest] sdUUID=4a088fb2-5497-4b4d-adec-4c91d998c56e imgUUID=8401cd0f-419c-417c-8da3-52da285fa47c volUUID = 7a295270-fdf5-4561-b688-033c5d312d1a legality = ILLEGAL (volume:399) 2017-11-07 09:20:41,667-0500 INFO (tasks/8) [storage.merge] Updating Vdsm metadata, syncing new chain: ['e77f8714-9e0f-4f39-8437-e75a3fa44c74', u'a6cb7a60-e500-4a75-9da8-9e7cce3e21ac'] (merge:311) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Current chain=e77f8714-9e0f-4f39-8437-e75a3fa44c74 < 7a295270-fdf5-4561-b688-033c5d312d1a < a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (top) (image:1162) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Unlinking subchain: ['7a295270-fdf5-4561-b688-033c5d312d1a'] (image:1172) 2017-11-07 09:20:41,709-0500 INFO (tasks/8) [storage.Image] Setting parent of volume a6cb7a60-e500-4a75-9da8-9e7cce3e21ac to e77f8714-9e0f-4f39-8437-e75a3fa44c74 (image:1185) 2017-11-07 09:20:41,824-0500 INFO (tasks/8) [storage.SANLock] Releasing Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:435) 2017-11-07 09:20:41,828-0500 INFO (tasks/8) [storage.SANLock] Successfully released Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:444) 2017-11-07 09:20:41,842-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] FINISH task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (threadPool:210) 2017-11-07 09:20:42,388-0500 INFO (jsonrpc/4) [api.virt] START diskReplicateFinish(srcDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u'4a08 8fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}, dstDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u '4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:46) 2017-11-07 09:20:42,388-0500 ERROR (jsonrpc/4) [virt.vm] (vmId='61d328bc-c920-4691-997f-09bbecb5de10') Drive not found (srcDisk: {u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f 25-c1184c10841a', u'domainID': u'4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) (vm:4386) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 13) in 0.01 seconds (__init__:630)** : * * *</error>* ** --------------106EB129C1BC1AEAACE8C19F Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a failure in test 004_basic_sanity.disk_operations. From the log I can see that vdsm has a KeyError: 'diskReplicate' and than reports the drive not found. Can you please check the issue? <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83641/">https://gerrit.ovirt.org/#/c/83641/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">engine log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">vdsm log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> From engine log: 2017-11-07 09:20:42,396-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Failed in 'VmReplicateDiskFinishVDS' method 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-basic-suite-mas ter-host-0 command VmReplicateDiskFinishVDS failed: Drive image file could not be found 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDisk FinishVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]' 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] HostName = lago-basic-suite-master-host-0 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'VmReplicateDiskFinishVDSCommand(HostName = lago-basic-sui te-master-host-0, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c9 1d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'})' execution failed: VDSGenericException: VDSErrorException: Fai led to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 2017-11-07 09:20:42,399-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorExcep tion: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand.executeVdsBrokerCommand(VmReplicateDiskFinishVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor244.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskFinish(LiveMigrateDiskCommand.java:374) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.handleDestDisk(LiveMigrateDiskCommand.java:681) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.endWithFailure(LiveMigrateDiskCommand.java:337) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:783) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:712) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1989) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:574) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:125) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:156) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:65) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:141) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:106) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2017-11-07 09:20:42,400-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] FINISH, VmReplicateDiskFinishVDSCommand, log id: 32c5a56e 2017-11-07 09:20:42,400-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: runVdsCommand, params: [VmReplicateDiskFinish, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'}], timeElapsed: 12ms 2017-11-07 09:20:42,400-05 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Replication end of disk '36837a9b-6396-4d65-a354-df97ce808f01' in vm '61d328bc-c920-4691-997f-09bbecb5de10' back to the source failed, skipping deletion of the target disk 2017-11-07 09:20:42,408-05 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Trying to release exclusive lock which does not exist, lock key: '61d328bc-c920-4691-997f-09bbecb5de10LIVE_STORAGE_MIGRATION' 2017-11-07 09:20:42,408-05 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Lock freed to object 'EngineLock:{exclusiveLocks='[61d328bc-c920-4691-997f-09bbecb5de10=LIVE_STORAGE_MIGRATION]', sharedLocks=''}' 2017-11-07 09:20:42,428-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: get, params: [038b0c9a-f64c-4aa7-8e36-72e435771dab], timeElapsed: 2ms 2017-11-07 09:20:42,430-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk vm0_disk0 to domain iscsi. vdsm log: 2017-11-07 09:20:41,178-0500 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeSize return={'truesize': '2415919104', 'apparentsize': '2415919104'} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e, task_id=042481eb-74ac-42a8-9d1e-8fb1c84db234 (api:52) 2017-11-07 09:20:41,193-0500 ERROR (jsonrpc/6) [api] FINISH diskReplicateFinish error='diskReplicate' (api:127) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 558, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4477, in diskReplicateFinish self._delDiskReplica(drive) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4528, in _delDiskReplica del dev['diskReplicate'] KeyError: 'diskReplicate' 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'General Exception: ("\'diskReplicate\'",)', 'code': 100}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 100) in 0.79 seconds (__init__:630) 2017-11-07 09:20:41,253-0500 INFO (jsonrpc/2) [vdsm.api] START finalizeMerge(spUUID=u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', subchainInfo={u'img_id': u'8401cd0f-419c-417c-8da3-52da285fa47c', u'sd_id': u'4a088fb2-5497-4b4d-adec-4c91d998c5 6e', u'top_id': u'7a295270-fdf5-4561-b688-033c5d312d1a', u'base_id': u'e77f8714-9e0f-4f39-8437-e75a3fa44c74'}) from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (ap i:46) 2017-11-07 09:20:41,267-0500 INFO (jsonrpc/2) [vdsm.api] FINISH finalizeMerge return=None from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (api:52) 2017-11-07 09:20:41,285-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.finalizeMerge succeeded in 0.03 seconds (__init__:630) 2017-11-07 09:20:41,285-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] START task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x2f0a560>>, args=None) (threadPool:208) 2017-11-07 09:20:41,295-0500 INFO (tasks/8) [storage.merge] Finalizing subchain after merge: <SubchainInfo sd_id=4a088fb2-5497-4b4d-adec-4c91d998c56e, img_id=8401cd0f-419c-417c-8da3-52da285fa47c, top_id=7a295270-fdf5-4561-b688-033c5d312d 1a, base_id=e77f8714-9e0f-4f39-8437-e75a3fa44c74 base_generation=None at 0x30a7990> (merge:254) 2017-11-07 09:20:41,332-0500 INFO (tasks/8) [storage.SANLock] Acquiring Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:377) 2017-11-07 09:20:41,345-0500 INFO (tasks/8) [storage.SANLock] Successfully acquired Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:415) 2017-11-07 09:20:41,372-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/e77f8714-9e0f-4f39-8437-e75a3fa44c74 (volume:558) 2017-11-07 09:20:41,377-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/7a295270-fdf5-4561-b688-033c5d312d1a (volume:558) 2017-11-07 09:20:41,395-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (volume:558) 2017-11-07 09:20:41,399-0500 INFO (tasks/8) [storage.VolumeManifest] sdUUID=4a088fb2-5497-4b4d-adec-4c91d998c56e imgUUID=8401cd0f-419c-417c-8da3-52da285fa47c volUUID = 7a295270-fdf5-4561-b688-033c5d312d1a legality = ILLEGAL (volume:399) 2017-11-07 09:20:41,667-0500 INFO (tasks/8) [storage.merge] Updating Vdsm metadata, syncing new chain: ['e77f8714-9e0f-4f39-8437-e75a3fa44c74', u'a6cb7a60-e500-4a75-9da8-9e7cce3e21ac'] (merge:311) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Current chain=e77f8714-9e0f-4f39-8437-e75a3fa44c74 < 7a295270-fdf5-4561-b688-033c5d312d1a < a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (top) (image:1162) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Unlinking subchain: ['7a295270-fdf5-4561-b688-033c5d312d1a'] (image:1172) 2017-11-07 09:20:41,709-0500 INFO (tasks/8) [storage.Image] Setting parent of volume a6cb7a60-e500-4a75-9da8-9e7cce3e21ac to e77f8714-9e0f-4f39-8437-e75a3fa44c74 (image:1185) 2017-11-07 09:20:41,824-0500 INFO (tasks/8) [storage.SANLock] Releasing Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:435) 2017-11-07 09:20:41,828-0500 INFO (tasks/8) [storage.SANLock] Successfully released Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:444) 2017-11-07 09:20:41,842-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] FINISH task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (threadPool:210) 2017-11-07 09:20:42,388-0500 INFO (jsonrpc/4) [api.virt] START diskReplicateFinish(srcDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u'4a08 8fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}, dstDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u '4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:46) 2017-11-07 09:20:42,388-0500 ERROR (jsonrpc/4) [virt.vm] (vmId='61d328bc-c920-4691-997f-09bbecb5de10') Drive not found (srcDisk: {u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f 25-c1184c10841a', u'domainID': u'4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) (vm:4386) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 13) in 0.01 seconds (__init__:630) : <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------106EB129C1BC1AEAACE8C19F--

2 2

[ OST Failure Report ] [ oVirt Master ] [ 06-11-2017 ] [ 002_bootstrap.verify_add_hosts ]
by Dafna Ron 07 Nov '17

07 Nov '17

This is a multi-part message in MIME format. --------------88DD9FCD3B3DDEA34D4EAEF4 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We failed test 002_bootstrap.verify_add_hosts I can see we only tried to install one of the hosts (host-0) and failed. the second host has no log which means we did not try to deploy it. The error suggests that we ovirt-imageio-daemon failed to start. However, there is another message that I think should be addressed about conflicting vdsm and libvirt configurations. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/83612/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact… * * *(Relevant) error snippet from the log: * * <error> \* 2017-11-06 02:56:46,526-0500 DEBUG otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921 execute-output: ('/usr/bin/vdsm-tool', 'configure', '--force') stdout: Checking configuration status... abrt is not configured for vdsm WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration lvm requires configuration libvirt is not configured for vdsm yet FAILED: conflicting vdsm and libvirt-qemu tls configuration. vdsm.conf with ssl=True requires the following changes: libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 qemu.conf: spice_tls=1. multipath requires configuration 2017-11-06 02:56:47,551-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stderr: Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details. 2017-11-06 02:56:47,552-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-R4R8gZhaQI/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 179, in _start self.services.state('ovirt-imageio-daemon', True) File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'ovirt-imageio-daemon' 2017-11-06 02:56:47,553-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon' ** *</error>* * * --------------88DD9FCD3B3DDEA34D4EAEF4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We failed test 002_bootstrap.verify_add_hosts I can see we only tried to install one of the hosts (host-0) and failed. the second host has no log which means we did not try to deploy it. The error suggests that we ovirt-imageio-daemon failed to start. However, there is another message that I think should be addressed about conflicting vdsm and libvirt configurations. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83612/">https://gerrit.ovirt.org/#/c/83612/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> \ <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-06 02:56:46,526-0500 DEBUG otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921 execute-output: ('/usr/bin/vdsm-tool', 'configure', '--force') stdout: Checking configuration status... abrt is not configured for vdsm WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration lvm requires configuration libvirt is not configured for vdsm yet FAILED: conflicting vdsm and libvirt-qemu tls configuration. vdsm.conf with ssl=True requires the following changes: libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 qemu.conf: spice_tls=1. multipath requires configuration </pre> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-06 02:56:47,551-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stderr: Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details. 2017-11-06 02:56:47,552-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-R4R8gZhaQI/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 179, in _start self.services.state('ovirt-imageio-daemon', True) File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'ovirt-imageio-daemon' 2017-11-06 02:56:47,553-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon'</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------88DD9FCD3B3DDEA34D4EAEF4--

5 7

[ OST Failure Report ] [ oVirt master ] [ 06-11-2017] [ 002_bootstrap.add_hosts ]
by Dafna Ron 07 Nov '17

07 Nov '17

This is a multi-part message in MIME format. --------------F7FAC86DB10D28BA8B32AB1D Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in test 002_bootstrap.add_hosts I can see that the hosts are added but there are errors in engine.log that suggest there is a problem with crypto. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/83367/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact/ ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact… (Relevant) error snippet from the log: <error> * 2017-11-06 10:29:55,367-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.DbFacadeUtils] (default task-7) [d5f76e37-ef60-4195-a735-400c1ed293be] Failed to decrypt password: javax.crypto.BadPaddingException: Decryption error at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:380) [rt.jar:1.8.0_151] at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:291) [rt.jar:1.8.0_151] at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:363) [sunjce_provider.jar:1.8.0_151] at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:389) [sunjce_provider.jar:1.8.0_151] at javax.crypto.Cipher.doFinal(Cipher.java:2165) [jce.jar:1.8.0_151] at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:179) [utils.jar:] at org.ovirt.engine.core.dal.dbbroker.DbFacadeUtils.decryptPassword(DbFacadeUtils.java:35) [dal.jar:] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper.mapRow(ProviderDaoImpl.java:137) [dal.jar:] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper.mapRow(ProviderDaoImpl.java:119) [dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:463) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:474) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:484) [spring-jdbc.jar:4.3.9.RELEASE] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl.getAllWithQuery(ProviderDaoImpl.java:189) [dal.jar:] at org.ovirt.engine.core.bll.SearchQuery.genericSearch(SearchQuery.java:337) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.searchProviders(SearchQuery.java:408) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.executeQueryCommand(SearchQuery.java:215) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:579) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:548) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.Final.jar:11.0.0.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.runQuery(BackendResource.java:82) at org.ovirt.engine.api.restapi.resource.BackendResource.getBackendCollection(BackendResource.java:152) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:48) at org.ovirt.engine.api.restapi.resource.openstack.BackendOpenStackNetworkProvidersResource.getBackendCollection(BackendOpenStackNetworkProvidersResource.java:70) at org.ovirt.engine.api.restapi.resource.openstack.BackendOpenStackNetworkProvidersResource.list(BackendOpenStackNetworkProvidersResource.java:50) at org.ovirt.engine.api.resource.openstack.OpenstackNetworkProvidersResource.doList(OpenstackNetworkProvidersResource.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:180) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:284) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] 2017-11-06 10:29:55,378-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-7) [d5f76e37-ef60-4195-a735-400c1ed293be] method: runQuery, params: [Search, SearchParameters:{refresh='false', filtered='false', searchType='Provider', searchPattern='Providers: type=EXTERNAL_NETWORK', caseSensitive='false', from='0', max='2147483647'}], timeElapsed: 49ms 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] Entered SsoRestApiAuthFilter 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter authenticating with sso 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter authenticating using BEARER header 2017-11-06 10:29:55,387-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter successfully authenticated using BEARER header 2017-11-06 10:29:55,387-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-8) [] Entered SsoRestApiNegotiationFilter 2017-11-06 10:29:55,389-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-8) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth ** *</error>* * * --------------F7FAC86DB10D28BA8B32AB1D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a failure in test 002_bootstrap.add_hosts I can see that the hosts are added but there are errors in engine.log that suggest there is a problem with crypto. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83367/">https://gerrit.ovirt.org/#/c/83367/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3636/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-06 10:29:55,367-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.DbFacadeUtils] (default task-7) [d5f76e37-ef60-4195-a735-400c1ed293be] Failed to decrypt password: javax.crypto.BadPaddingException: Decryption error at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:380) [rt.jar:1.8.0_151] at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:291) [rt.jar:1.8.0_151] at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:363) [sunjce_provider.jar:1.8.0_151] at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:389) [sunjce_provider.jar:1.8.0_151] at javax.crypto.Cipher.doFinal(Cipher.java:2165) [jce.jar:1.8.0_151] at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:179) [utils.jar:] at org.ovirt.engine.core.dal.dbbroker.DbFacadeUtils.decryptPassword(DbFacadeUtils.java:35) [dal.jar:] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper.mapRow(ProviderDaoImpl.java:137) [dal.jar:] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper.mapRow(ProviderDaoImpl.java:119) [dal.jar:] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:463) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:408) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:474) [spring-jdbc.jar:4.3.9.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:484) [spring-jdbc.jar:4.3.9.RELEASE] at org.ovirt.engine.core.dao.provider.ProviderDaoImpl.getAllWithQuery(ProviderDaoImpl.java:189) [dal.jar:] at org.ovirt.engine.core.bll.SearchQuery.genericSearch(SearchQuery.java:337) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.searchProviders(SearchQuery.java:408) [bll.jar:] at org.ovirt.engine.core.bll.SearchQuery.executeQueryCommand(SearchQuery.java:215) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:579) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:548) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.Final.jar:11.0.0.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:] at org.ovirt.engine.api.restapi.resource.BackendResource.runQuery(BackendResource.java:82) at org.ovirt.engine.api.restapi.resource.BackendResource.getBackendCollection(BackendResource.java:152) at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.getBackendCollection(AbstractBackendCollectionResource.java:48) at org.ovirt.engine.api.restapi.resource.openstack.BackendOpenStackNetworkProvidersResource.getBackendCollection(BackendOpenStackNetworkProvidersResource.java:70) at org.ovirt.engine.api.restapi.resource.openstack.BackendOpenStackNetworkProvidersResource.list(BackendOpenStackNetworkProvidersResource.java:50) at org.ovirt.engine.api.resource.openstack.OpenstackNetworkProvidersResource.doList(OpenstackNetworkProvidersResource.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.24.Final.jar:3.0.24.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:180) at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:284) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] 2017-11-06 10:29:55,378-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-7) [d5f76e37-ef60-4195-a735-400c1ed293be] method: runQuery, params: [Search, SearchParameters:{refresh='false', filtered='false', searchType='Provider', searchPattern='Providers: type=EXTERNAL_NETWORK', caseSensitive='false', from='0', max='2147483647'}], timeElapsed: 49ms 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] Entered SsoRestApiAuthFilter 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter authenticating with sso 2017-11-06 10:29:55,384-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter authenticating using BEARER header 2017-11-06 10:29:55,387-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-8) [] SsoRestApiAuthFilter successfully authenticated using BEARER header 2017-11-06 10:29:55,387-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-8) [] Entered SsoRestApiNegotiationFilter 2017-11-06 10:29:55,389-05 DEBUG [org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter] (default task-8) [] SsoRestApiNegotiationFilter Not performing Negotiate Auth</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------F7FAC86DB10D28BA8B32AB1D--

3 2

bookmarkable webadmin URLs
by Greg Sheremeta 06 Nov '17

06 Nov '17

Hi, Alexander fixed a long overlooked bug ( https://bugzilla.redhat.com/show_bug.cgi?id=1508033) where bookmarked webadmin main pages were not respected when a user was logged out and clicked the bookmark. Instead, the user was always redirected back to the dashboard. Now, navigating to /ovirt-engine/webadmin/#somePlace while logged out will properly redirect you to the "somePlace" main page after the login form is submitted. Thanks Alexander for fixing and Vojtech and Ravi for thoughts and reviews. And what we'd like to do next is make the UI *fully* bookmarkable by allowing very expressive links directly to specific entities. I filed an RFE for this: https://bugzilla.redhat.com/show_bug.cgi?id=1509065 There are lots of ways this could go, but we imaging some kind of synchronization with the search bar. /ovirt-engine/webadmin/#vms?name=my-vm or something like that. Thoughts are welcome. Best wishes, Greg -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

2 1