- Devel - oVirt List Archives

Default MAC address range
by Yedidyah Bar David 22 Nov '17

22 Nov '17

Hi all, The default range went through several changes during the lifetime of the project - from being small and constant until 3.2, to small and random in 3.3 until 3.5, back to fixed and small in 3.6, slightly enlarged (to 1024 addresses) in current master. In all cases, the address range was taken from the IANA-assigned range 001A4A of Qumranet (now Red Hat) [1]. I suggest to: 1. Default to a random range again - So that two or (a few) more setups using the same network segment are less likely to use the same range 2. One in the locally-administered address range [2] - So that we have a larger address range 3. Make it a bit larger, say 65000 addresses - So that admins do not have to handle this manually, even for large setups, unless they have specific needs If there is objection to (2.), we can still have the others - among the 24 bits in the 001A4A prefix range, choose among the first 8 a random value during engine-setup, and have the pool of size 16 bits. Comments/ideas/etc. are welcome. Best regards, [1] https://regauth.standards.ieee.org/standards-ra-web/pub/view.html [2] https://en.wikipedia.org/wiki/MAC_address#Universal_vs._local -- Didi

2 2

[ OST Failure Report ] [ oVirt 4.2 ] [ 22-11-2017 ] [ 002_bootstrap.add_dc ]
by Dafna Ron 22 Nov '17

22 Nov '17

This is a multi-part message in MIME format. --------------4E0BDA3C10A37D37E4CBA152 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I am not sure we should be alerting on 4.2 versions yet but since we are getting several cq failures for the same reasons and test, I decided this should be communicated. The failed test is 002_bootstrap.add_dc in OST 4.2. and all are pointing to one patch. Remove mandatory option for cluster in register operation - https://gerrit.ovirt.org/#/c/84359/* * can you please have a look? I was unable to determine from what log the error was coming from as well but looking at the failed test and the patch reported there seem to be a connection. ** *Link and headline of suspected patches: * ** Patch that is marked as root cause: Remove mandatory option for cluster in register operation - https://gerrit.ovirt.org/#/c/84359/* * Patches reported as failed: 1. Add discoveredTargets property to HostService - https://gerrit.ovirt.org/#/c/84489/ *2. Document the fact that sparse/raw isn't supported on block - https://gerrit.ovirt.org/#/c/84490/* 3. Update CHANGES.adoc for release 4.2.25 - https://gerrit.ovirt.org/#/c/84499/* * 4. [maven-release-plugin] prepare release 4.2.26 - https://gerrit.ovirt.org/#/c/84500/* * ** * Link to Job: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/* * * *Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/* * * *(Relevant) error snippet from the log: * * <error> * Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/002_bootstrap.py", line 109, in add_dc api = prefix.virt_env.engine_vm().get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 329, in get_api return self.get_api_v4() File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 338, in get_api_v4 self._api_v4 = self._get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 316, in _get_api while not testapi.test(): File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 744, in test six.reraise(*sys.exc_info()) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 741, in test self.system_service().get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 21865, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get context = self._connection.send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 370, in send return self.__send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 388, in __send self.authenticate() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 381, in authenticate self._sso_token = self._get_access_token() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 617, in _get_access_token sso_response = self._get_sso_response(self._sso_url, post_data) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 702, in _get_sso_response self._check_content_type(self.__JSON_CONTENT_TYPE_RE, 'JSON', header_lines) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 887, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=UTF-8' isn't the expected JSON ** *</error>* ** --------------4E0BDA3C10A37D37E4CBA152 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, I am not sure we should be alerting on 4.2 versions yet but since we are getting several cq failures for the same reasons and test, I decided this should be communicated. The failed test is 002_bootstrap.add_dc in OST 4.2. and all are pointing to one patch. Remove mandatory option for cluster in register operation - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84359/">https://gerrit.ovirt.org/#/c/84359/</a> can you please have a look? I was unable to determine from what log the error was coming from as well but looking at the failed test and the patch reported there seem to be a connection. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link and headline of suspected patches: Patch that is marked as root cause: Remove mandatory option for cluster in register operation - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84359/">https://gerrit.ovirt.org/#/c/84359/</a> Patches reported as failed: 1. Add discoveredTargets property to HostService - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84489/">https://gerrit.ovirt.org/#/c/84489/</a> 2. Document the fact that sparse/raw isn't supported on block - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84490/">https://gerrit.ovirt.org/#/c/84490/</a> 3. Update CHANGES.adoc for release 4.2.25 - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84499/">https://gerrit.ovirt.org/#/c/84499/</a> 4. [maven-release-plugin] prepare release 4.2.26 - <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84500/">https://gerrit.ovirt.org/#/c/84500/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/">http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/75/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/basic-suite-4.2/test-scenarios/002_bootstrap.py", line 109, in add_dc api = prefix.virt_env.engine_vm().get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 329, in get_api return self.get_api_v4() File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 338, in get_api_v4 self._api_v4 = self._get_api(api_ver=4) File "/usr/lib/python2.7/site-packages/ovirtlago/virt.py", line 316, in _get_api while not testapi.test(): File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 744, in test six.reraise(*sys.exc_info()) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 741, in test self.system_service().get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 21865, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get context = self._connection.send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 370, in send return self.__send(request) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 388, in __send self.authenticate() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 381, in authenticate self._sso_token = self._get_access_token() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 617, in _get_access_token sso_response = self._get_sso_response(self._sso_url, post_data) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 702, in _get_sso_response self._check_content_type(self.__JSON_CONTENT_TYPE_RE, 'JSON', header_lines) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 887, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=UTF-8' isn't the expected JSON</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------4E0BDA3C10A37D37E4CBA152--

3 2

Re: [ovirt-devel] Question about availability of image-io in ovirt master repo
by Nir Soffer 21 Nov '17

21 Nov '17

On Tue, Nov 21, 2017 at 9:38 PM Idan Shaby <ishaby(a)redhat.com> wrote: > So that what it gives me: > - el6 - nothing. > - el7: > ovirt-imageio-common-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211045.gitd4425c0.el7.centos.noarch > - fc27 - nothing. > - fc26: > ovirt-imageio-common-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211044.gitd4425c0.fc26.noarch > - fc25: > ovirt-imageio-common-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-daemon-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-proxy-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > ovirt-imageio-proxy-setup-0:1.2.0-0.201711211044.gitd4425c0.fc25.noarch > fc24 - nothing. > > So I guess it means we don't deliver imageio1.2.0 on el6, fc27 and fc24. > fc27 should be a problem, right? > I hope that tomorrow we will have also fc27 and fcraw, but these are not supported platforms yet. Looks like we can require 1.2 in vdsm. > > Regards, > Idan > > ---------- Forwarded message ---------- > From: Barak Korren <bkorren(a)redhat.com> > Date: Tue, Nov 21, 2017 at 9:27 PM > Subject: Re: Question about availability of image-io in ovirt master repo > To: Idan Shaby <ishaby(a)redhat.com> > Cc: infra <infra(a)ovirt.org> > > > On 21 November 2017 at 21:04, Idan Shaby <ishaby(a)redhat.com> wrote: > > Hi, > > > > How do I check if our ovirt master repos contain version 1.2.0 of > > ovirt-imageio-common and ovirt-imageio-daemon? > > I want to add these requirements to vdsm and I am not sure how to verify > it. > > This searches for a certain package in a certain repo: > > repoquery --repofrompath=r, > http://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7 > --repoid=r > <http://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7%0D--repoid=r> > list '*imageio*' > Adding devel, may be useful for people adding dependencies. > > You can change the url slightly to check for other distros and oVirt > versions. > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > >

1 0

[ OST Failure Report ] [ oVirt master ] [ 20-11-2017 ] [004_basic_sanity.vm_run ]
by Dafna Ron 20 Nov '17

20 Nov '17

This is a multi-part message in MIME format. --------------2EEBD942435F80A2268AD3F9 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in OST on test 004_basic_sanity.vm_run. it seems to be an error in vm type which is related to the patch reported. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/84343/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact (Relevant) error snippet from the log: <error> vdsm log: * *2017-11-20 07:40:12,779-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1341, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 508, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1664, in getStats stats.update(self._getConfigVmStats()) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1703, in _getConfigVmStats 'vmType': self.conf['vmType'], KeyError: 'vmType' * * * engine log: ***2017-11-20 07:43:07,675-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "5bf12e5a-4a09-4999-a6ce-a7dd639d3833", "error": {"message": "Internal JSON-RPC error: {'reason': \"'vmType'\"}", "code": -32603}} 2017-11-20 07:43:07,676-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'r eason': "'vmType'"}] 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Failed in 'GetAllVmStatsVDS' method 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdV DSCommandParametersBase:{hostId='1af28f2c-79db-4069-aa53-5bb46528c5e9'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 2017-11-20 07:43:07,676-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGeneric Exception: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42) [vdsbroker.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) * *</error>* ** --------------2EEBD942435F80A2268AD3F9 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a failure in OST on test 004_basic_sanity.vm_run. it seems to be an error in vm type which is related to the patch reported. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84343/">https://gerrit.ovirt.org/#/c/84343/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3922/artifact</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> vdsm log: 2017-11-20 07:40:12,779-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1341, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 508, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1664, in getStats stats.update(self._getConfigVmStats()) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1703, in _getConfigVmStats 'vmType': self.conf['vmType'], KeyError: 'vmType' engine log: 2017-11-20 07:43:07,675-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "5bf12e5a-4a09-4999-a6ce-a7dd639d3833", "error": {"message": "Internal JSON-RPC error: {'reason': \"'vmType'\"}", "code": -32603}} 2017-11-20 07:43:07,676-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'r eason': "'vmType'"}] 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Failed in 'GetAllVmStatsVDS' method 2017-11-20 07:43:07,676-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Command 'GetAllVmStatsVDSCommand(HostName = lago-basic-suite-master-host-0, VdsIdV DSCommandParametersBase:{hostId='1af28f2c-79db-4069-aa53-5bb46528c5e9'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 2017-11-20 07:43:07,676-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGeneric Exception: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'vmType'"}, code = -32603 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand.executeVdsBrokerCommand(GetAllVmStatsVDSCommand.java:23) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher.poll(VmsStatisticsFetcher.java:29) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.VmsListFetcher.fetch(VmsListFetcher.java:57) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher.poll(PollVmStatsRefresher.java:42) [vdsbroker.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------2EEBD942435F80A2268AD3F9--

2 1

[ OST Failure Report ] [ oVirt master ] [ 20-11-1017 ] [ 002_bootstrap.verify_add_all_hosts ]
by Dafna Ron 20 Nov '17

20 Nov '17

This is a multi-part message in MIME format. --------------FBD48B98231B5BEAA6AAA899 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in OST for test 002_bootstrap.verify_add_all_hosts. >From the logs I can see that vdsm on host0 was reporting that it cannot find the physical volume but eventually the storage was created and is reported as responsive. However, Host1 is reported to became non-operational with storage domain does not exist error and I think that there is a race. I think that we create the storage domain while host1 is being installed and if the domain is not created and reported as activated in time, host1 will become nonOperational. are we starting installation of host1 before host0 and storage are active? ** *Link to suspected patches: I do not think that the patch reported is related to the error* * ** ** https://gerrit.ovirt.org/#/c/84133/** Link to Job: ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/ Link to all logs: * *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/ * * * *(Relevant) error snippet from the log: * * <error> Lago log: * 2017-11-18 11:15:25,472::log_utils.py::end_log_task::670::nose::INFO:: # add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09) 2017-11-18 11:15:25,472::log_utils.py::start_log_task::655::nose::INFO:: # add_secondary_storage_domains: ESC[0mESC[0m 2017-11-18 11:16:47,455::log_utils.py::end_log_task::670::nose::INFO:: # add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21) 2017-11-18 11:16:47,456::log_utils.py::start_log_task::655::nose::INFO:: # import_templates: ESC[0mESC[0m 2017-11-18 11:16:47,513::testlib.py::stopTest::198::nose::INFO:: * SKIPPED: Exported domain generation not supported yet 2017-11-18 11:16:47,514::log_utils.py::end_log_task::670::nose::INFO:: # import_templates: ESC[32mSuccessESC[0m (in 0:00:00) 2017-11-18 11:16:47,514::log_utils.py::start_log_task::655::nose::INFO:: # verify_add_all_hosts: ESC[0mESC[0m 2017-11-18 11:16:47,719::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function <lambda> at 0x2909230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 430, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 129, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 149, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-basic-suite-master-host-1: non_operational 2017-11-18 11:16:47,722::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir 2017-11-18 11:16:47,722::workdir.py::resolve_workdir_path::361::lago.workdir::DEBUG::Checking if /dev/shm/ost/deployment-basic-suite-master is a workdir 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m vdsm host0: 2017-11-18 06:14:23,980-0500 INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3, guids=[u'360014059618895272774e97a2aaf5dd6'], checkStatus=False, options={}) from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff 364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:46) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014059618895272774e97a2aaf5dd6".'] (lvm:322) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM] getPV failed for guid: 360014059618895272774e97a2aaf5dd6 (hsm:1973) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1970, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014059618895272774e97a2aaf5dd6',)" 2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getDeviceList return={'devList': [{'status': 'unknown', 'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev': '4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '', 'pathlist': [{'initiatorname': u'default', 'connection': u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}, {'initiatorname': u'default', 'connection': u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}], 'logicalblocksize': '512', 'discard_max_bytes': 1073741824, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': 'SLIO-ORG_lun0_bdev_96188952-7277-4e97-a2aa-f5dd6aad6fc2', 'GUID': '360014059618895272774e97a2aaf5dd6', 'productID': 'lun0_bdev'}]} from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:52) 2017-11-18 06:14:31,788-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-66079a39f123', 'vgMetadataDevice': '360014059618895272774e97a2aaf5dd6', 'vguuid': '7ifbmt-0elj-uWZZ-zS LG-plA8-8hd3-JG298b', 'metadataDevice': '360014059618895272774e97a2aaf5dd6', 'state': 'OK', 'version': '4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data', 'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,45636, flow_id=2c1876 99, task_id=c2080b61-d4a5-4bdb-9d75-f81580a8257a (api: vdsm host1: 2017-11-18 06:16:34,315-0500 ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for c65437ce-339f-4b01-aeb5-45c1d486bf49 failed (monitor:329) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 177, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c65437ce-339f-4b01-aeb5-45c1d486bf49',) 2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.201.4,58722 (api:46) 2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:46) 2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'c65437ce-339f-4b01-aeb5-45c1d486bf49': {'code': 358, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.1', 'valid': False}, u'cc61e074-a3b6-4371-9185-66079a39f123': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00103987', 'lastCheck': '6.5', 'valid': True}} from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:52) engine log: 2017-11-18 06:15:54,040-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-45c1d486bf49:nfs' was reported with error code '358' 2017-11-18 06:15:54,041-05 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is in problem in host 'lago-basic-suite-master-host-1' 2017-11-18 06:15:54,045-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host lago-basic-suite-master-host-1 reports about one of the Active Storage Domains as Problematic.** ** *** * *</error>* ** --------------FBD48B98231B5BEAA6AAA899 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in OST for test 002_bootstrap.verify_add_all_hosts. From the logs I can see that vdsm on host0 was reporting that it cannot find the physical volume but eventually the storage was created and is reported as responsive. However, Host1 is reported to became non-operational with storage domain does not exist error and I think that there is a race. I think that we create the storage domain while host1 is being installed and if the domain is not created and reported as activated in time, host1 will become nonOperational. are we starting installation of host1 before host0 and storage are active? <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: I do not think that the patch reported is related to the error <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/84133/">https://gerrit.ovirt.org/#/c/84133/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3902/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> Lago log: 2017-11-18 11:15:25,472::log_utils.py::end_log_task::670::nose::INFO:: # add_master_storage_domain: ESC[32mSuccessESC[0m (in 0:01:09) 2017-11-18 11:15:25,472::log_utils.py::start_log_task::655::nose::INFO:: # add_secondary_storage_domains: ESC[0mESC[0m 2017-11-18 11:16:47,455::log_utils.py::end_log_task::670::nose::INFO:: # add_secondary_storage_domains: ESC[32mSuccessESC[0m (in 0:01:21) 2017-11-18 11:16:47,456::log_utils.py::start_log_task::655::nose::INFO:: # import_templates: ESC[0mESC[0m 2017-11-18 11:16:47,513::testlib.py::stopTest::198::nose::INFO:: * SKIPPED: Exported domain generation not supported yet 2017-11-18 11:16:47,514::log_utils.py::end_log_task::670::nose::INFO:: # import_templates: ESC[32mSuccessESC[0m (in 0:00:00) 2017-11-18 11:16:47,514::log_utils.py::start_log_task::655::nose::INFO:: # verify_add_all_hosts: ESC[0mESC[0m 2017-11-18 11:16:47,719::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function <lambda> at 0x2909230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 430, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 129, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 149, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-basic-suite-master-host-1: non_operational 2017-11-18 11:16:47,722::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir 2017-11-18 11:16:47,722::workdir.py::resolve_workdir_path::361::lago.workdir::DEBUG::Checking if /dev/shm/ost/deployment-basic-suite-master is a workdir 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m 2017-11-18 11:16:47,724::log_utils.py::__enter__::600::lago.prefix::INFO:: * Collect artifacts: ESC[0mESC[0m vdsm host0: 2017-11-18 06:14:23,980-0500 INFO (jsonrpc/0) [vdsm.api] START getDeviceList(storageType=3, guids=[u'360014059618895272774e97a2aaf5dd6'], checkStatus=False, options={}) from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff 364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:46) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.LVM] lvm pvs failed: 5 [] [' Failed to find physical volume "/dev/mapper/360014059618895272774e97a2aaf5dd6".'] (lvm:322) 2017-11-18 06:14:24,353-0500 WARN (jsonrpc/0) [storage.HSM] getPV failed for guid: 360014059618895272774e97a2aaf5dd6 (hsm:1973) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1970, in _getDeviceList pv = lvm.getPV(guid) File "/usr/lib/python2.7/site-packages/vdsm/storage/lvm.py", line 852, in getPV raise se.InaccessiblePhysDev((pvName,)) InaccessiblePhysDev: Multipath cannot access physical device(s): "devices=(u'360014059618895272774e97a2aaf5dd6',)" 2017-11-18 06:14:24,389-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getDeviceList return={'devList': [{'status': 'unknown', 'vendorID': 'LIO-ORG', 'capacity': '21474836480', 'fwrev': '4.0', 'discard_zeroes_data': 0, 'vgUUID': '', 'pvsize': '', 'pathlist': [{'initiatorname': u'default', 'connection': u'192.168.200.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}, {'initiatorname': u'default', 'connection': u'192.168.201.4', 'iqn': u'iqn.2014-07.org.ovirt:storage', 'portal': '1', 'user': u'username', 'password': '********', 'port': '3260'}], 'logicalblocksize': '512', 'discard_max_bytes': 1073741824, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sda', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf', 'capacity': '21474836480', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': 'SLIO-ORG_lun0_bdev_96188952-7277-4e97-a2aa-f5dd6aad6fc2', 'GUID': '360014059618895272774e97a2aaf5dd6', 'productID': 'lun0_bdev'}]} from=::ffff:192.168.201.4,45636, flow_id=ed8310a1-a7af-4a67-b351-8ff364766b8a, task_id=6ced0092-34cd-49f0-aa0f-6aae498af37f (api:52) 2017-11-18 06:14:31,788-0500 INFO (jsonrpc/0) [vdsm.api] FINISH getStorageDomainInfo return={'info': {'uuid': 'cc61e074-a3b6-4371-9185-66079a39f123', 'vgMetadataDevice': '360014059618895272774e97a2aaf5dd6', 'vguuid': '7ifbmt-0elj-uWZZ-zS LG-plA8-8hd3-JG298b', 'metadataDevice': '360014059618895272774e97a2aaf5dd6', 'state': 'OK', 'version': '4', 'role': 'Regular', 'type': 'ISCSI', 'class': 'Data', 'pool': [], 'name': 'iscsi'}} from=::ffff:192.168.201.4,45636, flow_id=2c1876 99, task_id=c2080b61-d4a5-4bdb-9d75-f81580a8257a (api: vdsm host1: 2017-11-18 06:16:34,315-0500 ERROR (monitor/c65437c) [storage.Monitor] Setting up monitor for c65437ce-339f-4b01-aeb5-45c1d486bf49 failed (monitor:329) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 326, in _setupLoop self._setupMonitor() File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 348, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 177, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 366, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'c65437ce-339f-4b01-aeb5-45c1d486bf49',) 2017-11-18 06:16:40,377-0500 INFO (jsonrpc/7) [api.host] START getStats() from=::ffff:192.168.201.4,58722 (api:46) 2017-11-18 06:16:40,378-0500 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:46) 2017-11-18 06:16:40,379-0500 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={u'c65437ce-339f-4b01-aeb5-45c1d486bf49': {'code': 358, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '6.1', 'valid': False}, u'cc61e074-a3b6-4371-9185-66079a39f123': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00103987', 'lastCheck': '6.5', 'valid': True}} from=::ffff:192.168.201.4,58722, task_id=8fb74944-08c0-491e-ad55-a7a9f0a11ef8 (api:52) engine log: 2017-11-18 06:15:54,040-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Domain 'c65437ce-339f-4b01-aeb5-45c1d486bf49:nfs' was reported with error code '358' 2017-11-18 06:15:54,041-05 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] Storage Domain 'nfs' of pool 'test-dc' is in problem in host 'lago-basic-suite-master-host-1' 2017-11-18 06:15:54,045-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29) [4ce8aff3] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host lago-basic-suite-master-host-1 reports about one of the Active Storage Domains as Problematic. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------FBD48B98231B5BEAA6AAA899--

2 2

Where is subprocess32 for EL7?
by Nir Soffer 17 Nov '17

17 Nov '17

I'm trying to update ioprocess to use subprocess32 instead of cpopen. The build fail on el7-* http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-x86_64/262/co… http://jenkins.ovirt.org/job/ioprocess_master_check-patch-el7-ppc64le/32/ I think I'm missing a secret repository. Who can tell me where subprocess32 is hiding? Nir

3 4

sos 3.5 has been released
by Sandro Bonazzola 17 Nov '17

17 Nov '17

Hi, sos 3.5 has been released upstream. I packaged it for Fedora >= 26 (and waiting karma[fc26][fc27]) being 25 almost dead. Is there interest in packaging it for CentOS Virt SIG considering CentOS 7.4 is still shipping sos 3.4? Among the improvements in sos 3.5 there's support for PostgreSQL from SCL, the same we are using in oVirt 4.2. Or are we fine waiting until sos 3.5 will be packaged and shipped within RHEL? [fc26] https://bodhi.fedoraproject.org/updates/FEDORA-2017-bef483c3a0 [fc27] https://bodhi.fedoraproject.org/updates/FEDORA-2017-4bd9528c50 Thanks, -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Adding new features to ovirt.org
by John Marks 16 Nov '17

16 Nov '17

Hi, Thanks to everyone who helped with the feature categorization effort on the ovirt feature page <https://ovirt.org/develop/release-management/features/>. We've made significant progress, and thanks also to Eldan's UX work, the page is looking much better organized. There are a *few simple things* you can do to help keep the page organized, and to improve it. 1. Adding a new feature to ovirt.org: Please add your feature to a suitable category: - Gluster (The location in the repository is: /develop/release-management/features/gluster) - Infra (Same pattern as above) - Integration - Metrics - Network - Node - SLA - Storage - UX - VDSM - Virt 2. ‎To help us all find which release each feature is associated with, please add a link to your feature under *Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release>. * The folder location in the source repository is under: source/develop/release-management/releases/x.x/feature.html.md x.x = release version 3. We want to continue to list all *existing features* under the correct category and under Features by Release <https://ovirt.org/develop/release-management/features/#features-by-release> . You can help by visiting this spreadsheet and completing the missing feature category and release version info: https://docs.google.com/spreadsheets/d/1c7Di6xz0N0j0rPFRzogBhErJj7rnTCatAZe… *Legend* Green*: *The feature is listed under the correct category and release *Brown**: *The feature needs to be moved to a different category / link needs to be added to the feature under the relevant release. Let me know if you have any questions. Thanks! John -- John Marks Community Manager & Technical Writer, oVirt redhat Israel Cell: +972 52 8644 491

4 6

[ OST Failure Report ] [ oVirt master ] [ 16-11-2017 ] [ 002_bootstrap.add_dc ]
by Dafna Ron 16 Nov '17

16 Nov '17

This is a multi-part message in MIME format. --------------D85A420919595D580C076AD7 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. ** *Link to suspected patches: * ** These are the reported patches: https://gerrit.ovirt.org/#/c/83790/ https://gerrit.ovirt.org/#/c/83779/ * ** Here is the original patch for the sso support: https://gerrit.ovirt.org/#/c/82997/ Link to Job: *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/ ** *Link to all logs:* *** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/ (Relevant) error snippet from the log: <error> * 2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException ** *</error>* ** --------------D85A420919595D580C076AD7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, we have a CQ failure in add_dc test. I can see at engine long that we have an issue with 'extension.aaa.jdbc.binding.api.AuthnExtension' I don't think that the patch reported was the actual cause of the failure. Yesterday we were failing on missing package ovirt-cockpit-sso which we added to our repos. and I think perhaps we now exposed an issue with sso support which was undiscovered before. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">These are the reported patches: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83790/">https://gerrit.ovirt.org/#/c/83790/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83779/">https://gerrit.ovirt.org/#/c/83779/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> Here is the original patch for the sso support: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82997/">https://gerrit.ovirt.org/#/c/82997/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/</a><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/"> </a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3877/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3878/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-15 16:50:07,115-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authn' 2017-11-15 16:50:07,118-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authn' loaded 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.profile.name: internal 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.aaa.authn.authz.plugin: internal-authz 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authn 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,118-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authn.internal-authn] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,119-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Loading extension 'internal-authz' 2017-11-15 16:50:07,120-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Extension 'internal-authz' loaded 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config BEGIN 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.provides: org.ovirt.engine.api.extensions.aaa.Authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.class: org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthzExtension 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.bindings.method: jbossmodule 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] config.datasource.file: /etc/ovirt-engine/aaa/internal.properties 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.name: internal-authz 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] ovirt.engine.extension.binding.jbossmodule.module: org.ovirt.engine.extension.aaa.jdbc 2017-11-15 16:50:07,120-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager.trace."ovirt-engine-extension-aaa-jdbc".authz.internal-authz] (ServerService Thread Pool -- 45) [] Config END 2017-11-15 16:50:07,121-05 INFO [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Initializing extension 'internal-authn' 2017-11-15 16:50:07,182-05 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Unexpected Exception invoking: EXTENSION_INITIALIZE[e5ae1b7f-9104-4f23-a444-7b9175ff68d2] 2017-11-15 16:50:07,182-05 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (ServerService Thread Pool -- 45) [] Exception:: org.postgresql.util.PSQLException: FATAL: the database system is shutting down at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) [postgresql.jar:] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) [postgresql.jar:] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) [postgresql.jar:] at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) [postgresql.jar:] at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) [postgresql.jar:] at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) [postgresql.jar:] at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) [postgresql.jar:] at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:] at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:] at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:] at java.sql.DriverManager.getConnection(DriverManager.java:664) [rt.jar:1.8.0_151] at java.sql.DriverManager.getConnection(DriverManager.java:247) [rt.jar:1.8.0_151] at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:] at com.sun.proxy.$Proxy177.getConnection(Unknown Source) at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:41) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.ExtensionUtils.checkDbVersion(ExtensionUtils.java:146) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.doInit(AuthnExtension.java:114) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:56) [ovirt-engine-extension-aaa-jdbc.jar:] at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73) at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109) at org.ovirt.engine.core.extensions.mgr.ExtensionsManager.initialize(ExtensionsManager.java:315) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.initialize(SsoExtensionsManager.java:91) at org.ovirt.engine.core.sso.utils.SsoExtensionsManager.<init>(SsoExtensionsManager.java:28) at org.ovirt.engine.core.sso.context.SsoContextListener.contextInitialized(SsoContextListener.java:28) at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:205) at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:174) at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:239) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.jboss.threads.JBossThread.run(JBossThread.java:320) 2017-11-15 16:50:07,185-05 ERROR [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Error in activating extension 'internal-authn': FATAL: the database system is shutting down 2017-11-15 16:50:07,185-05 DEBUG [org.ovirt.engine.core.extensions.mgr.ExtensionsManager] (ServerService Thread Pool -- 45) [] Class: class org.ovirt.engine.core.extensions.mgr.ExtensionInvokeCommandFailedException</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------D85A420919595D580C076AD7--

4 4

[ OST Failure Report ] [ oVirt master ] [ 14-11-2017 ] [ 002_bootstrap.add_hosts ]
by Dafna Ron 15 Nov '17

15 Nov '17

This is a multi-part message in MIME format. --------------5185435FD38913B821CED967 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. ** *Link to suspected patches: As I said,**I do not think its related, but this is the patch that was reported. * * https://gerrit.ovirt.org/#/c/83854/ Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/* * * *Link to all logs:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/* * * *(Relevant) error snippet from the log: * * <error> ** _*Test error: *_** ** * Error Message False != True after 900 seconds Stacktrace Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds ** ** ** **_lago log: _* * *2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::ssh::81::lago.ssh::DEBUG::Command bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::INFO::ESC[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') * * * **_from host: _* * ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb)** * * *</error>* ** --------------5185435FD38913B821CED967 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not seeing any error that can suggest on an issue in engine. I can see in the host messages host that we have stopped writing to the log for 15 minutes and it may suggest that there is something that is keeping the host from starting which causes us to fail the test on timeout. However, i can use some help in determining the cause for this failure and weather its connected to the bootstrap_add_host test in upgrade. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: As I said, I do not think its related, but this is the patch that was reported. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83854/">https://gerrit.ovirt.org/#/c/83854/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Test error: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Error Message</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">False != True after 900 seconds</pre> <h3 style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: Helvetica, Arial, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Stacktrace</h3> <pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py", line 187, in add_hosts testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 900 seconds</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> lago log: 2017-11-13 15:31:23,212::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:29,228::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:30,230::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 2 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:31,233::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:32,234::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-host0: [Errno None] Unable to connect to port 22 on 192.168.200.2 2017-11-13 15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error connecting to lago-upgrade-from-release-suite-master-engine: [Errno None] Unable to connect to port 22 on 192.168.200.3 2017-11-13 15:31:33,237::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for lago-upgrade-from-release-suite-master-engine: 2017-11-13 15:31:34,238::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a> exception while sshing to lago-upgrade-from-release-suite-master-engine: Timed out (in 1 s) trying to ssh to lago-upgrade-from-release-suite-master-engine 2017-11-13 15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,639::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb49adfa on lago-upgrade-from-release-suite-master-host0: true 2017-11-13 15:31:34,665::ssh.py::<a class="moz-txt-link-freetext" href="ssh::81::lago.ssh::DEBUG::Command">ssh::81::lago.ssh::DEBUG::Command</a> bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0 2017-11-13 15:31:34,665::ssh.py::wait_for_<a class="moz-txt-link-freetext" href="ssh::153::lago.ssh::DEBUG::Wait">ssh::153::lago.ssh::DEBUG::Wait</a> succeeded for ssh to lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,665::log_utils.py::__exit__::611::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[32mSuccessESC[0m (in 0:00:11) 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.prefix::<a class="moz-txt-link-freetext" href="INFO::ESC">INFO::ESC</a>[0mESC[0m 2017-11-13 15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 100 tries for lago-upgrade-from-release-suite-master-host0 2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for lago-upgrade-from-release-suite-master-host0: 2017-11-13 15:31:34,906::ssh.py::<a class="moz-txt-link-freetext" href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a> bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/') from host: ions. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/ebtables.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64 Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 3. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 4 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 4. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Created slice user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting user-0.slice. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Started Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd: Starting Session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: New session 5 of user root. Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind: Removed session 5. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Removed slice user-0.slice. Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd: Stopping user-0.slice. Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Starting Cleanup of Temporary Directories... Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd: Started Cleanup of Temporary Directories. Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb) Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb) <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------5185435FD38913B821CED967--

3 4