- Devel - oVirt List Archives

[ OST Failure Report ] [ oVirt master ] [ 2017-11-13 ] [add_master_storage_domain]
by Barak Korren 13 Nov '17

13 Nov '17

Test failed: [ add_master_storage_domain ] Link to suspected patches: - https://gerrit.ovirt.org/#/c/83849/4 This seems to be a fairly consistent regression as all patches that follow the patch above exhibit the same issue when tested, this list of similarly failing patches includes: - https://gerrit.ovirt.org/#/c/83940/2 - https://gerrit.ovirt.org/#/c/83608/7 - https://gerrit.ovirt.org/#/c/83815/3 The following patch is an exception because it simply failed to build: - https://gerrit.ovirt.org/#/c/83370/9 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3770/artifact… Error snippet from log: <error> Fault reason is "Operation Failed". Fault detail is "[Failed to attach Storage due to an error on the Data Center master Storage Domain. -Please activate the master Storage Domain first.]". HTTP response code is 409. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

3 8

failing check-patch on ovirt-engine
by Greg Sheremeta 13 Nov '17

13 Nov '17

Hi, I'm seeing a weird issue in ovirt-engine check-patch. There's no way this failure is related to my patch [ https://gerrit.ovirt.org/#/c/83950/] which is a 1 character typo fix in the constants file. Anyone have any ideas or also seeing this? It seems random, too -- sometimes 1 or 2 of the jobs will succeed [el7, fc25] but the other will fail. Etc. *14:48:08* + automation/packaging-setup-tests.sh*14:48:08* + trap popd 0*14:48:08* +++ readlink -f automation/packaging-setup-tests.sh*14:48:08* ++ dirname /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/packaging-setup-tests.sh*14:48:08* + pushd /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/automation/..*14:48:08* ~ ~*14:48:08* + export PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + PYTHONPATH=packaging/pythonlib:packaging/setup:*14:48:08* + python -m pytest packaging/setup*14:48:08* ============================= test session starts ==============================*14:48:08* platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.4.34, pluggy-0.4.0*14:48:08* rootdir: /home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine, inifile:*14:48:08* collected 0 items / 1 errors*14:48:08* *14:48:08* ==================================== ERRORS ====================================*14:48:08* ERROR collecting packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py *14:48:08* ImportError while importing test module '/home/jenkins/workspace/ovirt-engine_master_check-patch-fc26-x86_64/ovirt-engine/packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py'.*14:48:08* Hint: make sure your test modules/packages have valid Python names.*14:48:08* Traceback:*14:48:08* packaging/setup/tests/ovirt_engine_setup/engine_common/test_database.py:19: in <module>*14:48:08* import ovirt_engine_setup.engine_common.database as under_test # isort:skip # noqa: E402*14:48:08* packaging/setup/ovirt_engine_setup/engine_common/database.py:30: in <module>*14:48:08* from otopi import base*14:48:08* E ImportError: No module named otopi*14:48:08* !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!*14:48:08* =========================== 1 error in 0.33 seconds ============================*14:48:08* + popd*14:48:08* ~ ---------- Forwarded message ---------- From: Code Review <gerrit(a)ovirt.org> Date: Sun, Nov 12, 2017 at 10:31 AM Subject: Change in ovirt-engine[master]: engine: typo fix snaphot -> snapshot To: Greg Sheremeta <gshereme(a)redhat.com> Jenkins CI *posted comments* on this change. View Change <https://gerrit.ovirt.org/83950> Patch set 1:Continuous-Integration -1 Build Failed http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc25-x86_64/17231/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-fc26-x86_64/157/ : FAILURE http://jenkins.ovirt.org/job/ovirt-engine_master_check- patch-el7-x86_64/33193/ : SUCCESS To view, visit change 83950 <https://gerrit.ovirt.org/83950>. To unsubscribe, visit settings <https://gerrit.ovirt.org/settings>. Gerrit-Project: ovirt-engine Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b4d75855c0abe2f0eddd8b282c172205db2e42 Gerrit-Change-Number: 83950 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Alexander Wels <awels(a)redhat.com> Gerrit-Reviewer: Greg Sheremeta <gshereme(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Tal Nisan <tnisan(a)redhat.com> Gerrit-Reviewer: Vojtech Szocs <vszocs(a)redhat.com> Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org> Gerrit-Comment-Date: Sun, 12 Nov 2017 15:31:27 +0000 Gerrit-HasComments: No -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>

3 2

oVirt 4.1.9 will be the last 4.1 release
by Sandro Bonazzola 13 Nov '17

13 Nov '17

Hi, we are now targeting oVirt 4.2.0 GA around December 12th and oVirt 4.1.8 around December 5th. Against tradition that would have set 4.1.8 as last 4.1 release, this round we'll release also oVIrt 4.1.9 being it already in development just before 4.2.0 GA. We suggest to be prepared to upgrade to 4.2 since 4.1 won't receive any further update after 4.1.9 release. -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Forking 'ansible' into oVirt org on GitHub
by Barak Korren 13 Nov '17

13 Nov '17

Guys, I'm not sure who did that, but somebody forked the core 'ansible' repo in to the ovirt organisation on GitHub. IMO this does not look good. Why was this done? -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

[ FIXED ] Missing release RPMs in resources.ovirt.org/pub/yum-repo/
by Daniel Belenky 12 Nov '17

12 Nov '17

Hi all, Some of you might have encountered an error while trying to download the latest ovirt release (4.1/4.1/master) from resources.ovirt.org/yum-repo/ During the weekend we had an issue with our nightly publisher's job. Those jobs are responsible to move artifacts from Jenkins to the matching snapshot repo in resources.ovirt.org. Since the publishers failed, ovirt-release-master.rpm and ovirt-release-4.x-snapshot.rpms (which are symlinks to ovirt-${version}-snapshot) were unavailable. The nightly publisher is fixed now and, the rpms were properly deployed to the repo. Sorry for any inconvenience, -- DANIEL BELENKY RHV DEVOPS EMEA VIRTUALIZATION R&D <https://red.ht/sig>

3 2

[ OST Failure Report ] [ oVirt 4.1 ] [ 09-11-2017 ] [ 001_upgrade_engine.test_initialize_engine ]
by Dafna Ron 09 Nov '17

09 Nov '17

This is a multi-part message in MIME format. --------------1E382631B1CFE8CFC947FDB5 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had a failure in installing engine with a dependency error between ovirt-engine-backend and vdsm-jsonrpc-java Please note that we have other issues which are not related to this patch so it might be hard to find this issue. I also added the lago log where you can find the package dependency error. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/82995/* * Link to Job:http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291 * http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex… * * *Link to all logs: http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact* * (Relevant) error snippet from the log: <error> * [ INFO ] Checking for an update for Setup... [ ERROR ] Yum [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Yum Performing yum transaction rollback [ ERROR ] Failed to execute stage 'Environment customization': [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20171109042111-tpncix.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20171109042115-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed ** *</error>* ** --------------1E382631B1CFE8CFC947FDB5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had a failure in installing engine with a dependency error between ovirt-engine-backend and vdsm-jsonrpc-java Please note that we have other issues which are not related to this patch so it might be hard to find this issue. I also added the lago log where you can find the package dependency error. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82995/">https://gerrit.ovirt.org/#/c/82995/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job:<a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291</a> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex…">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact/ex…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1291/artifact</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">[ INFO ] Checking for an update for Setup... [ ERROR ] Yum [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Yum Performing yum transaction rollback [ ERROR ] Failed to execute stage 'Environment customization': [u'ovirt-engine-backend-4.1.7.7-0.0.master.20171109084643.gitcd8d74c.el7.centos.noarch requires vdsm-jsonrpc-java >= 1.3.14'] [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20171109042111-tpncix.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20171109042115-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed </pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------1E382631B1CFE8CFC947FDB5--

3 3

VM start fails due to missing disk
by Viktor Mihajlovski 08 Nov '17

08 Nov '17

Hi, I ran into a problem after building ovirt-engine from current master to verify my last outstanding patch: but now I can't start my VM anymore, because the <disk> element is omitted from the final libvirt XML created by vdsm. The xml argument sent by the engine for the VM.create call contains a valid disk definition, but it disappears from the final libvirt XML. The preceding prepareImage call returns a seemingly valid path info... I am currently at loss and would appreciate any hints. -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski

1 1

Infra related events message - review needed
by Dafna Ron 08 Nov '17

08 Nov '17

This is a multi-part message in MIME format. --------------C2C1B6C457A9CCD4A0E14F7B Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We had some confusion last week when I sent a message about an infrastructure related issue we have found and people thought I was reporting an error on OST so I decided that perhaps it would be a good idea to create a general message that would alert people on an on-going infra event and not just OST related issues. iBelow is the format I was thinking of using: ############################################## Subject: Ovirt CI - [Infrastructure Incident] [$Date] [$TIME] Hello, We encountered an incident that can suggest there is an ongoing infrastructure incident. Our team is currently investigating the issue and we will send a mail once we have more information or the issue is resolved. Issue summary: ################################################ Please let me know what you're ideas on this are and if the message is at all lacking in some way. Thanks. Dafna --------------C2C1B6C457A9CCD4A0E14F7B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We had some confusion last week when I sent a message about an infrastructure related issue we have found and people thought I was reporting an error on OST so I decided that perhaps it would be a good idea to create a general message that would alert people on an on-going infra event and not just OST related issues. iBelow is the format I was thinking of using: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">############################################## <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Subject: Ovirt CI - [Infrastructure Incident] [$Date] [$TIME] <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Hello, <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">We encountered an incident that can suggest there is an ongoing infrastructure incident. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Our team is currently investigating the issue and we will send a mail once we have more information or the issue is resolved. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Issue <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">summary<span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">################################################ Please let me know what you're ideas on this are and if the message is at all lacking in some way. Thanks. Dafna </body> </html> --------------C2C1B6C457A9CCD4A0E14F7B--

2 1

[ OST Failure Report ] [ oVirt master ] [ 07-11-2017 ] [ 004_basic_sanity.disk_operations ]
by Dafna Ron 08 Nov '17

08 Nov '17

This is a multi-part message in MIME format. --------------106EB129C1BC1AEAACE8C19F Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in test 004_basic_sanity.disk_operations. >From the log I can see that vdsm has a KeyError: 'diskReplicate' and than reports the drive not found. Can you please check the issue? ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/83641/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/ engine log: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact… vdsm log: ** http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact… (Relevant) error snippet from the log: <error> >From engine log: * 2017-11-07 09:20:42,396-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Failed in 'VmReplicateDiskFinishVDS' method 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-basic-suite-mas ter-host-0 command VmReplicateDiskFinishVDS failed: Drive image file could not be found 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDisk FinishVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]' 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] HostName = lago-basic-suite-master-host-0 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'VmReplicateDiskFinishVDSCommand(HostName = lago-basic-sui te-master-host-0, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c9 1d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'})' execution failed: VDSGenericException: VDSErrorException: Fai led to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 2017-11-07 09:20:42,399-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorExcep tion: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand.executeVdsBrokerCommand(VmReplicateDiskFinishVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor244.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskFinish(LiveMigrateDiskCommand.java:374) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.handleDestDisk(LiveMigrateDiskCommand.java:681) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.endWithFailure(LiveMigrateDiskCommand.java:337) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:783) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:712) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1989) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:574) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:125) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:156) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:65) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:141) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:106) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2017-11-07 09:20:42,400-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] FINISH, VmReplicateDiskFinishVDSCommand, log id: 32c5a56e 2017-11-07 09:20:42,400-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: runVdsCommand, params: [VmReplicateDiskFinish, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'}], timeElapsed: 12ms 2017-11-07 09:20:42,400-05 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Replication end of disk '36837a9b-6396-4d65-a354-df97ce808f01' in vm '61d328bc-c920-4691-997f-09bbecb5de10' back to the source failed, skipping deletion of the target disk 2017-11-07 09:20:42,408-05 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Trying to release exclusive lock which does not exist, lock key: '61d328bc-c920-4691-997f-09bbecb5de10LIVE_STORAGE_MIGRATION' 2017-11-07 09:20:42,408-05 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Lock freed to object 'EngineLock:{exclusiveLocks='[61d328bc-c920-4691-997f-09bbecb5de10=LIVE_STORAGE_MIGRATION]', sharedLocks=''}' 2017-11-07 09:20:42,428-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: get, params: [038b0c9a-f64c-4aa7-8e36-72e435771dab], timeElapsed: 2ms 2017-11-07 09:20:42,430-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk vm0_disk0 to domain iscsi.** ** * * * * vdsm log: 2017-11-07 09:20:41,178-0500 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeSize return={'truesize': '2415919104', 'apparentsize': '2415919104'} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e, task_id=042481eb-74ac-42a8-9d1e-8fb1c84db234 (api:52) 2017-11-07 09:20:41,193-0500 ERROR (jsonrpc/6) [api] FINISH diskReplicateFinish error='diskReplicate' (api:127) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 558, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4477, in diskReplicateFinish self._delDiskReplica(drive) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4528, in _delDiskReplica del dev['diskReplicate'] KeyError: 'diskReplicate' 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'General Exception: ("\'diskReplicate\'",)', 'code': 100}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 100) in 0.79 seconds (__init__:630) 2017-11-07 09:20:41,253-0500 INFO (jsonrpc/2) [vdsm.api] START finalizeMerge(spUUID=u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', subchainInfo={u'img_id': u'8401cd0f-419c-417c-8da3-52da285fa47c', u'sd_id': u'4a088fb2-5497-4b4d-adec-4c91d998c5 6e', u'top_id': u'7a295270-fdf5-4561-b688-033c5d312d1a', u'base_id': u'e77f8714-9e0f-4f39-8437-e75a3fa44c74'}) from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (ap i:46) 2017-11-07 09:20:41,267-0500 INFO (jsonrpc/2) [vdsm.api] FINISH finalizeMerge return=None from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (api:52) 2017-11-07 09:20:41,285-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.finalizeMerge succeeded in 0.03 seconds (__init__:630) 2017-11-07 09:20:41,285-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] START task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x2f0a560>>, args=None) (threadPool:208) 2017-11-07 09:20:41,295-0500 INFO (tasks/8) [storage.merge] Finalizing subchain after merge: <SubchainInfo sd_id=4a088fb2-5497-4b4d-adec-4c91d998c56e, img_id=8401cd0f-419c-417c-8da3-52da285fa47c, top_id=7a295270-fdf5-4561-b688-033c5d312d 1a, base_id=e77f8714-9e0f-4f39-8437-e75a3fa44c74 base_generation=None at 0x30a7990> (merge:254) 2017-11-07 09:20:41,332-0500 INFO (tasks/8) [storage.SANLock] Acquiring Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:377) 2017-11-07 09:20:41,345-0500 INFO (tasks/8) [storage.SANLock] Successfully acquired Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:415) 2017-11-07 09:20:41,372-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/e77f8714-9e0f-4f39-8437-e75a3fa44c74 (volume:558) 2017-11-07 09:20:41,377-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/7a295270-fdf5-4561-b688-033c5d312d1a (volume:558) 2017-11-07 09:20:41,395-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (volume:558) 2017-11-07 09:20:41,399-0500 INFO (tasks/8) [storage.VolumeManifest] sdUUID=4a088fb2-5497-4b4d-adec-4c91d998c56e imgUUID=8401cd0f-419c-417c-8da3-52da285fa47c volUUID = 7a295270-fdf5-4561-b688-033c5d312d1a legality = ILLEGAL (volume:399) 2017-11-07 09:20:41,667-0500 INFO (tasks/8) [storage.merge] Updating Vdsm metadata, syncing new chain: ['e77f8714-9e0f-4f39-8437-e75a3fa44c74', u'a6cb7a60-e500-4a75-9da8-9e7cce3e21ac'] (merge:311) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Current chain=e77f8714-9e0f-4f39-8437-e75a3fa44c74 < 7a295270-fdf5-4561-b688-033c5d312d1a < a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (top) (image:1162) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Unlinking subchain: ['7a295270-fdf5-4561-b688-033c5d312d1a'] (image:1172) 2017-11-07 09:20:41,709-0500 INFO (tasks/8) [storage.Image] Setting parent of volume a6cb7a60-e500-4a75-9da8-9e7cce3e21ac to e77f8714-9e0f-4f39-8437-e75a3fa44c74 (image:1185) 2017-11-07 09:20:41,824-0500 INFO (tasks/8) [storage.SANLock] Releasing Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:435) 2017-11-07 09:20:41,828-0500 INFO (tasks/8) [storage.SANLock] Successfully released Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:444) 2017-11-07 09:20:41,842-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] FINISH task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (threadPool:210) 2017-11-07 09:20:42,388-0500 INFO (jsonrpc/4) [api.virt] START diskReplicateFinish(srcDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u'4a08 8fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}, dstDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u '4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:46) 2017-11-07 09:20:42,388-0500 ERROR (jsonrpc/4) [virt.vm] (vmId='61d328bc-c920-4691-997f-09bbecb5de10') Drive not found (srcDisk: {u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f 25-c1184c10841a', u'domainID': u'4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) (vm:4386) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 13) in 0.01 seconds (__init__:630)** : * * *</error>* ** --------------106EB129C1BC1AEAACE8C19F Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We have a failure in test 004_basic_sanity.disk_operations. From the log I can see that vdsm has a KeyError: 'diskReplicate' and than reports the drive not found. Can you please check the issue? <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83641/">https://gerrit.ovirt.org/#/c/83641/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">engine log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">vdsm log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3653/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> From engine log: 2017-11-07 09:20:42,396-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Failed in 'VmReplicateDiskFinishVDS' method 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lago-basic-suite-mas ter-host-0 command VmReplicateDiskFinishVDS failed: Drive image file could not be found 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDisk FinishVDSCommand' return value 'StatusOnlyReturn [status=Status [code=13, message=Drive image file could not be found]]' 2017-11-07 09:20:42,399-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] HostName = lago-basic-suite-master-host-0 2017-11-07 09:20:42,399-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Command 'VmReplicateDiskFinishVDSCommand(HostName = lago-basic-sui te-master-host-0, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c9 1d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'})' execution failed: VDSGenericException: VDSErrorException: Fai led to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 2017-11-07 09:20:42,399-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorExcep tion: VDSGenericException: VDSErrorException: Failed to VmReplicateDiskFinishVDS, error = Drive image file could not be found, code = 13 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:81) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:223) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:193) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand.executeVdsBrokerCommand(VmReplicateDiskFinishVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:112) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:387) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor244.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_151] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) [vdsbroker.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.replicateDiskFinish(LiveMigrateDiskCommand.java:374) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.handleDestDisk(LiveMigrateDiskCommand.java:681) [bll.jar:] at org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand.endWithFailure(LiveMigrateDiskCommand.java:337) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:783) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:712) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1989) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:574) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:125) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:156) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:65) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:141) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:106) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_151] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_151] : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_151] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_151] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) 2017-11-07 09:20:42,400-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.VmReplicateDiskFinishVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] FINISH, VmReplicateDiskFinishVDSCommand, log id: 32c5a56e 2017-11-07 09:20:42,400-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: runVdsCommand, params: [VmReplicateDiskFinish, VmReplicateDiskParameters:{hostId='038b0c9a-f64c-4aa7-8e36-72e435771dab', vmId='61d328bc-c920-4691-997f-09bbecb5de10', storagePoolId='b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', srcStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', targetStorageDomainId='4a088fb2-5497-4b4d-adec-4c91d998c56e', imageGroupId='36837a9b-6396-4d65-a354-df97ce808f01', imageId='8a1660d7-f153-4aca-9f25-c1184c10841a'}], timeElapsed: 12ms 2017-11-07 09:20:42,400-05 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Replication end of disk '36837a9b-6396-4d65-a354-df97ce808f01' in vm '61d328bc-c920-4691-997f-09bbecb5de10' back to the source failed, skipping deletion of the target disk 2017-11-07 09:20:42,408-05 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Trying to release exclusive lock which does not exist, lock key: '61d328bc-c920-4691-997f-09bbecb5de10LIVE_STORAGE_MIGRATION' 2017-11-07 09:20:42,408-05 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] Lock freed to object 'EngineLock:{exclusiveLocks='[61d328bc-c920-4691-997f-09bbecb5de10=LIVE_STORAGE_MIGRATION]', sharedLocks=''}' 2017-11-07 09:20:42,428-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] method: get, params: [038b0c9a-f64c-4aa7-8e36-72e435771dab], timeElapsed: 2ms 2017-11-07 09:20:42,430-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [4bcdd94e] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), User admin@internal-authz have failed to move disk vm0_disk0 to domain iscsi. vdsm log: 2017-11-07 09:20:41,178-0500 INFO (jsonrpc/6) [vdsm.api] FINISH getVolumeSize return={'truesize': '2415919104', 'apparentsize': '2415919104'} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e, task_id=042481eb-74ac-42a8-9d1e-8fb1c84db234 (api:52) 2017-11-07 09:20:41,193-0500 ERROR (jsonrpc/6) [api] FINISH diskReplicateFinish error='diskReplicate' (api:127) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 558, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4477, in diskReplicateFinish self._delDiskReplica(drive) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4528, in _delDiskReplica del dev['diskReplicate'] KeyError: 'diskReplicate' 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'General Exception: ("\'diskReplicate\'",)', 'code': 100}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:41,195-0500 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 100) in 0.79 seconds (__init__:630) 2017-11-07 09:20:41,253-0500 INFO (jsonrpc/2) [vdsm.api] START finalizeMerge(spUUID=u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', subchainInfo={u'img_id': u'8401cd0f-419c-417c-8da3-52da285fa47c', u'sd_id': u'4a088fb2-5497-4b4d-adec-4c91d998c5 6e', u'top_id': u'7a295270-fdf5-4561-b688-033c5d312d1a', u'base_id': u'e77f8714-9e0f-4f39-8437-e75a3fa44c74'}) from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (ap i:46) 2017-11-07 09:20:41,267-0500 INFO (jsonrpc/2) [vdsm.api] FINISH finalizeMerge return=None from=::ffff:192.168.204.4,49380, flow_id=56c9f051-c091-49cd-9fd5-cc98e92ef05f, task_id=9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (api:52) 2017-11-07 09:20:41,285-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.finalizeMerge succeeded in 0.03 seconds (__init__:630) 2017-11-07 09:20:41,285-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] START task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x2f0a560>>, args=None) (threadPool:208) 2017-11-07 09:20:41,295-0500 INFO (tasks/8) [storage.merge] Finalizing subchain after merge: <SubchainInfo sd_id=4a088fb2-5497-4b4d-adec-4c91d998c56e, img_id=8401cd0f-419c-417c-8da3-52da285fa47c, top_id=7a295270-fdf5-4561-b688-033c5d312d 1a, base_id=e77f8714-9e0f-4f39-8437-e75a3fa44c74 base_generation=None at 0x30a7990> (merge:254) 2017-11-07 09:20:41,332-0500 INFO (tasks/8) [storage.SANLock] Acquiring Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:377) 2017-11-07 09:20:41,345-0500 INFO (tasks/8) [storage.SANLock] Successfully acquired Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) for host id 1 (clusterlock:415) 2017-11-07 09:20:41,372-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/e77f8714-9e0f-4f39-8437-e75a3fa44c74 (volume:558) 2017-11-07 09:20:41,377-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/7a295270-fdf5-4561-b688-033c5d312d1a (volume:558) 2017-11-07 09:20:41,395-0500 INFO (tasks/8) [storage.VolumeManifest] Volume: preparing volume 4a088fb2-5497-4b4d-adec-4c91d998c56e/a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (volume:558) 2017-11-07 09:20:41,399-0500 INFO (tasks/8) [storage.VolumeManifest] sdUUID=4a088fb2-5497-4b4d-adec-4c91d998c56e imgUUID=8401cd0f-419c-417c-8da3-52da285fa47c volUUID = 7a295270-fdf5-4561-b688-033c5d312d1a legality = ILLEGAL (volume:399) 2017-11-07 09:20:41,667-0500 INFO (tasks/8) [storage.merge] Updating Vdsm metadata, syncing new chain: ['e77f8714-9e0f-4f39-8437-e75a3fa44c74', u'a6cb7a60-e500-4a75-9da8-9e7cce3e21ac'] (merge:311) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Current chain=e77f8714-9e0f-4f39-8437-e75a3fa44c74 < 7a295270-fdf5-4561-b688-033c5d312d1a < a6cb7a60-e500-4a75-9da8-9e7cce3e21ac (top) (image:1162) 2017-11-07 09:20:41,690-0500 INFO (tasks/8) [storage.Image] Unlinking subchain: ['7a295270-fdf5-4561-b688-033c5d312d1a'] (image:1172) 2017-11-07 09:20:41,709-0500 INFO (tasks/8) [storage.Image] Setting parent of volume a6cb7a60-e500-4a75-9da8-9e7cce3e21ac to e77f8714-9e0f-4f39-8437-e75a3fa44c74 (image:1185) 2017-11-07 09:20:41,824-0500 INFO (tasks/8) [storage.SANLock] Releasing Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/images/8401cd0 f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:435) 2017-11-07 09:20:41,828-0500 INFO (tasks/8) [storage.SANLock] Successfully released Lease(name='e77f8714-9e0f-4f39-8437-e75a3fa44c74', path=u'/rhev/data-center/mnt/192.168.204.4:_exports_nfs_share1/4a088fb2-5497-4b4d-adec-4c91d998c56e/im ages/8401cd0f-419c-417c-8da3-52da285fa47c/e77f8714-9e0f-4f39-8437-e75a3fa44c74.lease', offset=0) (clusterlock:444) 2017-11-07 09:20:41,842-0500 INFO (tasks/8) [storage.ThreadPool.WorkerThread] FINISH task 9ce348bc-e3fb-4f4e-aea0-f8c79e7ae204 (threadPool:210) 2017-11-07 09:20:42,388-0500 INFO (jsonrpc/4) [api.virt] START diskReplicateFinish(srcDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u'4a08 8fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}, dstDisk={u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f25-c1184c10841a', u'domainID': u '4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:46) 2017-11-07 09:20:42,388-0500 ERROR (jsonrpc/4) [virt.vm] (vmId='61d328bc-c920-4691-997f-09bbecb5de10') Drive not found (srcDisk: {u'device': u'disk', u'poolID': u'b20f0eb7-6bea-4994-b87b-e1080c4fd9e5', u'volumeID': u'8a1660d7-f153-4aca-9f 25-c1184c10841a', u'domainID': u'4a088fb2-5497-4b4d-adec-4c91d998c56e', u'imageID': u'36837a9b-6396-4d65-a354-df97ce808f01'}) (vm:4386) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [api.virt] FINISH diskReplicateFinish return={'status': {'message': 'Drive image file could not be found', 'code': 13}} from=::ffff:192.168.204.4,49358, flow_id=4bcdd94e (api:52) 2017-11-07 09:20:42,389-0500 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 13) in 0.01 seconds (__init__:630) : <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------106EB129C1BC1AEAACE8C19F--

2 2

[ OST Failure Report ] [ oVirt Master ] [ 06-11-2017 ] [ 002_bootstrap.verify_add_hosts ]
by Dafna Ron 07 Nov '17

07 Nov '17

This is a multi-part message in MIME format. --------------88DD9FCD3B3DDEA34D4EAEF4 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We failed test 002_bootstrap.verify_add_hosts I can see we only tried to install one of the hosts (host-0) and failed. the second host has no log which means we did not try to deploy it. The error suggests that we ovirt-imageio-daemon failed to start. However, there is another message that I think should be addressed about conflicting vdsm and libvirt configurations. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/83612/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/ http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact… * * *(Relevant) error snippet from the log: * * <error> \* 2017-11-06 02:56:46,526-0500 DEBUG otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921 execute-output: ('/usr/bin/vdsm-tool', 'configure', '--force') stdout: Checking configuration status... abrt is not configured for vdsm WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration lvm requires configuration libvirt is not configured for vdsm yet FAILED: conflicting vdsm and libvirt-qemu tls configuration. vdsm.conf with ssl=True requires the following changes: libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 qemu.conf: spice_tls=1. multipath requires configuration 2017-11-06 02:56:47,551-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stderr: Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details. 2017-11-06 02:56:47,552-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-R4R8gZhaQI/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 179, in _start self.services.state('ovirt-imageio-daemon', True) File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'ovirt-imageio-daemon' 2017-11-06 02:56:47,553-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon' ** *</error>* * * --------------88DD9FCD3B3DDEA34D4EAEF4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, We failed test 002_bootstrap.verify_add_hosts I can see we only tried to install one of the hosts (host-0) and failed. the second host has no log which means we did not try to deploy it. The error suggests that we ovirt-imageio-daemon failed to start. However, there is another message that I think should be addressed about conflicting vdsm and libvirt configurations. <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/83612/">https://gerrit.ovirt.org/#/c/83612/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact/</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"> <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact…">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3626/artifact…</a> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error> \ <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-06 02:56:46,526-0500 DEBUG otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921 execute-output: ('/usr/bin/vdsm-tool', 'configure', '--force') stdout: Checking configuration status... abrt is not configured for vdsm WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration lvm requires configuration libvirt is not configured for vdsm yet FAILED: conflicting vdsm and libvirt-qemu tls configuration. vdsm.conf with ssl=True requires the following changes: libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 qemu.conf: spice_tls=1. multipath requires configuration </pre> <pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;">2017-11-06 02:56:47,551-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stderr: Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details. 2017-11-06 02:56:47,552-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/tmp/ovirt-R4R8gZhaQI/pythonlib/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 179, in _start self.services.state('ovirt-imageio-daemon', True) File "/tmp/ovirt-R4R8gZhaQI/otopi-plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'ovirt-imageio-daemon' 2017-11-06 02:56:47,553-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon'</pre> <span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error> </body> </html> --------------88DD9FCD3B3DDEA34D4EAEF4--

5 7