------=_Part_9128865_1887831902.1433250537047 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Indeed.=20 This fix will be released in oVirt 3.5.4=20 ----- Original Message -----
From: "Tim Macy" <macytd@gmail.com> To: users@ovirt.org Sent: Friday, May 1, 2015 5:59:41 AM Subject: Re: [ovirt-users] 3.5.2 live merge
Seems very similar to this bug - https://bugzilla.redhat.com/show_bug.cgi?id=3D1213157
On Thu, Apr 30, 2015 at 3:54 PM, Patrick Russell < patrick_russell@volusion.com > wrote:
Our packages in my earlier email were incorrect. Here are the correct packages and some more logs: =20
# rpm -qa |grep ovirt =20 ovirt-engine-cli-3.5.0.5-1.el6.noarch =20 ovirt-engine-setup-plugin-ovirt-engine-3.5.2-1.el6.noarch =20 ovirt-engine-setup-plugin-allinone-3.5.2-1.el6.noarch =20 ovirt-engine-setup-3.5.2-1.el6.noarch =20 ovirt-engine-dbscripts-3.5.2-1.el6.noarch =20 ovirt-engine-backend-3.5.2-1.el6.noarch =20 ovirt-guest-tools-3.5.0-0.5.master.noarch =20 ovirt-host-deploy-1.3.1-1.el6.noarch =20 ovirt-engine-sdk-python-3.5.2.1-1.el6.noarch =20 ovirt-engine-setup-plugin-ovirt-engine-common-3.5.2-1.el6.noarch =20 ovirt-engine-restapi-3.5.2-1.el6.noarch =20 ovirt-engine-userportal-3.5.2-1.el6.noarch =20 ovirt-engine-3.5.2-1.el6.noarch =20 ovirt-host-deploy-offline-1.3.1-1.el6.x86_64 =20 ovirt-engine-setup-plugin-websocket-proxy-3.5.2-1.el6.noarch =20 ovirt-engine-websocket-proxy-3.5.2-1.el6.noarch =20 ovirt-iso-uploader-3.5.2-1.el6.noarch =20 ovirt-engine-extensions-api-impl-3.5.2-1.el6.noarch =20 ovirt-engine-jboss-as-7.1.1-1.el6.x86_64 =20 ovirt-guest-tools-iso-3.5-7.noarch =20 ovirt-image-uploader-3.5.1-1.el6.noarch =20 ovirt-engine-lib-3.5.2-1.el6.noarch =20 ovirt-engine-setup-base-3.5.2-1.el6.noarch =20 ovirt-release35-003-1.noarch =20 ovirt-engine-tools-3.5.2-1.el6.noarch =20 ovirt-engine-webadmin-portal-3.5.2-1.el6.noarch =20 ovirt-host-deploy-java-1.3.1-1.el6.noarch =20
Logs: =20
[4cbab1c6] Lock freed to object EngineLock [exclusiveLocks=3D key: f2afb8c7-6b1d-4822-a43a-0889178de719 value: VM =20 , sharedLocks=3D ] =20 2015-04-30 19:50:24,762 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (pool-7-thread-7) [598dbb1e] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected := ID: 00000000-0000-0000-0000-000000000000 Type: Storage =20 2015-04-30 19:50:24,771 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (pool-7-thread-8) [5c8285eb] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected := ID: 00000000-0000-0000-0000-000000000000 Type: Storage =20 2015-04-30 19:50:24,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: 4cbab1c6, Job ID: b0057939-d15a-4da7-b782-c835127a1413, Call Stack: null, Custom Event ID= : -1, Message: Snapshot 'test_snap1' deletion for VM '3.5.2.wintest1' was initiated by patrick_russell@corp.volusion.com . =20 2015-04-30 19:50:29,642 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-13) [5c8285eb] Executing Live Merge comm= and step MERGE =20 2015-04-30 19:50:29,665 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-13) Waiting on Live Merge child commands= to complete =20 2015-04-30 19:50:29,671 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-9) [e10f6a3] Running command: MergeCommand internal: tru= e. Entities affected : ID: 58ef50e1-2efb-4081-a3ca-493c2ee7e7b6 Type: Stor= age =20 2015-04-30 19:50:29,679 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-13) [598dbb1e] Executing Live Merge comm= and step MERGE =20 2015-04-30 19:50:29,697 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [4898b008] Running command: MergeCommand internal: t= rue. Entities affected : ID: 58ef50e1-2efb-4081-a3ca-493c2ee7e7b6 Type: Stor= age =20 2015-04-30 19:50:29,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] START, MergeVDSCommand(HostName =3D aus02gdkvm01.dev.volusion.com , MergeVDSCommandParameters{HostId =3D cfc227ef-c9ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, storagePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3Dbb8359c6-1dba-45e2-97cb-929d1c83a7c8, imageId=3D87f26ef2-6a84-4f7c-9771-be76650c703c, baseImageId=3D3f90ced5-58ea-43cc-955a-559e8d89fbb2, topImageId=3D87f26ef2-6a84-4f7c-9771-be76650c703c, bandwidth=3D0}), log= id: 5859dc0 =20 2015-04-30 19:50:29,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] START, MergeVDSCommand(HostName =3D aus02gdkvm01.dev.volusion.com , MergeVDSCommandParameters{HostId =3D cfc227ef-c9ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, storagePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3D99e44df8-fa01-4255-bc1a-a4c240eb01e2, imageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3ed4129, baseImageId=3D7ad1cf7d-f2e6-41d4-a3c4-57458e80c43d, topImageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3ed4129, bandwidth=3D0}), log= id: 160b45fb =20 2015-04-30 19:50:29,819 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Failed in MergeVDS method =20 2015-04-30 19:50:29,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand return value =20 StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52, mMessage=3DMerge failed]] =20 2015-04-30 19:50:29,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] HostName =3D aus02gdkvm01.dev.volusion.com =20 2015-04-30 19:50:29,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Command MergeVDSCommand(HostName =3D aus02gdkvm01.dev.volusion.com , MergeVDSCommandParameters{HostId =3D cfc227ef-c9ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, storagePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3Dbb8359c6-1dba-45e2-97cb-929d1c83a7c8, imageId=3D87f26ef2-6a84-4f7c-9771-be76650c703c, baseImageId=3D3f90ced5-58ea-43cc-955a-559e8d89fbb2, topImageId=3D87f26ef2-6a84-4f7c-9771-be76650c703c, bandwidth=3D0}) exec= ution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D Merge failed, code =3D= 52 =20 2015-04-30 19:50:29,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] FINISH, MergeVDSCommand, log id: 5859dc0 =20 2015-04-30 19:50:29,828 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-9) [e10f6a3] Command org.ovirt.engine.core.bll.MergeComm= and throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D M= erge failed, code =3D 52 (Failed with error mergeErr and code 52) =20 2015-04-30 19:50:29,834 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-9) [e10f6a3] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeCommand. =20 2015-04-30 19:50:29,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] Failed in MergeVDS method =20 2015-04-30 19:50:29,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] Command org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand return value =20 StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52, mMessage=3DMerge failed]] =20 2015-04-30 19:50:29,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] HostName =3D aus02gdkvm01.dev.volusion.co= m =20 2015-04-30 19:50:29,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] Command MergeVDSCommand(HostName =3D aus02gdkvm01.dev.volusion.com , MergeVDSCommandParameters{HostId =3D cfc227ef-c9ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, storagePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3D99e44df8-fa01-4255-bc1a-a4c240eb01e2, imageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3ed4129, baseImageId=3D7ad1cf7d-f2e6-41d4-a3c4-57458e80c43d, topImageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3ed4129, bandwidth=3D0}) exec= ution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D Merge failed, code =3D= 52 =20 2015-04-30 19:50:29,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] FINISH, MergeVDSCommand, log id: 160b45fb =20 2015-04-30 19:50:29,865 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [4898b008] Command org.ovirt.engine.core.bll.MergeCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D M= erge failed, code =3D 52 (Failed with error mergeErr and code 52) =20 2015-04-30 19:50:29,871 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [4898b008] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeCommand. =20 2015-04-30 19:50:39,698 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-73) [5c8285eb] Failed child command stat= us for step MERGE =20 2015-04-30 19:50:39,701 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-73) [5c8285eb] Waiting on Live Merge chi= ld commands to complete =20 2015-04-30 19:50:39,710 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-73) [598dbb1e] Failed child command stat= us for step MERGE =20 2015-04-30 19:50:49,720 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-36) [5c8285eb] Merging of snapshot 470b525e-fb30-4956-a702-f4c6389bcb60 images 3f90ced5-58ea-43cc-955a-559e8d89fbb2..87f26ef2-6a84-4f7c-9771-be76650c7= 03c failed. Images have been marked illegal and can no longer be previewed = or reverted to. Please retry Live Merge on the snapshot to complete the operation. =20 2015-04-30 19:50:49,725 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-36) [4cbab1c6] All Live Merge child comm= ands have completed, status FAILED =20 2015-04-30 19:50:49,734 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-36) [598dbb1e] Merging of snapshot 470b525e-fb30-4956-a702-f4c6389bcb60 images 7ad1cf7d-f2e6-41d4-a3c4-57458e80c43d..2ca5b861-52a8-4c0f-92a2-a8f0a3ed4= 129 failed. Images have been marked illegal and can no longer be previewed = or reverted to. Please retry Live Merge on the snapshot to complete the operation. =20 2015-04-30 19:50:59,787 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (DefaultQuartzScheduler_Worker-16) [4cbab1c6] Ending command with failu= re: org.ovirt.engine.core.bll.RemoveSnapshotCommand =20 2015-04-30 19:50:59,837 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4cbab1c6] Correlation ID: 4cbab1c6, Call Stack: null, Custom Event ID: -1, Message: Failed to delete snapshot 'test_snap1' for VM '3.5.2.wintest1=E2=80=99. =20
[4cbab1c6] Running command: RemoveSnapshotCommand internal: false. Enti= ties affected : ID: f2afb8c7-6b1d-4822-a43a-0889178de719 Type: VMAction grou= p MANIPULATE_VM_SNAPSHOTS with role type USER =20 2015-04-30 19:50:24,723 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-=
[4cbab1c6] Lock Acquired to object EngineLock [exclusiveLocks=3D key: f2afb8c7-6b1d-4822-a43a-0889178de719 value: VM =20 , sharedLocks=3D ] =20 2015-04-30 19:50:24,689 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-=
2015-04-30 19:50:24,578 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (ajp--127.0.0.1-8702-=
On Apr 30, 2015, at 9:54 AM, Patrick Russell < patrick_russell@volusion.com
wrote: =20 =20
Hi everyone, =20 =20
We=E2=80=99re not seeing live merge working as of the 3.5.2 update. W= e=E2=80=99ve tested using fibre channel and NFS attached storage. Both throwing the same error code. Are other people seeing success with live-merge after the updat= e? =20 =20
Here=E2=80=99s the environment: =20 =20
Engine Running on CentOS 6x64 updated to 3.5.2 via yum update (standa= lone physical box, dual socket hex core + hyperthreading, 16GB memory) =20 =20
# rpm -qa |grep ovirt =20 =20 ovirt-engine-cli-3.5.0.5-1.el6.noarch =20 =20 ovirt-engine-3.5.1.1-1.el6.noarch =20 =20 ovirt-engine-setup-plugin-ovirt-engine-3.5.2-1.el6.noarch =20 =20 ovirt-engine-setup-plugin-allinone-3.5.2-1.el6.noarch =20 =20 ovirt-engine-setup-3.5.2-1.el6.noarch =20 =20 ovirt-guest-tools-3.5.0-0.5.master.noarch =20 =20 ovirt-host-deploy-1.3.1-1.el6.noarch =20 =20 ovirt-engine-sdk-python-3.5.2.1-1.el6.noarch =20 =20 ovirt-engine-setup-plugin-ovirt-engine-common-3.5.2-1.el6.noarch =20 =20 ovirt-engine-backend-3.5.1.1-1.el6.noarch =20 =20 ovirt-engine-userportal-3.5.1.1-1.el6.noarch =20 =20 ovirt-engine-dbscripts-3.5.1.1-1.el6.noarch =20 =20 ovirt-engine-tools-3.5.1.1-1.el6.noarch =20 =20 ovirt-host-deploy-offline-1.3.1-1.el6.x86_64 =20 =20 ovirt-engine-setup-plugin-websocket-proxy-3.5.2-1.el6.noarch =20 =20 ovirt-engine-websocket-proxy-3.5.2-1.el6.noarch =20 =20 ovirt-iso-uploader-3.5.2-1.el6.noarch =20 =20 ovirt-engine-extensions-api-impl-3.5.2-1.el6.noarch =20 =20 ovirt-engine-jboss-as-7.1.1-1.el6.x86_64 =20 =20 ovirt-engine-webadmin-portal-3.5.1.1-1.el6.noarch =20 =20 ovirt-engine-restapi-3.5.1.1-1.el6.noarch =20 =20 ovirt-guest-tools-iso-3.5-7.noarch =20 =20 ovirt-image-uploader-3.5.1-1.el6.noarch =20 =20 ovirt-engine-lib-3.5.2-1.el6.noarch =20 =20 ovirt-engine-setup-base-3.5.2-1.el6.noarch =20 =20 ovirt-release35-003-1.noarch =20 =20 ovirt-host-deploy-java-1.3.1-1.el6.noarch =20 =20
Hypervisors are running ovirt-node, upgraded from ISO : http://resources.ovirt.org/pub/ovirt-3.5/iso/ovirt-node/el7-3.5.2/ovi= rt-node-iso-3.5-0.999.201504280931.el7.centos.iso =20 =20
Here=E2=80=99s a snippet from the logs: =20 =20
2015-04-29 18:47:16,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [48eb0b1d] FINISH, MergeVDSCommand, log id: 5121ecc= 9 =20 =20 2015-04-29 18:47:16,947 ERROR [org.ovirt.engine.core.bll.MergeCommand= ] (pool-7-thread-2) [48eb0b1d] Command org.ovirt.engine.core.bll.MergeCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D= Merge failed, code =3D 52 (Failed with error mergeErr and code 52) =20 =20 2015-04-29 18:47:16,954 ERROR [org.ovirt.engine.core.bll.MergeCommand= ] (pool-7-thread-2) [48eb0b1d] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeCommand. =20 =20 2015-04-29 18:47:16,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-3) [5495bde7] Failed in MergeVDS method =20 =20 2015-04-29 18:47:16,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-3) [5495bde7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand return valu= e =20 =20 StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52, mMessage=3DMerge failed]] =20 =20
_______________________________________________ =20 =20 Users mailing list =20 =20 Users@ovirt.org =20 =20 http://lists.ovirt.org/mailman/listinfo/users =20 =20
_______________________________________________ =20 Users mailing list =20 Users@ovirt.org =20 http://lists.ovirt.org/mailman/listinfo/users =20
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
------=_Part_9128865_1887831902.1433250537047 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: times new roman,new york,times,serif= ; font-size: 12pt; color: #000000"><div>Indeed. <br></div><div>This fi= x will be released in oVirt 3.5.4</div><div><br></div><hr id=3D"zwchr"><blo= ckquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left= :5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;f= ont-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Tim Ma= cy" <macytd@gmail.com><br><b>To: </b>users@ovirt.org<br><b>Sent: </b>= Friday, May 1, 2015 5:59:41 AM<br><b>Subject: </b>Re: [ovirt-users] 3.5.2 l= ive merge<br><div><br></div><div dir=3D"ltr">Seems very similar to this bug= - <a href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1213157" t= arget=3D"_blank">https://bugzilla.redhat.com/show_bug.cgi?id=3D1213157</a>&= nbsp;</div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu= , Apr 30, 2015 at 3:54 PM, Patrick Russell <span dir=3D"ltr"><<a href=3D= "mailto:patrick_russell@volusion.com" target=3D"_blank">patrick_russell@vol= usion.com</a>></span> wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word"> Our packages in my earlier email were incorrect. Here are the correct packa= ges and some more logs: <div><br> </div> <div><span class=3D""> <div># rpm -qa |grep ovirt</div> <div>ovirt-engine-cli-3.5.0.5-1.el6.noarch</div> </span><div>ovirt-engine-setup-plugin-ovirt-engine-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-plugin-allinone-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-3.5.2-1.el6.noarch</div> <div>ovirt-engine-dbscripts-3.5.2-1.el6.noarch</div> <div>ovirt-engine-backend-3.5.2-1.el6.noarch</div> <div>ovirt-guest-tools-3.5.0-0.5.master.noarch</div> <div>ovirt-host-deploy-1.3.1-1.el6.noarch</div> <div>ovirt-engine-sdk-python-3.5.2.1-1.el6.noarch</div> <div>ovirt-engine-setup-plugin-ovirt-engine-common-3.5.2-1.el6.noarch</div> <div>ovirt-engine-restapi-3.5.2-1.el6.noarch</div> <div>ovirt-engine-userportal-3.5.2-1.el6.noarch</div> <div>ovirt-engine-3.5.2-1.el6.noarch</div><span class=3D""> <div>ovirt-host-deploy-offline-1.3.1-1.el6.x86_64</div> <div>ovirt-engine-setup-plugin-websocket-proxy-3.5.2-1.el6.noarch</div> <div>ovirt-engine-websocket-proxy-3.5.2-1.el6.noarch</div> <div>ovirt-iso-uploader-3.5.2-1.el6.noarch</div> <div>ovirt-engine-extensions-api-impl-3.5.2-1.el6.noarch</div> <div>ovirt-engine-jboss-as-7.1.1-1.el6.x86_64</div> </span><span class=3D""><div>ovirt-guest-tools-iso-3.5-7.noarch</div> <div>ovirt-image-uploader-3.5.1-1.el6.noarch</div> <div>ovirt-engine-lib-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-base-3.5.2-1.el6.noarch</div> <div>ovirt-release35-003-1.noarch</div> </span><div>ovirt-engine-tools-3.5.2-1.el6.noarch</div> <div>ovirt-engine-webadmin-portal-3.5.2-1.el6.noarch</div> <div>ovirt-host-deploy-java-1.3.1-1.el6.noarch</div> <div><br> </div> <div>Logs:</div> <div><br> </div> <div> <div>2015-04-30 19:50:24,578 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommand] (ajp--127.0.0.1-8702-6) [4cbab1c6] Lock Acquired to object E= ngineLock [exclusiveLocks=3D key: f2afb8c7-6b1d-4822-a43a-0889178de719 valu= e: VM</div> <div>, sharedLocks=3D ]</div> <div>2015-04-30 19:50:24,689 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommand] (ajp--127.0.0.1-8702-6) [4cbab1c6] Running command: RemoveSn= apshotCommand internal: false. Entities affected : ID: f2afb8c7-6b1d-= 4822-a43a-0889178de719 Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER</div> <div>2015-04-30 19:50:24,723 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommand] (ajp--127.0.0.1-8702-6) [4cbab1c6] Lock freed to object Engi= neLock [exclusiveLocks=3D key: f2afb8c7-6b1d-4822-a43a-0889178de719 value: = VM</div> <div>, sharedLocks=3D ]</div> <div>2015-04-30 19:50:24,762 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotSingleDiskLiveCommand] (pool-7-thread-7) [598dbb1e] Running command: = RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : &nb= sp;ID: 00000000-0000-0000-0000-000000000000 Type: Storage</div> <div>2015-04-30 19:50:24,771 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotSingleDiskLiveCommand] (pool-7-thread-8) [5c8285eb] Running command: = RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : &nb= sp;ID: 00000000-0000-0000-0000-000000000000 Type: Storage</div> <div>2015-04-30 19:50:24,805 INFO [org.ovirt.engine.core.dal.dbbroker= .auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID:= 4cbab1c6, Job ID: b0057939-d15a-4da7-b782-c835127a1413, Call Stack: null, = Custom Event ID: -1, Message: Snapshot 'test_snap1' deletion for VM '3.5.2.wintest1' was initiated by <a= href=3D"mailto:patrick_russell@corp.volusion.com" target=3D"_blank"> patrick_russell@corp.volusion.com</a>.</div> <div>2015-04-30 19:50:29,642 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-13) [5c8285eb] = Executing Live Merge command step MERGE</div> <div>2015-04-30 19:50:29,665 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommandCallback] (DefaultQuartzScheduler_Worker-13) Waiting on Live M= erge child commands to complete</div> <div>2015-04-30 19:50:29,671 INFO [org.ovirt.engine.core.bll.MergeCom= mand] (pool-7-thread-9) [e10f6a3] Running command: MergeCommand internal: t= rue. Entities affected : ID: 58ef50e1-2efb-4081-a3ca-493c2ee7e7b6 Typ= e: Storage</div> <div>2015-04-30 19:50:29,679 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-13) [598dbb1e] = Executing Live Merge command step MERGE</div> <div>2015-04-30 19:50:29,697 INFO [org.ovirt.engine.core.bll.MergeCom= mand] (pool-7-thread-10) [4898b008] Running command: MergeCommand internal:= true. Entities affected : ID: 58ef50e1-2efb-4081-a3ca-493c2ee7e7b6 T= ype: Storage</div> <div>2015-04-30 19:50:29,703 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] START, MergeVDSCommand= (HostName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a>, MergeVDSCommandParameters{HostId =3D cfc227ef-c9= ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, sto= ragePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3Dbb8= 359c6-1dba-45e2-97cb-929d1c83a7c8, imageId=3D87f26ef2-6a84-4f7c-9771-be7665= 0c703c, baseImageId=3D3f90ced5-58ea-43cc-955a-559e8d89fbb2, topImageId=3D87= f26ef2-6a84-4f7c-9771-be76650c703c, bandwidth=3D0}), log id: 5859dc0</div> <div>2015-04-30 19:50:29,728 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] START, MergeVDSComma= nd(HostName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a>, MergeVDSCommandParameters{HostId =3D cfc227ef-c9= ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, sto= ragePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3D99e= 44df8-fa01-4255-bc1a-a4c240eb01e2, imageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3= ed4129, baseImageId=3D7ad1cf7d-f2e6-41d4-a3c4-57458e80c43d, topImageId=3D2c= a5b861-52a8-4c0f-92a2-a8f0a3ed4129, bandwidth=3D0}), log id: 160b45fb</div> <div>2015-04-30 19:50:29,819 ERROR [org.ovirt.engine.core.vdsbroker.vdsbrok= er.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Failed in MergeVDS method</= div> <div>2015-04-30 19:50:29,820 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Command org.ovirt.engi= ne.core.vdsbroker.vdsbroker.MergeVDSCommand return value </div><span c= lass=3D""> <div> StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52= , mMessage=3DMerge failed]]</div> </span><div>2015-04-30 19:50:29,821 INFO [org.ovirt.engine.core.vdsbr= oker.vdsbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] HostName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a></div> <div>2015-04-30 19:50:29,822 ERROR [org.ovirt.engine.core.vdsbroker.vdsbrok= er.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] Command MergeVDSCommand(Hos= tName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a>, MergeVDSCommandParameters{HostId =3D cfc227ef-c9= ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, sto= ragePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3Dbb8= 359c6-1dba-45e2-97cb-929d1c83a7c8, imageId=3D87f26ef2-6a84-4f7c-9771-be7665= 0c703c, baseImageId=3D3f90ced5-58ea-43cc-955a-559e8d89fbb2, topImageId=3D87= f26ef2-6a84-4f7c-9771-be76650c703c, bandwidth=3D0}) execution failed. Exception: VDSErrorException: VDSGenericException: VDSEr= rorException: Failed to MergeVDS, error =3D Merge failed, code =3D 52</div> <div>2015-04-30 19:50:29,827 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-9) [e10f6a3] FINISH, MergeVDSComman= d, log id: 5859dc0</div> <div>2015-04-30 19:50:29,828 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-9) [e10f6a3] Command org.ovirt.engine.core.bll.MergeCommand= throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.eng= ine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D Merg= e failed, code =3D 52 (Failed with error mergeErr and code 52)</div> <div>2015-04-30 19:50:29,834 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-9) [e10f6a3] Transaction rolled-back for command: org.ovirt= .engine.core.bll.MergeCommand.</div> <div>2015-04-30 19:50:29,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbrok= er.MergeVDSCommand] (pool-7-thread-10) [4898b008] Failed in MergeVDS method= </div> <div>2015-04-30 19:50:29,859 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] Command org.ovirt.en= gine.core.vdsbroker.vdsbroker.MergeVDSCommand return value </div><span= class=3D""> <div> StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52= , mMessage=3DMerge failed]]</div> </span><div>2015-04-30 19:50:29,860 INFO [org.ovirt.engine.core.vdsbr= oker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] HostName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a></div> <div>2015-04-30 19:50:29,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbrok= er.MergeVDSCommand] (pool-7-thread-10) [4898b008] Command MergeVDSCommand(H= ostName =3D <a href=3D"http://aus02gdkvm01.dev.volusion.com" target=3D"_blank">aus02gdk= vm01.dev.volusion.com</a>, MergeVDSCommandParameters{HostId =3D cfc227ef-c9= ab-4c35-aa22-138c169f2ba6, vmId=3Df2afb8c7-6b1d-4822-a43a-0889178de719, sto= ragePoolId=3Dedca3dc6-d9ca-410f-af84-9e24d49df3e0, storageDomainId=3D58ef50e1-2efb-4081-a3ca-493c2ee7e7b6, imageGroupId=3D99e= 44df8-fa01-4255-bc1a-a4c240eb01e2, imageId=3D2ca5b861-52a8-4c0f-92a2-a8f0a3= ed4129, baseImageId=3D7ad1cf7d-f2e6-41d4-a3c4-57458e80c43d, topImageId=3D2c= a5b861-52a8-4c0f-92a2-a8f0a3ed4129, bandwidth=3D0}) execution failed. Exception: VDSErrorException: VDSGenericException: VDSEr= rorException: Failed to MergeVDS, error =3D Merge failed, code =3D 52</div> <div>2015-04-30 19:50:29,864 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-10) [4898b008] FINISH, MergeVDSComm= and, log id: 160b45fb</div> <div>2015-04-30 19:50:29,865 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-10) [4898b008] Command org.ovirt.engine.core.bll.MergeComma= nd throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.e= ngine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D Merg= e failed, code =3D 52 (Failed with error mergeErr and code 52)</div> <div>2015-04-30 19:50:29,871 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-10) [4898b008] Transaction rolled-back for command: org.ovi= rt.engine.core.bll.MergeCommand.</div> <div>2015-04-30 19:50:39,698 ERROR [org.ovirt.engine.core.bll.RemoveSnapsho= tSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-73) [5c8285eb] Faile= d child command status for step MERGE</div> <div>2015-04-30 19:50:39,701 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommandCallback] (DefaultQuartzScheduler_Worker-73) [5c8285eb] Waitin= g on Live Merge child commands to complete</div> <div>2015-04-30 19:50:39,710 ERROR [org.ovirt.engine.core.bll.RemoveSnapsho= tSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-73) [598dbb1e] Faile= d child command status for step MERGE</div> <div>2015-04-30 19:50:49,720 ERROR [org.ovirt.engine.core.bll.RemoveSnapsho= tSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-36) [5c8285eb] Mergi= ng of snapshot 470b525e-fb30-4956-a702-f4c6389bcb60 images 3f90ced5-58ea-43= cc-955a-559e8d89fbb2..87f26ef2-6a84-4f7c-9771-be76650c703c failed. Images have been marked illegal and can no longer be previewed or = reverted to. Please retry Live Merge on the snapshot to complete the operat= ion.</div> <div>2015-04-30 19:50:49,725 INFO [org.ovirt.engine.core.bll.RemoveSn= apshotCommandCallback] (DefaultQuartzScheduler_Worker-36) [4cbab1c6] All Li= ve Merge child commands have completed, status FAILED</div> <div>2015-04-30 19:50:49,734 ERROR [org.ovirt.engine.core.bll.RemoveSnapsho= tSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-36) [598dbb1e] Mergi= ng of snapshot 470b525e-fb30-4956-a702-f4c6389bcb60 images 7ad1cf7d-f2e6-41= d4-a3c4-57458e80c43d..2ca5b861-52a8-4c0f-92a2-a8f0a3ed4129 failed. Images have been marked illegal and can no longer be previewed or = reverted to. Please retry Live Merge on the snapshot to complete the operat= ion.</div> <div>2015-04-30 19:50:59,787 ERROR [org.ovirt.engine.core.bll.RemoveSnapsho= tCommand] (DefaultQuartzScheduler_Worker-16) [4cbab1c6] Ending command with= failure: org.ovirt.engine.core.bll.RemoveSnapshotCommand</div> <div>2015-04-30 19:50:59,837 ERROR [org.ovirt.engine.core.dal.dbbroker.audi= tloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4cbab1c6= ] Correlation ID: 4cbab1c6, Call Stack: null, Custom Event ID: -1, Message:= Failed to delete snapshot 'test_snap1' for VM '3.5.2.wintest1=E2=80=99.</div> </div> <div><br> </div> <div><br> </div> <div> <blockquote><div><div class=3D"h5"> <div>On Apr 30, 2015, at 9:54 AM, Patrick Russell <<a href=3D"mailto:pat= rick_russell@volusion.com" target=3D"_blank">patrick_russell@volusion.com</= a>> wrote:</div> <br> </div></div><div><div><div class=3D"h5"> <div style=3D"word-wrap:break-word"> <div>Hi everyone,</div> <div><br> </div> We=E2=80=99re not seeing live merge working as of the 3.5.2 update. We=E2= =80=99ve tested using fibre channel and NFS attached storage. Both throwing= the same error code. Are other people seeing success with live-merge after= the update? <div><br> </div> <div>Here=E2=80=99s the environment:</div> <div><br> </div> <div>Engine Running on CentOS 6x64 updated to 3.5.2 via yum update (standal= one physical box, dual socket hex core + hyperthreading, 16GB memory)</div> <div><br> </div> <div> <div># rpm -qa |grep ovirt</div> <div>ovirt-engine-cli-3.5.0.5-1.el6.noarch</div> <div>ovirt-engine-3.5.1.1-1.el6.noarch</div> <div>ovirt-engine-setup-plugin-ovirt-engine-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-plugin-allinone-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-3.5.2-1.el6.noarch</div> <div>ovirt-guest-tools-3.5.0-0.5.master.noarch</div> <div>ovirt-host-deploy-1.3.1-1.el6.noarch</div> <div>ovirt-engine-sdk-python-3.5.2.1-1.el6.noarch</div> <div>ovirt-engine-setup-plugin-ovirt-engine-common-3.5.2-1.el6.noarch</div> <div>ovirt-engine-backend-3.5.1.1-1.el6.noarch</div> <div>ovirt-engine-userportal-3.5.1.1-1.el6.noarch</div> <div>ovirt-engine-dbscripts-3.5.1.1-1.el6.noarch</div> <div>ovirt-engine-tools-3.5.1.1-1.el6.noarch</div> <div>ovirt-host-deploy-offline-1.3.1-1.el6.x86_64</div> <div>ovirt-engine-setup-plugin-websocket-proxy-3.5.2-1.el6.noarch</div> <div>ovirt-engine-websocket-proxy-3.5.2-1.el6.noarch</div> <div>ovirt-iso-uploader-3.5.2-1.el6.noarch</div> <div>ovirt-engine-extensions-api-impl-3.5.2-1.el6.noarch</div> <div>ovirt-engine-jboss-as-7.1.1-1.el6.x86_64</div> <div>ovirt-engine-webadmin-portal-3.5.1.1-1.el6.noarch</div> <div>ovirt-engine-restapi-3.5.1.1-1.el6.noarch</div> <div>ovirt-guest-tools-iso-3.5-7.noarch</div> <div>ovirt-image-uploader-3.5.1-1.el6.noarch</div> <div>ovirt-engine-lib-3.5.2-1.el6.noarch</div> <div>ovirt-engine-setup-base-3.5.2-1.el6.noarch</div> <div>ovirt-release35-003-1.noarch</div> <div>ovirt-host-deploy-java-1.3.1-1.el6.noarch</div> </div> <div><br> </div> <div>Hypervisors are running ovirt-node, upgraded from ISO : <a href= =3D"http://resources.ovirt.org/pub/ovirt-3.5/iso/ovirt-node/el7-3.5.2/ovirt= -node-iso-3.5-0.999.201504280931.el7.centos.iso" target=3D"_blank">http://r= esources.ovirt.org/pub/ovirt-3.5/iso/ovirt-node/el7-3.5.2/ovirt-node-iso-3.= 5-0.999.201504280931.el7.centos.iso</a></div> <div><br> </div> <div><br> </div> <div>Here=E2=80=99s a snippet from the logs: <div><br> </div> <div> <div>2015-04-29 18:47:16,947 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-2) [48eb0b1d] FINISH, MergeVDSComma= nd, log id: 5121ecc9</div> <div>2015-04-29 18:47:16,947 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-2) [48eb0b1d] Command org.ovirt.engine.core.bll.MergeComman= d throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.en= gine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error =3D Merg= e failed, code =3D 52 (Failed with error mergeErr and code 52)</div> <div>2015-04-29 18:47:16,954 ERROR [org.ovirt.engine.core.bll.MergeCommand]= (pool-7-thread-2) [48eb0b1d] Transaction rolled-back for command: org.ovir= t.engine.core.bll.MergeCommand.</div> <div>2015-04-29 18:47:16,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbrok= er.MergeVDSCommand] (pool-7-thread-3) [5495bde7] Failed in MergeVDS method<= /div> <div>2015-04-29 18:47:16,982 INFO [org.ovirt.engine.core.vdsbroker.vd= sbroker.MergeVDSCommand] (pool-7-thread-3) [5495bde7] Command org.ovirt.eng= ine.core.vdsbroker.vdsbroker.MergeVDSCommand return value </div> <div> StatusOnlyReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D52= , mMessage=3DMerge failed]]</div> </div> </div> <div><br> </div> <div><br> </div> </div></div></div> _______________________________________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br=
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank"=
http://lists.ovirt.org/mailman/listinfo/users</a><br> </div> </blockquote> </div> <br> </div> </div>
<br>_______________________________________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br=
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank"=
http://lists.ovirt.org/mailman/listinfo/users</a><br> <br></blockquote></div><br></div> <br>_______________________________________________<br>Users mailing list<b= r>Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></blo= ckquote><div><br></div></div></body></html> ------=_Part_9128865_1887831902.1433250537047--