--=_55a563500b973bf81693afb3908ca577 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8; format=flowed Hi, We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are: 2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52) I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm). What could be the reason for this error? Thanks. --=_55a563500b973bf81693afb3908ca577 Content-Transfer-Encoding: base64 Content-Type: application/x-gzip; name=snapshot-deletion.tar.gz Content-Disposition: attachment; filename=snapshot-deletion.tar.gz; size=29925 H4sIALPbbFcAA+w82XLbuJZ5nXwFHlIlu0akuS/q1q3rSHKiHm/XctI9lXapQBK0mFAkLxfZbpf/ /R6ApEVZlEQllpOeiR5sCQTOjrOAAMKZF6ccCa69gPB+eP1qBx8BPpqm0P+irgrV//BRRVlXX4mS LsuiqooitIuSpiivkLALYp5+siTFMUKvAs8O1/Xb9Pxv+pEEUeMEjZMUJOgdVejISluWVTQ8PTpD 6FMYX/MhNRG+MBE7jAlv+T5/QabhjIwCHCWTMO2F0ykOnCu05xAXZ36KUpx84UTR2EefTEfQTGJd oePQ/oIO7X9nXkwclIYotD4TO0WtAQNOH3fuya3tZ4k3Yz+TbuuTLhiyIUuYwyKWOEWydM7URJeT iGkLju3oRDS6v348aaPD3uXw7HR8+b/ng/HR4fB40B+fvf1t0LscH5/1/mfQ/8dVq42SCQbsBfAg 8/3WQ+t1nRhUVduNGC6yIPCCa2TnvTuoFgjygpTEAfY7yMV+Qng0CFIv9UiCsOuC2ECEHYSG/Q5q IiF0eReRDvp4cminXhig6zjMInRyeDo8/3B8eDkYfzwZj04Pz0fvzy5H6MZLJygOfYJSGIY+jAYX 9TIyzB2aihuTv4GdaIqwrQxGoH6f9L3kyzEwMJdIFIY+p3PpJCbY4UwQiK4TxxUVZaPV1IKs2FAa Z+tMSCg+XM2f8lOY0CgNY3xN6mWh6+tl4WCfdywrDr+QmMeZ46UQdCZAqg/084e04Ti87oODsAFN vZFcoV4Yx8THzJAp9YXZtNFvocUasGCrsmbJnOMaNqfYrslhyZQ5Ituirbs61kXSRj3s+8AOBnNC VMPQAuEgnKLBjAQpA8SJbXRCkgQYBs4LWaPWYZaG3DUJSIypEN0wRlTspWzQiXcdM/JayCE+YYTS Th9PUOvjCe+EU+wFIJFpC93gBNQEWmGQrDuEnakX/LPUHF8jZ7UtSF/roinPFnAMou3nov1XhuP0 r5E9IU7mk3j8exiDcjhNBVFrrqLJMJWu0O8YSATzA0YYpyckBj7tiec7pUkmdKrC9wgYrjMPIFvd 2l2smioNiJ/PncEtsTNGfpX2Yo4kKYnAtfxxOTjt181wINsUN5PNgA5uUwIyXjGhRSBKBkckuZZR M6GXITSfv6Ysm7oh2JysaTqnuMTlMBZlzrSILRimqtuqu3H+7orTtzghiLKThhHypnSCJN5flIsY AswEfuEp+QUFYeUhyiIHZgSKSZ4u1JCrAbnyy9iTsbU9nQwu3g3qzElri5LUUMgrxStQckwTSxKR V1nS97Ehyt4GpcycpAgA82+M4o/9UUOOR5eHF5dt9GTU3vswSU/BllAXTeCrVHG0S53PcQw9QTRJ 5z7OgsPkLrC7LSoiCP509NDptmxZdGzIJziiWhBHZFfjTEVwOQ0kRWTHsQhRoPtsSjs3SUBoYpHL 7hz4o6NcDGWPo8icQjSDUwzN5UzXdDhMZCKbombbijsf1WcM0XFNtAXj2Ix6R5M9OkhTsWAJjsKp tqlxiuNAXJRFC74JBlEErGhYLQcxJITYJjFcTjYc6IUtl7NEW+FMycUmMOsalgj9LZjgw3KMqhmW 5Oo6JziOCzJTAIeiUjpd2xKIQ2xLgzHgC4ZbowmcG89JJ92W0HrYbyPIHZAH9i66BnEwUWrtUVIl NLi4OLvYoT0eYc8ndKrBxC9GthBY1ySsd1wS1BZrado6T1pL3tN8qUh2tkx/gKmTpXn16PZcJoLC 9xS/VvK+a/9Q+r7WV2NoQeRJszhAM+xnBLVATmmWnAX+3QVrPwrjP6b+RWSjTwl71M17VNrt0CFd VWqDITAJdquiubqqDw4vIZ01bvLrrPU5NfbTp/9/9uktRPLcihZMhUcB/b6jpZZnD25tEtFnrHEQ x2FcaSqcMBQhpVm0EaF9wHSqU6+N6MyERlX6blPwaHg6HL1fMuDGIW3DhNw2g8wXVkD2hTAhK4/D mwDdQH0H+Thk+jS/nFbz3A6qR00fhwHPBJ/wOeCKkpYamou4RuU7NA20Vwxia2F5ZyYAgM2KGdZT lfZf/xdOV3BB1fDRSd7TuA0MsPhNoOGCJFBr7FUefcYz3BFFCbRCB33GcedqE+D+6C0TzFEcBrQY G0LlzUMtAGBL51nfhyGT5S1wFfBoIcfHCxiqTxhcSTCVLSBXrZTP5375c2/hGQOubCOgKmkF5N9B l2GWXsY4SDBbCF1mQJRE9duw5Eusw6CCZmSHEanBJavm16thAUMNcNNoKi5wuH7Cp3NofJX2LIpC OiDnbhickpsq4pquuSUwJ5ODfl4CLopFgZWoRcXcEepckSvxsgXKZnhXGE+dkZjacxjkMmRZN7YA /BYqBXAh1PIKgGVLPjeNbeb9BmC6vExZkgV8TFwfSh7+Xbn2esKKrEPbhhw7jEVRl3gvmIHL2/sQ fAloDBuFWWwTgNYReZ0XxqIgLsPrE59cY7qkswiQ+csC4No+uQioT2W2ED9FRp/zPobirUSZAymB F78YGE3QVsKhEv1shUnC44QnLNxGYQD1Gn+CA0i9nAvikpgENskhDunqD42DYcxHcUgpHgJGmxWD e5vHMIKqUS5H7j3C4Cu9ezTI3KY5IpieNY9yuxPMNQB/J3hGNhC+3CcnVH4K96mUvPmQhP+QkLgC 4wjTYvrujViDsL5roa8lrLsUz/KoN3M6N8JWBL1GRDfEh+j72eJ/S2LJNN96LOFLqvJ1cusnj21U Kqu757OYBoAbz3dc/46jODiDl3iRP/Lomw1q3ZXfNfbdhKzwic02IMvYPVnLBrSBJlP+Npp+Gvkq I6+PPwsiqqyPDR2I7RCIqnLgMdTakDvnnnpT5yJmf034gszzZ/TKI9Gx5xL7zva/KoytGZw7gO1M 72c8e3bxUDY+W/ICIyWKKUmhoE/4QbkSdOlNyXrxrew6N+zSt1K02/jWv4mXy2E/RqZDK4GKxk4H v72lxRJJUla40FR7lruuVe5tq5G5kdJymYUs6l05DxwKJ4FAtYqAJUFUOFHgBBmJQkdQ10XVwWdr M+YaG2gyLJ/+3xJrdzIV6IwO7AwCC8yE3uPXYuh6vtd2zw1Frdo/+XGsn+568ddzV9OpWFx4SRfe m2DIHjb48JpOuTtdInWtE++VzX0viXBqTxbTizoDWDtgRSB5AZeesL0WKWAYld8eaR0GSYohVB9C DmJ7Dab31iDKyvUHcv2liNJbvndyeXlbJTfPpIbBaXh5u7f0NLd5TfgGbtYhT/JFrGQFYtk0d4S4 xprrWf+xYnhd+lKdwrk/nnPVzJE3GJX7c/FHl8XTVI7uHtycxdX1+gETuJLhBHLO2Evv+FHxpZGW 13TOFwLUH5HXFYY+mmRpH8rUhkXH6u5FpFT+Rrwfh9fXEJPWa7umUx6XvsWh7iwFLfmk+y0glbBJ I4te1ztnVnjRXC3v2vNxkhyH2NmcPq3rX9rld60K6VuS1cAkbQ6sMKm5b5qypZGY/x0eHPl3pfc5 KZqdsDch9hegdUWHwv+uozdfSWJwNs6H1X1XpMo/qqCXgJ3H3szzyTVx6EtmSFItKAY2mN7mMeV6 9f+dWuejR25GJJ55NnlDv5dLiJX2YgnVUJvA6pPEjj227l8bc570KWG/6ButHYv0PA5v7+aQyj0l hWBXPC1eszZaJ3fBuSflK9viLfIZO5NVbmB58+bNDAStVt7nLq1gN301TI/6JOVr7HxlD2TxZG/I 08dFgbTNzo16PJh5qDeyINTjULbZsFGLAszUxr6/ggV51/CXRcRW9+mmheoq1FGWZjG5BATMKVZ+ FqKW1r8teArvkm08ozskHwkCuPn5ir2ah8V+CrZ29U1o3uQ4GBer8GjiBjTs3Uc+ugKomEQraKzZ uqe3RdHY+cktWi0pjixjXbef5eQWkC01ILu6X2s7auc7ERfP0kxw8khXfsyObWNNULMdqTzfbBtq DctGW5Q3nGV8tsNqiwc9mx4uGo8uDy8/jFYQrzU9Y5RvW1+1T5QqRzJsUwYOVh00WoDwwseNjLa0 6chp3VbSo8z3j70kXbNhd5Hz4sDR8rj1+9OX+z/rFnUn6bYo/k9LmJtAuCp2uTc9PH21cOZGt2RJ VVy3Vim6seEc4bMopdxEvTywXZze6KBP99iOvEGALZ90qZjbiEwzn717x+AEA9KNbM5TQDi3XDwh vg51t9BGked0Rd3Q6OmN6TuYuBjsmjgj7y/SVUGIbcR2DEaYRp332TU5p36pwOB4SeTju/MwTruq KUuPLay6ImW73EZJBPnuKPF7XjQh8SjzUtLtD44OPxyDtdlRRu2/e0om2CdgTsk06hptdE1fcp1m U3waOoDz0/E8ROVZ2S//1DA2iG3QIeBzbBw7hQjYXQIAmp306d47hCbcY1F2sW5KIkc0VeEUEWPO EGXCEUPUREhJBEW3i642xpIuu9DV0TVOMbHNWRIx6NlsyXVEwxQ0pftx2me9O/cgxlb5a+h0CoTM yBsA2u4cxgN0z+F3W1ng3dITCVR+rd77w9PTwTE9bhCG6VkMNT49bkDFT2w2I+kcoPMBO04M6V+3 dW9lSVeg+8GDNA59cNf0FwNHjdkLuYTEHvbBUqg2RYo7r7XZ+Q8QMjREfnbNGor5TW2YHiGa98jJ PfQ9DDjtCQ4C4lPCcv1A6h6RmHrPgrykCDRUJOxOAjYfPUj3qAt6bMvom5eRnXgXBKjM34KWOB9K pM+t9KKraRiG6yoqhGOTcIqKVc7SFImzTU1WLcmRbN1qZB9NAP1t7EN6RvsQfyD7aKTJJoC+XpOe Q+aKPDu9vDg7Ph5cbKXLxA/TrnAriDCIqvVWAF3m0ZT9EErVRrbXRm4WsAoXHj3LtM8Z+FE0upsZ X3S1VAKphwsJu2BQqI7MGa5OOBHrqmFCESDoTiOTagLo602KReXZ1H5BByE/o4OQdmtOD1SuLDH5 Mpuy/IhlRYZgaixF6UGCl5xDNhPaXwiERtqdQV+4+aSNAiprgR6DZOdnP0RUALcnBThFNBVZUHKJ n5AgW0xgmGabKJZqILo7x1A5LWSBRUY2jLqiwIsCpH0aXzCQF/eUBcoKZYBlXLSuO8FRBKLp3j+U uRvL6Hq54JNuQtlrg/SiLE3aCTxLwrjNMNFquJ3EBPJfp50U6NtJBukvcTzo9JiqgYvxfMLOmLgk XqAaZHYCSZ/fjVMfpq7ZjmZt9IXcWSGMO8Z3YZZ2ScBlCbTOpgtDIy+9IF5A80OqzUKOBR2nJL2B arTLMvXp9TQtLas+vZR0A1JwIj+O7/77Fiw59abkzHUT0Do9OQ7a/NiLQLUiyNX2PfpyO+qWCqdv eruKoeuiISuG8Kh88MV0xMNV01rDFBoehtym1J2fGPRpFc5O/lHKPd+nx/xnoZ9N6SoKqBuKjCar EW3UZC3iqoZHE8p5ZTOPz7EWwc4vzdciCiksLBYV6mOrMY2WIxShLQkvRL++QD7VOlv6ch8dH2oR E0u2o2IOQpgMQVAWOUMD94EVSXEdWSLYllu7WWcqTpnyaJhDn2AwLSjFA7B9IN9BYF3kGvv5+VIc 0Ctx/DC4JjH0QlFM6Bo7dAPJw3dw7MxCeXTuE3rFDtS88V111Ygd5CVz3iurfOwBDQ75G4oarYmg tZ2vVUpy9da3Q5hcq9combQeVwNLN4Ja+RVudYYntSVR39bwmpqaWaV8kJ+QLqfIinsnViJr5aeM qXlANKnTBrBiqM98Z8gW/H2nG9fmXphdo1Yx5W2uYKu/eY1//b0v3/wBPjMnme7q3tfyw+5/Verv fxUFWZJU7fH+V0WSXwmiKInSz/tfX+KTp7qcIalQF3Y6/cHbD+86nSX3AwWwbGqdDs1LR/1OR9Tl TqeYafyJl9g8ubV7U6fT2bsmkGZipw/e4m4fHWRJfGB5wQF7U0lSxHF2lHG+l6RI4ER13sGB6Od2 D+IJmR04OMWcTeibhINpkB4UGbpM/3XGkNhyNHVjXyARE8fkltZQB8SBqg/o5CyV3r0C/gfKW8fk BEl2CcRwQxCdA/AA46lzMCUppmgAqY+vuw7zipAodA8g7z2gvglZSVeBmgacehZAGYP27BsHnYYB 2X/dXGxQn36F2EYfer3BaNRBv5I4/gfqopbw3yLKawd6XeSfQfU3JP1/BrJqIusuhaxij359u0/r Ho/GSYGHiaaoBpAFZQFk7ujL24Pkz6D1C/o1til04TXgpkEB5blwp0PD/gqGdLXTieIwDe3Qd8An 03Gdjg5iOC9a+0Ur2xlENzLQM2bAY37/xRiz1n2UP83DJlRP+b20bgzRALwBD1TzYkcVFV1com6d vA25jjyjjrz+4+O98ZjewTke76MPtPR9vHZwTO8i7IrLJKwRUD0FomisJ6GQDgW/j/J2iGslIHQ7 9Z9HOFZ+BBXgxZENdMkCyOaPqb9MSsIK+H2GGnKWGE3SNEJOSRqjZq/1SE+rjRhF+6+LU65/nBxf nPc2yqqkhE4OiEd8PozuBSILcmFne/Zp+hCnpY6gBeUdYhbmnwqnmKiSIMEUMNaTosxJod9rSBkX u3ZKYpJ9dLGZCJqo0lR9iZi1mlKWNSVpEiMLKLmJcQTJ+35RVqNPjzivOh26+wPNpu9IOgwvs4Cc h75n3+Up5l7DK6320f3DdgSrdQSbSwSX16rVk3ff8iptLSitr9qolWf68Ou+VdylBt9bfXDFYHMt ehEP/BYelglep25trm5Tel51h8Cqs5XL0F/Yp0pbuQ2z1uW/pE+tp2BHPnV74ezap0rb+FRT26VP lRYnmdmYlOf0qVK9TzU3aEp7Pp/6rd50PamqUBOom3nTRT9KPWjl+33rJvZSMmaJ4jghNvWbx+BD 0zDF/tgLo4VWSm994xKAHO7TvjnYxc4P0B7hdAI/stZyvt/k7sSDJnuHDvIlvIMm9yMeNLytsBXg KWGEzxzcevj62LR22qiCupvYJG0dm8S28OL5vtzc/QJ53znfpxSYLywg5dup21Hk3FJ15u4jp9w4 chYE7S5ybicc0/jedl1PwQ7tWl3wkUDFJup2kVrItakFJWattmpM+XnKtf+wdzU9ktzI9e5fMTdJ cLObZPBzgDVgeOHbwsBq92QYXn7uyh5pBM1qbR/2v5vMrhpNVTGZyeyMrAbUwBxmumuqXkUkHyOC 8YJMgIwpKGLKrlQ2rcCIpcwSyCxTrhPNFhoBxgJgRukrS9f67maU4WyJMLsldq2n7rw4GW1mjEi0 frvhdI2j8WldfPNyYyEymXq1rpSvzpVyYIdm1GDu0OLVILnaCtlqKHtuhaK9FbKFZwZtK+QlW9NR lUxOQckPuShboc2+fLwOzGQjOA2trZD3rWdxrCfb1usdFxU0rLkv72E94CbzQHNJl6FWKmq6bIGR 7GOCUPJebuPJeiMUAfetDc4iQOLPW2ZfMg42f6oR/gSJyVrqH/7r08cfyps/nuWVT6prIF0LIfVo +P17Weuo57PRqjM9TyD4w/Trr//z+UKw2sOXvnlXf/ibP4GTSmpHSZgeaWCCeGc1Cd5G4ZIFJcSf 3r///uPfKv7JfjU2TdOFe+/IP53+9eNP6Uf3U3lJE/68NSf0Hz7++Y913vf79/XAIX4ewfXlYv39 zz9MLV314Sq+fF9M+ePH+k0+ff1xUqV/+s3zufLg59eY+ZfPr8Q8+vkP756vsyk/+fRjwZFK0P7z V4ULQKjgCRhaD9KDJEYFTQz1KnJtqRQwhffnWL6E9WVR/Ow+lH/8Yeo2/ao8TJ9q9837dzD99nl9 //L7WE+6a15Qj6eZYEbWG0S++uA+/XUaD1F/ZR5rK/NXf3Mfvjv/z78/vPt53T0pu+IDCuwGnn60 TXhlJ/LJWUZU9KnWhDlxXkfiygPJfKJRKLceHl2GV/xhlLq1n3qUTYAGhNBMRRJDtERQqFuBKVCd o4aKLK3b17/1lmcq+A1A+aibANd0cuxrwak/4gYfe4QmPu5sojIIAi4Us5kMxPOQSQBflZXaADP7 PoC0JOe3+OwjbS8QVRCGXJZF8MV+XlhiXSoLmCnFc0ysgN8Xny0LRDbw3T6Afx/eJCrNPW8SjFnW 3SWeqXx0g8hlP/j0l3rX0Bv5vZHfG/m9kd9rJL9DI+TPMfEvYfKZJTd9hRJfTnOATpjfv7fii17I 31/+9vHf/ueHKYL9KU3Khn/+8OGbd9PPHn/5yTkf+VQy1nfnt6//2Amfocv4Qp0/++ELeJ9/8CW6 F/i8ZOh9n8cUfkp50N3lf7yj75z/+JzY/WuVgn2uUvSwVUFFyV79h5Kp1oZWwTlSH3Dtwl2zkPdp 5F0ubJVvzl9bh1u3llQBN/td7gm4U8eseDXOGY/4fMZzAQb6YJCKqqpdFoQFV5r9W5e2nix2yqkV qcVxohzuXTmBOfTgR79qdGbomYO3HrReD9qaPPHcg5aFUxxCJD7X9jJjc9nEysrLyTvDsw0ps6eQ tbNeB+KMFiUxEZz4mqLELCBFBzamuFMP2sIeD2LLHn8rWmH/SC9EK1/+exKtTNvySbUiHtm7/76S rXBR6EZX2Qp71O9+tyhb6X6p+zbUVARHtmLdMlHXOAe0YunVBxUFkKCYBxX6kghFHwrDiQR0OxIQ fU9VNHtHAgNHgwMRlOA4kYAab9mpaJpdFkcu/zYCpOVvxozTCC/3Xv5maPmjdleYsRXXjL3f4qBz HLSinnqOgxgtj3vJOkqOkUsclJknXiRPas1YR+AlB0lPj4/rXqd1Fjrr8nuoQRXlnlgRQmEtDilx ppz3O/bsS9l/YpHaSEx7l5ALz+yOTTgvbb9Z2F2R8kXdTvoX7Cbfcp7eWl9T+j+vdRMjYyJWSU09 pwKniPEl8ZE5lo8o2Y309gk0CJ6yISaV5S10UsRbT4kQzsqopMyQj1vDUuA8i+bmWew/hrrmKgcV WVe5dJ8i68KXrlXvV5F1Mg6c8Zp1wiM0ss7z9+nVi+FB0uOcuKoWsY8TF750/f3rcKKkHKA6ka8q HczRAjyodrc/Yp3Org/eCzwm7prZVARHdmAOGueADky7OrM5AcLLbOzldtdpH6xQkNTZth2ydsFo QNp77fC5QQVzsObVDJQyCzxxX81rRdDkHJwVP2qcxuSUnVd8vZBr9YqfAKGteEMvF5leDWXHFW9o e8XrBU81D9T30DpYiMIpCySnWj8Aw4n3gZGUlMvOC4jto80lwK/uQH3B3TgH6pO7hzlVHCwnNgOj sQo8eV9J5ywCJE4dNA6+DtCsnxt2AoTHqZdzw2RnkNQVlD05tT03TPbGcM146qWFv61sugBVNU+y 3mptG2ptFpiOLkRCjZflXYMiPshIkrciW0iFst2TBqe9VkCSCJoIIacRvYkE4YwFq22OYs9aW3/Z KJzTQcNuam29Ljl4MGZTs8O2hsY1Y4v2KdP0v7Slr6XDg1FL6anWZl5UprHq6KRtYE5cgWfYnQOM GQRIAcagcTh+gLF+iF4FhDmY1FwO0ZOdaWAFiuU4AUZ7iJ7sTaaraJonZLskbcrEKLUkMU6D2qC2 e/Oy/1ledstcdjHlWmHGEuDmJNV7Jm0L7sYZTGrGh79VMAdPTjEDQ7LEA61ihDtyakVw4LDnUePg D3s268erVUCYpe+GcXpQDj7SMQMjlAo8uPtzDc3nGtFAlxNvVGcaWEWHVMdsD39TvVlqJzS759zr N8ORB0vcd1pLRXBklWtg2tUJGjZhyvWDy4421urpVxUYav1t0G0HFCfFN0OM0IL0VtraJJ1hLiiR HCM+CkcEJEeM5KkOai5/NKdA45PMWshcS5LSciK8YcR6rgnTVFiZGfXipW1kA2sDs3e5rNKLB7Gj qKxQcJpSTXu2mer1Bp3Q4KSOOQsdpbXEGSPKoyfLAxdceaCcc0Z4wUSzNXUhypAUJxODZmvqkvXk 5nI5Uh5bPdezHo5sxrSnEqpewbOiaRSYdpqrl5QKuXxkSLqQWTCGWCiBmwggmI+OU9EqWyysXIlU Hh+X0VYwRycDasy3r63Cs7QycMb7GzEujCpo1H3r0rMIEB+v9SrtWXRIMfjAeMkK7YCq+frxkidA eKHPFSt01IcVCuBsP+35DaqnpT+h2btEsDXoWYK6ucnpV5G7jFw9I6KmoJUgWiZadbWGeKU8yVY6 oKFeNi6fslLlrROQwEVxo1aBOChZTHJeaMdigfHSY/kRTtH03tuBPpJwB2TyJ2jYhLteJn8ChEe4 lzJ51RHyVShIx5RtmbzqiXZPaHaXyW+M9PsblQacaFC1s8wlu71VrvZifxpsVJ4n4icBpARKjCjE 7kCLDNIYH30VO696nbbOZh8yMakOa9LeE+MpL04N4HIILCm7Y/PW0lo3OM/srWi3F6zIB17Pp1/p pdTfffr4tGYY6q43Ui/YC+r1AkfcRi24+dz1Vf56cxu1UJzL2vVlpW1cR33+Nj2yUg+0DgN+pd7/ a/r+x3WjEXZ1f99grF4rechl5EJ+voy8/PX2MnLOylNRbyNXL7mNXD+AOvogf2CwSoGn76vNnEWA FE2PGwc7mjbrz1sroqNbSO3aWP8EDi/Wv5yJozqDEioUHOWoaQ87Ub3BARXNjpqHqxuHNNUpZEuc LmBEKDGh486S6BlIC0Ln1Iz4lwBvnii0Y8F6ZKXe+ZLVWQR4K9MO7UJtdEgkOyCAP0HDJtn1AvgK CFUOa0dozOBozs3tvJcJTEcAX8FIHE5tq/FVbwToCQ0Op0qZmZIlJ4KaGAltSjoUDSOeKy1CTjIm 1eLUJcCvTZu75G4kbe78vIOe9ex9DwFnESAS/Bi6ZkMuDsE3tp6u6/APAS0diqLbxkJ0JR/Yfr5U 0+y+/djLaQyqI8+vUHCOKG17GoPqDTc4oUHqkMk+auEDSYw5IjxoYnwKRAgqs/LBZiNbjL8E+NW1 fSy4G6ftw85PY+hYT9Bjzvm6EO47kncWASJTrRepVHTNiRlIm86g6/AHBtv1AyFOgPBo/XIghOoo 2ysUnKZb2x4IoXpTFk5odp8Eu74sMvJQtUVZR/JBGwHSihtQSJ+gYa84PhLmHWusAenjQcZaL32s gDD1MpYPMQJ7O+Lfa+6KDgl8hERYpqIQYQlqJyLMenoPLaNlT06qQJmXRE8KlqwT8TaWTwXqsytU GdhL73V5LQ8iXD6IHaF7hYJzw5ZtzzVQvTEBJzR775NbS139+ILhTEy3t5NzKhjd0chcgdnTiW0l ru7p7k5odu/62pi99p438yBarYRvLLyp0SrQxGUxMvEm1eYKiMQ6U/upNFBQlPGonmo3rVXOESp9 rrdFRGKyUcTRLES03lml92yg6nBf9T6OvMfy2TXcfRY3V1J+Fc/iiFyVKm9N7fMSPBbK9xqISVzX O98Uz9YlBXxq+tvzdVlCCCFyQnNtMwuWEhuMLGGG8tLFJClcNgf+pT7bD+9+5b7KFqzMnBMmROFz WjjeO6bL8og8mBLH5UirD3Z9HdeaumQTkU6Xb1GIi5T/ZojS2efogrE6XvnKv4CHett3Xfk4jZwW hmWG5kGao88NBoayFHj2vsqIWQRIGfe4cbAzbrE6466AMJUR9lKFrzta3isoe8bIbSW07unIT2iw lNA0KgGe1IirxGweSjZc74uq15I7L71uXiO4CLgp5bjjOc+Su3EEHXZe3tsD02z/ROTUgdkt5kHR JuUfyKkzCJA4ddw42JwqBzh1AoTHqZcTwOoj2oOCdHbenmxSwXQ9hSDv3cqmS1Df5L175Q/ac6lM EiSrem2uqtprGnPxg/Pe0vJqzp4iLelbjpHIWvgVwmViNEskBqGDko4yqXasOywtG6R4f3ysSAEj j275GpiuUOE1r+s+cm9SR8b7Y8ZRB8T760dPnADh7U2Xoyd0R9F7BWXPvak9eqIPRmNl+PO3cPfA NJcU4oofkPcXeOa+DUkVwZEtP4PGOaDlZ/3sgxMgvBV/OftAd/TQV1D2XPHt2Qe6p/KsaNDm7Emh qLSWEy6tJoJZR7yQnHidy6NgTWQlKGnEpAuAW8Wj+2b4fXd/WdDZk1P1Fk49vHF+QORZ4d07ippB gMSp48bB5lQzwKmGokZRlxpT3RFnXUHZk1PbGlPdk2ye0OzeHrKRTZegNun/LcPfkOHHOkDYl3fI ktcBXoIS71lJ+E0WWTsTkgj15G/X1wklClapCHOxPBY8lC9ArSkAM4tBieyjv60Y/OpPc5VV0dMs CI+K1gk55WU5OALesLLSIICZTtR3fV0AJQN1ufxei7KEAYiTXpCCzDMNJoRsrnzl33z1BJpT7X0g MSVbXss8MdrIQoTRewfCyDz5YNfXAc3e81TLqAWl4NQSJzQlkDjzFqyTgV35KrykEtff3hhSB5AZ jiHtXNqJGEMOaNgLPHtfYcAsAjQDSbp+/MAsOqQId9B1+Jcz2PXjB06A8CLcy/EDuiMBv4KyZ4Tb VvzrnoC+otmxAfqyaqCEoF7YRHhioTbAe+KjTkTSIJTmyRremqKyCHhznItVNei6+yKh2ZPxNyj+ K5r7TqaqCI4dtiTpDuhQOLXF9l3XNWZz7MupBdEAp06AsDi1GudikXVE1hUKymSqAqLNqT2J+gkN Dqf6JI3JyRDNTMlOvPKkBOmeZMqooVwwL5q1gyXAm7VhWJy64G6U061nd49zavsyyCM5tY0AkVNv RazD6JA4dUDQfoKGzanrBe0VEOKYrGqci0XWEZxdQdmTU9uCdt2Tr57Q7F2J3RqhLkF9U9ruprTV koLOgaTkNRHWRGKNCCRW+Z1VKnAIT1E4zouFSTDVjVYKUnzASBKci1xCZcdeeg3cCKfw+w7VmkWA RLgD8wxO0LAJd/2N7xUQYnNrNc4A4X4JZc+oZkbh25EqXoHZk/3bMm3dk81WNAidtltj6QWosLmr 4Y39r9g/+ehBSkZClV+LYCyxkDXhnvtMXdRRpacqnIeYyhsG7YiIwpQ3LPsEc4JSy71mdE+F78Ky AZQekedlM1TfZ/SBt4d9IaYF68e1TPCaxw9HbZTzCJA2ynHjYG+Uq2fZTIAs5kZ5OULEzMtXr6Hs uTe1p0+Yjoy+omEv7xH59npvCk56ZYGRnOsptMpVLpwCkQx89jQa73hx4r//x832tA1tZ3v69np7 2kSbCx5lSLQ5LEg+gzmUNtdrbid4d22tqwj4cQKFUeNwdIFCQTRCmxyxta4a52KRzStUr6HsSZtt QbLp6HsnNC8XJN/QplUmRqklidGqEgtCCRgdL6Gh5SUyzZCEcjO0uQktPm0ueBQpYxzWHJ/BHEqb 62W1FR7cdY7DhODIssygcQ4oy6zWHJ8B4dHmpea4PgMroexJm23NcQXT9RTfnzbXDaxr0uYmtPi0 ueBRlFENzx4dpk04+uxuveKzwhN3rWbPI0CizUHjNB7vvWlztRz2DAiPNi/lsGZegXoNZU/abMth TecGzAnNywvIN7S57ib2Jm1uQotPmwseRbk36dmjw7Qpjp1wI+l62WyFJ5vB8JG02UaARJuDxjmg 62K1pvgMCI82LzXFZl5keg1lT9psa4pNR6I7oXl518UNbVLBvObOkCRKoCm0FMRyJgnQyJRNJlk9 V9vchBafNhc8ikSbw7LhCczR3bfrlbEVnrprK908AiTaHDQO+sW5BdEIbSrUZrVL2bCZ11VdQ9mT NtuyYdPR4k5oXn6N4g1tckGdjnUSmAKoIlJGrM2+fLAOzGQj+Gy0uQktPm0ueFTh0OawUu4M5lDa XC+3qvDaF1gfSZttBEi0OW4cbNpcrUU7A8KjzUstmpkXJ01QkHQTbS2a6Ui7zmh2p01KS5YOknAd JBFCArEsCsKYDZprC0KIGdrchBafNhc8inLB7LNHh2lTHyswlmy9oqrCM3cVGM8jwKHNDcZBpk22 Wm42AUKU8FbjXCyyef3RNZQdaZO15Wamo946o9mbNpkAGVNQxAjniYihRJuUWQKZZcp1otnCDG0u oW0KjvFpc8GjOIoyNq8o64E5dl6iHLjAtcKzd52XOI8AiTbHjYNNm6sVZRMgxHmJ1TgXi2xe4HAN ZU/abCvKTEemdUazewMSROGUBZJTSc0FGE68L9yZknLZeQFxljaX0I6OPdiJNrseBZzhBs8eHaVN OLrdna2XPk3w7t3uPoMAiTYHjYM+ErEgGqBNQG13Z5e6MDOvKbmGsidttqVYpqNvqmgw2t2ZslyC 9kSq6bJFUS9bjPV2LK0CCAZSzyXpm9Di02bfowyJNreohODodnc2JISBu7e7zyBAos0x4xzQ7s5G VEKA2u7OLlVCtqspAaR2d9ZWCdm+7gYw2t2Bm8wDzUTkehMJrxelTqIhH1MdVBl5vSumRZvb0KLT 5pJHOQ5tblEJwdHt7mxICAN3b3efQYBEm4PGwW93ZyMqIUBtd2eXKiHb1ZQAUrs7a6uEbF93Axjt 7poaMMAdccxV2vSaWMUy4ckGGkPUiZkZ2tyEFp82FzyKRJtbVEJwdLs7GxLCwN3b3WcQINHmuHGw aXNEJQSo7e7sUiVUV85KKHvSZlslVMF0PYVAm+vuLWjS5hLa+7S7L3kURyXEtqiE4Oh2dzYkhIG7 t7vPIECizXHjYNPmiEoIUNvd2aVKyHY1JYDU7s7aKiHb191AS5jw4iR91YWuTdpcQnufdvclj+K0 u7MtKiE4ut2dDQlh4O7t7jMIkGhz3DjYtDmiEpoA4dHmpUrIdjUlgNTuztoqIdvX3UBLmPBy2lQq 5PJhIWmos98MsaBl+VtJQ3x0nIqZUR7b0OLT5oJHJQ5tblEJwdHt7mxICAP6rpcYzCNAos1x42DT 5ohKaAKER5uXKiHb1ZRcQNmTNtsqIdvX3UBLmPBS2lw31LRJm5vQ4tPmgkdxVEJsi0oIdLMtEZE2 h4QwYO6tEppBgESbg8Y5oN19RCU0AcKjzUuVkO1qSi6g7EmbbZWQ7etuAKPdfd1NAE3a3IQWnzYX PIrU7r5FJQQHX0Mo+ZAQBu57DeE8Ahza3GAcZNrkIyohwLzorxrnYpF1NSWAc9FfAdGmzb7uBlrC hBeLKzXVKWRLnC4wRFCZOO4siZ6BtCB0TnO0uYT2PiqhJY/i0Cb/RSV0AmO41AunZqL2bebvPqRv f1tsUx/1b8taK1/z8XfffQqP6X/Dv3xfFu/Xf05/LQs6/jZ9cP/3zf+zd629jeVG9q/oW2cQ0eL7 0ZsOsBlk98tmF8gk3wbo8Jnxjts2/JgZIJj/nuK1PN2SeKlL+fLKQBsIJrKsls8tksWqYp3D1ebx /m4Dht082Psf7+PDCiF/+4iuLgEfRkR8/kAIq8v04VDl/9P1w4YAcKwuWP6/9x9//OkTur25uRpe /HRzRT7GX25v7h42MWAVwEUgJ/KtAJYapEkwCFOWYlJKYxI24ebTx09h8yk+2Pxn4I9e2X9+COAr /cPqBhAEAHD9CBPQ3X/gsLGBc3+8fvhAVr/zP4fV/8KAftNiNspPMNt3f//22z9/99371R/i3d0f Vx9W7/DvCUw+f3MX7leX199ff/nzzePD99ewAFbD9Qmr3+WXf4KVdXN7GcN6BeOOsTCUS7G6X68k lqsf/7S5//763X+s/nDn89d/9jO18II+ieO5K3Cr+XF47knrMgvyGBgJS9snWOLesdykncV1o4Ih lZKmEAkWfp5hPPbQw8FH80MfjiH5Pd4Zwy9/HsZwwLkdRH5BYJh2R5FoRnKFbb2iF3T1l4NBnLrH 07XQC3dh0+kRSEa3cNcObWCPADyzTNdOFcJ529RHEXSKEhvHpz+7h9JvXq2xmmaSxP17+ul0BukW UL+Qmr4aJLtcVlNhPu5BmTO4L5OyTI0dmtG8nF0wcuPstJtIDuP7CgFqi7eH9cpMYFMjiWU0L++W G7HeNK3DwXrTnZdiC9eZaQMpCeDx89aZRxF08u7txunt3VnLVqj4woe9lE/3+AO4fh6/bejEeY+d RxF0mtcNlJottN7zmn/OEGuHlRmNWCwtnnSh4Txp8YRIQDWLOfa+yv3I5qtkq4xaD8DTHZLsWl/m DSHoDpQ5q5F0vxqZwRBc4WbugZkzoiuybQcw1Un1cm2XkYhumlRBa0Qnlz4COyT+1NAtrEdFZUMu MaDrsQyK1yYfnXnq5Ot/+/hfgivkwIy3UyZWpHsOYKrW65bHTtMB3c9jj3m9vAZ6zD1WnnvHrFfM Y880945EaHnvfB01fMJg8HMNn13ICTX86kP1UcZrWc6dTijHub41e5y542wUQcet65Dj0IyuUz7X wPXdQuudz4mmOsWSWrm0geG3hdbbWHIGY3Wc97ohh+ra2gjzvAFJ12xO7npuOhnKnIFYkUA+gKnO 6Jd3C/33frfQaUcJBFf4hhlpH1FRWmSQDmCqdpsx8ZzhEKEG1SxDPqxA0Pi8ahujCDptKg38xy20 3puKatlUdP7jixmrgfW0hdbbWNNZTwCIdN3jPpNFaykhW4ucWC1UCNeBMi69Q0zj3O0HabaWXiGN nQyQYWPB2QsL4RPND8/95RW/85tf7+4JFWbnHpQ5N6giV3cAU5sQ7OVc3YMNqqEmugO1Qu1iTycZ PexWJOsNYCp2GzmmeJndTqhJ1Zf7U937VVRVBAeTPXVGlqoqDVN2pJg/pay1W9DKpawvXv/r3c93 lw/x4/AUH++jzwWs/1mv3j3cPNirj5c3tzvvZrzlNw++4Ol79z/79LW7H/4V3r+1Dz/AD4/vDnut kyVChNw7HaVGXMuETDIB2cgiM0R6z9O0A8jLTzAJ7jeCJK2DdEi6oBA33iIdMMw4i6lMOMoU04Yn Jj3RERkJoSbXlCOdLEM0GR6cCjKZrI/37tp+igPwn4J99+vpBepjK69V+f7rGv5Jbdnb4efBK5L1 FIQgeWQFZGAGa2Q4CcFLQ7iPG66s9gYyNCVgenClYCIkDTmHzwIMRNokxZzDX8uo2EsOVL6K4W9Z /SRq7nNIFiWFBDxpggwEBEjhAGaWgoUoN4TYCH4h3y7pYI44+GptGEbMGaciU8kFPOfw1woRb8N/ bPgnBd3b4deECOsxLGzsCeKMUWSpcMgw4rxn4C9M3DjGYJYQjTwmAWISp5BLIiAdIR5hjjkf1ZzD X42dd04n5zzOUOXztUqlqSMYWQZTKReyfiePogymHqqrPlKJVB+AqXsLM1z3sBAdijGjNPaISQmh FAROyNqswO6ix9oICOnTTH1fRx6avpqjVKyMUk9HqboS9Nd2fLFmhL1aYuPl/c0GYiEXrSFIBhez fDS4UYiKkTVMEhdx4NLOymo8Zq9Ml1uC0chhPT6PPrw8YDRiqfOKXa805wVG49RCo3iqWi16etWg FQLwzqytP4qgUyW23Ti9K7HThVS2gPqVAs3uPlmR3chQOjVbFYVUjoCRolM4064BksEsTDBlDY09 AE+e96BqFEGfFX+CcTqveDZdA2QLqNuKZ3vFy4piRIZCu6x4VtQAGcBUR6rYIDhHe+W0OzUOjwCO AW5Vhu/eTXtkuPuIH7P2a4QzmIV731jD4T/AU+cVPx5F0Mmnthunt0+dTgLfAurnU/d6XCu8lwyl j4onK5KHBzDVkepwoHqqNz0CVbfe6vZ11VInVXS2tVQpLHaQ6iPhjUQ8BIMsIw5eYR05tlxasTEx ehN1QkwH+JV1CTnic/dWsgYGOGlH5qyl1peN7lMxZIc8mdp5rlwPe+grre88xE+3G804V0QGBCvN II5ZvlZRRxhAizXmSRjL5i3wHDFYvslyEckqLn6TrIKXhwUezdlwpi+VKRR4/v/+5hpm18Wff4n+ ERBualQptcYMPE8ezffvRVb5fX6ov8Fbf7HX8PJueJ3n5eMtGDnChvIQv1nlNz/8QxJnYQkLFEUi +aAin3xEgbBlHEsSwUXSf7x//+nmp7wFDVtgXlVxleOHFfrj9qfbu3hr7+AjRfhjy2mL/urmn39/ uLwC58lh7YXLe3BB/odhKf3mTP/6eL2CtTPEBzA33oPJb2/yk9z/7uY2R1L3H54mw4v+fk7mW//+ Ghb34Ojv4v0t4Ijgbh7fTTlRGhzTsxcChwRxzaO9gh/+dvcY4WfwJvfwZPAGG377FKJ9/n3IUzR7 tM+tItmvXdn7h29/iP7H/Ct2wfJ7P9mry+d/CU7+8d0ULz0vvqGqXcBHivimlIen48PH8W0LrwcA yYUqApzi3mY14NZxHADEF6YIcIps4MwWHMT4DgDKC1kEOOXMfd4pOHDUDvDxsSUyoSVkXnyDDl4B nz7A92vzNsHx8zZBSBZ6q+wTT868dYtIsCPc/xDDm/t7c39v7u/N/b1W97dolPxbXPw5VH72kyc9 AsSYN493Pm4xQwrKv0hk/rr724v/+/l6iGLv4lW09/E/r66+WQ3vXXx+57msdL/616+r56/PP8yE T+Pj+Ly99vHqC3i/vfEluheMuTGsPuYh+ruYGocb/sUKr6y7earP/Ze9uo+TC6dqTcXC7FJ2KDda Q7ewmAprkBEDeOq89wmMIuhooAYpMYCnz6uzNoqg08FA2+zR/XXWGJt8MJABiZ4HA2y3wllRAtqD MufBwIgGVU3YLKOZkTG0p2XDsVUh3yErIXrllOdm7eTgzytPNETgFPvS8cAxwK1X1XQ/bD0y3H20 Thhr1zoBNKbYs7ak1yoj6OS1GgTAt9B6e60GAfCMaMmz37b9zyxw9stnMFbHgGGyhscWXL8NaLI8 eUbS86ZLtqMNSUj1dGAHypxbYVGefABTndEznpHvyTFyJiAFkkhzm09fPWyFmBjEEkmYqoiTYYWt kFRUtDparyyKR2qk3ozm5feEjliPUZ2oxwnxxHKfQa4jDgItLkTmmQ40303fpiyi1gyf9y6MUQSd vHuDsNYWWm/vLhoC+AFQP68lpjsKvWb8NYgBT16brxJwZV/IePtoBrOyZjCpcNj2wMzpZsuSV6RG o8poOkg3nZip1banLdIegzgu8lgDs7A4MDuU5ntN6Bo0ngCe6C8Yemzaizd1jbn49ToEQnjI/X75 bJZZCEudSUikAH8Ccyqc2TDFOI1JIx0jRKwqSuSMw4hza0SQQiSWZmwKPOKCRR/xbCbaayiAptyx vljgOIqgU+DYIDKaofUXGWUNIqNbRIsZq82xLmEsNTnK3gLqF2XLV4NE7XqfiqJChtIpACxrd5Ia j3qLZnZptBPrExVZjI52K0vKkVp7ckbTIXBuqEw0DDHHJ/PJvopgp0VNJnErKfMBuZTJDRriHE1g oqforKbJ+JjIxidljVMeWa044hACIZfbrkLiLAbLTIhhzmCnPlM5LtYh34b/efinEB+etaTA2hp8 G3gyyGd5Ig45DgFtbv9TgVHwdHFzcTHtc0olrpKC37M8lzB1yHDvYbVTFiMl0jo35zSp7kociz4x cVnnp+7qOZZ9wKgTcv1BjGvRbLpBBFg/XX5x1vRBqSXrzo3GWaDuPF0heQuoXxy6K9FLKvpVe1Dm jKfKUrOkJngJaHRR8m6O8x7DAuzahsEmnZ0x07AdOwhJY5Q2WQfetxyNHgN8Mq+0VxG8Pty9mJD6 FJ+qlz5bb5DzAXjmvMonowg6+dRG4yzQ1jJd62gLqJ9P3dU6IhV5oT0oc/rUstZRBtM6Ui/NUU/1 psegntyG91UkKS0FecOIChZyVKydgG+F9NR5yEWiMzwZFrnGdqOYVU5JhiKHVJdzYZClJiLPrTbM KJMCnzP5OLJsOp3ttatymTUuF3j77U28QXgK4KnzNoqPIuizN51gnM57E5+uypUB9WwU57uqXKQi 07QHZca9iZdVuUhN5Cqj6dYobqQOQSiBQhhUSlhmwlFwnYaCo03gAKUt7VDHAL+2RvFjw92n6YW3 q3JlMAuzk3iD8BTAK18cuKRPXfrqQj6dXJbRFTs2Onn8xqErLMu5Pf50zbAtoH4ef6/NpSJ+tAdl To9f1gwjNSGujKbDidmpvv4IVPOmGTbXiRmxXvJoCXKQN0K+GC3SgsasDgb/UxQzHDYiKS5SziKF oYi7fEeHowoRhbkRiWDHX9oe1OJTzHk7ckYRdNwOWItfMX2OZzhp758y6+ECi7OO1giCTttjAwdt C6339tjAQcuIlsweG2jGW2i9jTWdZrwF1C+WmMzy6o5kz/tUGLAZSqc8doTlVeMPb9F04ilFKX2C P+nz5WXca43yXWTwinHigqWYl2IbWuFCdLRemS5Oa9SXjKZbFSAlroIwBlmtOUQzAmIYbyFGsdZq 7jjhZRJCZbgpXhNx8s1cM1YBJnrbAe/CB218Ous3w8tDfb69exxBp+2o3Ti9tyM+dTt6BtRvE+BT N4EM5Us57DlD0DKli45Tk/bBzOlTy8zZ2nW4A5oZd6QZ9qJ2qGesqVY2zwFvp8RnXHyjBqZYlOvo 2kXbNHyjWM2lu469CdLRiNxAsRIMI80NRZYpnpjQ2gWX204nfU4Za5LzCemYm/OVc0g7TGFQPbPJ exKlmfHk94jr1LjPihpnddbmrD5r4WccQadwZDqHf4DWn13EJ3P4nwH1C0f2vB2dDGXOCKBM6q7d jTk2Ui+NAE7Mp45CfbvnfK5dggeFmZIcKRFxZg9p5KR0KBlhGfbaGyY2SUr46siQpxyGUUkPuwNs EbAXcGVJABhz9v0cWTamk/c/uM9XUzF0w1WmIqf61d7QEX+5vbl72EwRSZ71jo4pZuNkoXs6hPnt ng54eXBPB2ECP93TYUr3dDzPykofIqVPIq9L3aU8QbF5pqtWjjx09rev4y5lbTQleRDpBS3cpTw1 kKJrzorHDB2zo+m89gyPn7fwlREseWjVaJwFDq3k5EhzC6hfpCl3t8xxnt4+lDkjzTJ7vA5GdLr5 kcvmekgGs3QL1XRxhgzvvH274wg6rfh24/Re8ZOVKwZAXft2d/Ui6DgZdh/KnCu+rHtQu5ZtQNNP 4FlhFX2CDFMBGO4hIbIQIaHgILQzjKsUizXmY4BfW9/useHuw4XgzdznZzCL+tTp9N4M77x9uwOC JTtjG42zQGfsZO7zM6B+PnWX+0zHybD7UOb0qWXuM61QiQc0PUQYT/SmR6C+dcbOxtNTPjIXWEQk YQ4jFP3TCCU1fIcSwZCNFdJj4gRSQ/tsUhE5E+CvMuyShTH0ZE4tmSPL5kt665x7kz44da9Pw0H0 YqEyzaQhnadMc+Shh7X3Kso0hGthcpmGXcgXlGnYmuuFpUr5dK57hmfOKq4yjqCfgcR0wiXAE/i8 Edgogj4R2AnG6RyBCTy5+XppY7UtNYEXCFcny0o8A+oWrsJMeiVI+K7ABR1n6u9DmTFwFmUScfXg IqOZMXDea75OLijuPIqEQNzlmELaQYTGORZJOm+SFqXweZyTO+DtI7nEy/IgtMJofkbTx3pCJCIh V4CYCf4DMRIErEETCJyk4j5FEaIcuWJj3HqSFjVxOm6F05moGR4r6sQtuBVKtqBoUKtxCnnj3Fvh ZJruM6B+3v0zTbfWJcnW6rSj15MSmUmszXkSmQm+uyjNd966bN1djijUnRNwdaseBOo6JOsCl1vk xymj+2DmjBvKVHRa4XcPaArMirlIWzhIzhwCEAHx5BiyNF+FwqyQ1gmnRi+XqllvYRaPmKwOcRZ0 rCHmGtD1qFmZ0WVQnXknizH3ciJH1m0fWWDRTvXOaDrpMk7cq19N/Q2iaGZUrr+RCzGh/lZ9qLNe WzeOoKP7OGThNaPrFFNP5/Y/Q+sdU0/n9i9urOncfoCmcSF6m9tYk7n9z4D6JSBTuf39kext1hVa Z4bSKUwsc/tpjey9RTM7k/K0whKrEBM72q3M6mc1WmdGM2N4PUNJqQq1z110033mCIJOPrOBgJ6h Fcqrc/vM6QT0LaB+nqolLtB06XO5BrIewGPnPpcbQdBpXjcap/9RkxCfq2G0isbk/HUpyskUOtnL SoCTF7NhXc/XdumbrMJC3oMy5/ZZFnBg9TzTsB5Xi02vS+1ApeN242sp+9BexSHttbaGMpCT+oF6 tMZQgYV8ao1hhdR8chgFD6Xe7m2bixvrcaQCjIycjvlqLxaQsToLJSiGmcyLQm4yBdZIaxEWLuUL uQLSSUtkceI8GGeNVDP22lXdQB7+t3vb5hIhVY4KqSNHSeaLFWXmreOQYByscwbDpynZBKypSyEg kbssObcJaUUiCp4rL4XFRMg5h7+S0eXR71R4ZeWydS0TzmiKrI+3uXjCXMTSGZho+X7PAImsUwzp SFW+TFLSZGyEKTqIucz5uSSY9z5QhJODue0NRsZrgZyRTtgQBWa7oi8/5Lm9Xn3lY5UMMyJRigjn HnYE55GzRMHyCNRrak0KOI/BrJ+jSmEbTUTCKngK2LcQ/DONpErgnqzXRoW9sXIv3IZqfqjPNQKC F/0QOxLnlUWlZmlZ4hILYyiiwsDiIcYix8H0TiVIDY0OBKxXaPiq1DMz3j43zYuyTN0x6726DoYj 6UWvDgbRTIXLYOjCBZ8GzjzAM+ftPhtF0Kng026c3gWf6YICGVDX7rNdQQFW4fDvQZkzhy8LCrDK XeXPaGY/OjjNtR+BqsraxW8x8QlxVsgXPTj4hgRJOoQ8HCPnCKRrOvGkrPaR+xw/zfo5LjlgFRIR G2BaUA8PgCHO0jiR4CVPLhTu3/7qY2JpZHA4cUSDxFk/Ej6WvEXM6azcwTzTQ14y6+c8k8JjyMi5 UhyWMGPICscRIHNEMe190ntj5d7GasMUxcpBlhFiNPBZ4pBWkPZREZyzjGtIRvIYzPo5BjmmozEX mAElp9ggyxVGLFLiIJOxwpO9sfIvyV+q25vCfeQURLtEDV9runQM2aDCAvDOzGAYRdAphmw0zgIM hukSNVtA/WLIXYkaVtEs2YMyZwxZlqhhNcWXjKZbd7fkHDtuIqKR+MzGd8gFFZHAnktFo9G0JKpw FPDJBz298vIjw91H9ku0S9RswSzqUxtUWADemYX+RhF0NNAh77eGbkEZwtah6y9DKKYL6GwB9fP4 uwI6rKIEsgdlTo9fFtBhNWWMjKZ4VPqyqkGDr2+ZVKIPhaFhxZURdFpxDRoAW2i9V9x0DYAtoH4r zrRN8pOF7L6KelOT9JISmKnkUYxOIW50QEZzSGtz97WR0lPmN4FbSsHCyOvsAIzgSBNFUOSU8kS4 seSll5LuDH+FzZlnIu3jcMvSAaxG3t2i6RNiuyi0TlEjRTRG3EmHYPQdSphgjSknjheLtfXtSvQh AYpD4apJ1nt18f6Rudcp3v83e1fbG8eNpP+Kv3mNEyW+vxjIAYfc3mEPGyyQTfLlcEjIbnKjPdkS JDtZYJH/fsXWjO2e4bCbra4eHSLAgEej0cwzRfJhFVlPlVvi76uNW+PqBj0dwNNn7UZ7GgHO/r7A OMj7u6Yt+7vG9Kj1QbpnRaecoeA0kdPlyjqiVuxgh2Ztj3optU9ANS8lKdfyi2Log1CKkS7r0mRn HXEiGcIDD4n63vQ6XlGqregjvGFnPJG9tPCG4EExLyl1PBhGV02TrS8bw1H2Jn1c5aJ2HawuRM4E eqbtY64fbq+SlSFm6ZzucwZgThj0wcBccUKzEGkvtV+1d8yUvfKx1BZ9YyS3nxL24eFh3xgpgY1s Tth3yhT6xszdeNWFlhufROqGEkwAT533JPIkAiTPpN042J7J/PpUO0B4ngkbU2ylBkiGgnPWp8u1 e0StgsoOzRM9k78eeiadV0E7wUhKOQdBp5xyGzuimAgp0N4Gz2EQ//t/jp2TKbStIo6/HjonyzbN iRHFERfo01VdJsBsSpsNVTYA3pkDuoxgw74trcbB79uieQttfhlFrU+bfLzIKrnfB1DWpM1yLQtR U/BkNE/vCn5Em07bvldGkb53GiIBkTvYcQgMHIe4JIkIPuUJ2lyEFp82J0YUp/235ktoc+t2V7qh 3o56VC+elzbNhq2UW41TmN5r0+b8YkRqrDZcnzbHJYBkRfh4AGVN2iyXspE1DbY6JXx8Gm3OK2VT os1laNFpc2pEcSRkWiyhTVMst4NImw0ldwDemTvQZwQb1m1pNQ5+PwU9vx7RDhAebcrxIqvoNA+g rEmb5RImsla7IKMp5io8jTbndaAv0uYitPi0OTGiOJnrWi6hza379OiGik4Az523jNtJBEi02W4c bNpULbTpMCs/6XHlp3zNUINicWjzuILRHkx1pJ4ubz+iTSpZMNxbEiU4mtIoSRxnigjaM+2ijc6c ONtchhafNidGFEdyr9tF4+rC0KKeBZE2G3TR6lEcdVbaPIEAiTbbjYNNm/NF4xmQw6TNsWhcVlR1 B1DWpM2yaFzWr5INK9a1eBptckm96XOFPC1ElhAz4lwK8MGmYzZZyU96m4vQ4tNmfUQZxaHNdp1k BrNxpyfdIAXM8M5bM/8kAiTabDMOx6+1oefrJHeA8GhzrJOUFeHcAZQ1abOsk5Q12WFG8/RaG8e0 SSlE6UIRbjpFpFSCONZLwpjrDDdOSClP0OYitPi0OTGiSOln7VLIHZhNabNBTwfwziwvP4kAiTYb jYMvfdLzxYY7QHi0ORYbyop64wDKmrRZFhvKmg4roymkij2VNsE7U33sNLHSByL7DrxNyhwRiSXK TaTJiRO0uQgtPm1OjCiOHkfbJbQpNu5fphtEkQBPnlcxmhFsmbfZaJwN8jbnK0Z3gPBoc6wYlRXZ 1gGUNWmzLBmUNdFbRoOQt+lEL712gqQIobkUlpMQgDtj1D75IEV/kjYXocWnzYkRxRHi6XYhXgaz cd6madCaATx17rzNEwhwaHOBcZBp08wX4u0AodGmGQvxZEVRlKHg5G2ashBP1tRtOzSre5vacSVM IEoPPchk7kHW5wrTRndCMqHMqSB9EVp82pwYUZx0d0OX0KbaOG/TNAlhTDnZfEva3DLdfYFxsGmz RSVkUNPdzVglJKuaEqNx8jZNWSUk67obg5HuLrhNvKOJyJS7iPDca2YQDYU+5jKlPc89lIq0uQgt Pm1OjChOurtZohIyeuO8TdMkhDHlZPMtaXPLdPcFxsGmzRaVkDGYeZtmrBKSVU3JCMqatFlWCcm6 7sZgpLsbaoUV3BPPfKbNYIjTLBEeXUf7rjeR2RO0OYX2THmbEyOKk+5ulqiEjNm4TalpEsKYcrL5 lrS5Zbr7AuNg02aLSshYzLxNM1YJqaqmZARlTdosq4RUXXdjMNLd53WtKNHmJNrz5G1OjShOurtZ ohIyW6e7myYhjDl7uvsJBEi02W4cbNpsUQmZL3PM16fNsUpIVTUlBind3ZRVQqquuzE5VXv1IH1W o+MibS5Ci0+b1RG1FCdv0yxRCdmt091NkxDGnj3d/QQCJNpsNw42bbaohCxqursZq4RytaWZUNak zbJKKIOpjhQKbWrdJfiwLhqRK/9Z4oRR8EhIFnrPqTxRymMK7ZnS3SdGFCnd3SxRCdmt091NkxDG nj3d/QQCJNpsM84G6e6mRSVkUdPdzVglpKqaEouU7m7KKiFV191YjHT3eSVti7S5CC0+bU6MKMeh zSUqoc27qZkmIcy5u6mpx7Zcm9Fmo3Hw091Ni0oIt5uaGauEVFVTgtVNzZRVQqquu1mjm9oRbc7r rVOkzUVo8WlzYkRx0t3NEpWQ3Trd3TQJYezZ091PIECizUbjFKb32rTZohKyqOnuZqwSUlVNiUVK dzdllZCq626KzeOeLK401MQuOeINwJCdTsRz70gfmFBOSJPiKdpchBafNidGFCfd3Rx37antKvqC 5d8/09r+H+K7uysrchl73ZO+6x2RVOTMNBuJ9B7CEJmU82Ld4v4TBsvtmLYo7i+k2hf3zw8Pi/sL 64Rwubi/tfoJxf3thbEb75q2wZ8GeO68u2ZGUIzGcHbNVuPg75p2frCxA4S2a9qDYKMiKTqAsuKu aU8EG1UwjuEcL9h2PzmD2VjfZI9dweeEbn5X3zOgcy1cvj26BpUBwOPbKNdqEMonR9tC2ObWoQZB njfN5iQCpE3Vzd/BBmB4O1iDBjxjwb+5tu5N07BteM1foL4zG8vR52usRgLYwFiszVgb3oQWHJKq sQppCWsvw/lnUztAaBzlxppiVVGgHkBZ08sul2JQNZUuoFFPr5d4onvzPGXxcDw1fxwV5o22G4sc VUUSdwBlxXF0ZW24qskGM5qn32ifGMd5ZdyOjxkrE0/QC+pa+1NgNL6eN/H2cPGcnIPD7dOiogEK jt7BldW1qiK82qPBmXjzqlUXJ17NejjaAttcN2UPZtPok88mlQFea5ozxhqd63cAYEbPqiE7jQBx RI91ZDV0G8aQbr4weA8N27/mx/51DRI7a/2LAcGGQu7CTKoaB1/I7cSbuQ5ZRsRxahS4z6VdZrsL Iyzr+6ljFtenxXSHUNaMN8p34boiOBzQoLkL8xr3HrsLdf/mBODn6qcOcPEm3liXq0/LkQYoOAIz Vy5nUHeaR2DWJAd2lD+wt0x1Uj29uct/LqvkNcz/hh2Ib6x+dbKNTv6/eajirPcIpxEg+RTz5bp7 aNg+hZx9xnraWIjTf75QL8Mr193ccjaVESDNpnbjYM8m9abFFZM4xQetXeChKsw2v4cszmdDWdNR KBfwyGBq00Y9vVzc0d7ccIbaMIKoMcZYFKtPa4EOoaw5guVaArqil9qjWXsEFx5G6tNigAEp0lFu WUysK5maezRr221hVDblQKvFft/Y48u+3heP//n61/vrD/HHIY/0x4fYZQ/vz+Dtfbj94G9+vL69 Gz2b8ZafPHqDx/c9fO3j245f/Bs8f+c//Aw/fHx9nAicPDBULwWRUVsirU7EJdcTH0UUjumuk+mK exep6iQRvovwoiRI4F0inQghRmmsYPbq+h24tA9XiiVrex2IDr0h0nWe2J4CN3nKdaJRp5iuZBK6 YzYSp2HmS8slsckLwpOTfTC9Ti4Xbnv93r+LA/Bfev/6tyYvumWm6lYh+e9r+G3PhdRdIMJSeKWH PcfqDuJBGnQP2w1VUuyH3zKmfEcpMbRjRArBiecqwCploesE08BuV0GIaCWzpKMM2E8GQ0JSPbER 9i4RROiiWXP46wcLGkew73g5lp/wF3SrOvf3NRedBjLqUiK+CyIfSzjifIS5CDOLpz4y4Kn9XJR9 Z1guOqEUyyyjYLNw1BInWd932jHZxStpvO0cbCZGAVVJY4CUkoXtuMtVKpj2Sas15+KEs/Ey/Kvt RAw4pss8FTUHXyFZRpwSEpipN4ZqJfqorxjzEfao3IIzwBwJ8NbWCUpEcMFEYVLo6ZrDXw9dNE6W uhNlKqp7kxpHWupUGUw9JNA4gi0nD8FYrgblQmWRSrlEg7SNaCv+4+72/sNV7CkEiE6ToPKK8twB Q/aOUC7A/zLGwsa7qmxrjtnyhdsm0i3lPkm34OGhdEtSOeg2snTLLJduwTeCSGTT8zxN5yeZAjxD z33jfAIBynneEuPgnucBorlJpQMgxCO0bJwR256Wxx5CWe8gAUCUDxIqEuIBzYoHMONL3nkFxI+P E6YAt5bBxbhGahluh7G5Pg53axLbAGZTTm1KLDLsrJUeTyNA4tR242Bz6ux2DHtAeJx6kDhzWoF6 CGVNTi3nL+hKF8IBzdMrPR4dzi5l0wmouRjXS0h8+kRECAf+fEeE1gbC3JiI97mBUIgdtU6ZTqV9 SKyVp4H2kqgOYgPZQ0TgBcttgamNknqpvbpyMXYu2kSE7eFXPiQSWJcvKJJ3MMDJBrZmSFxfNhwl 8HtcNuMotDYN+QU3+WTmBigrhy8yT0qUqC/HXLMOudaptjHxpa1a8qWPYzb2L3QUs3358xCzDTh3 QZu8ZBCWjaM2pmG+qxy08Uv56pujoO3vD7fvYY5c/vEfsfsIEK9qehF+IXK+aR6St2+VU5+/1Xfw 1Df+PTy8Hx7n2fXxDswcYVv4EN+8yk9+9VP0KWgRHFF9vobKvRQsN5pY4XiMVMgu0J/evn13+0ve SIaNLK+N+Cp7Aa/Iv+5+uruPd/4eXlKEf2pR7NDf3P7t+w/XN0CBWUvYXz8AkXQ/DwviEyV++/H9 K1gBwy4Ps+Mt2PzuNn+Thz/c3mV/6OGrx+nQ+vn6y8/PYqHWz7+AJTrQ9X18uAMcEUjj4+s5Nw0D vey5BGgFvJOP/gZ++O7+Y4SfgRMe4JvBE2L47aOj9fn3fZ6jmZfytOLaAHtmdrrxDx++/jl2/5t/ ZS+HM99f/M31/i+Bqj++nsO1q+JjXFGlj/CZS13El6wM0TtGdB9i3oE58cHAhuGEZiEC82s/Hx+d xkdzdSBrCwBpEeCcckKrGnBXqOcIoL7kRYBzDs5WtuBwHHUEkF26IsA5J+DrTkEmrTo2oDu1RGbs XeviG3aGI3y0MAN/a98m9H6bYCwXQqrsE49k3rpFJNgRHn6O/Qv9vdDfC/290N9zpb9NveRPfvFn V3nPk4u+AviYtx/vu7jDDIGk/OL26dvxby//8uv7wYu9jzfRP8R/u7l582p47vLzM/vDoYdX//zt 1f7t8w8r4bN0Gl/n33fx5gt4n574Et0Txtw5UR/zPnb3MTUON/zFK/rKh9vHU7b/8DcP8VM4WjtY 57tye9vE4LMyDdaJwetf2tJnE4MD18jh4lRcqkIMPvcMW1wIvW2dNM3m133K8Mw2ddIqBiojwDnk X2Ac5EN+5mYf8mdAiK3qs3FGp5WVg5EDKCse8rNyNR5do/KMZkXtwUEVF0m96XPzOw0+rOQyJ1Cl AB9vOmbBD+e0Kx31TwF+DlVcWoYbpSHz43C3XZxmMNtW/9B8fnm4DM+eV1yWEWzXkLnZOOgNmQFR C6daRDVQNs5okVUKMRxAWZFTebkylq4VscloEFQtS9l0Aqp7ySVeK5fY9j1jss93ovnkS3gIa4JL RKUePoJKroK7As9I8pgssTEGIk3UJLgAoZD0DmIipZJIa16c1pfNl+W3V9ybOF2wN8mNG2pp3rIx yHJZZER0DVk1AO/MJZMHCOfNesoItitPWpo9VeOglycFRPPrDGxtrMbJvIGx5qeI7QDheTqsAYlE rEaah2m0eVTq2xxAWdPnOpEAXCuatEODVFVWCtXHDnZz6XMaVAeeF2WOiMQS5SbS5ETB8zKVwgwZ L0qbPRjCovVMrajNDg2O9QS3iXc0EZlETvjLV4GDODv0UXTC9tz1B9arCbHFhcknzhsd8M5StT7t gPfmOvxyff/hj7+AXf98e3tX/e6AAobkHThrIk8Z+LvL/NMfbt9/+93X3w+3K29gKP7Uf/WTAAMb GToS2VCbSeTaTLEDR5OqpEPnklU/vX37Pv766v5DB3BTNgo3lOqWdWdLEomzHsdMTPXHo/LnBLhO s5ai1Bh8pNljiV2dtdDA8AUBgy2XFkV0yecXPs3otj5qkzMJ1Gb1xPO4LOKPutN8WVRK2GzwGS3b pspZDQI/r5QuI9j4yowfl8dqRocUcTSUlt1Bw444REPEYTGruWbjjDi/UvsiQ0GpGAQgyp7qBHmV yy8+7Wy1wcNvmVTi7HwgtlxxDYUXd9CwV1xD4cWtjdVQV3AjY6kWehKo9CSbGEEsvhD+XVxhtFRm StIDL3c9CSnLuqxLxDLgxRSDtzy5LiZ21SXjXTAd8dZIIqnkJORU1T5JEXsvXB/7J15hNExExOLT eZWOJiKfDWXNfbJckTCDqa4KhDvIBWc5A9RK2ZaMFEf0yssVCSecHYFStPvR2SlE2xODWC6P+EJt e2qb03h8X+mJwk4Obh44dQmoLbFAgoyBZDmA6QUHpy9eXV7Oe50xSZpk4Pci8yTlgTjZdbAQuIiR M+1DWPEWd4J4FE5pBn5U92jGanaW4YBRC06InN36Slm3uHXuvG3TBwRFAyH5vI3GKUj61/Z5dYPP 6zDbpmfjjBZZpexahiJwtkxd3jJrpVF3aHAuj5zowSN1AhzQTMbCgqsZIDqPUfvkA7Bv+eptCvCz u7OYGG6UIu6Pw93MqW7bdmOam/m0IS8oPW8KaUawZQppo3E2SCE1szl1BwiPU814kVUKXB1AWZNT TZlTa4KejAYhfFvKphNQWZH+X4KUBSmkEEyb3nc9oTYoeNdOk9BBLBKDk8mJKC31V0Z4E4wWJMrO 5M4OjnjuIumkt04441Iv1ww+6suGIe1NpnlvkqeaTSLuTfN7lGd45d6KG+5NjG25NzUaZ4O9aXYD 9z0gvL1p3H/ZVApcZShIR4vlhoqmVvBphwbJ39e275VRpO+H2mMiK+M5UKfjQLQJCFD70g41Bfi5 1dqcGm6kAx27hFM53ZhTG5Sm8rGv4Xk5tYwAiVMbjbNBIvV8Ge4OEB6njmW4pqLLPICyJqeWZbim pmoFNGLF9OWnsukU1MWnJ78Lf7/lvpX5TsvoGQkQmUFEFj2xisdcVRP+GU4F7a9UMlKlHKcpx4kM uUdF4IYwQ6VTidEg15SMTSwbgeTvu6PLhtrdmLzQZrs08FnlU9ep8zHxpQcN8DNJ3ZTUzk/dPDWj 1AUr30HgORiiIbFUXXBa9H82czBOIsBxMBYYB9nBEPM16RkQQ3QwxFiTbiri2gMoKzoYoqxJNzWh d0ZTuE5bSeETte4SfGSX+2jJzlqS22LBIyFZ6D2nsuhmTAFefKyIFbRNDDdOEWrRrqXOYDZOBBcN mfIAj523dtJJBIgGOlbANqNDYvzGocNviSPmt8TZAcJj/HFLHFORah1AWZPxy4pYU1PmZTQIXaUb uL5lUvHzJsJkBBsmwhS4oGoc/EQYMV8NvwOEt+J40yR/afyxWuMP2rleBx5JGOrXKEGJlY4TL4xM Qlkb+pAzJGe9zjjvUugSsTFLakwIxAbKYVA74VPXsajdiocWtiJbzbMEJ19KlMX2tqZA3qHBccVT kqZXzhFvrYSJoWA6dN4Q6723MkiI2UtFoia2NaT2KqKsQJ603nNL3pqaezgHZqJdMp3BbJy8JRpk mADvzIq5kwiQ/IB242D7AfM1qhkQZjNKMdao2ops4wDKmgRf1qjamiJth2Ztz3shtU9CXXw1/uI/ HfhPsjdUGC2JUZFmdZQlQetAklNe0M52TqirpDW8dRSk4xKG0egO/CZwnsBLksazHmCsmbw1sWy+ LOG05t50rHaqtbRXF0LqZ9st+/rh9mpO05BVW2VP2WurNtmS20+3QPDwsE22ppw9FvAwjBXaZB/1 Fah9LX3Bsq+0383fwpPw8v1b/Bf8/+1d93lePuRH3+43lq+BNvO+8vqHby7fxXtYHPCFXoX76/5v e6b6CCxx98Ptzfff/+nfh6U1s4khrMF3u7+Y170SXtjfX/8y8CA8vH3nr9/vP3JOd5zhI29vPr6L rUDzzN39zRx6G/5mYK/dH83p/fh66GESYO3/et3v2JUO7/T32/DJtiJ6baPQsGJgz5I+wTspCWun U4HqPmlPHz8++If45aAobQNPxhDa94lIIeEPpcoGS12gsY9d0K8/k1xN6QETSppnyysf4ru7eQrG VYildSlmd+UJbV1gn1M+wg5Iuc3zPMBCga2RWyZ72B6T6sTTmh9WVIg79IuaH8JM+OFx6b1Pt3/4 P/aetLltHNnv+ytU+aJkn2ADIIjDVX5V2SSzO/VylZOd92FqKgOSoM2NrtXhid9U/vvrBklbByWR sih7Zj0fMjIvdDcafQLd0wT58rz2up2Oi+drrr5scFm8UGvl9TrX+Uo5r7lOep3R9haOu5ggqGg8 w4zZ3HimEMhLXhk88x7MmfNfy5fqUPNX+BIg+msciIiFOiaGJRZWiAtxs0xMBDM8MAbAZhFwUmmT /GannYFNkJE6Xb9Op1d24k7BshmcTvPxT6+mg5PxTbcDGsN1ugELTbdjZ53u0uR3D0OtUIQbqXWG DYQuQay4SXkHFtDkBtfEbNQp7902D+p0m1Cw6w0z3CzSmd2M4WVPieRAeEnMdzbB62JPJLIpyAfn TkCO/eaxQerALJfodJ6zDjjRoG87c7AJ9mByUYEeD2ox+eXEDmfH4fC/41DguRd2+F5o5gLdV3Ta ItBLYrwsuoo1FOpfbntdlR846zShS+d5PrP7zeR+banqYYZtqdhqW6qmmgmjE17IAJR0gcl+Ki6i MYLy50UH/y2n+6xT6KJaFCz1Sh2tUuoUMNjr6JRO81kpe1OVGMqAbTHE3hVmTmM7bMEomn7Nxueh RnsoZHzPfXxbvLMdVqfax5vdwzsLRFh6Z/hz1TsDzwDkCnpnhtIDeGfK7DmRe+w/xGkrUMOfq6gp Sjn3bcZUcAjHE1EbAAJwW67gBMj4Dolgj75+/TcE6UUH0Djr/LyCBYZhVvDAS40x6f4ChmT2f+6s A883FS+YjLgVL4HaJl5qrYY6EuS0ltzIUJqBMv/9LpLV/fA/SKHcQ8a/a5rZ6B2jSYOv/Pj+85uL 9y/f+g+5aTzJvMWLt35/9jqbfn3Zz+z02dkzMP2/frHjL3iNPev5e6/vXoAnnn3v+pDhBJw4fJ8W /5GKf8r/8A0wswbWv/Hqw//iBe9S4981rfoumC8+3NdlAuODQorA4wMglnj6Xqx9dwmsOPONPd++ +XuO9aB82QMDjo1HADnIf5FrLQxlOm9BG19l/QTulxkv9EnxMX/Pjm1cfJ2D6yx0IIXOcZzPfY/Q 2n5HdzYBtVUNQ4nSp48vLz696Va0n9zN43t1ll+y76u6y3eRSX2I9YlHn3h0kUermXSbSkEu3bMt eD1rdKEt+BPfPvHt4fn2qIG/Rv2cd6HAjtnPed4soAFW8G0s5AIcymZv98CE+gnNUZhjbNjd7Yyi f8Fv9E+6/73HbFeSqs3W0vsAJNnmaMzdrFXH0PytfUNoBwIfs+wNwL/ILyR7B/6e08WQ2LR5JKUa DbEj5LeGxr0ifj0UALgxAT0p8OHBlrsBLsfbv9nMR2AwwJnNTpqjx6vQC3kj9N6POhXg7Dllvc4r +Pwk3xrj/cf9sGo7/LXWlX1/IPdRL4JRGzMmMAEbEBEBLY12AdGxtUprFilm2swr6T99Xqlmrvl+ eSX98HmlxEidCgNETIS3tUJgosARTZ0NAqFMFD+WvFIltf4EeaVKvP48eSX9wHmlmhx+37ySvnde qaZQf7i8kt43r1QTs0PklbB1+mPNK9XRKc3zSvqB8kqSHjyv1BR1Ix9NJkbqQBvm8xfCHCATg6g9 TCZmHZN7ZWKMeoBMTK2V1lIm5u2blz9URAqX4351Q1wHjvtJGlClDhX3CwOpFTWM04OE/Wqa3Eth v2UQ7ptRQV79Q2VUnnjt0fBa8xAfctt+mZGa9lT7mZEn/vtT8N9RQ1CHzHAY/ZThuBepHjDDUQnQ HyfDUQ3+Hy7DgcHGP3GGoxq9P3qGA7FqOxBTK8Ox2Zb2ZXeuB2dnwuDho+tsMjvBP5/7U2kvOteD H5PzX+scJPsVESyOS/uX83NsxSmr83pHrCqhf3Nx8eFiE/iyBD8M7wv+W1yfOeipzfqw3J8D9Ged OpC/+MvniY1dZOOvneeD0XSGXIOnpf0B6b6dzl6c/aXT+QE+23m2GpBHsE+vMRj/rJdH4wEd1sPg vQcHXux0MLADgmP7N7JJbicufEpq/6HUf2TiZp3zjp3NJs//aieX017nr3/9+hv+WvlyP4tOxzez q9GQn6jTaTZzBMylrxZPDvvx4AEc8q6cwsKQjAs/ZuGILYyc3nfYOfp8i0MFLKgaC/26dD6Mn2fD 6QyG2zWqFBvHLTBdJGlBUx/TejUaDLKZHxdEokfzvEPYWWdiM9BzxdtvJhOQRc+78Pu1n6G/3b37 /EXBbyB0YPrOp64P/LT44lk+FAaAbsAWy4ClcnVy1kGjueOPXXdsH7d73yBkhbLJ3wImrl1TQvVC lFpHLbrRoBOz6kn6sBXUN0LQUtGN5sRpu+iGqF10owCovaIby52XkQe2gdJOBfWguuGt3lYauYCm napKXFHlYlRSCoARsQTzhltDkogFoQmESl1VgdNdALPKku8PWcho+3Szdjp7BqJxISMEprJeX4sy tUHDdgCvumnGMWVqNQQtydTmxGlbptbvZl8A1J5MXW4irrd4DwhKS0Wjq5th623+VgHNoQsZ7StN d4FaWXPpqZDRHl3oVOyCKAkcYSkVMEMuzmcoVf4bKkwMO7WhjCmLQqJ864pUORKZBEYNaJRamMOY JYcsZLRj2ch2dNNd1+k1P3pbV0Td41gt/J6lbP4BXu8JZtje/TDv999m01lVUZvrgb8DbFuzPs0v FSHLHchgtHQBmaA+MkVGsA4uPwMyNh5nb4Y26uc8g1kAXyXGDeZ9C9LrnY2vsmF+cxyTTAiafiOw JPrqhJ/4/MDYZxq6TGnpXx24wd/n1m9ycsmnPKUQBkb1/PftcJrnO/4xv3Qfkf0XB8YMyrhvbz6O JrP8hIYJ+ML1T0C+iVu8G/gxp+Msdp+m/VfZ+MpNPs1hpfsHXr/54eU/3372z8Tj+ec8lTHvvndX tu8G+buDsb+WFxK6RDK+nw/se1gXUy+c5t0h/P5xmLhvuFCKb+Vw0x7r8V7QE72wJ3uqJMBocpN/ kxrpFyEOA+ogtpNkgdwpxrty4ObA94NuXsDIXQM2X1iQWmU4I06GIBWYtUQzEBJOM7DGAkOFiotH Y2u5ClJ4NFGSCGNjEDZOE7A+eJowbagUOOCznwav/Rtnv2fJebf868fk7Pek+HXerfWxXh6YqrcA 8ChS/v3z7nyYfYM/Mal03n31j5fv37/BpFg0Gs0+TBI3OfdZsenYxR+BhwbT8+7Pv8AFmyQTkCXn 3d+j+fQc5gCMx9lk1Acpgn/5z2GwIRsRWA+Z7fc6Y+CSc4ZjD3yoEqAt6T3uzy/9hYLtUOx9GPZv 7p7IwfXHrYAiIK+Gro+A5fMEBh3otlnmSvCmQzueXo1mSBIsDASX+qPLDFSy305YXpujlfkpnmYX DletRdu3HPP7M1/HqpW5Lx41Wus0FSGxwoBOCm1IIik4iQ0I3ognPFZRbTap9bE/CpvwA7IJe1xs UntCa31s7wnNEnc3nx/ef7748Pbtm4tGUzrtj2bn9Btl8BLO7jdKfbjRZkP/By1neBxnPR879YSj 3w4iBHIEHtHEtrP+i0ej0CWRS1MiqMavJgHRaHIyq0JtjLFUJbU5q9bH9uYsr/uvB/ERxUVwQHHB W+aq73lBxVvT5+v1oLRSCuMMjZTCEHo1Anp8BAvKxwD882VFxvelX/HTu5MiRROP8k8NAaluWZZw Yd/PP8f5SPbbu9vBBDMioMJXIoTZeeeG80qDCJmh26gEJKYWPtrpzK3bs6X5+OPYW6p5oUAuT4ws LcDc9UHUkQS3iOfmIJ6ufgfuLJAZLbTvdxanN0df5ROZG4RTpEwPZmQ8n017U7g/HU16fnRMrfWm eS62Ny1A6k3n02jikgweujURPVRY0OYz2supm1SgBER/B2ap3w00mfVBPJje+Nq/+dXdRCP4zFt7 gztx8S03JDAnvXz+qwz+cTa7cNnwX3lAbmUyCljfu9lvo8lXf3OE62JwOZh178RWbi53Z3Zy6ZlH c0OLub5bermNOyhAn3fz9YVFLrsWF4a/GIFLNhoNab7lC799y7/FrYVhC0axIY0CCTLQxEFChOCU RCELSOAMSwNGRUzzspnFqs8BQaWS8+43mnNDVJr234rnbzeu5deKq7MSINA2+XwVCqd8MK/bmYPo r8GKWwR+tooRPA8AZYWfUaymO5pMYYXT7joxv6/T6E6MrZPJOCysayTRLo2JSLQjxgSKxExaC/cc /LWdTEG7ZEI8c7mxwiPk9s6sAtGceuukuSPgkiSvYq1tZJNBHDNuNAlTagnwkiHY7JXE2oSMM5BS 0Q6yiXbJtoRdIyrBu/DTPxlwJXNpWl6RYRjk6+IKBeSiSrgEjZA/xGSgRXeV2okbVVHZ31gncKwS o0MeEelEhKG1iFgJwj6VwqUupDZM+HYC83YJ/O9vK2TNEUGKdmf9aVVQYpnK2wX6/gpsu3iuEA+X Ezu+yuJphXAIhJaRdETLMM5b32pHFVEgJ6xgImVWdlfXK2rCZcosDNAdL4Vycv5b0F0wAdeFlRC/ hJnNyU/PmD0T9kxG+EPH/ol+Nvz60m8LgIdwp6K3PRaQVjz8gnGxV4VdsjoB2TAazYfJrQ4H5Xh3 4Xuhc2duUoRPpwMyHBGAi0zHo1GK2n+Jw2HoKv7OMEic2oImS+QNY64SGWtC4xR4XNCYmACoTXkY gnSRLk3S7Tyu2uXxPDy4/PIiPgvRZcCeqzWdla/BzD/uX06WX7tK4mUiwgOUMFKl2OZ3QfgcUDTR RmBQd8udquvEx80k63RPJBXOSka41gHYkFoQS3VCQsWk48pakdh1updUZsXqvBWidxW4b4mU1x/3 l0rrp3xmPszKP1fJHSeTklXzM7DLZleBfndlKEQxp3vz8uNlnfNyk/4+xdLxGwXBcxD9Rnc7dR/S dOpRZ1IpxZmkdwPiCLU3o/uvfSywr3/iz7/mhU7jGuvbMz5YXrzoAt2oQ8XBjpDlnJMthqDv1lhh QVecJTBSokUsmKy2G/coRr+6Dn8ouHXjQqzg7KXDBiswztZX8j4sOnH/LgfIV+HdmY949FuFeVcX 942SOWxTMpfUOGp/lcNxrxdmk9HYXlqwbnA/YE6lUZqru7y4he0vc8lSJrMMGPhwzqIVukffhnWf AMlLC1Pq64Ub97PYzir7SUhuTZwC1eII9zULQ4x1imgmJU8Tx2gY7xSz9b+xQ8wyEwZKSnE4MVsD tCOJ2TqQtCFm1yRFN7bxlZ+FYZ4y35PpkLXK+fRYempsWRX7NBdpQWI8yETsqZuKmXl1lcvcn59W 3h9o5bU3WVpoQ7Vemay6x0X/oyarXjugX77vXmtPzsQjdibam6xQqpDSA661P+9k1V9rvy9GDebT aNmILS4seWDLwezbQ9zLVzc6M+yQzgzvfl9FoQgLLYWBHjMKbB2FYofKAg4L24TWMgv5rQU0lq5V hp1ojbDTetD/NtzaBGT2OECuYJSF5P462PxxgB0A2L8spEmRSW+TBdngTjh2Sc6UA/vtp1flDkIf 9+jmG7rz8H73Non+uSgUIbRSTAdC0/J9TKgDs0+9yP3+S+2DCqbH2LFbUsv6hxUAPP6wLalN3hTz aCc5GhKnYtP9oU9yyNonOQqA2jvJIZe3pIvaoBzyJIesPsmxFRguWmpCLxuf3TI9iXv5j7riVROm lliS6EFX/AYIWiSQbgRd5dG7luRRw6nj7csj9eL+xGpxKk19aSmxhkZ70lItC6iwNiiHlJaqWlpu O+9SQNPOWeIgjRIlopg4xiwRUaCIBn8Nk+9hKqPYpDqsOv22C2Dx2M4S75julvSR2qiPtlBP0eOc 1t0KwsMWYdgIQXuSStD7Q9eS0llXh7umrm2lo5soneMSyzQjVkWVhkMTyzTwGNRiEYvD60D9aCAx y+JZbgWF0Xa0sa7WxnI7z1QXyjiINnY0kSKICGU8ISKNAmK5FCQMbChtFEbKJFXaWG2nXktn+E01 9dQO6lXYxIehXhimTIaGkcDCP0LphNhEMxJxqUScujBxsqBeExHBKwNCx1SF1RC0qAoPAF070r1C Se+aupalu6BNpDt2pGhNpjYkjnjouMIGCFrinGb2r6ioNHJozmFNjCgtj1y1TfAmBNNKPTA3aXVE K7Mpcdo3yQVvxE362N5V0Ihg5qF17gYIWuKmhsRR7XNT8KI06jQPldlSOk7QnsDOKHh859NrAEUF W1q/AJ7Ja9e3N43b3pxX7z4oDvsE+L+zL1+vBwT3Bfof16M+++K+YSLy1CVUJYGRJApxQ5/lhmiW GEJ5kLpUKU1ZcpqMBl8GyV1nnKVeO6OiuQ6eKsaGO4IaWdU7p64ngXRTlbbwQ4bJthjvHuDKQORD ArzZcSzhbSOup2/jevVsPw+LaNP2Y8tk0bVBOaBHKGi1R6h3MFVlqbT7VXVrENeuqRRKSI+qNdeL wz4m6MK6UQgPXUsl0cw+S7FVN4wvk8XUBuWQS5FVL0WzYylWGKoHCs4ISUNjOOGhUUQwY0kkQk4i lYLtYnTCcIvbYmirjuGB8DY2PNabztH/YktN5xb/9k3ngtCUrdrw52qrtoBqJXyrNnbCO+/Wms7V te4QJX2cDT7bQKhOEh/L+t4MQUvWd/0K3CVobVvforYv5yGq3L/SErHql9YtQWubWLVL6wJAutXS uuDHNRC2urqa7QMa04Zupx1vR1XxSlWFwGylXmt7ImD90QhLgHHHYiwlGxEwJh0JaSyk4s5oXlkR eKui9/C2YP94Rd9oy5sHplLJtGgqyiYegX4qnrz1zKFOeCBkHJFAUwxmxCHRMgYPh0Yy4crQUATl SQsqI+DXCEuDJQwcIRUQ7ThYYNpInhrrZMBPT04O+1waBnEcJ5zQNMI2OoYSE+uQREZGoU1cSAOz dIT5Cosx9zr/4XOVmsCEKeeECQGCh4L7GlmmYHkkPAahY9KE4hwc9DmuFLUOpF1oFWDBwSaH1zSR Ko3SxMbaqGRlrqJ7FM7eKdYrgptHV4oN7Ik2N0k0s0x18NBm+wYIWrJE6x8N8KAdwRKVTcx2D9FR dbBqECLUgWjHXKG35koDy7MtYHg1MGw7MO0cpBBBtRnMdvD1AYOny2Zw5EKtU6eJYpqCpseKcyyM SEoZ1ZQLFokyYlNfXgZtHohZCUjuJF3F7pCH9b/4dtrJdhivur/VjtCbDioV9T2j9vvtf9u1YFU7 0iNovLkboXnYbTebIWhJS9c/MFOC1raWrn9gZjOxWtTSdbfoeuB4m9J02Wc3mzverIJySOFU3SjK bOm+5KE5YHBoj+2lDWawzSNPKxaf2XwmcxWUQ85g9QHRXepF7H2C6CkEtBJWUBEPpXaCpDLgREiL jyUpzIONIkM5dlI4Tajm4O8nJMTWA0LYlGjFHElioWIs18FCuVZ17h5hALHP9N9zATdOQeagbj46 5iFtyR2oPim4w0Rs6Vy1ENWO0pZTgCU0h57EBsH5RoZH5e7ao1pllRC0ZJXVP1FWgta2VaabZPHC VrN4epnJt+4wM6wlu6f6aJLZvqHQsBbsngZxgAYm2gZQnxT8HpVoY+p4CEQmkXagu22QEGO1gz9V QAOJzrQ8tZFQRlpLaBhhqyFwsHWqJbE0FSIxkTVSHVLBb3UXTEsNMkW4j6IyVW7nEy/u16xVhTRQ aUyci4BvwaoiRouYJLjj0kgZ8yA+TYTlHChMYo1K3ISCaKYYcYJzkTJhLEsPyYs7pKbYexv20/Sv TL+LkigIQ0Zi3HIrYm2ICVJFeATeBbWJSqQ7pVTqIAGHJIiVJSIRGj4IjMKsoNTwSDF6UFG0XYOL ljZm6I2iaBswLWU6VDUw270wU23APy2MPZzwhCkqIvhCCpqaCC4oiSIGPrlOwYmxOnYixpz9QZ8T UhgsOQorK4HVyGNAgBr9/+2dTW7DIBCFe6CiAsPvObKsujDY3P8IhUiREocQ/+Eu+r41iyeQH6Px zJAFJjFGo1L+Wp8/tH9fh2G8GUMOi1iOm/Le2nyRuRQHRsEJrQxFctdamEPXRTI68iGHZtaWNy2J 2KCDYllZEJZcjMnNzirgrL7ISm5DyAHGNPm8VgTmrNNM6jGEgZTT6XoGh64jnkKQU/k7lVUqyctk WssZTVIET37QUczOKu5MljXuDNcp6bN6mJ4Sn4Kf3YSyfN5LkSeqRRen5XleKuiU51m5OZWk69F5 nsXDcG6C+uV5HkfQ+Eb70kzKkXme+hAV3+rDLGr2Z1Yv8zxPzNeN8SRYSuWWMqkUAk6RaUEhBT66 MJSJwt8/z6meTWobYeRlHkZus803J9opBeG32ObZU4f1iqYk8ddTh18r6GObazen/9RhvXjKzE1Q N9vMm/PwkTXKCEW3qcO63mnsW30FRU214mufbXpTHrq1mo3j9SkDKk+MSMXIyxzpJpqUGV7Y5ia1 /W3zzYn2qabSfIttyurUmY62uaKRMsujajB8pm3WFXSyzZWb0384lxZrbPO+vvJ423wc0OAbzWIz KUfaZr0r3LcaFYua/aWeT7a5rJrqZpsfAAAAAAAAAAAAAAAAAAAAAAAAAAAAgDt+AfLBejkAsAQA --=_55a563500b973bf81693afb3908ca577--
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error?
@Nir, that looks like some issue in VDSM please have a look jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job
Thanks.<snapshot-deletion.tar.gz>_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Jun 24, 2016 at 11:39 AM, Vinzenz Feenstra <vfeenstr@redhat.com> wrote:
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error?
@Nir,
that looks like some issue in VDSM please have a look
jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job
This means there was a previous attempt to merge, and the block job did not complete yet. Engine should detect that there is an active block job and wait until it completes. This is probably engine bug, please file a bug and attach complete vdsm and engine logs. Ala, can you look at this? Nir
Done, you can find it in [1]. Feel free to add/modify anything you consider inaccurate. Thank you. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1349950 El 24/06/16 a las 12:39, Nir Soffer escribió:
On Fri, Jun 24, 2016 at 11:39 AM, Vinzenz Feenstra <vfeenstr@redhat.com> wrote:
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error? @Nir,
that looks like some issue in VDSM please have a look
jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job This means there was a previous attempt to merge, and the block job did not complete yet.
Engine should detect that there is an active block job and wait until it completes.
This is probably engine bug, please file a bug and attach complete vdsm and engine logs.
Ala, can you look at this?
Nir
Hello, I have the same problem but i use the ovirt version 4.0.4.4-1.el7.centos . My logs. Engine.log 2016-09-28 08:18:00,947 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler1) [7013b545] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:08,010 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,169 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: eb73a967-1908-46e9-9de2-9706bf29643a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2016-09-28 08:18:09,185 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock freed to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,265 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (pool-7-thread-3) [354939e9] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2016-09-28 08:18:09,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-54) [] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Backup the VM' deletion for VM 'SRV-ActPrint' was initiated by admin@internal. 2016-09-28 08:18:10,197 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler4) [354939e9] Executing Live Merge command step 'EXTEND' 2016-09-28 08:18:10,254 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Running command: MergeExtendCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:10,255 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Refreshing volume c08d86ed-46f1-44bc-9476-0cc2c6aed367 on host f22d87b9-4449-4a71-8529-58095dd81b6f 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Running command: RefreshVolumeCommand internal: true. 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] START, RefreshVolumeVDSCommand(HostName = Host04, RefreshVolumeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367'}), log id: 77b9ded4 2016-09-28 08:18:11,245 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] FINISH, RefreshVolumeVDSCommand, log id: 77b9ded4 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Successfully refreshed volume 'c08d86ed-46f1-44bc-9476-0cc2c6aed367' on host 'f22d87b9-4449-4a71-8529-58095dd81b6f' 2016-09-28 08:18:12,267 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [354939e9] Waiting on Live Merge command step 'EXTEND' to finalize 2016-09-28 08:18:14,294 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Executing Live Merge command step 'MERGE' 2016-09-28 08:18:14,347 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Running command: MergeCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:14,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] START, MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'}), log id: 544f03ae 2016-09-28 08:18:15,335 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:15,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Failed in 'MergeVDS' method 2016-09-28 08:18:15,628 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [15ef379f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM Host04 command failed: Merge failed 2016-09-28 08:18:15,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=52, message=Merge failed]]' 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] HostName = Host04 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] FINISH, MergeVDSCommand, log id: 544f03ae 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:139) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:729) [bll.jar:] at sun.reflect.GeneratedMethodAccessor675.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.2.Final.jar:2.3.2.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) at org.ovirt.engine.core.bll.interfaces.BackendCommandObjectsHandler$$$view3.runAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.executeCommand(CommandExecutor.java:57) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.lambda$executeAsyncCommand$0(CommandExecutor.java:46) [bll.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-28 08:18:16,038 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler10) [354939e9] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:16,341 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Waiting on Live Merge command step 'MERGE' to complete 2016-09-28 08:18:16,351 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler6) [15ef379f] Merge command (jobId = null) has completed for images 'c08d86ed-46f1-44bc-9476-0cc2c6aed367'..'95fefce5-7599-460f-b38c-377323659b52' 2016-09-28 08:18:18,372 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Executing Live Merge command step 'MERGE_STATUS' 2016-09-28 08:18:18,418 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Running command: MergeStatusCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:18,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] START, FullListVDSCommand(HostName = Host04, FullListVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vds='Host[Host04,f22d87b9-4449-4a71-8529-58095dd81b6f]', vmIds='[eb73a967-1908-46e9-9de2-9706bf29643a]'}), log id: 62d7ca75 2016-09-28 08:18:19,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.2.0, afterMigrationStatus=, vmId=eb73a967-1908-46e9-9de2-9706bf29643a, memGuaranteedSize=1740, transparentHugePages=true, displaySecurePort=5907, timeOffset=-7088, cpuType=Nehalem, smp=2, guestNumaNodes=[Ljava.lang.Object;@1f96e8d8, hypervEnable=true, custom={device_66bcfc2f-fca9-4890-8bf5-74095d059da9device_de626997-b95c-45be-81f2-b2de374b5f8adevice_0faaec88-ceff-467b-85fd-6f131cfae9fadevice_ad363387-214a-4435-ab86-486feeea6ad3=VmDevice:{id='VmDeviceId:{deviceId='ad363387-214a-4435-ab86-486feeea6ad3', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5-74095d059da9=VmDevice:{id='VmDeviceId:{deviceId='66bcfc2f-fca9-4890-8bf5-74095d059da9', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5-74095d059da9device_de626997-b95c-45be-81f2-b2de374b5f8a=VmDevice:{id='VmDeviceId:{deviceId='de626997-b95c-45be-81f2-b2de374b5f8a', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5-74095d059da9device_de626997-b95c-45be-81f2-b2de374b5f8adevice_0faaec88-ceff-467b-85fd-6f131cfae9fa=VmDevice:{id='VmDeviceId:{deviceId='0faaec88-ceff-467b-85fd-6f131cfae9fa', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}}, vmType=kvm, smpThreadsPerCore=1, memSize=1740, smpCoresPerSocket=2, vmName=SRV-ActPrint, nice=0, status=Up, maxMemSize=4194304, bootMenuEnable=false, guestDiskMapping={}, pid=14894, displayIp=192.168.144.104, displayPort=-1, smartcardEnable=false, maxMemSlots=16, nicModel=rtl8139,pv, kvmEnable=true, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@1c3c34e6, maxVCpus=32, clientIp=, statusTime=7115868520, display=qxl}], log id: 62d7ca75 2016-09-28 08:18:19,538 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Failed to live merge, still in volume chain: [c08d86ed-46f1-44bc-9476-0cc2c6aed367, 95fefce5-7599-460f-b38c-377323659b52] 2016-09-28 08:18:20,422 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Failed child command status for step 'MERGE_STATUS' 2016-09-28 08:18:21,449 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler8) [354939e9] Merging of snapshot '275c0570-32b6-4d50-9529-b2428f6a8a0b' images 'c08d86ed-46f1-44bc-9476-0cc2c6aed367'..'95fefce5-7599-460f-b38c-377323659b52' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2016-09-28 08:18:21,459 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5' child commands '[fd866748-3211-4d48-9908-12eb6078a69e]' executions were completed, status 'FAILED' 2016-09-28 08:18:22,564 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler10) [354939e9] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2016-09-28 08:18:22,634 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [354939e9] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Failed to delete snapshot 'Backup the VM' for VM 'SRV-ActPrint'. Vdsm.log periodic/5740::DEBUG::2016-09-28 08:01:31,574::vm::4519::virt.vm::(queryBlockJobs) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Still waiting for block job 4dd2b885-2452-4520-b20a-928edea50836 to be synchronized .... jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,783::vm::4427::virt.vm::(trackBlockJob) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Cannot add block job 50a359bd-097d-4c07-a7ae-5f4a9db23b81. A block job with id 4dd2b885-2452-4520-b20a-928edea50836 already exists for image 9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,784::vm::4600::virt.vm::(merge) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::A block job is already active on this disk Thanks. Marcelo Leandro 2016-06-24 12:49 GMT-03:00 Nicolás <nicolas@devels.es>:
Done, you can find it in [1]. Feel free to add/modify anything you consider inaccurate.
Thank you.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1349950
El 24/06/16 a las 12:39, Nir Soffer escribió:
On Fri, Jun 24, 2016 at 11:39 AM, Vinzenz Feenstra <vfeenstr@redhat.com>
wrote:
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error?
@Nir,
that looks like some issue in VDSM please have a look
jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job
This means there was a previous attempt to merge, and the block job did not complete yet.
Engine should detect that there is an active block job and wait until it completes.
This is probably engine bug, please file a bug and attach complete vdsm and engine logs.
Ala, can you look at this?
Nir
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Marcelo, This error indicates that the image you are trying to delete doesn't exist. When do you get this error? When running Live Merge or Live Storage Migration (LSM)? Please note that we fixed in LSM area where the VM went down while we tried to delete the auto-generated snapshot. See https://bugzilla.redhat.com/1368203 . -Ala On Wed, Sep 28, 2016 at 3:07 PM, Marcelo Leandro <marceloltmm@gmail.com> wrote:
Hello, I have the same problem but i use the ovirt version 4.0.4.4-1.el7.centos .
My logs.
Engine.log
2016-09-28 08:18:00,947 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler1) [7013b545] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:08,010 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,169 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: eb73a967-1908-46e9-9de2-9706bf29643a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2016-09-28 08:18:09,185 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock freed to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,265 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (pool-7-thread-3) [354939e9] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2016-09-28 08:18:09,302 INFO [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (default task-54) [] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Backup the VM' deletion for VM 'SRV-ActPrint' was initiated by admin@internal. 2016-09-28 08:18:10,197 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler4) [354939e9] Executing Live Merge command step 'EXTEND' 2016-09-28 08:18:10,254 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Running command: MergeExtendCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:10,255 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Refreshing volume c08d86ed-46f1-44bc-9476-0cc2c6aed367 on host f22d87b9-4449-4a71-8529-58095dd81b6f 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Running command: RefreshVolumeCommand internal: true. 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] START, RefreshVolumeVDSCommand(HostName = Host04, RefreshVolumeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367'}), log id: 77b9ded4 2016-09-28 08:18:11,245 INFO [org.ovirt.engine.core.bll. ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] FINISH, RefreshVolumeVDSCommand, log id: 77b9ded4 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Successfully refreshed volume 'c08d86ed-46f1-44bc-9476-0cc2c6aed367' on host 'f22d87b9-4449-4a71-8529- 58095dd81b6f' 2016-09-28 08:18:12,267 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [354939e9] Waiting on Live Merge command step 'EXTEND' to finalize 2016-09-28 08:18:14,294 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Executing Live Merge command step 'MERGE' 2016-09-28 08:18:14,347 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Running command: MergeCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:14,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] START, MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'}), log id: 544f03ae 2016-09-28 08:18:15,335 INFO [org.ovirt.engine.core.bll. ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:15,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Failed in 'MergeVDS' method 2016-09-28 08:18:15,628 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [15ef379f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM Host04 command failed: Merge failed 2016-09-28 08:18:15,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'org.ovirt.engine.core. vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=52, message=Merge failed]]' 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] HostName = Host04 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] FINISH, MergeVDSCommand, log id: 544f03ae 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase. executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase. executeActionInTransactionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInNewTransaction(TransactionSupport.java:204) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInRequired(TransactionSupport.java:139) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport. executeInScope(TransactionSupport.java:107) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:729) [bll.jar:] at sun.reflect.GeneratedMethodAccessor675.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceMethodIntercep tor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor. delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor. doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor. processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ee.component.interceptors. UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors. CorrelationIdTrackerInterceptor.aroundInvoke( CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceLifecycleMetho dInterceptor.processInvocation(ManagedReferenceLifecycleMetho dInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics. ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat ionInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.2.Final.jar:2.3.2.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter ceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor. processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation( InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors. ComponentDispatcherInterceptor.processInvocation( ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton. SingletonComponentInstanceAssociationInterceptor.processInvocation( SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors. CurrentInvocationContextInterceptor.processInvocation( CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.0.0.Final. jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics. WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor. processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors. ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor. processInvocation(LoggingInterceptor.java:66) [wildfly-ejb3-10.0.0.Final. jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor. processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor. processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor. processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerIntercep tor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke( ViewService.java:195) at org.jboss.as.ee.component.ViewDescription$1.processInvocation( ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke( ProxyInvocationHandler.java:73) at org.ovirt.engine.core.bll.interfaces. BackendCommandObjectsHandler$$$view3.runAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.executeCommand(CommandExecutor.java:57) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.lambda$ executeAsyncCommand$0(CommandExecutor.java:46) [bll.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-28 08:18:16,038 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler10) [354939e9] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:16,341 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Waiting on Live Merge command step 'MERGE' to complete 2016-09-28 08:18:16,351 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler6) [15ef379f] Merge command (jobId = null) has completed for images 'c08d86ed-46f1-44bc-9476- 0cc2c6aed367'..'95fefce5-7599-460f-b38c-377323659b52' 2016-09-28 08:18:18,372 INFO [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Executing Live Merge command step 'MERGE_STATUS' 2016-09-28 08:18:18,418 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Running command: MergeStatusCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:18,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] START, FullListVDSCommand(HostName = Host04, FullListVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vds='Host[Host04,f22d87b9-4449-4a71-8529-58095dd81b6f]', vmIds='[eb73a967-1908-46e9-9de2-9706bf29643a]'}), log id: 62d7ca75 2016-09-28 08:18:19,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.2.0, afterMigrationStatus=, vmId=eb73a967-1908-46e9-9de2-9706bf29643a, memGuaranteedSize=1740, transparentHugePages=true, displaySecurePort=5907, timeOffset=-7088, cpuType=Nehalem, smp=2, guestNumaNodes=[Ljava.lang. Object;@1f96e8d8, hypervEnable=true, custom={device_66bcfc2f-fca9- 4890-8bf5-74095d059da9device_de626997-b95c-45be-81f2- b2de374b5f8adevice_0faaec88-ceff-467b-85fd-6f131cfae9fadevice_ad363387- 214a-4435-ab86-486feeea6ad3=VmDevice:{id='VmDeviceId:{ deviceId='ad363387-214a-4435-ab86-486feeea6ad3', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890- 8bf5-74095d059da9=VmDevice:{id='VmDeviceId:{deviceId=' 66bcfc2f-fca9-4890-8bf5-74095d059da9', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890- 8bf5-74095d059da9device_de626997-b95c-45be-81f2- b2de374b5f8a=VmDevice:{id='VmDeviceId:{deviceId='de626997-b95c-45be-81f2-b2de374b5f8a', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890- 8bf5-74095d059da9device_de626997-b95c-45be-81f2- b2de374b5f8adevice_0faaec88-ceff-467b-85fd-6f131cfae9fa= VmDevice:{id='VmDeviceId:{deviceId='0faaec88-ceff-467b-85fd-6f131cfae9fa', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}}, vmType=kvm, smpThreadsPerCore=1, memSize=1740, smpCoresPerSocket=2, vmName=SRV-ActPrint, nice=0, status=Up, maxMemSize=4194304, bootMenuEnable=false, guestDiskMapping={}, pid=14894, displayIp=192.168.144.104, displayPort=-1, smartcardEnable=false, maxMemSlots=16, nicModel=rtl8139,pv, kvmEnable=true, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@1c3c34e6, maxVCpus=32, clientIp=, statusTime=7115868520, display=qxl}], log id: 62d7ca75 2016-09-28 08:18:19,538 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Failed to live merge, still in volume chain: [c08d86ed-46f1-44bc-9476-0cc2c6aed367, 95fefce5-7599-460f-b38c- 377323659b52] 2016-09-28 08:18:20,422 ERROR [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Failed child command status for step 'MERGE_STATUS' 2016-09-28 08:18:21,449 ERROR [org.ovirt.engine.core.bll.snapshots. RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler8) [354939e9] Merging of snapshot '275c0570-32b6-4d50-9529-b2428f6a8a0b' images 'c08d86ed-46f1-44bc-9476-0cc2c6aed367'..'95fefce5-7599-460f-b38c-377323659b52' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2016-09-28 08:18:21,459 INFO [org.ovirt.engine.core.bll. ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5' child commands '[fd866748-3211-4d48-9908-12eb6078a69e]' executions were completed, status 'FAILED' 2016-09-28 08:18:22,564 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler10) [354939e9] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2016-09-28 08:18:22,634 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [354939e9] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Failed to delete snapshot 'Backup the VM' for VM 'SRV-ActPrint'.
Vdsm.log periodic/5740::DEBUG::2016-09-28 08:01:31,574::vm::4519::virt.vm::(queryBlockJobs) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Still waiting for block job 4dd2b885-2452-4520-b20a-928edea50836 to be synchronized .... jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,783::vm::4427::virt.vm::(trackBlockJob) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Cannot add block job 50a359bd-097d-4c07-a7ae-5f4a9db23b81. A block job with id 4dd2b885-2452-4520-b20a-928edea50836 already exists for image 9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,784::vm::4600::virt.vm::(merge) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::A block job is already active on this disk
Thanks.
Marcelo Leandro
2016-06-24 12:49 GMT-03:00 Nicolás <nicolas@devels.es>:
Done, you can find it in [1]. Feel free to add/modify anything you consider inaccurate.
Thank you.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1349950
El 24/06/16 a las 12:39, Nir Soffer escribió:
On Fri, Jun 24, 2016 at 11:39 AM, Vinzenz Feenstra <vfeenstr@redhat.com>
wrote:
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error?
@Nir,
that looks like some issue in VDSM please have a look
jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job
This means there was a previous attempt to merge, and the block job did not complete yet.
Engine should detect that there is an active block job and wait until it completes.
This is probably engine bug, please file a bug and attach complete vdsm and engine logs.
Ala, can you look at this?
Nir
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Ala, When running Live Merge. I create the live snapshot and when I try delete snapshot and has this error. I saw the bug https://bugzilla.redhat.com/1368203 , but I not move the disk to another domain. Thanks. Marcelo Leandro 2016-09-28 9:19 GMT-03:00 Ala Hino <ahino@redhat.com>:
Hi Marcelo,
This error indicates that the image you are trying to delete doesn't exist. When do you get this error? When running Live Merge or Live Storage Migration (LSM)?
Please note that we fixed in LSM area where the VM went down while we tried to delete the auto-generated snapshot. See https://bugzilla.redhat. com/1368203 .
-Ala
On Wed, Sep 28, 2016 at 3:07 PM, Marcelo Leandro <marceloltmm@gmail.com> wrote:
Hello, I have the same problem but i use the ovirt version 4.0.4.4-1.el7.centos .
My logs.
Engine.log
2016-09-28 08:18:00,947 INFO [org.ovirt.engine.core.vdsbrok er.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler1) [7013b545] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:08,010 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock Acquired to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,169 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Running command: RemoveSnapshotCommand internal: false. Entities affected : ID: eb73a967-1908-46e9-9de2-9706bf29643a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2016-09-28 08:18:09,185 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-54) [5a27e364] Lock freed to object 'EngineLock:{exclusiveLocks='[eb73a967-1908-46e9-9de2-9706bf29643a=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2016-09-28 08:18:09,265 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (pool-7-thread-3) [354939e9] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2016-09-28 08:18:09,302 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (default task-54) [] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Snapshot 'Backup the VM' deletion for VM 'SRV-ActPrint' was initiated by admin@internal. 2016-09-28 08:18:10,197 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler4) [354939e9] Executing Live Merge command step 'EXTEND' 2016-09-28 08:18:10,254 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Running command: MergeExtendCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:10,255 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-7-thread-7) [57c94fc3] Refreshing volume c08d86ed-46f1-44bc-9476-0cc2c6aed367 on host f22d87b9-4449-4a71-8529-58095dd81b6f 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Running command: RefreshVolumeCommand internal: true. 2016-09-28 08:18:10,275 INFO [org.ovirt.engine.core.vdsbrok er.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] START, RefreshVolumeVDSCommand(HostName = Host04, RefreshVolumeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367'}), log id: 77b9ded4 2016-09-28 08:18:11,245 INFO [org.ovirt.engine.core.bll.Con currentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.vdsbrok er.vdsbroker.RefreshVolumeVDSCommand] (pool-7-thread-7) [47625ba4] FINISH, RefreshVolumeVDSCommand, log id: 77b9ded4 2016-09-28 08:18:11,810 INFO [org.ovirt.engine.core.bll.RefreshVolumeCommand] (pool-7-thread-7) [47625ba4] Successfully refreshed volume 'c08d86ed-46f1-44bc-9476-0cc2c6aed367' on host 'f22d87b9-4449-4a71-8529-58095dd81b6f' 2016-09-28 08:18:12,267 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [354939e9] Waiting on Live Merge command step 'EXTEND' to finalize 2016-09-28 08:18:14,294 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Executing Live Merge command step 'MERGE' 2016-09-28 08:18:14,347 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Running command: MergeCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:14,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] START, MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'}), log id: 544f03ae 2016-09-28 08:18:15,335 INFO [org.ovirt.engine.core.bll.Con currentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' (id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5') waiting on child command id: 'fd866748-3211-4d48-9908-12eb6078a69e' type:'RemoveSnapshotSingleDiskLive' to complete 2016-09-28 08:18:15,625 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Failed in 'MergeVDS' method 2016-09-28 08:18:15,628 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [15ef379f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM Host04 command failed: Merge failed 2016-09-28 08:18:15,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=52, message=Merge failed]]' 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] HostName = Host04 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] Command 'MergeVDSCommand(HostName = Host04, MergeVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vmId='eb73a967-1908-46e9-9de2-9706bf29643a', storagePoolId='77e24b20-9d21-4952-a089-3c5c592b4e6d', storageDomainId='6e5cce71-3438-4045-9d54-607123e0557e', imageGroupId='9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa', imageId='95fefce5-7599-460f-b38c-377323659b52', baseImageId='c08d86ed-46f1-44bc-9476-0cc2c6aed367', topImageId='95fefce5-7599-460f-b38c-377323659b52', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-09-28 08:18:15,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [15ef379f] FINISH, MergeVDSCommand, log id: 544f03ae 2016-09-28 08:18:15,629 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [15ef379f] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsComman d(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2171) [bll.jar:] at org.ovirt.engine.core.bll.MergeCommand.executeCommand(MergeCommand.java:45) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1305) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransac tionScope(CommandBase.java:1447) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2075) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.e xecuteInNewTransaction(TransactionSupport.java:204) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.e xecuteInRequired(TransactionSupport.java:139) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.e xecuteInScope(TransactionSupport.java:107) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1490) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:398) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:729) [bll.jar:] at sun.reflect.GeneratedMethodAccessor675.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor. processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInte rception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInte rception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvoc ation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-10.0.0.Final.jar :10.0.0.Final] at org.jboss.as.ee.component.interceptors.UserInterceptorFactor y$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerI nterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInt erceptor.processInvocation(ManagedReferenceLifecycleMethodIn terceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeI nterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed( InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationIntercep tor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.2.Final.jar:2.3.2.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor. processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.proc essInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(In itialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherIn terceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInst anceAssociationInterceptor.processInvocation(SingletonCompon entInstanceAssociationInterceptor.java:53) [wildfly-ejb3-10.0.0.Final.jar :10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationCo ntextInterceptor.processInvocation(CurrentInvoc ationContextInterceptor.java:41) [wildfly-ejb3-10.0.0.Final.jar :10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterc eptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.proces sInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptor Factory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor. processInvocation(LoggingInterceptor.java:66) [wildfly-ejb3-10.0.0.Final.jar:10.0.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.proces sInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processIn vocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorConte xt.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecke d(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessCheckingInterceptor.processInvoca tion(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorConte xt.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor. processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewServic e.java:195) at org.jboss.as.ee.component.ViewDescription$1.processInvocatio n(ViewDescription.java:185) at org.jboss.invocation.InterceptorContext.proceed(InterceptorC ontext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch ainedInterceptor.java:61) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(Prox yInvocationHandler.java:73) at org.ovirt.engine.core.bll.interfaces.BackendCommandObjectsHa ndler$$$view3.runAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.executeCommand(CommandExecutor.java:57) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandExecutor.lambda$execu teAsyncCommand$0(CommandExecutor.java:46) [bll.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] 2016-09-28 08:18:16,038 INFO [org.ovirt.engine.core.vdsbrok er.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler10) [354939e9] VM Job [4dd2b885-2452-4520-b20a-928edea50836]: In progress (no change) 2016-09-28 08:18:16,341 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Waiting on Live Merge command step 'MERGE' to complete 2016-09-28 08:18:16,351 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler6) [15ef379f] Merge command (jobId = null) has completed for images 'c08d86ed-46f1-44bc-9476-0cc2c 6aed367'..'95fefce5-7599-460f-b38c-377323659b52' 2016-09-28 08:18:18,372 INFO [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler6) [354939e9] Executing Live Merge command step 'MERGE_STATUS' 2016-09-28 08:18:18,418 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Running command: MergeStatusCommand internal: true. Entities affected : ID: 6e5cce71-3438-4045-9d54-607123e0557e Type: Storage 2016-09-28 08:18:18,506 INFO [org.ovirt.engine.core.vdsbrok er.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] START, FullListVDSCommand(HostName = Host04, FullListVDSCommandParameters:{runAsync='true', hostId='f22d87b9-4449-4a71-8529-58095dd81b6f', vds='Host[Host04,f22d87b9-4449-4a71-8529-58095dd81b6f]', vmIds='[eb73a967-1908-46e9-9de2-9706bf29643a]'}), log id: 62d7ca75 2016-09-28 08:18:19,528 INFO [org.ovirt.engine.core.vdsbrok er.vdsbroker.FullListVDSCommand] (pool-7-thread-1) [42d41e81] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.2.0, afterMigrationStatus=, vmId=eb73a967-1908-46e9-9de2-9706bf29643a, memGuaranteedSize=1740, transparentHugePages=true, displaySecurePort=5907, timeOffset=-7088, cpuType=Nehalem, smp=2, guestNumaNodes=[Ljava.lang.Object;@1f96e8d8, hypervEnable=true, custom={device_66bcfc2f-fca9-4 890-8bf5-74095d059da9device_de626997-b95c-45be-81f2-b2de374b 5f8adevice_0faaec88-ceff-467b-85fd-6f131cfae9fadevice_ ad363387-214a-4435-ab86-486feeea6ad3=VmDevice:{id='VmDeviceId:{deviceId=' ad363387-214a-4435-ab86-486feeea6ad3', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5-74095d059da9=VmDevice:{id=' VmDeviceId:{deviceId='66bcfc2f-fca9-4890-8bf5-74095d059da9', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5 -74095d059da9device_de626997-b95c-45be-81f2-b2de374b5f8a= VmDevice:{id='VmDeviceId:{deviceId='de626997-b95c-45be-81f2-b2de374b5f8a', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}, device_66bcfc2f-fca9-4890-8bf5-74095d059da9device_de626997- b95c-45be-81f2-b2de374b5f8adevice_0faaec88-ceff-467b-85fd- 6f131cfae9fa=VmDevice:{id='VmDeviceId:{deviceId='0faaec88-ceff-467b-85fd-6f131cfae9fa', vmId='eb73a967-1908-46e9-9de2-9706bf29643a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false', hostDevice='null'}}, vmType=kvm, smpThreadsPerCore=1, memSize=1740, smpCoresPerSocket=2, vmName=SRV-ActPrint, nice=0, status=Up, maxMemSize=4194304, bootMenuEnable=false, guestDiskMapping={}, pid=14894, displayIp=192.168.144.104, displayPort=-1, smartcardEnable=false, maxMemSlots=16, nicModel=rtl8139,pv, kvmEnable=true, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@1c3c34e6, maxVCpus=32, clientIp=, statusTime=7115868520, display=qxl}], log id: 62d7ca75 2016-09-28 08:18:19,538 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-1) [42d41e81] Failed to live merge, still in volume chain: [c08d86ed-46f1-44bc-9476-0cc2c6aed367, 95fefce5-7599-460f-b38c-377323 659b52] 2016-09-28 08:18:20,422 ERROR [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [354939e9] Failed child command status for step 'MERGE_STATUS' 2016-09-28 08:18:21,449 ERROR [org.ovirt.engine.core.bll.sna pshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler8) [354939e9] Merging of snapshot '275c0570-32b6-4d50-9529-b2428f6a8a0b' images 'c08d86ed-46f1-44bc-9476-0cc2c6aed367'..'95fefce5-7599-460f-b38c-377323659b52' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2016-09-28 08:18:21,459 INFO [org.ovirt.engine.core.bll.Con currentChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [354939e9] Command 'RemoveSnapshot' id: '18613dc9-d8c8-45c4-9fbe-a298e701ead5' child commands '[fd866748-3211-4d48-9908-12eb6078a69e]' executions were completed, status 'FAILED' 2016-09-28 08:18:22,564 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler10) [354939e9] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2016-09-28 08:18:22,634 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [354939e9] Correlation ID: 5a27e364, Job ID: 661f8f55-30c6-4735-bb0a-fdcd3ac4004a, Call Stack: null, Custom Event ID: -1, Message: Failed to delete snapshot 'Backup the VM' for VM 'SRV-ActPrint'.
Vdsm.log periodic/5740::DEBUG::2016-09-28 08:01:31,574::vm::4519::virt.vm::(queryBlockJobs) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Still waiting for block job 4dd2b885-2452-4520-b20a-928edea50836 to be synchronized .... jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,783::vm::4427::virt.vm::(trackBlockJob) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::Cannot add block job 50a359bd-097d-4c07-a7ae-5f4a9db23b81. A block job with id 4dd2b885-2452-4520-b20a-928edea50836 already exists for image 9fc0b2f6-d786-4a21-8f5c-b22b23df4aaa jsonrpc.Executor/5::ERROR::2016-09-28 08:33:04,784::vm::4600::virt.vm::(merge) vmId=`eb73a967-1908-46e9-9de2-9706bf29643a`::A block job is already active on this disk
Thanks.
Marcelo Leandro
2016-06-24 12:49 GMT-03:00 Nicolás <nicolas@devels.es>:
Done, you can find it in [1]. Feel free to add/modify anything you consider inaccurate.
Thank you.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1349950
El 24/06/16 a las 12:39, Nir Soffer escribió:
On Fri, Jun 24, 2016 at 11:39 AM, Vinzenz Feenstra <vfeenstr@redhat.com>
wrote:
On Jun 24, 2016, at 9:10 AM, nicolas@devels.es wrote:
Hi,
We're trying to delete an auto-generated live snapshot that has been created after migrating an online VM's storage to a different domain. oVirt version is 3.6.6 and VDSM version is 4.17.28. Most relevant log lines are:
2016-06-24 07:50:36,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Failed in 'MergeVDS' method 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (pool-7-thread-10) [799a22e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host2.domain.com command failed: Merge failed 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-10) [799a22e3] Command 'MergeVDSCommand(HostName = host2.domain.com, MergeVDSCommandParameters:{runAsync='true', hostId='c31dca1a-e5bc-43f6-940f-6397e3ddbee4', vmId='7083832a-a1a2-42b7-961f-2e9c0dcd7e18', storagePoolId='fa155d43-4e68-486f-9f9d-ae3e3916cc4f', storageDomainId='9339780c-3667-4fef-aa13-9bec08957c5f', imageGroupId='65a0b0d4-5c96-4dd9-a31b-4d08e40a46a5', imageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', baseImageId='568b2f77-0ddf-4349-a45c-36fcb0edecb6', topImageId='9eec9e8f-38db-4abf-b1c4-92fa9383f8b1', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2016-06-24 07:50:36,256 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-10) [799a22e3] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52)
I'm attaching relevant logs (both for ovirt-engine and SPM's vdsm).
What could be the reason for this error?
@Nir,
that looks like some issue in VDSM please have a look
jsonrpc.Executor/3::ERROR::2016-06-24 07:50:36,216::vm::4955::virt.vm::(merge) vmId=`7083832a-a1a2-42b7-961f-2e9c0dcd7e18`::Live merge failed (job: 3ea68e36-6d99-4af9-a54e-4c5b06df6a0f) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 4951, in merge flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 668, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: block copy still active: disk 'vda' already in active block job
This means there was a previous attempt to merge, and the block job did not complete yet.
Engine should detect that there is an active block job and wait until it completes.
This is probably engine bug, please file a bug and attach complete vdsm and engine logs.
Ala, can you look at this?
Nir
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (6)
-
Ala Hino -
Marcelo Leandro -
nicolas＠devels.es -
Nicolás
-
Nir Soffer -
Vinzenz Feenstra