<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 10 May 2016, at 16:20, Luciano Natale <<a href="mailto:luchonat@gmail.com" class="">luchonat@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Ok, here are the relevant vdsm logs!</div></div></blockquote><div><br class=""></div>it seems to be indeed storage-related problem. I could only find that the export task failed due to</div><div><div><br class=""></div><div>106774fb-1093-495a-a996-48f48e066a6f::ERROR::2016-05-07 18:04:15,998::blockVolume::429::Storage.Volume::(validateImagePath) Unexpected error</div><div>Traceback (most recent call last):</div><div> File "/usr/share/vdsm/storage/blockVolume.py", line 427, in validateImagePath</div><div> os.mkdir(imageDir, 0o755)</div><div>OSError: [Errno 17] File exists: '/rhev/data-center/58aa23b5-9680-4ff2-991c-8f8952cfa13c/2a1cc76e-9ba2-4586-a612-049894467470/images/63e58057-00fd-4597-a258-64558d45c155’</div><div><br class=""></div><div>Tal?</div><div><br class=""></div><div>Thanks,</div><div>michal</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Thanks!</div><div class="">Luciano.</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, May 10, 2016 at 5:46 AM, Michal Skrivanek <span dir="ltr" class=""><<a href="mailto:michal.skrivanek@redhat.com" target="_blank" class="">michal.skrivanek@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On 09 May 2016, at 23:09, Luciano Natale <<a href="mailto:luchonat@gmail.com" target="_blank" class="">luchonat@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">Ok, i've filtered out saturday. At midnight it starts to backup automatically with a custom made script. At that run, I had trouble with vm named "operaciones-ad". Then i started working on the problem (among other things, i added a new backup storage domain) and you can se around 8 PM that another VM failed the backup, this one called "biblioteca". Other VM's i remeber failing where "NS1" and "NS2”.</div></div></blockquote><div class=""><br class=""></div></span><div class="">Tal, can someone take a look and investigate why the task is failing?</div><div class="">engine.log failure excerpt below</div><div class=""><br class=""></div><div class="">Luciano, I suppose vdsm.log from that time would help further</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">michal</div><div class=""><br class=""></div><div class="">2016-05-07 18:04:52,386 INFO [org.ovirt.engine.core.bll.ExportVmCommand] (org.ovirt.thread.pool-8-thread-15) [7a94a972] Running command: ExportVmCommand internal: fals</div><div class="">e. Entities affected : ID: c9bddba0-c553-4a90-b97c-bdc1e88a333e Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN</div><div class="">2016-05-07 18:04:52,388 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-15) [7a94a972] START, SetVmStatusVDSCommand( vmId</div><div class="">= 59e1be99-c37e-431d-9b4c-1d039fd667a7, status = ImageLocked, exit status = Normal), log id: 7261d03c</div><div class="">2016-05-07 18:04:52,392 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-15) [7a94a972] FINISH, SetVmStatusVDSCommand, log</div><div class="">id: 7261d03c</div><div class="">2016-05-07 18:04:52,453 INFO [org.ovirt.engine.core.bll.ExportVmCommand] (org.ovirt.thread.pool-8-thread-15) [7a94a972] Lock freed to object EngineLock [exclusiveLocks</div><div class="">= key: 59e1be99-c37e-431d-9b4c-1d039fd667a7 value: VM</div><div class="">2016-05-07 18:04:52,465 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] Running command: CopyImageGroupCommand in</div><div class="">ternal: true. Entities affected : ID: c9bddba0-c553-4a90-b97c-bdc1e88a333e Type: Storage</div><div class="">2016-05-07 18:04:52,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MoveImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] START, MoveImageGroupV</div><div class="">DSCommand( storagePoolId = 58aa23b5-9680-4ff2-991c-8f8952cfa13c, ignoreFailoverLimit = false, storageDomainId = 2a1cc76e-9ba2-4586-a612-049894467470, imageGroupId = 63e</div><div class="">58057-00fd-4597-a258-64558d45c155, dstDomainId = c9bddba0-c553-4a90-b97c-bdc1e88a333e, vmId = 59e1be99-c37e-431d-9b4c-1d039fd667a7, op = Copy, postZero = false, force =</div><div class=""> true), log id: 781cc2f3</div><div class="">2016-05-07 18:04:53,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MoveImageGroupVDSCommand] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] FINISH, MoveImageGroup</div><div class="">VDSCommand, log id: 781cc2f3</div><div class="">2016-05-07 18:04:53,233 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] CommandAsyncTask::Adding CommandMultiAsy</div><div class="">ncTasks object for command 1711f190-c188-419c-acc2-25edc8b8d1cf</div><div class="">2016-05-07 18:04:53,234 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] CommandMultiAsyncTasks::AttachTask: Atta</div><div class="">ching task 106774fb-1093-495a-a996-48f48e066a6f to command 1711f190-c188-419c-acc2-25edc8b8d1cf.</div><div class="">2016-05-07 18:04:53,421 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] Adding task 106774fb-1093-495a-a996-48f4</div><div class="">8e066a6f (Parent Command ExportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.</div><div class="">2016-05-07 18:04:54,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] Correlation ID: 7a94a972, Job ID: a312c127-45ef-44b4-8bee-b36fe7f251ee, Call Stack: null, Custom Event ID: -1, Message: Starting export Vm operaciones-ad to vms-backups</div><div class="">2016-05-07 18:04:54,166 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-8-thread-15) [6f66db4a] BaseAsyncTask::startPollingTask: Starting to poll task 106774fb-1093-495a-a996-48f48e066a6f.</div><div class="">2016-05-07 18:04:54,945 WARN [org.ovirt.engine.core.bll.scheduling.policyunits.EvenGuestDistributionBalancePolicyUnit] (DefaultQuartzScheduler_Worker-53) [662bbf2a] There is no host with less than 4 running guests</div><div class="">2016-05-07 18:04:54,946 WARN [org.ovirt.engine.core.bll.scheduling.PolicyUnitImpl] (DefaultQuartzScheduler_Worker-53) [662bbf2a] All hosts are over-utilized, cant balance the cluster main</div><div class="">2016-05-07 18:05:00,647 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now</div><div class="">2016-05-07 18:05:00,656 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] Failed in HSMGetAllTasksStatusesVDS method</div><div class="">2016-05-07 18:05:00,657 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] SPMAsyncTask::PollTask: Polling task 106774fb-1093-495a-a996-48f48e066a6f (Parent Command ExportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'.</div><div class="">2016-05-07 18:05:00,709 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] BaseAsyncTask::logEndTaskFailure: Task 106774fb-1093-495a-a996-48f48e066a6f (Parent Command ExportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:</div><div class="">2016-05-07 18:05:00,712 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] CommandAsyncTask::endActionIfNecessary: All tasks of command 1711f190-c188-419c-acc2-25edc8b8d1cf has ended -> executing endAction</div><div class="">2016-05-07 18:05:00,714 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler_Worker-90) [3b5cd3b6] CommandAsyncTask::endAction: Ending action for 1 tasks (command ID: 1711f190-c188-419c-acc2-25edc8b8d1cf): calling endAction .</div><div class="">2016-05-07 18:05:00,716 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-8-thread-30) [3b5cd3b6] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction ExportVm, executionIndex: 0</div><div class="">2016-05-07 18:05:00,820 ERROR [org.ovirt.engine.core.bll.ExportVmCommand] (org.ovirt.thread.pool-8-thread-30) [3b5cd3b6] Ending command with failure: org.ovirt.engine.core.bll.ExportVmCommand</div><div class="">2016-05-07 18:05:00,876 ERROR [org.ovirt.engine.core.bll.CopyImageGroupCommand] (org.ovirt.thread.pool-8-thread-30) [6f66db4a] Ending command with failure: org.ovirt.engine.core.bll.CopyImageGroupCommand</div><div class="">2016-05-07 18:05:00,881 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-30) START, SetVmStatusVDSCommand( vmId = 59e1be99-c37e-431d-9b4c-1d039fd667a7, status = Down, exit status = Normal), log id: 311b6c1c</div><div class="">2016-05-07 18:05:00,886 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-8-thread-30) FINISH, SetVmStatusVDSCommand, log id: 311b6c1c</div><div class="">2016-05-07 18:05:00,987 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) Correlation ID: 7a94a972, Call Stack: null, Custom Event ID: -1, Message: Failed to export Vm operaciones-ad to vms-backups</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><span class=""><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Luciano.</div></div><div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br class=""><div class="gmail_quote">On Mon, May 9, 2016 at 5:05 AM, Michal Skrivanek<span class=""> </span><span dir="ltr" class=""><<a href="mailto:michal.skrivanek@redhat.com" target="_blank" class="">michal.skrivanek@redhat.com</a>></span><span class=""> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br class="">> On 08 May 2016, at 02:14, Luciano Natale <<a href="mailto:luchonat@gmail.com" target="_blank" class="">luchonat@gmail.com</a>> wrote:<br class="">><br class="">> Hi everyone. I've been having trouble when exporting VM's. I get error when moving image. I've created a whole new storage domain exclusive for this issue, and same thing happens. It's not always the same VM that fails, but once it fails on a certain storage domain, I cannot export it anymore. Please tell me which logs are relevant so i can post them and any other relevant iformation I can provide, and maybe someone can help me get through this problem.<br class=""><br class=""></span>Hi,<br class="">please send /var/log/ovirt-engine/engine.log and let’s see.<br class=""><br class="">Thanks,<br class="">michal<br class=""><span class=""><br class="">><br class="">> Ovirt version is 3.5.4.2-1.el6. Hosted engines is CentOS 6. Hosts are CentOS 7. VM's are all CentOS 7, except for two that are CentOS 6 and Windows 7.<br class="">><br class="">> Please excuse my bad english!<br class="">> Thanks in advance!<br class="">><br class="">> --<br class="">> Luciano Natale<br class=""></span>> _______________________________________________<br class="">> Users mailing list<br class="">><span class=""> </span><a href="mailto:Users@ovirt.org" target="_blank" class="">Users@ovirt.org</a><br class="">><span class=""> </span><a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/mailman/listinfo/users</a><br class=""><br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>--<span class=""> </span><br class=""><div class="">Luciano Natale</div></div></span><span class=""><engine.log></span></div></blockquote></div><br class=""></div></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature">Luciano Natale</div>
</div>
<span id="cid:BB27D3B0-3CC9-4ED4-87D5-D842D7E37913@brq.redhat.com"><hyper1-vdsm.log.33.xz></span><span id="cid:FE6BA9C8-78B7-48F8-B2CD-68916D46AC62@brq.redhat.com"><hyper2-vdsm.log.65.xz></span></div></blockquote></div><br class=""></body></html>