<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 4, 2017 at 12:18 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class=""><blockquote type="cite"><div>On 3 Jul 2017, at 15:35, Shlomo Ben David <<a href="mailto:sbendavi@redhat.com" target="_blank">sbendavi@redhat.com</a>> wrote:</div><br class="m_4588124764219751948Apple-interchange-newline"><div><div dir="ltr">Hi,<br><br>Test failed: [ 006_migrations.migrate_vm ]<br>Link to suspected patches: N/A<br>Link to Job: <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7431/" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/7431/</a><br>Link to all logs: <br>Error snippet from the log: <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7431/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/" target="_blank">http://jenkins.ovirt.org/<wbr>job/test-repo_ovirt_<wbr>experimental_master/7431/<wbr>artifact/exported-artifacts/<wbr>basic-suit-master-el7/test_<wbr>logs/basic-suite-master/post-<wbr>006_migrations.py/</a><br><br><error><br><br> "Fault reason is "Operation Failed". Fault detail is "[Cannot migrate VM. There is no host that satisfies current scheduling constraints. See below for details:, The host lago-basic-suite-master-host0 did not satisfy internal filter CPUOverloaded because its CPU is too loaded.]"<div><br></div><div></error><br></div><div><br></div><div><engine log><br></div><div><br></div><div><div>2017-07-02 16:43:22,829-04 INFO [org.ovirt.engine.core.bll.<wbr>MigrateVmToServerCommand] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr>2b34910d-cef2-44d6-a274-<wbr>30e8473eb5d9=VM]', sharedLocks=''}'</div><div>2017-07-02 16:43:22,833-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Compiled stored procedure. Call string is [{call getdiskvmelementspluggedtovm(?<wbr>)}]</div><div>2017-07-02 16:43:22,833-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] SqlCall for procedure [GetDiskVmElementsPluggedToVm] compiled</div><div>2017-07-02 16:43:22,843-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Compiled stored procedure. Call string is [{call getattacheddisksnapshotstovm(?<wbr>, ?)}]</div><div>2017-07-02 16:43:22,843-04 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>PostgresDbEngineDialect$<wbr>PostgresSimpleJdbcCall] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] SqlCall for procedure [GetAttachedDiskSnapshotsToVm] compiled</div><div>2017-07-02 16:43:22,919-04 INFO [org.ovirt.engine.core.bll.<wbr>scheduling.SchedulingManager] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Candidate host 'lago-basic-suite-master-<wbr>host0' ('46bdc63d-98f5-4eee-81aa-<wbr>2fb88b8f7cbe') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPUOverloaded' (correlation id: null)</div><div>2017-07-02 16:43:22,920-04 WARN [org.ovirt.engine.core.bll.<wbr>MigrateVmToServerCommand] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Validation of action 'MigrateVmToServer' failed for user admin@internal-authz. Reasons: VAR__ACTION__MIGRATE,VAR__<wbr>TYPE__VM,SCHEDULING_ALL_HOSTS_<wbr>FILTERED_OUT,VAR__FILTERTYPE__<wbr>INTERNAL,$hostName lago-basic-suite-master-host0,<wbr>$filterName CPUOverloaded,VAR__DETAIL__<wbr>CPU_OVERLOADED,SCHEDULING_<wbr>HOST_FILTERED_REASON_WITH_<wbr>DETAIL</div></div></div></div></blockquote></span><div><br><div><br></div><div>This has nothing to do with migration<br>The CPUOverload is a scheduling policy, unless there was any change in that area the obvious explanation would be that the host has a CPU overload condition.<br>I briefly looked at logs and see ""cpuUser": "83.40", "cpuSys": "16.59", "cpuIdle": “0.08”” which indeed suggests an overload, from the same sample I can see it’s vdsm ("cpuUserVdsmd": “77.38”, cpuSysVdsmd": “18.44"<br><br></div>Since similar values are consistently being reported for some time, and there is a setupNetworks and storage rescan prior to the the failure, and there is no other indication of anything wrong, I’d just say the environment or the order of tests or timing has changed, but nothing wrong with the oVirt code</div><div>Did any of that changed recently? Does it reproduce locally?</div></div></div></blockquote><div><br></div><div>AFAIK, no significant environment changes or tests were done. </div><div>We will try to reproduce it locally and also on the manual job, but from what it looks it is very consistent (unlike other race failures we've seen lately ) and continues to fails on the same tests, so its either a change in oVirt or something else that we're not thinking on.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><br></div><div>Thanks,</div><div>michal</div><div><br></div><blockquote type="cite"><div><span class=""><div dir="ltr"><div><div>2017-07-02 16:43:22,920-04 INFO [org.ovirt.engine.core.bll.<wbr>MigrateVmToServerCommand] (default task-27) [87508047-fdc5-4a2f-9692-<wbr>c83f7b55bbc2] Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr>2b34910d-cef2-44d6-a274-<wbr>30e8473eb5d9=VM]', sharedLocks=''}'</div><div>2017-07-02 16:43:22,929-04 DEBUG [org.ovirt.engine.core.utils.<wbr>timer.FixedDelayJobListener] (DefaultQuartzScheduler7) [] Rescheduling <a href="http://DEFAULT.org" target="_blank">DEFAULT.org</a>.ovirt.engine.core.<wbr>bll.<wbr>ColdRebootAutoStartVmsRunner.<wbr>startFailedAutoStartVms#-<wbr>9223372036854775733 as there is no unfired trigger.</div><div>2017-07-02 16:43:22,932-04 ERROR [org.ovirt.engine.api.restapi.<wbr>resource.<wbr>AbstractBackendResource] (default task-27) [] Operation Failed: [Cannot migrate VM. There is no host that satisfies current scheduling constraints. See below for details:, The host lago-basic-suite-master-host0 did not satisfy internal filter CPUOverloaded because its CPU is too loaded.]</div><div>2017-07-02 16:43:23,331-04 DEBUG [org.ovirt.engine.core.utils.<wbr>timer.FixedDelayJobListener] (DefaultQuartzScheduler2) [] Rescheduling <a href="http://DEFAULT.org" target="_blank">DEFAULT.org</a>.ovirt.engine.core.<wbr>bll.HaAutoStartVmsRunner.<wbr>startFailedAutoStartVms#-<wbr>9223372036854775793 as there is no unfired trigger.</div><div>2017-07-02 16:43:23,332-04 DEBUG [org.ovirt.engine.core.utils.<wbr>timer.FixedDelayJobListener] (DefaultQuartzScheduler2) [] Rescheduling <a href="http://DEFAULT.org" target="_blank">DEFAULT.org</a>.ovirt.engine.core.<wbr>bll.tasks.<wbr>CommandCallbacksPoller.<wbr>invokeCallbackMethods#-<wbr>9223372036854775783 as there is no unfired trigger.</div><div><br></div><engine log><br><br><div><br></div><br clear="all"><div><div class="m_4588124764219751948gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Best Regards,</div><div dir="ltr"><br></div><div dir="ltr">Shlomi Ben-David | Software Engineer <span style="font-size:small">| </span><span style="font-size:12.8px">Red Hat ISRAEL</span></div><div>RHCSA | <span style="font-size:small">RHCVA | </span><span style="font-size:small">RHCE</span></div><div dir="ltr">IRC: shlomibendavid <span style="font-size:small">(on #rhev-integ, #rhev-dev, #rhev-ci)</span><br><br>OPEN SOURCE - 1 4 011 && 011 4 1<br><br></div></div></div></div></div></div></div></div></div></div></div>
</div></div></span>
______________________________<wbr>_________________<br>Devel mailing list<br><a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/devel" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a></div></blockquote></div><br></div><br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div>
</div></div>