[ovirt-devel] Re: callback hanging making commands executing serial

On Mon, Oct 19, 2020 at 2:58 PM Arik Hadas <ahadas(a)redhat.com&gt; wrote: > > > On Mon, Oct 19, 2020 at 10:44 AM Benny Zlotnik <bzlotnik(a)redhat.com&gt; > wrote: > >> IMHO (without knowing well the ansible related stuff), a sensible >> solution would be to make CreateOva async, this would require allowing >> to use AnsibleExecutor in an async manner, but this seems possible >> since AnsibleExecutor#runCommand already performs polling. >> So if the polling is extracted and will instead run in a callback of >> CreateOvaCommand (similar to VM jobs handling in MergeCommand) it >> should work properly. But then again, I'm not very familiar with the >> ansible stuff so it might not be feasible. >> > > +1 > That's neat! > It wasn't possible before when we invoked the playbook with > ProcessBuilder but now that we use Ansible-runner, that sounds like the > right approach to me (which minimizes the execution time of the 'execute' > phase so it could also fix [1]) > I will just say that yes - making ansible execute async sounds like a good approach to solve the specific command. I think that doing so is necessary, but it won't solve the callback problem. On each child command that has a synchronized part it will hang the other commands on the engine. If it happens quickly it might be a very short hang, otherwise it can be felt. Even without feeling it as a user, it probably gives less performance from the engine side monitoring parallel commands. I don't have such a case in my head right now(non ansible, sync child command), but it is possible we have some and will have in the future. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1797553 > > > >> >> On Mon, Oct 19, 2020 at 10:27 AM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> > >> > Adding Artur ... >> > >> > On Mon, Oct 19, 2020 at 9:06 AM Liran Rotenberg <lrotenbe(a)redhat.com&gt; >> wrote: >> >> >> >> Hi all, >> >> Investigating a bug about serial execution of the same command more >> than once shows a problem with our callbacks. >> >> >> >> The problem is felt when we have synchronized operation (such as >> running ansible runner service) that takes a bit time to be executed. >> >> The problem is, when running multiple commands and one has >> synchronized long operation within a child command, it will hang out all >> the commands running on the engine using callbacks. >> >> The good example is given in the bug, since export OVA command using >> ansible(sync) and the pack_ova script takes time to finish. >> >> This is not really noticeable with short synchronized commands, but >> it does execute serial for them as well instead of parallel. >> >> Running export to OVA command and then other commands with callbacks >> will get them hanging if the export command callback started and it reached >> to performNextOperation. >> >> >> >> Some technical details: >> >> We have one thread in CommandCallbacksPoller[1] that runs, collects >> the current command on the engine and processes them. >> >> Once we have the above scenario (let's say 2 commands), the first one >> will go into callback.doPolling in invokeCallbackMethodsImpl. [2] >> >> In that case it will go to ChildCommandsCallbackBase::doPolling, >> eventually to childCommandsExecutingEnded. [3] >> >> While there are more actions to perform we will do >> performNextOperation [4], which calls executeNextOperation (in the bug case >> [5]). >> >> When the next operation is long and synchronized, this will block the >> CommandCallbacksPoller thread and only when it finishes the thread is >> released and the callbacks continue working. >> >> >> >> Any idea how to solve this issue? >> >> >> >> Regards, >> >> Liran >> >> >> >> [1] - >> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... >> >> [2] - >> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... >> >> [3] - >> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... >> >> [4] - >> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... >> >> [5] - >> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules... >> >> >> > >> > >> > -- >> > Martin Perina >> > Manager, Software Engineering >> > Red Hat Czech s.r.o. >> > _______________________________________________ >> > Devel mailing list -- devel(a)ovirt.org >> > To unsubscribe send an email to devel-leave(a)ovirt.org >> > Privacy Statement: https://www.ovirt.org/privacy-policy.html >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GBOXNWVC3AT... >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/W45G7FUW736... >> >