Re: [ovirt-devel] [VDSM][sampling] thread pool status and handling of stuck calls

I share most of Nir's answers, let me just add some information here and there ----- Original Message ----- Indeed. My biggest reason for that actually boils down to what Federico suggested on his last email. By acknowledging stuck calls in the upper layer I believe we can manage them better on overall. I think is more akin to a flow error, cannot be recovered on the lower layers, nor the connection layer can provide a reliable error handling, so let's just handle this on application level. Well I see little benefit for additional timeout. And surely (sorry for the repetition, but IMO worth stressing out) we need to be nice with libvirt, thus wait is doing the less harm, because we want to avoid to clog libvirt by issuing more call that we will know in advance they would fail (since one is still stuck holding the QEMU monitor). If we patiently wait for libvirt call to get unstuck we also know precisely when is safe to restart sampling/calling libvirt - right after the call returned. Otherwise we'll have to implement some polling, or just call and hope (not good). Having said that, let me mention the timeout machinery we already have in place: vdsm/virt/vm.py :: NotifyingVirDomain Please note that this code is not sampling or stuck-handling specific and was introduced long ago. This code doesn't change anything we said or did because the timeout will be raised when 1. a call is already stuck inside the qemu monitor 2. another call tries to enter on the same monitor The above behaviour (and maybe a timeout) does'nt change the picture, or the 'be nice with libvirt' requirement (well, is probably more like 'do not blindly drop the ball to libvirt, because this will just make things worse'). It could just help us to detect earlier that a VM is stuck. Saggi is right when he says the async-ness is one of the core issues here, but the another, maybe also more important, is the stuck-ness of calls. And unfortunately, 1. we will know a call is stuck only *after* it happened (timeout, supervisor, anything else) 2. there is no reliable way to *cancel* a stuck call/task, even dropping the connection won't help, as Federico reminded. In the design we are evolving, we shall 1. for the VM *already* blocked, and somehow detected as that, suspend sampling until libvirt unlocks. This state has to be reported to Engine, but that's it. 2. for the *others* VM, replace the stuck worker threads with fresh ones (fancy threadpool approach) otherwise everything stops when the pool is exausted, even if some VM continues to behave correctly. - http://gerrit.ovirt.org/#/c/29191 Lastly, about to asyncness of the libvirt API, maybe worth (re)mentioning https://www.redhat.com/archives/libvir-list/2014-July/msg00139.html relevant snippet: "[...] The libvirt-gobject binding to libvirt provides async APIs to libvirt APIs. It does this by using threads internally. [...]" Indeed. More precisely, until we know the QEMU monitor for a VM is safe again, there is no point in try to use it again. Agreed Agreed again. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani