Re: [ovirt-devel] [VDSM][sampling] thread pool status and handling of stuck calls

----- Original Message ----- My review is almost done, but I'd like to bring the discussion in a place more friendly and visible than gerrit. Scheduler (29607) It is a nice code and could be an useful building block, but some important pieces are still out of the picture (the biggest: how to react to detect blocked calls? How to react to them?). On the other hand, I think I got all the bits in my threadpool patchset concept (29189). I surely recognize that I did in a very convoluted and unclear manner, but I'm not sure what will lead to the best outcome between to build on 29607 and add the missing pieces, or to deentangle 29189 and remove all the cruft from there. I think it could be useful anyway to map the concepts in play on this effort. At very least, this can act as summary/recap and can help us to converge on a shared vocabulary for the further discussion, so let me try. Tyring to be semi-formal: * Sampling: is a snippet of code, usually a method of the Vm class, that updates a set of fields of a Vm object, describing the state of a running Vm, with fresher data. This involves one or more calls to libvirt. * VM Samplings: Synonims "Samplings for a VM" or "VM sampling set" all the samplings that a VM need to fully update its state. It is worth to be noted that the samplings are 1. periodic 2. with different periods from each other because some data change more frequently than other * Safeness/Unsafeness In the context of the libvirt calls, and JUST and ONLY from the PoV of the sampling responsiveness, is useful to distinguish between 1. safe: except for yet to be seen catastrophic failures, they don't get stuck or block for indefinite amount of time 2. unsafe: they need to enter in the QEMU monitor and/or touch storage, and we *know* they can block, likely because it happened in the past. * Scheduler Is the agent which decides when to invoke a sampling of a VM. A Scheduler *must* know about wall time and about samplings intervals. * Worker Is the runnable entity (usually, but not necessarily a python thread) which perform a sampling. In the current solution, we have 1. One thread per VM which acts both as worker and as scheduler. 2. The scheduler logic is implicit and mixed with the worker code (see vdsm/virt/sampling.py - AdvancedStatsThread.collect() - sampling.py:~432) 3. The worker/scheduler doesn't handle unsafe calls in any way. 4. Each worker/scheduler carries all the VM samplings. 5. As implict side effect of 3 and 4 above, if one sampling blocks, all the VM samplings block automatically. There is some good in this behaviour we may want to preserve 6. As side effect of 1 above, all the VM samplings are independent which each other. If one blocks, the other (try to) continue to run. This is actually a nice behaviour we want to preserve. Drawbacks: a. len(worker) == len(VMs) -> do not scale b. no detection of unsafe calls blocking c. no reaction (including just signaling) of unsafe calls blocking Niceties: a. VM samplings are indpendent from each other b. VM samplings are related and part of a group In our ideal solution should have: 1. Fixed and low-ish (~10?) number of threads. Scalable to hundreds/thousands of VM. Must scale roughly like libvirt does (e.g. not be a bottleneck) 2. Explicit scheduler logic, deentangled from the worker 3. Handling of unsafe calls. Detection of a blocked call, and propagate its consequences on the VM health. At least mark it as not-responding. Better: stop (unsafe only?) sampling until the VM returns healthy. 4. Grouping among sampling to allow extension. If an unsafe sampling failed, should all the other samplings for different VMs continue to run? What about the other unsafe samplings of the same VM? Maybe not immediate need, but nice to have in near future 5. Do not waste libvirt resources nor just shift the burden on it. Do not issue calls when we don't know they can fail. Do not deplete the libvirt resource pool. Long story short: do not disconnect/reconnect, be nice with libvirt :) +++ Since my mail is already too long, I'd just stop here and ask if my above analysis is right, especially the description of the ideal solution, and/or if I missed something big. Thanks and bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani