[ovirt-devel] [VDSM] VDSM threads as in vdsm 4.15.0

[Nir Soffer]

----- Original Message -----
> From: "Francesco Romani" <fromani at redhat.com>
> To: devel at ovirt.org
> Sent: Friday, July 4, 2014 1:09:44 PM
> Subject: [ovirt-devel] [VDSM] VDSM threads as in vdsm 4.15.0
> 
> Hi,
> 
> I'm trying to enumerate all the daemon threads VDSM uses, as in 4.15.0
> (to be exact, as in master 3228f5dad8f488192ea62f4e3e39172832435e80)
> 
> So far I got:
> 
> ===
> 
> SocketServer: 1 service, short lived thread per request?

We don't have such thread, nor short lived threads per requests (but see bellow)

> 
> VDSM alone, with no VM running
> 
> num what                where
> --------------------------------------------------------
> 1   libVirtEventLoop    lib/vdsm/libvirtconnection.py
> P   TaskManager         vdsm/storage/hsm.py

This is not a thread - you mean the pool of threadPool.WorkerThread

> 1   StorageServer       vdsm/storage/storageServer.py

I guess you mean storageServer.ConnectionMonitor thread

> 1   StorageRefresh      vdsm/storage/hsm.py

This thread lives only during startup and does not exists when vdsm is ready.

> 1   ChannelListener     vdsm/virt/vmchannels.py
> 1   HostStatsThread     vdsm/virt/sampling.py
> 1   MomThread           vdsm/momIF.py

There is also ksm.KsmMonitorThread

> 1   clientIF            vdsm/clientIF.py

This is not a thread, don't let the interface confuse you :-)

The start() method starts the bindings threads and the protocol detector
you list bellow.

This is another short lived thread the exit when running vms are recovered
and does not exists when vdsm is ready.

> 1   HostMonitor         mom/HostMonitor.py
> 1   StompReactor        vdsm/rpc/BindingJsonRpc.py
> 1   JsonRpcServer       vdsm/rpc/BidningJsonRpc.py

This thread create short living threads for each request, without any limit.
If you send 100 requests, vdsm will start 100 threads the handle the requests.
(since Jun 11)

> 1   GuestManager        mom/GuestManager.py
> 1   PolicyEngine        mom/PolicyEngine.py
> 1   XmlRpcServer?       vdsm/rpc/BindingXMLRPC.py

This server create new thread per connection. Since ovirt-3.4. This thread
is long living since we use keep-alive (since 3.4).

There is no limit to the number of connection threads. On and idle machine
we see 3-4 such threads. So you should add 4 to the baseline for these.

> 1   Detector            vdsm/clientIF.py
> 1   RPCServer           mom/RPCServer.py
> 1   SocketServer        stdlib, triggered by vdsm/rpc/BindingXMLRPC.py

There is no such thread - you counted it in XmlRpcServer.

> 1   IOProcess?          vdsm/storage/outOfProcess.py
> S   DomainMonitor       vdsm/storage/domainMonitor.py

When domain monitor changes state, it starts a new thread for notifying
observers and this thread may start new thread for each notification.
These thread exit after the notifications are done. 

> P/2 HSM_MailMonitor     vdsm/storage/storage_mailbox.py

These run on all hosts but the spm

> P/2 SPM_MailMonitor     vdsm/storage/storage_mailbox.py

These run only on the spm - so we have only pool_size / 2 mailbox threads.

> 
> P = pool size (config/irs/thread_pool_size)
> 
> Total: 17 + number(StorageDomains) + P * 2
> Total with defaults (P=10) = 37 + number(StorageDomains)
> Minimum storage domains: 2 (ISO + data)

You don't need iso domain, so minimum is 1 storage domain thread.

But the minimum is not interesting - we have customers with 30-40 domains,
so assume 40 of these threads.

> Baseline: 39 threads

Based on my corrections:

14 long living threads
10 task threads
5 mailbox threads
3-? xmlprc connection threads or ? jsonrpc requests threads
1-40 domain monitor threads
0-600 vm threads (2 per vm)

Total: 33-?

Nir