[ovirt-users] vdsm using 100% CPU, rapidly filling logs with _handle_event messages

[Robert Story]

I'm running oVirt 3.5.x with a hosted engine. This morning I noticed that 2
of my 5 hosts were showing 99-100% cpu usage. Logging in to them, vdsmd
seemed to be the culprit, and it was filling the log file with these
messages:

VM Channels Listener::DEBUG::2015-11-12
08:09:26,292::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 75
VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,294::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12 08:09:26,294::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll.

I googled to see how to change the debug level to turn of DEBUG messages
for vdsm, which referred me to libvirtd.conf, but the debug level there was
not set, which should have meant a log level of 3 (warnings and errors), so
I'm not sure why the log was filling up with DEBUG/INFO messages.

I restarted vdsmd, which resulted in those nodes being marked as
'disconnected', but they did eventually recover and loads went back to
normal.

This may or may not be related to the fact that the 3 hosts where this did
not happen can't seem to keep their ha brokers up. I'll be starting a new
thread on that shortly.


Robert

-- 
Senior Software Engineer @ Parsons
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151112/c0703201/attachment-0001.sig>