[ovirt-users] vdsm using 100% CPU, rapidly filling logs with _handle_event messages

[Robert Story]

On Wed, 18 Nov 2015 07:28:23 -0500 Robert wrote:
RS> On Wed, 18 Nov 2015 12:35:17 +0100 Vinzenz wrote:
RS> VF> On 11/12/2015 03:16 PM, Robert Story wrote:
RS> VF> > On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote:
RS> VF> > DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
RS> VF> > DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I
RS> VF> > DK> > noticed that 2 of my 5 hosts were showing 99-100% cpu usage.
RS> VF> > DK> > Logging in to them, vdsmd seemed to be the culprit, and it
RS> VF> > DK> > was filling the log file with these messages:
RS> VF> > DK>
RS> VF> > DK> You're probably seeing
RS> VF> > DK>     Bug 1226911 - vmchannel thread consumes 100% of CPU
RS> VF> > DK>
RS> VF> > DK> which was closed due to missing information. Do you have any
RS> VF> > DK> information on when this pops up? Is it reproducible? Would
RS> VF> > DK> you be bale to test a suggested patch
RS> VF> > DK>
RS> VF> > DK>     https://gerrit.ovirt.org/#/c/42570/
RS> VF> >
RS> VF> > Hi Dan,
RS> VF> >
RS> VF> > Thanks for the pointers. If it comes up again, I'll try this
RS> VF> > patch and report back on the bug...
RS> VF> >
RS> VF> Out of curiosity, did you happen again to see that happening again?
RS> 
RS> No, I have not.

So naturally it shows up again during a holiday. I came in today to find 1
of my 5 nodes (the SPM and host where hosted engine is running) with two
vdsmd threads chewing up 90-100% of the CPU. I applied the patch from above
and restarted vdsmd. This resulted in another node being selected as the
SPM, and within about 15 minutes, that node had the same issue. Applied the
patch to the new node, and restarted vdsmd, and the SPM went back to the
previous (now patched) node. Hopefully things will stay stable.

I've attached snippets of the logs from the SPM when the problem started,
along with the server/engine log snippets from the hosted engine around the
same time..




Robert

-- 
Senior Software Engineer @ Parsons
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm-runaway.tgz
Type: application/x-compressed-tar
Size: 8512 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151127/cb50af57/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151127/cb50af57/attachment.sig>