On 06/01/2015 03:56 AM, Simone Tiraboschi wrote:
----- Original Message -----
From: "Douglas Schilling Landgraf" <dougsland@redhat.com> To: "Simone Tiraboschi" <stirabos@redhat.com>, devel@ovirt.org Cc: "Fabian Deutsch" <fdeutsch@redhat.com> Sent: Saturday, May 30, 2015 11:28:38 PM Subject: Re: oVirt node 3.6 and CPU load indefinitely stuck on 100% while vdsmd indefinitely tries to restart
On 05/29/2015 06:44 AM, Simone Tiraboschi wrote:
Hi, I tried to have hosted-engine deploying the engine appliance over oVirt node. I think it will be quite a common scenario. I tried with an oVirt node build from yesterday.
Unfortunately I'm not able to conclude the setup cause oVirt node got the CPU load indefinitely stuck on 100% and so it's almost unresponsive.
The issue seams to be related to vdsmd daemon witch couldn't really start and so it retries indefinitely using all the available CPU power (it also runs with niceless -20...).
[root@node36 admin]# grep "Unit vdsmd.service entered failed state." /var/log/messages | wc -l 368 It tried 368 times in a row in a few minutes.
With journalctl I can read: May 29 10:06:45 node36 systemd[1]: Unit vdsmd.service entered failed state. May 29 10:06:45 node36 systemd[1]: vdsmd.service holdoff time over, scheduling restart. May 29 10:06:45 node36 systemd[1]: Stopping Virtual Desktop Server Manager... May 29 10:06:45 node36 systemd[1]: Starting Virtual Desktop Server Manager... May 29 10:06:45 node36 vdsmd_init_common.sh[13697]: vdsm: Running mkdirs May 29 10:06:45 node36 vdsmd_init_common.sh[13697]: vdsm: Running configure_coredump May 29 10:06:45 node36 vdsmd_init_common.sh[13697]: vdsm: Running configure_vdsm_logs May 29 10:06:45 node36 vdsmd_init_common.sh[13697]: vdsm: Running wait_for_network May 29 10:06:45 node36 vdsmd_init_common.sh[13697]: vdsm: Running run_init_hooks May 29 10:06:46 node36 vdsmd_init_common.sh[13697]: vdsm: Running upgraded_version_check May 29 10:06:46 node36 vdsmd_init_common.sh[13697]: vdsm: Running check_is_configured May 29 10:06:46 node36 vdsmd_init_common.sh[13697]: vdsm: Running validate_configuration May 29 10:06:47 node36 vdsmd_init_common.sh[13697]: vdsm: Running prepare_transient_repository May 29 10:06:49 node36 vdsmd_init_common.sh[13697]: vdsm: Running syslog_available May 29 10:06:49 node36 vdsmd_init_common.sh[13697]: vdsm: Running nwfilter May 29 10:06:50 node36 vdsmd_init_common.sh[13697]: vdsm: Running dummybr May 29 10:06:51 node36 vdsmd_init_common.sh[13697]: vdsm: Running load_needed_modules May 29 10:06:51 node36 vdsmd_init_common.sh[13697]: vdsm: Running tune_system May 29 10:06:51 node36 vdsmd_init_common.sh[13697]: vdsm: Running test_space May 29 10:06:51 node36 vdsmd_init_common.sh[13697]: vdsm: Running test_lo May 29 10:06:51 node36 systemd[1]: Started Virtual Desktop Server Manager. May 29 10:06:51 node36 systemd[1]: vdsmd.service: main process exited, code=exited, status=1/FAILURE May 29 10:06:51 node36 vdsmd_init_common.sh[13821]: vdsm: Running run_final_hooks May 29 10:06:52 node36 systemd[1]: Unit vdsmd.service entered failed state. May 29 10:06:52 node36 systemd[1]: vdsmd.service holdoff time over, scheduling restart. May 29 10:06:52 node36 systemd[1]: Stopping Virtual Desktop Server Manager... May 29 10:06:52 node36 systemd[1]: Starting Virtual Desktop Server Manager... repeated a lot of times
/var/log/vdsm/vdsm.log is empty.
while [root@node36 admin]# /usr/share/vdsm/daemonAdapter -0 /dev/null -1 /dev/null -2 /dev/null /usr/share/vdsm/vdsm; echo $? 1
Thanks for the report Simone. From my tests you are facing:
non-root user cannot `from ovirtnode import ovirtfunctions`: permission denied: '/var/log/ovirt-node.log' and '/var/log/ovirt.log https://bugzilla.redhat.com/show_bug.cgi?id=1224400
We should handle this bug very soon. The workaround is chmod o+rw in /var/log/ovirt.log /var/log/ovirt-node.log
OK. I tried [root@node36 admin]# chmod o+rw /var/log/ovirt.log /var/log/ovirt-node.log
but now I'm getting: [root@node36 admin]# systemctl status -l vdsmd vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled) Active: active (running) since Mon 2015-06-01 07:53:09 UTC; 17s ago Process: 4040 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 4049 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 4164 (vdsm) CGroup: /system.slice/vdsmd.service └─4164 /usr/bin/python /usr/share/vdsm/vdsm
Jun 01 07:53:07 node36 vdsmd_init_common.sh[4049]: vdsm: Running nwfilter Jun 01 07:53:08 node36 vdsmd_init_common.sh[4049]: vdsm: Running dummybr Jun 01 07:53:09 node36 vdsmd_init_common.sh[4049]: vdsm: Running load_needed_modules Jun 01 07:53:09 node36 vdsmd_init_common.sh[4049]: vdsm: Running tune_system Jun 01 07:53:09 node36 vdsmd_init_common.sh[4049]: vdsm: Running test_space Jun 01 07:53:09 node36 vdsmd_init_common.sh[4049]: vdsm: Running test_lo Jun 01 07:53:09 node36 systemd[1]: Started Virtual Desktop Server Manager. Jun 01 07:53:10 node36 vdsm[4164]: vdsm vds ERROR failed to init clientIF, shutting down storage dispatcher Jun 01 07:53:10 node36 vdsm[4164]: vdsm vds ERROR Exception raised Traceback (most recent call last): File "/usr/share/vdsm/vdsm", line 154, in run serve_clients(log) File "/usr/share/vdsm/vdsm", line 93, in serve_clients cif = clientIF.getInstance(irs, log) File "/usr/share/vdsm/clientIF.py", line 166, in getInstance File "/usr/share/vdsm/clientIF.py", line 112, in __init__ File "/usr/share/vdsm/clientIF.py", line 170, in _createAcceptor File "/usr/share/vdsm/clientIF.py", line 183, in _createSSLContext File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 149, in __init__ File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 174, in _initContext File "/usr/lib/python2.7/site-packages/vdsm/sslutils.py", line 153, in _loadCertChain File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Context.py", line 100, in load_cert_chain SSLError: No such file or directory Jun 01 07:53:20 node36 vdsm[4164]: vdsm vds ERROR Vm's recovery failed Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 416, in _recoverExistingVms File "/usr/share/vdsm/caps.py", line 177, in __init__ File "/usr/share/vdsm/caps.py", line 209, in _getCpuTopology File "/usr/share/vdsm/caps.py", line 199, in _getFreshCapsXMLStr File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 162, in get File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 99, in open_connection File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1008, in retry File "/usr/lib64/python2.7/site-packages/libvirt.py", line 105, in openAuth libvirtError: authentication failed: polkit: polkit\56retains_authorization_after_challenge=1 Authorization requires authentication but no agent is available.
Was it just a partial workaround or am I facing a different issue?
It should be a different one, I will try to catch this one locally. -- Cheers Douglas