9:53 a.m.
I have a three node cluster that worked fine upon setup, but randomly (Its
happened three times now) when a host is rebooted it loses its network
config. I backed up the ifcfg*, route, and rule* files to quickly put
things back in place, but even then vdsm fails to start and host reinstall
from the engine errors on vdsm startup. The only way that I have found to
recover the node is yum remove vdsm* libvirt*, and rm -Rf /etc/vdsm
& /var/lib/vdsm. Then I can run host reinstall from the engine.
Where the error occurs in host-deploy log:
2015-01-20 01:21:58 DEBUG otopi.plugins.otopi.services.systemd
plugin.execute:866 execute-output: ('/bin/systemctl', 'start',
'vdsmd.service') stderr:
Job for vdsmd.service failed. See 'systemctl status vdsmd.service' and
'journalctl -xn' for details.
2015-01-20 01:21:58 DEBUG otopi.context context._executeMethod:152 method
exception
Traceback (most recent call last):
File "/tmp/ovirt-6z1CuMPhsV/pythonlib/otopi/context.py", line 142, in
_executeMethod
method['method']()
File
"/tmp/ovirt-6z1CuMPhsV/otopi-plugins/ovirt-host-deploy/vdsm/packages.py",
line 219, in _start
self.services.state('vdsmd', True)
File "/tmp/ovirt-6z1CuMPhsV/otopi-plugins/otopi/services/systemd.py",
line 138, in state
'start' if state else 'stop'
File "/tmp/ovirt-6z1CuMPhsV/otopi-plugins/otopi/services/systemd.py",
line 77, in _executeServiceCommand
raiseOnError=raiseOnError
File "/tmp/ovirt-6z1CuMPhsV/pythonlib/otopi/plugin.py", line 871, in
execute
command=args[0],
RuntimeError: Command '/bin/systemctl' failed to execute
2015-01-20 01:21:58 ERROR otopi.context context._executeMethod:161 Failed
to execute stage 'Closing up': Command '/bin/systemctl' failed to execute
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:490
ENVIRONMENT DUMP - BEGIN
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/error=bool:'True'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>,
RuntimeError("Command '/bin/systemctl' failed to execute",),
<traceback
object at 0x259cbd8>)]'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:504
ENVIRONMENT DUMP - END
2015-01-20 01:21:58 INFO otopi.context context.runSequence:417 Stage:
Pre-termination
2015-01-20 01:21:58 DEBUG otopi.context context.runSequence:421 STAGE
pre-terminate
2015-01-20 01:21:58 DEBUG otopi.context context._executeMethod:138 Stage
pre-terminate METHOD otopi.plugins.otopi.core.misc.Plugin._preTerminate
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:490
ENVIRONMENT DUMP - BEGIN
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/aborted=bool:'False'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/debug=int:'0'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/error=bool:'True'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>,
RuntimeError("Command '/bin/systemctl' failed to execute",),
<traceback
object at 0x259cbd8>)]'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/executionDirectory=str:'/root'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/log=bool:'True'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/pluginGroups=str:'otopi:ovirt-host-deploy'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/pluginPath=str:'/tmp/ovirt-6z1CuMPhsV/otopi-plugins'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
BASE/suppressEnvironmentKeys=list:'[]'
2015-01-20 01:21:58 DEBUG otopi.context context.dumpEnvironment:500 ENV
COMMAND/chkconfig=str:'/sbin/chkconfig'