12:33 p.m.
Hi,
I have some issues with hosted engine (oVirt 3.4 prerelease repo on
CentOS 6.5).
My setups is the following:
2 hosts (will be 4 in the future) with 4 GlusterFS shares:
- engine (for hosted engine)
- iso (for ISO domain)
- ovirt (oVirt storage domain)
I had a split-brain situation today (after rebooting both nodes) on
hosted-engine.lockspace file on engine GlusterFS volume which I resolved.
hosted engine used engine share via NFS (TCP) as glusterfs isn't
supported for oVirt hosted engine, yet. I'll switch to GlusterFS as soon
as oVirt will support it (I hope this will be soon as RHEV 3.3 is
already supporting GlusterFS for hosted engine).
First of all ovirt-ha-agent fails to start on both nodes:
# service ovirt-ha-agent start
Starting ovirt-ha-agent: [ OK ]
# service ovirt-ha-agent status
ovirt-ha-agent dead but subsys locked
MainThread::INFO::2014-03-03
11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
ovirt-hosted-engine-ha agent 1.1.0 started
MainThread::INFO::2014-03-03
11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
Found certificate common name: 10.0.200.101
MainThread::CRITICAL::2014-03-03
11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could
not start ha-agent
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
97, in run
self._run_agent()
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
154, in _run_agent
hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 152, in __init__
"STOP_VM": self._stop_engine_vm
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py",
line 56, in __init__
logger, actions)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
line 93, in __init__
self._logger = FSMLoggerAdapter(logger, self)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
line 16, in __init__
super(FSMLoggerAdapter, self).__init__(logger, None)
TypeError: super() argument 1 must be type, not classobj
If I want to start my hosted engine, I receive the following error in
vdsm logs, which makes absolutly no sense to me, as there is plenty of
disk space available:
Thread-62::DEBUG::2014-03-03
11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown
libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to acquire
lock: No space left on device
Thread-62::DEBUG::2014-03-03
11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released
Thread-62::ERROR::2014-03-03
11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/vm.py", line 3159, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py",
line 92, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in
createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed',
conn=self)
libvirtError: Failed to acquire lock: No space left on device
Thread-62::DEBUG::2014-03-03
11:24:46,286::vm::2720::vm.Vm::(setDownStatus)
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down:
Failed to acquire lock: No space left on device
# df -h | grep engine
ovirt-host01:/engine 281G 21G 261G 8%
/rhev/data-center/mnt/ovirt-host01:_engine
# sudo -u vdsm dd if=/dev/zero
of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test
bs=512 count=100
100+0 records in
100+0 records out
51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s
Could you give me some information on how to fix the ovirt-ha-agent and
then hosted-engine storage issue? Thanks a lot.
Btw, I had some issues during installation which I will explain in
separate emails.
--
Best Regards
René Koch
Senior Solution Architect
============================================
LIS-Linuxland GmbH
Brünner Straße 163, A-1210 Vienna
Phone: +43 1 236 91 60
Mobile: +43 660 / 512 21 31
E-Mail: rkoch(a)linuxland.at
============================================