[Users] hosted engine issues

[René Koch]

Hi,

I have some issues with hosted engine (oVirt 3.4 prerelease repo on 
CentOS 6.5).
My setups is the following:
2 hosts (will be 4 in the future) with 4 GlusterFS shares:
- engine (for hosted engine)
- iso (for ISO domain)
- ovirt (oVirt storage domain)

I had a split-brain situation today (after rebooting both nodes) on 
hosted-engine.lockspace file on engine GlusterFS volume which I resolved.

hosted engine used engine share via NFS (TCP) as glusterfs isn't 
supported for oVirt hosted engine, yet. I'll switch to GlusterFS as soon 
as oVirt will support it (I hope this will be soon as RHEV 3.3 is 
already supporting GlusterFS for hosted engine).


First of all ovirt-ha-agent fails to start on both nodes:

# service ovirt-ha-agent start
Starting ovirt-ha-agent:                                   [  OK  ]
# service ovirt-ha-agent status
ovirt-ha-agent dead but subsys locked


MainThread::INFO::2014-03-03 
11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) 
ovirt-hosted-engine-ha agent 1.1.0 started
MainThread::INFO::2014-03-03 
11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) 
Found certificate common name: 10.0.200.101
MainThread::CRITICAL::2014-03-03 
11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could 
not start ha-agent
Traceback (most recent call last):
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
97, in run
     self._run_agent()
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
154, in _run_agent
     hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", 
line 152, in __init__
     "STOP_VM": self._stop_engine_vm
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", 
line 56, in __init__
     logger, actions)
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", 
line 93, in __init__
     self._logger = FSMLoggerAdapter(logger, self)
   File 
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", 
line 16, in __init__
     super(FSMLoggerAdapter, self).__init__(logger, None)
TypeError: super() argument 1 must be type, not classobj



If I want to start my hosted engine, I receive the following error in 
vdsm logs, which makes absolutly no sense to me, as there is plenty of 
disk space available:

Thread-62::DEBUG::2014-03-03 
11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown 
libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to acquire 
lock: No space left on device
Thread-62::DEBUG::2014-03-03 
11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm) 
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released
Thread-62::ERROR::2014-03-03 
11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm) 
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed
Traceback (most recent call last):
   File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm
     self._run()
   File "/usr/share/vdsm/vm.py", line 3159, in _run
     self._connection.createXML(domxml, flags),
   File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", 
line 92, in wrapper
     ret = f(*args, **kwargs)
   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in 
createXML
     if ret is None:raise libvirtError('virDomainCreateXML() failed', 
conn=self)
libvirtError: Failed to acquire lock: No space left on device
Thread-62::DEBUG::2014-03-03 
11:24:46,286::vm::2720::vm.Vm::(setDownStatus) 
vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: 
Failed to acquire lock: No space left on device

# df -h | grep engine
ovirt-host01:/engine     281G   21G  261G   8% 
/rhev/data-center/mnt/ovirt-host01:_engine

# sudo -u vdsm dd if=/dev/zero 
of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test 
bs=512 count=100
100+0 records in
100+0 records out
51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s


Could you give me some information on how to fix the ovirt-ha-agent and 
then hosted-engine storage issue? Thanks a lot.

Btw, I had some issues during installation which I will explain in 
separate emails.


-- 
Best Regards

René Koch
Senior Solution Architect

============================================
LIS-Linuxland GmbH
Brünner Straße 163, A-1210 Vienna

Phone:   +43 1 236 91 60
Mobile:  +43 660 / 512 21 31
E-Mail:  rkoch at linuxland.at
============================================