Hi René, Then I guess the traceback is my fault... See http://gerrit.ovirt.org/#/c/25269/ for the fix. I will try to get it into the soonest release possible. Me neither. Is everything Read-Write? Read-Only FS might report no space left as well in some cases. Other than that, I do not know. Regards -- Martin Sivák msivak(a)redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message ----- > On 03/03/2014 12:05 PM, Martin Sivak wrote: > > Hi René, > > > > thanks for the report. > > > >>> TypeError: super() argument 1 must be type, not classobj > > What Python version are you using? > > > > > > You can debug a crash of this version of ha-agent using: > > > > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon --pdb > > This gives me the same information as in vdsm.log > > > > > But this exception is trying to tell you that > > FSMLoggerAdapter(logging.LoggerAdapter) does not have object in the > > ancestor list. And that is very weird. > > > > It can be related to the disk space issues. > > > >>> libvirtError: Failed to acquire lock: No space left on device > > > > Check the free space on all your devices, including /tmp and /var. Or post > > the output of "df -h" command here > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/vg0-lv_root 5.0G 1.1G 3.6G 24% / > tmpfs 16G 0 16G 0% /dev/shm > /dev/sda1 243M 45M 185M 20% /boot > /dev/mapper/vg0-lv_data 281G 21G 261G 8% /data > /dev/mapper/vg0-lv_tmp 2.0G 69M 1.9G 4% /tmp > /dev/mapper/vg0-lv_var 5.0G 384M 4.3G 9% /var > ovirt-host01:/engine 281G 21G 261G 8% > /rhev/data-center/mnt/ovirt-host01:_engine > > > Thanks, > René > > > > > > Regards > > > > -- > > Martin Sivák > > msivak(a)redhat.com > > Red Hat Czech > > RHEV-M SLA / Brno, CZ > > > > ----- Original Message ----- > >> Il 03/03/2014 11:33, René Koch ha scritto: > >>> Hi, > >>> > >>> I have some issues with hosted engine (oVirt 3.4 prerelease repo on > >>> CentOS > >>> 6.5). > >>> My setups is the following: > >>> 2 hosts (will be 4 in the future) with 4 GlusterFS shares: > >>> - engine (for hosted engine) > >>> - iso (for ISO domain) > >>> - ovirt (oVirt storage domain) > >>> > >>> I had a split-brain situation today (after rebooting both nodes) on > >>> hosted-engine.lockspace file on engine GlusterFS volume which I resolved. > >> > >> How did you solved it? By switching to NFS only? > >> > >> > >>> hosted engine used engine share via NFS (TCP) as glusterfs isn't > >>> supported > >>> for oVirt hosted engine, yet. I'll switch to GlusterFS as soon as oVirt > >>> will support it (I hope this will be soon as RHEV 3.3 is already > >>> supporting > >>> GlusterFS for hosted engine). > >>> > >>> > >>> First of all ovirt-ha-agent fails to start on both nodes: > >>> > >>> # service ovirt-ha-agent start > >>> Starting ovirt-ha-agent: [ OK ] > >>> # service ovirt-ha-agent status > >>> ovirt-ha-agent dead but subsys locked > >>> > >>> > >>> MainThread::INFO::2014-03-03 > >>> 11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> ovirt-hosted-engine-ha agent 1.1.0 started > >>> MainThread::INFO::2014-03-03 > >>> 11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > >>> Found > >>> certificate common name: 10.0.200.101 > >>> MainThread::CRITICAL::2014-03-03 > >>> 11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> Could not start ha-agent > >>> Traceback (most recent call last): > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> line 97, in run > >>> self._run_agent() > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> line 154, in _run_agent > >>> hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > >>> line 152, in __init__ > >>> "STOP_VM": self._stop_engine_vm > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", > >>> line 56, in __init__ > >>> logger, actions) > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>> line 93, in __init__ > >>> self._logger = FSMLoggerAdapter(logger, self) > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>> line 16, in __init__ > >>> super(FSMLoggerAdapter, self).__init__(logger, None) > >>> TypeError: super() argument 1 must be type, not classobj > >>> > >>> > >>> > >>> If I want to start my hosted engine, I receive the following error in > >>> vdsm > >>> logs, which makes absolutly no sense to me, as there is plenty of disk > >>> space available: > >>> > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown > >>> libvirterror: ecode: 38 edom: 42 level: 2 message: Failed > >>> to acquire lock: No space left on device > >> > >> seems like a vdsm failure in starting monitor the hosted engine storage > >> domain. > >> Can you attach vdsm logs? > >> > >> > >> > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released > >>> Thread-62::ERROR::2014-03-03 > >>> 11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed > >>> Traceback (most recent call last): > >>> File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm > >>> self._run() > >>> File "/usr/share/vdsm/vm.py", line 3159, in _run > >>> self._connection.createXML(domxml, flags), > >>> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", > >>> line > >>> 92, in wrapper > >>> ret = f(*args, **kwargs) > >>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in > >>> createXML > >>> if ret is None:raise libvirtError('virDomainCreateXML() failed', > >>> conn=self) > >>> libvirtError: Failed to acquire lock: No space left on device > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,286::vm::2720::vm.Vm::(setDownStatus) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: > >>> Failed > >>> to acquire lock: No space left on device > >>> > >>> # df -h | grep engine > >>> ovirt-host01:/engine 281G 21G 261G 8% > >>> /rhev/data-center/mnt/ovirt-host01:_engine > >>> > >>> # sudo -u vdsm dd if=/dev/zero > >>> of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test > >>> bs=512 count=100 > >>> 100+0 records in > >>> 100+0 records out > >>> 51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s > >>> > >>> > >>> Could you give me some information on how to fix the ovirt-ha-agent and > >>> then hosted-engine storage issue? Thanks a lot. > >>> > >>> Btw, I had some issues during installation which I will explain in > >>> separate > >>> emails. > >>> > >>> > >> > >> > >> -- > >> Sandro Bonazzola > >> Better technology. Faster innovation. Powered by community collaboration. > >> See how it works at redhat.com > >> >