[ovirt-devel] VDSM crashed while adding host to newly created cluster

[Christopher Pereira]

Continuing with the 3.6 Night Builds testing...

While hosted-engine-setup was adding the host to the newly created 
cluster, VDSM crashed, probably because the gluster engine storage 
disappeared as in BZ 1201355 [1]

Facts:
     - the engine storage (/rhev/data-center/mmt/...) was umounted 
during this process
     - another mount of the same volume was still mounted after the VDSM 
crash (maybe the problem is not related with gluster)

After doing a "hosted-engine --connect-storage", the volume is mounted 
again.
Now, when trying to restart VDSM, I get an "invalid lockspace":

     Thread-46::ERROR::2015-03-26 
19:24:31,843::vm::1237::vm.Vm::(_startUnderlyingVm) 
vmId=`191045ac-79e4-4ce8-aad7-52cc9af313c5`::The vm start process failed
     Traceback (most recent call last):
       File "/usr/share/vdsm/virt/vm.py", line 1185, in _startUnderlyingVm
         self._run()
       File "/usr/share/vdsm/virt/vm.py", line 2253, in _run
         self._connection.createXML(domxml, flags),
       File 
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 126, 
in wrapper
         ret = f(*args, **kwargs)
       File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3427, 
in createXML
         if ret is None:raise libvirtError('virDomainCreateXML() 
failed', conn=self)
     libvirtError: Failed to acquire lock: No space left on device
     Thread-46::INFO::2015-03-26 
19:24:31,844::vm::1709::vm.Vm::(setDownStatus) 
vmId=`191045ac-79e4-4ce8-aad7-52cc9af313c5`::Changed state to Down: 
Failed to acquire lock: No space left on device (code=1)
     Thread-46::DEBUG::2015-03-26 
19:24:31,844::vmchannels::214::vds::(unregister) Delete fileno 60 from 
listener.
     VM Channels Listener::DEBUG::2015-03-26 
19:24:32,346::vmchannels::121::vds::(_do_del_channels) fileno 60 was 
removed from listener.

In sanlock.log we have:

     2015-03-26 19:24:30+0000 7589 [752]: cmd 9 target pid 9559 not found
     2015-03-26 19:24:31+0000 7589 [764]: r7 cmd_acquire 2,8,9559 
invalid lockspace found -1 failed 935819904 name 
7ba46e75-51af-4648-becc-5a469cb8e9c2

(All 3 lease files are present)

This problem is similar to BZ 1201355 reported by Sandro [1].

About the hosted-engine VM not being resumed after restarting VDSM, 
please check [2] and [3] (duplicated).
I confirmed that QEMU is not reopening the file descriptors when 
resuming a paused VMs, which explains those issues.

Now, how can I fix the "invalid lockspace"?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1201355
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1172905
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1058300