On 29-04-2015 11:56, Liron Aravot wrote:
Can you please attach the engine/vdsm logs from the time the issue occurred? thanks.
Hi, relevant VDSM logs were published here: https://bugzilla.redhat.com/show_bug.cgi?id=1183977
And engine says "Failed in 'ReconstructMasterVDS' method" because of "Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error'))":
If reconstruct master fails, it is expected that the old master will be restored. Your probably have storage issues, revealed by sanlock - it reads and write to all storage domains every 10 seconds, so flaky storage will cause failure to acquire a host id. Please attach to the bug these logs: Hypervisor: /var/log/sanlock.log /var/log/messages /var/log/glusterfs/<glusterhost>:_<volumename>.log Gluster server: server logs showing what happened when sanlock failed access the gluster volume.
How do I specify the dataDict in the vdsClient command?
I guess it should be something like:
vdsClient -s 0 reconstructMaster '00000001-0001-0001-0001-00000000007e' 'Default' 'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'
Looking in cleint/vdsClient.py:920 - it should be: ad08adbc-b3a7-4198-a1cb-4ad70c724454=status,15674257-e14a-4693-9b95-cc4bb92aa765=status,... Where status is the string vdsm expects. You can check what vdsm expects or what engine is sending. Nir