<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
After some indirect path to getting here, I believe that the cause
of my problem is with sanlock. (See thread on 'Data Center Status
on the GUI flips between "Non Responsive" and
<br>
"Contending" '
for how I got here).<br>
<br>
My sanlock.log file is full of:<br>
<br>
<pre>2014-01-26 21:36:19-0500 36474 [2905]: s4576 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-26 21:36:19-0500 36474 [26767]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre>
<pre>2014-01-26 21:36:19-0500 36474 [26767]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-26 21:36:20-0500 36475 [2905]: s4576 add_lockspace fail result -90</pre>
<pre>2014-01-26 21:36:26-0500 36481 [2906]: s4577 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-26 21:36:26-0500 36481 [26809]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre>
<pre>2014-01-26 21:36:26-0500 36481 [26809]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-26 21:36:27-0500 36482 [2906]: s4577 add_lockspace fail result -90</pre>
<br>
The only other thing is after rebooting, when it says:<br>
<br>
<pre>2014-01-26 11:28:53-0500 29 [2897]: sanlock daemon started 2.8 host 09423427-2f70-4f68-9696-e41a93937ef0.office2a.l</pre>
<pre>2014-01-26 11:29:22-0500 57 [2905]: s1 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-26 11:29:22-0500 57 [3569]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c001000 result 0:0 match len 512</pre>
<pre>2014-01-26 11:29:22-0500 57 [3569]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-26 11:29:23-0500 58 [2905]: s1 add_lockspace fail result -90</pre>
<pre>2014-01-26 11:29:32-0500 67 [2905]: s2 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-26 11:29:32-0500 67 [3731]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre>
<pre>2014-01-26 11:29:32-0500 67 [3731]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-26 11:29:33-0500 68 [2905]: s2 add_lockspace fail result -90</pre>
<pre>2014-01-26 11:29:42-0500 77 [2905]: s3 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:1:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-26 11:29:42-0500 77 [3806]: 0322a407 aio collect 0 0x7f196c0008c0:0x7f196c0008d0:0x7f196c101000 result 0:0 match len 512</pre>
<pre>2014-01-26 11:29:42-0500 77 [3806]: read_sectors delta_leader offset 0 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-26 11:29:43-0500 78 [2905]: s3 add_lockspace fail result -90</pre>
<br>
Background:<br>
Setup is 2 hosts on Centos 6.5 with gluster storage, engine on
separate Centos 6.5, all up to date.<br>
Originally set up running:<br>
<blockquote>Data Center: default<br>
Cluster: default<br>
Master storage domain: VM<br>
</blockquote>
log had entries like:<br>
<blockquote>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res</pre>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
<br>
</blockquote>
storage 'VM' had a Split-Brain. In trying to fix split-brain, file
dom_md/ids was erased. I never revived the Data Center 'default'
after that.<br>
<blockquote>log is full of: <br>
<pre>2014-01-16 17:23:21-0500 62902 [29262]: open error -13 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2014-01-16 17:23:21-0500 62902 [29262]: s3 open_disk /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids error -13</pre>
<pre>2014-01-16 17:23:22-0500 62903 [3368]: s3 add_lockspace fail result -19</pre>
<pre>2014-01-16 17:23:31-0500 62912 [3368]: s4 lockspace 9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88:1:/rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids:0</pre>
<br>
</blockquote>
Jan. 16: Created new: Data Center: mill; Cluster: one; Master
storage domain: 'VM2' All was working.<br>
<blockquote>Logs looked like this:<br>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -103:0 match res</pre>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 delta_renew read rv -103 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:39-0500 4510 [3318]: s1 renewal error -103 delta_length 9 last_success 4481</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:40-0500 4510 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:40-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: 9b6e6bf1 aio collect 0 0x7fd3b40008c0:0x7fd3b40008d0:0x7fd3c59ae000 result -107:0 match res</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 delta_renew read rv -107 offset 0 /rhev/data-center/mnt/10.41.65.4:_VM/9b6e6bf1-29a1-4abf-8e85-bf9c4a329c88/dom_md/ids</pre>
<pre>2013-12-18 23:20:41-0500 4511 [3318]: s1 renewal error -107 delta_length 0 last_success 4481</pre>
</blockquote>
For a while, very little was logged at all, and this were working
fine. Then, <br>
<br>
Jan 22 logs went from:<br>
<br>
<blockquote>
<pre>2014-01-22 00:44:56-0500 521397 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0</pre>
<pre>2014-01-22 00:44:56-0500 521397 [30933]: leader4 sn rn ts 0 cs 0</pre>
<pre>2014-01-22 00:44:56-0500 521397 [30933]: s123 delta_renew verify_leader error -223</pre>
<pre>2014-01-22 00:44:56-0500 521397 [30933]: s123 renewal error -223 delta_length 0 last_success 521369</pre>
<pre>2014-01-22 00:44:57-0500 521398 [3362]: s123 kill 32599 sig 15 count 9</pre>
<pre>2014-01-22 00:44:57-0500 521398 [30933]: verify_leader 2 wrong magic 0 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-22 00:44:57-0500 521398 [30933]: leader1 delta_renew error -223 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c host_id 2</pre>
<pre>2014-01-22 00:44:57-0500 521398 [30933]: leader2 path /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids offset 0</pre>
<pre>2014-01-22 00:44:57-0500 521398 [30933]: leader3 m 0 v 0 ss 0 nh 0 mh 0 oi 0 og 0 lv 0</pre>
</blockquote>
<br>
to:<br>
<br>
<blockquote>
<pre>2014-01-22 00:45:21-0500 521422 [17199]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512</pre>
<pre>2014-01-22 00:45:21-0500 521422 [17199]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-22 00:45:22-0500 521423 [3368]: s125 add_lockspace fail result -90</pre>
<pre>2014-01-22 00:45:25-0500 521426 [3368]: s126 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
<pre>2014-01-22 00:45:25-0500 521426 [17243]: 0322a407 aio collect 0 0x7f87ac0008c0:0x7f87ac0008d0:0x7f87ac101000 result 0:0 match len 512</pre>
<pre>2014-01-22 00:45:25-0500 521426 [17243]: read_sectors delta_leader offset 512 rv -90 /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids</pre>
<pre>2014-01-22 00:45:26-0500 521427 [3368]: s126 add_lockspace fail result -90</pre>
<pre>2014-01-22 00:45:36-0500 521437 [3368]: s127 lockspace 0322a407-2b16-40dc-ac67-13d387c6eb4c:2:/rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids:0</pre>
</blockquote>
<br>
and the Data Center has been down ever since.<br>
<br>
Best I can figure is that something is supposed to be in dom_md/ids,
but that file is empty:<br>
<br>
<pre>ls /rhev/data-center/mnt/glusterSD/10.41.65.2\:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ -l</pre>
<pre>total 1029</pre>
<pre>-rw-rw---- 1 vdsm kvm 0 Jan 22 00:44 ids</pre>
<pre>-rw-rw---- 1 vdsm kvm 0 Jan 16 18:50 inbox</pre>
<pre>-rw-rw---- 1 vdsm kvm 2097152 Jan 21 18:20 leases</pre>
<pre>-rw-r--r-- 1 vdsm kvm 491 Jan 21 18:20 metadata</pre>
<pre>-rw-rw---- 1 vdsm kvm 0 Jan 16 18:50 outbox
</pre>
Any hints as to how to put whatever is needed into 'ids', or
reinitialize the sanlock system--or a better diagnosis and
solution--gladly accepted.<br>
<br>
Ted Miller<br>
Elkhart, IN, USA<br>
<br>
</body>
</html>