<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 09/29/2016 05:18 PM, Sahina Bose
wrote:<br>
</div>
<blockquote
cite="mid:CACjzOvfCdjG0H+pi7hf_jVNxhJfv-Qi2r3u-yTqnYGPt1W0O2w@mail.gmail.com"
type="cite">
<div dir="ltr">Yes, this is a GlusterFS problem. Adding gluster
users ML<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Sep 29, 2016 at 5:11 PM, Davide
Ferrari <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:davide@billymob.com" target="_blank">davide@billymob.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hello<br>
<br>
</div>
maybe this is more glustefs then ovirt related
but since OVirt integrates Gluster management
and I'm experiencing the problem in an ovirt
cluster, I'm writing here.<br>
<br>
</div>
The problem is simple: I have a data domain
mappend on a replica 3 arbiter1 Gluster volume
with 6 bricks, like this:<br>
<span style="font-family:monospace,monospace"><br>
Status of volume: data_ssd<br>
Gluster process <wbr>
TCP Port RDMA Port Online Pid<br>
------------------------------<wbr>------------------------------<wbr>------------------<br>
Brick vm01.storage.billy:/gluster/<wbr>ssd/data/<br>
brick <wbr>
49153 0 Y 19298<br>
Brick vm02.storage.billy:/gluster/<wbr>ssd/data/<br>
brick <wbr>
49153 0 Y 6146 <br>
Brick vm03.storage.billy:/gluster/<wbr>ssd/data/<br>
arbiter_brick <wbr>
49153 0 Y 6552 <br>
Brick vm03.storage.billy:/gluster/<wbr>ssd/data/<br>
brick <wbr>
49154 0 Y 6559 <br>
Brick vm04.storage.billy:/gluster/<wbr>ssd/data/<br>
brick <wbr>
49152 0 Y 6077 <br>
Brick vm02.storage.billy:/gluster/<wbr>ssd/data/<br>
arbiter_brick <wbr>
49154 0 Y 6153 <br>
Self-heal Daemon on localhost
N/A N/A Y 30746<br>
Self-heal Daemon on vm01.storage.billy
N/A N/A Y 196058<br>
Self-heal Daemon on vm03.storage.billy
N/A N/A Y 23205<br>
Self-heal Daemon on vm04.storage.billy
N/A N/A Y 8246 </span><br>
<br>
<br>
</div>
Now, I've put in maintenance the vm04 host, from
ovirt, ticking the "Stop gluster" checkbox, and
Ovirt didn't complain about anything. But when I
tried to run a new VM it complained about "storage
I/O problem", while the storage data status was
always UP.<br>
<br>
</div>
Looking in the gluster logs I can see this:<br>
<br>
<span style="font-family:monospace,monospace">[2016-09-29
11:01:01.556908] I [glusterfsd-mgmt.c:1596:mgmt_<wbr>getspec_cbk]
0-glusterfs: No change in volfile, continuing<br>
[2016-09-29 11:02:28.124151] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_<wbr>txn_refresh_done]
0-data_ssd-replicate-1: Failing READ on gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d:
split-brain observed. [Input/output error]<br>
[2016-09-29 11:02:28.126580] W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_<wbr>txn]
0-data_ssd-replicate-1: Unreadable subvolume -1
found with event generation 6 for gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d. (Possible
split-brain)<br>
[2016-09-29 11:02:28.127374] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_<wbr>txn_refresh_done]
0-data_ssd-replicate-1: Failing FGETXATTR on gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d:
split-brain observed. [Input/output error]<br>
[2016-09-29 11:02:28.128130] W [MSGID: 108027]
[afr-common.c:2403:afr_<wbr>discover_done]
0-data_ssd-replicate-1: no read subvols for (null)<br>
[2016-09-29 11:02:28.129890] W
[fuse-bridge.c:2228:fuse_<wbr>readv_cbk]
0-glusterfs-fuse: 8201: READ => -1
gfid=bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d
fd=0x7f09b749d210 (Input/output error)<br>
[2016-09-29 11:02:28.130824] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_<wbr>txn_refresh_done]
0-data_ssd-replicate-1: Failing FSTAT on gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d:
split-brain observed. [Input/output error]<br>
</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
Does `gluster volume heal data_ssd info split-brain` report that the
file is in split-brain, with vm04 still being down? <br>
If yes, could you provide the extended attributes of this gfid from
all 3 bricks:<br>
getfattr -d -m . -e hex /path/to/brick/bf/59/<span
style="font-family:monospace,monospace">bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d</span><br>
<br>
If no, then I'm guessing that it is not in actual split-brain (hence
the 'Possible split-brain' message). If the node you brought down
contains the only good copy of the file (i.e the other data brick
and arbiter are up, and the arbiter 'blames' this other brick), all
I/O is failed with EIO to prevent file from getting into actual
split-brain. The heals will happen when the good node comes up and
I/O should be allowed again in that case.<br>
<br>
-Ravi<br>
<br>
<br>
<blockquote
cite="mid:CACjzOvfCdjG0H+pi7hf_jVNxhJfv-Qi2r3u-yTqnYGPt1W0O2w@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div><span style="font-family:monospace,monospace">[2016-09-29
11:02:28.133879] W [fuse-bridge.c:767:fuse_attr_<wbr>cbk]
0-glusterfs-fuse: 8202: FSTAT()
/ba2bd397-9222-424d-aecc-<wbr>eb652c0169d9/images/f02ac1ce-<wbr>52cd-4b81-8b29-f8006d0469e0/<wbr>ff4e49c6-3084-4234-80a1-<wbr>18a67615c527
=> -1 (Input/output error)<br>
The message "W [MSGID: 108008]
[afr-read-txn.c:244:afr_read_<wbr>txn]
0-data_ssd-replicate-1: Unreadable subvolume -1
found with event generation 6 for gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d. (Possible
split-brain)" repeated 11 times between [2016-09-29
11:02:28.126580] and [2016-09-29 11:02:28.517744]<br>
[2016-09-29 11:02:28.518607] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_<wbr>txn_refresh_done]
0-data_ssd-replicate-1: Failing STAT on gfid
bf5922b7-19f3-4ce3-98df-<wbr>71e981ecca8d:
split-brain observed. [Input/output error]<br>
</span><br>
</div>
Now, how is it possible to have a split brain if I
stopped just ONE server which had just ONE of six
bricks, and it was cleanly shut down with maintenance
mode from ovirt?<br>
<br>
</div>
I created the volume originally this way:<br>
<span style="font-family:monospace,monospace"># gluster
volume create data_ssd replica 3 arbiter 1
vm01.storage.billy:/gluster/<wbr>ssd/data/brick
vm02.storage.billy:/gluster/<wbr>ssd/data/brick
vm03.storage.billy:/gluster/<wbr>ssd/data/arbiter_brick
vm03.storage.billy:/gluster/<wbr>ssd/data/brick
vm04.storage.billy:/gluster/<wbr>ssd/data/brick
vm02.storage.billy:/gluster/<wbr>ssd/data/arbiter_brick<br>
# gluster volume set data_ssd group virt<br>
# gluster volume set data_ssd storage.owner-uid 36
&& gluster volume set data_ssd storage.owner-gid
36<br>
# gluster volume start data_ssd</span><span
class="HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<br>
<br>
<br>
<blockquote
cite="mid:CACjzOvfCdjG0H+pi7hf_jVNxhJfv-Qi2r3u-yTqnYGPt1W0O2w@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><span class="HOEnZb"><font color="#888888"><br
clear="all">
<div>
<div>
<div>
<div>
<div>
<div>
<div><br>
-- <br>
<div>
<div dir="ltr">
<div>Davide Ferrari<br>
</div>
Senior Systems Engineer<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>