1:24 p.m.
This is a multi-part message in MIME format.
--------------050502080707090302020409
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Hi,
I am not able to resume a VM which was paused because of gluster
client quorum issue. Here is what happened in my setup.
1. Created a gluster storage domain which is backed by gluster volume
with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3
setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs
moved to paused state.
" server 10.70.45.17:49217 has not responded in the last 42
seconds, disconnecting."
"vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE as quorum is not met"
more gluster mount logs at http://pastebin.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the
the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error
in vdsm log.
http://pastebin.com/aXiamY15
Regards,
Ramesh
--------------050502080707090302020409
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
I am not able to resume a VM which was paused because of gluster
client quorum issue. Here is what happened in my setup. <br>
<br>
1. Created a gluster storage domain which is backed by gluster
volume with replica 3. <br>
2. Killed one brick process. So only two bricks are running in
replica 3 setup.<br>
3. Created two VMs<br>
4. Started some IO using fio on both of the VMs<br>
5. After some time got the following error in gluster mount and VMs
moved to paused state.<br>
"
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<span style="color: rgb(51, 51, 51); font-family: monospace;
font-size: 11px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height: 13.2px;
orphans: auto; text-align: left; text-indent: 0px; text-transform:
none; white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important; float:
none; background-color: rgb(255, 255, 255);">server
10.70.45.17:49217 has not responded in the last 42 seconds,
disconnecting."<br>
"</span><span style="color: rgb(51, 51, 51); font-family:
monospace; font-size: 11px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal; line-height:
13.2px; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline
!important; float: none; background-color: rgb(255, 255, 255);"><span
style="color: rgb(51, 51, 51); font-family: monospace;
font-size: 11px; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
13.2px; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px; display:
inline !important; float: none; background-color: rgb(255, 255,
255);">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is
not met</span>"<br>
more gluster mount logs at <a class="moz-txt-link-freetext"
href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiU...
</span>6. After some time gluster quorum is active and I am able to
write the the gluster file system.<br>
7. When I try to resume the VM it doesn't work and I got following
error in vdsm log.<br>
<a class="moz-txt-link-freetext"
href="http://pastebin.com/aXiamY15">http://pastebin.com/aXia...
<br>
<br>
Regards,<br>
Ramesh<br>
<br>
</body>
</html>
--------------050502080707090302020409--
3:13 p.m.
what are the gluster-quorum-type and gluster.server-quorum-ratio settings
on the volume?
On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
Hi,
I am not able to resume a VM which was paused because of gluster client
quorum issue. Here is what happened in my setup.
1. Created a gluster storage domain which is backed by gluster volume with
replica 3.
2. Killed one brick process. So only two bricks are running in replica 3
setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved
to paused state.
" server 10.70.45.17:49217 has not responded in the last 42
seconds, disconnecting."
"vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing
WRITE as quorum is not met"
more gluster mount logs at http://pastebin.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the the
gluster file system.
7. When I try to resume the VM it doesn't work and I got following error
in vdsm log.
http://pastebin.com/aXiamY15
Regards,
Ramesh
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:25 p.m.
This is a multi-part message in MIME format.
--------------000009040208060202040809
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the gluster-quorum-type and gluster.server-quorum-ratio
settings on the volume?
*cluster.server-quorum-type*:server
*cluster.quorum-type*:auto
*gluster.server-quorum-ratio is not set.*
One brick process is purposefully killed but remaining two bricks are
up and running.
Regards,
Ramesh
On 22 September 2015 at 06:24, Ramesh Nachimuthu
<rnachimu(a)redhat.com
<mailto:rnachimu@redhat.com>> wrote:
Hi,
I am not able to resume a VM which was paused because of
gluster client quorum issue. Here is what happened in my setup.
1. Created a gluster storage domain which is backed by gluster
volume with replica 3.
2. Killed one brick process. So only two bricks are running in
replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and
VMs moved to paused state.
" server 10.70.45.17:49217 <http://10.70.45.17:49217> has
not responded in the last 42 seconds, disconnecting."
"vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE as quorum is not met"
more gluster mount logs at http://pastebin.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write
the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following
error in vdsm log.
http://pastebin.com/aXiamY15
Regards,
Ramesh
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--------------000009040208060202040809
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 09/22/2015 05:43 PM, Alastair Neil
wrote:<br>
</div>
<blockquote
cite="mid:CA+SarwpUsiNz5MsUz66LYmEua-G=P5v_mbzOMkiErJZWwdE2MQ@mail.gmail.com"
type="cite">
<div dir="ltr">what are the gluster-quorum-type
and gluster.server-quorum-ratio settings on the volume?</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
<br>
<div style="outline-style:none;"
__gwt_cell="cell-gwt-uid-89049">
<div class="" style="overflow: hidden; text-overflow: ellipsis;
white-space: nowrap;"
id="SubTabVolumeParameterView_table_content_col1_row9"><b>cluster.server-quorum-type</b>:server<br>
<div title="" tabindex="0" style="outline-style:
none;"
__gwt_cell="cell-gwt-uid-89187">
<div class="" style="overflow: hidden; text-overflow:
ellipsis; white-space: nowrap;"
id="SubTabVolumeParameterView_table_content_col1_row13"><b>cluster.quorum-type</b>:auto<br>
<b>gluster.server-quorum-ratio is not set.</b><br>
<br>
</div>
</div>
One brick process is purposefully killed but remaining two
bricks are up and running.<br>
<br>
Regards,<br>
Ramesh<br>
</div>
</div>
<br>
<blockquote
cite="mid:CA+SarwpUsiNz5MsUz66LYmEua-G=P5v_mbzOMkiErJZWwdE2MQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 22 September 2015 at 06:24, Ramesh
Nachimuthu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank">rnachimu(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
I am not able to resume a VM which was paused because
of gluster client quorum issue. Here is what happened in
my setup. <br>
<br>
1. Created a gluster storage domain which is backed by
gluster volume with replica 3. <br>
2. Killed one brick process. So only two bricks are
running in replica 3 setup.<br>
3. Created two VMs<br>
4. Started some IO using fio on both of the VMs<br>
5. After some time got the following error in gluster
mount and VMs moved to paused state.<br>
" <span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">server
<a moz-do-not-send="true"
href="http://10.70.45.17:49217"
target="_blank">10.70.45.17:49217</a>
has not responded in the last 42 seconds,
disconnecting."<br>
"</span><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as
quorum is not met</span>"<br>
more gluster mount logs at <a
moz-do-not-send="true"
href="http://pastebin.com/UmiUQq0F"
target="_blank"><a class="moz-txt-link-freetext"
href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiU...
</span>6. After some time gluster quorum is active and I
am able to write the the gluster file system.<br>
7. When I try to resume the VM it doesn't work and I got
following error in vdsm log.<br>
<a moz-do-not-send="true"
href="http://pastebin.com/aXiamY15"
target="_blank">http://pastebin.com/aXiamY15</a><br...
<br>
<br>
Regards,<br>
Ramesh<br>
<br>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>
--------------000009040208060202040809--
3:27 p.m.
You need to set the gluster.server-quorum-ratio to 51%
On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the gluster-quorum-type and gluster.server-quorum-ratio settings
on the volume?
*cluster.server-quorum-type*:server
*cluster.quorum-type*:auto
*gluster.server-quorum-ratio is not set.*
One brick process is purposefully killed but remaining two bricks are up
and running.
Regards,
Ramesh
On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
> Hi,
>
> I am not able to resume a VM which was paused because of gluster
> client quorum issue. Here is what happened in my setup.
>
> 1. Created a gluster storage domain which is backed by gluster volume
> with replica 3.
> 2. Killed one brick process. So only two bricks are running in replica 3
> setup.
> 3. Created two VMs
> 4. Started some IO using fio on both of the VMs
> 5. After some time got the following error in gluster mount and VMs moved
> to paused state.
> " server 10.70.45.17:49217 has not responded in the last 42
> seconds, disconnecting."
> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
> Failing WRITE as quorum is not met"
> more gluster mount logs at <http://pastebin.com/UmiUQq0F>
> http://pastebin.com/UmiUQq0F
> 6. After some time gluster quorum is active and I am able to write the
> the gluster file system.
> 7. When I try to resume the VM it doesn't work and I got following error
> in vdsm log.
> http://pastebin.com/aXiamY15
>
>
> Regards,
> Ramesh
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
3:38 p.m.
This is a multi-part message in MIME format.
--------------040609060800000303090805
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%
I did that. But still I am facing the same issue. VM get paused when I
do some I/O using fio on some disks backed by gluster. I am not able to
resume the VM after this. Now only way is to bring down the VM and run
again. It runs successfully on the same host without any issue.
Regards,
Ramesh
On 22 September 2015 at 08:25, Ramesh Nachimuthu
<rnachimu(a)redhat.com
<mailto:rnachimu@redhat.com>> wrote:
On 09/22/2015 05:43 PM, Alastair Neil wrote:
> what are the gluster-quorum-type and gluster.server-quorum-ratio
> settings on the volume?
>
*cluster.server-quorum-type*:server
*cluster.quorum-type*:auto
*gluster.server-quorum-ratio is not set.*
One brick process is purposefully killed but remaining two bricks
are up and running.
Regards,
Ramesh
> On 22 September 2015 at 06:24, Ramesh Nachimuthu
> <rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>> wrote:
>
> Hi,
>
> I am not able to resume a VM which was paused because of
> gluster client quorum issue. Here is what happened in my setup.
>
> 1. Created a gluster storage domain which is backed by
> gluster volume with replica 3.
> 2. Killed one brick process. So only two bricks are running
> in replica 3 setup.
> 3. Created two VMs
> 4. Started some IO using fio on both of the VMs
> 5. After some time got the following error in gluster mount
> and VMs moved to paused state.
> " server 10.70.45.17:49217
> <http://10.70.45.17:49217> has not responded in the last 42
> seconds, disconnecting."
> "vmstore-replicate-0:
> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum
> is not met"
> more gluster mount logs at http://pastebin.com/UmiUQq0F
> 6. After some time gluster quorum is active and I am able to
> write the the gluster file system.
> 7. When I try to resume the VM it doesn't work and I got
> following error in vdsm log.
> http://pastebin.com/aXiamY15
>
>
> Regards,
> Ramesh
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
--------------040609060800000303090805
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 09/22/2015 05:57 PM, Alastair Neil
wrote:<br>
</div>
<blockquote
cite="mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail.com"
type="cite">
<div dir="ltr">You need to set the gluster.server-quorum-ratio to
51%</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
<br>
I did that. But still I am facing the same issue. VM get paused when
I do some I/O using fio on some disks backed by gluster. I am not
able to resume the VM after this. Now only way is to bring down the
VM and run again. It runs successfully on the same host without any
issue.<br>
<br>
Regards,<br>
Ramesh<br>
<br>
<blockquote
cite="mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 22 September 2015 at 08:25, Ramesh
Nachimuthu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank">rnachimu(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span
class=""> <br>
<br>
<div>On 09/22/2015 05:43 PM, Alastair Neil wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">what are the gluster-quorum-type
and gluster.server-quorum-ratio settings on the
volume?</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
<br>
</span>
<div style="outline-style:none">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"><b>cluster.server-quorum-type</b>:server<br>
<div title="" style="outline-style:none">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"><b>cluster.quorum-type</b>:auto<br>
<b>gluster.server-quorum-ratio is not
set.</b><br>
<br>
</div>
</div>
One brick process is purposefully killed but
remaining two bricks are up and running.<br>
<br>
Regards,<br>
Ramesh<br>
</div>
</div>
<span class=""> <br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 22 September 2015 at
06:24, Ramesh Nachimuthu <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
I am not able to resume a VM which was
paused because of gluster client quorum issue.
Here is what happened in my setup. <br>
<br>
1. Created a gluster storage domain which is
backed by gluster volume with replica 3. <br>
2. Killed one brick process. So only two
bricks are running in replica 3 setup.<br>
3. Created two VMs<br>
4. Started some IO using fio on both of the
VMs<br>
5. After some time got the following error in
gluster mount and VMs moved to paused state.<br>
" <span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">server
<a moz-do-not-send="true"
href="http://10.70.45.17:49217"
target="_blank">10.70.45.17:49217</a>
has
not responded in the last 42 seconds,
disconnecting."<br>
"</span><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE as quorum is not
met</span>"<br>
more gluster mount logs at <a
moz-do-not-send="true"
href="http://pastebin.com/UmiUQq0F"
target="_blank"><a
class="moz-txt-link-freetext"
href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiU...
</span>6. After some time gluster quorum is
active and I am able to write the the gluster
file system.<br>
7. When I try to resume the VM it doesn't work
and I got following error in vdsm log.<br>
<a moz-do-not-send="true"
href="http://pastebin.com/aXiamY15"
target="_blank">http://pastebin.com/aXiamY15</a><br...
<br>
<br>
Regards,<br>
Ramesh<br>
<br>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>
--------------040609060800000303090805--
12:08 a.m.
--Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved =
in the upcoming ovirt 3.6.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the =
fix involved setting up a special cgroup for the mount, but i can=E2=80=99=
t find the exact details atm.
On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu
<rnachimu(a)redhat.com> =
wrote:
=20
=20
=20
On 09/22/2015 05:57 PM, Alastair Neil wrote:
> You need to set the gluster.server-quorum-ratio to 51%
>=20
=20
I did that. But still I am facing the same issue. VM get paused when I =
do some
I/O using fio on some disks backed by gluster. I am not able to =
resume the VM after this. Now only way is to bring down the VM and run =
again. It runs successfully on the same host without any issue.
=20
Regards,
Ramesh
=20
> On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)redhat.com =
<mailto:rnachimu@redhat.com>> wrote:
>=20
>=20
> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>> what are the gluster-quorum-type and gluster.server-quorum-ratio =
settings on the volume?
>>=20
>=20
> cluster.server-quorum-type:server
> cluster.quorum-type:auto
> gluster.server-quorum-ratio is not set.
>=20
> One brick process is purposefully killed but remaining two bricks =
are up and
running.
>=20
> Regards,
> Ramesh
>=20
>> On 22 September 2015 at 06:24, Ramesh Nachimuthu < =
<mailto:rnachimu@redhat.com>rnachimu@redhat.com =
<mailto:rnachimu@redhat.com>> wrote:
>> Hi,
>>=20
>> I am not able to resume a VM which was paused because of gluster =
client quorum issue. Here is what happened in my setup.=20
>>=20
>> 1. Created a gluster storage domain which is backed by gluster =
volume
with replica 3.=20
>> 2. Killed one brick process. So only two bricks are running
in =
replica 3 setup.
>> 3. Created two VMs
>> 4. Started some IO using fio on both of the VMs
>> 5. After some time got the following error in gluster mount and VMs =
moved
to paused state.
>> " server 10.70.45.17:49217
<http://10.70.45.17:49217/> has =
not responded in the last 42 seconds,
disconnecting."
>> "vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6: =
Failing WRITE as quorum is not met"
>> more gluster mount logs at =
<http://pastebin.com/UmiUQq0F>http://pastebin.com/UmiUQq0F =
<http://pastebin.com/UmiUQq0F>
>> 6. After some time gluster quorum is active and I am able to
write =
the the gluster file system.
>> 7. When I try to resume the VM it doesn't work and I got
following =
error in vdsm log.
>> http://pastebin.com/aXiamY15
<http://pastebin.com/aXiamY15>
>>=20
>>=20
>> Regards,
>> Ramesh
>>=20
>>=20
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>>=20
>>=20
>=20
>=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">This is a known issue in overt 3.5.x and below. It=E2=80=99s =
been solved in the upcoming ovirt 3.6.<div class=3D""><br =
class=3D""></div><div class=3D"">Related
to <a =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905" =
class=3D"">https://bugzilla.redhat.com/show_bug.cgi?id=3D117...;,
=
the fix involved setting up a special cgroup for the mount, but i =
can=E2=80=99t find the exact details atm.</div><div
class=3D""><br =
class=3D""></div><div class=3D""><br
class=3D""><div><blockquote =
type=3D"cite" class=3D""><div class=3D"">On Sep 23,
2015, at 7:38 AM, =
Ramesh Nachimuthu <<a href=3D"mailto:rnachimu@redhat.com" =
class=3D"">rnachimu(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D"">
=20
<meta content=3D"text/html; charset=3Dutf-8" =
http-equiv=3D"Content-Type" class=3D"">
=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 09/22/2015 05:57 PM, Alastair Neil
wrote:<br class=3D"">
</div>
<blockquote =
cite=3D"mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail=
.com" type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">You need to set the =
gluster.server-quorum-ratio to
51%</div>
<div class=3D"gmail_extra"><br class=3D"">
</div>
</blockquote>
<br class=3D"">
I did that. But still I am facing the same issue. VM get paused when
I do some I/O using fio on some disks backed by gluster. I am not
able to resume the VM after this. Now only way is to bring down the
VM and run again. It runs successfully on the same host without any
issue.<br class=3D"">
<br class=3D"">
Regards,<br class=3D"">
Ramesh<br class=3D"">
<br class=3D"">
<blockquote =
cite=3D"mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail=
.com" type=3D"cite" class=3D"">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">On 22 September 2015 at 08:25, Ramesh
Nachimuthu <span dir=3D"ltr" class=3D""><<a
=
moz-do-not-send=3D"true" href=3D"mailto:rnachimu@redhat.com" =
target=3D"_blank"
class=3D"">rnachimu(a)redhat.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D""><span =
class=3D""> <br class=3D"">
<br class=3D"">
<div class=3D"">On 09/22/2015 05:43 PM, Alastair Neil =
wrote:<br class=3D"">
</div>
<blockquote type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">what are =
the gluster-quorum-type
and gluster.server-quorum-ratio settings =
on the
volume?</div>
<div class=3D"gmail_extra"><br
class=3D"">
</div>
</blockquote>
<br class=3D"">
</span>
<div style=3D"outline-style:none" class=3D"">
<div =
style=3D"overflow:hidden;text-overflow:ellipsis;white-space:nowrap" =
class=3D""><b
class=3D"">cluster.server-quorum-type</b>:server<br =
class=3D"">
<div title=3D"" style=3D"outline-style:none"
class=3D"">=
<div =
style=3D"overflow:hidden;text-overflow:ellipsis;white-space:nowrap" =
class=3D""><b
class=3D"">cluster.quorum-type</b>:auto<br class=3D"">
<b class=3D"">gluster.server-quorum-ratio is not =
set.</b><br class=3D"">
<br class=3D"">
</div>
</div>
One brick process is purposefully killed but
remaining two bricks are up and running.<br class=3D"">
<br class=3D"">
Regards,<br class=3D"">
Ramesh<br class=3D"">
</div>
</div>
<span class=3D""> <br class=3D"">
<blockquote type=3D"cite" class=3D"">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">On 22 September 2015 at
06:24, Ramesh Nachimuthu <span dir=3D"ltr" =
class=3D""><<a moz-do-not-send=3D"true" =
href=3D"mailto:rnachimu@redhat.com" target=3D"_blank"
class=3D""></a><a =
class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:rnachimu@redhat.com">rnachimu@redhat.com</a>></span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
=
class=3D""> Hi,<br class=3D"">
<br class=3D"">
I am not able to resume a VM =
which was
paused because of gluster client quorum issue.
Here is what happened in my setup. <br =
class=3D"">
<br class=3D"">
1. Created a gluster storage domain which is
backed by gluster volume with replica 3. <br =
class=3D"">
2. Killed one brick process. So only two
bricks are running in replica 3 setup.<br =
class=3D"">
3. Created two VMs<br class=3D"">
4. Started some IO using fio on both of the
VMs<br class=3D"">
5. After some time got the following error in
gluster mount and VMs moved to paused =
state.<br class=3D"">
=
"
<span =
style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty=
le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin=
e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-=
space:normal;word-spacing:0px;display:inline!important;float:none;backgrou=
nd-color:rgb(255,255,255)" class=3D"">server
<a moz-do-not-send=3D"true" =
href=3D"http://10.70.45.17:49217/" target=3D"_blank" =
class=3D"">10.70.45.17:49217</a> has
not responded in the last 42 seconds,
disconnecting."<br class=3D"">
"</span><span =
style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty=
le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin=
e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-=
space:normal;word-spacing:0px;display:inline!important;float:none;backgrou=
nd-color:rgb(255,255,255)" class=3D""><span =
style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty=
le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin=
e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-=
space:normal;word-spacing:0px;display:inline!important;float:none;backgrou=
nd-color:rgb(255,255,255)" class=3D"">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE as quorum is not =
met</span>"<br class=3D"">
more
gluster =
mount logs at <a moz-do-not-send=3D"true" =
href=3D"http://pastebin.com/UmiUQq0F" target=3D"_blank"
class=3D""></a><a =
class=3D"moz-txt-link-freetext" =
href=3D"http://pastebin.com/UmiUQq0F">http://pastebin.com/Um...
=
class=3D"">
</span>6. After some time gluster quorum is
active and I am able to write the the gluster
file system.<br class=3D"">
7. When I try to resume the VM it doesn't work
and I got following error in vdsm log.<br =
class=3D"">
<a =
moz-do-not-send=3D"true" href=3D"http://pastebin.com/aXiamY15" =
target=3D"_blank"
class=3D"">http://pastebin.com/aXiamY15</a><br =
class=3D"">
<br class=3D"">
<br class=3D"">
Regards,<br class=3D"">
Ramesh<br class=3D"">
<br class=3D"">
</div>
<br class=3D"">
=
_______________________________________________<br class=3D"">
Users mailing list<br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer" =
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... =
class=3D"">
<br class=3D"">
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</span></div>
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50--
7:37 a.m.
This is a multi-part message in MIME format.
--------------050104050809050706040602
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
On 09/24/2015 02:38 AM, Darrell Budic wrote:
This is a known issue in overt 3.5.x and below. It’s been solved in
the upcoming ovirt 3.6.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1172905, the
fix involved setting up a special cgroup for the mount, but i can’t
find the exact details atm.
I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am
not sure above bug 1172905
<https://bugzilla.redhat.com/show_bug.cgi?id=1172905> fixes this correctly.
Regards,
Ramesh
> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com
> <mailto:rnachimu@redhat.com>> wrote:
>
>
>
> On 09/22/2015 05:57 PM, Alastair Neil wrote:
>> You need to set the gluster.server-quorum-ratio to 51%
>>
>
> I did that. But still I am facing the same issue. VM get paused when
> I do some I/O using fio on some disks backed by gluster. I am not
> able to resume the VM after this. Now only way is to bring down the
> VM and run again. It runs successfully on the same host without any
> issue.
>
> Regards,
> Ramesh
>
>> On 22 September 2015 at 08:25, Ramesh Nachimuthu
>> <rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>> wrote:
>>
>>
>>
>> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>>> what are the gluster-quorum-type
>>> and gluster.server-quorum-ratio settings on the volume?
>>>
>>
>> *cluster.server-quorum-type*:server
>> *cluster.quorum-type*:auto
>> *gluster.server-quorum-ratio is not set.*
>>
>> One brick process is purposefully killed but remaining two
>> bricks are up and running.
>>
>> Regards,
>> Ramesh
>>
>>> On 22 September 2015 at 06:24, Ramesh Nachimuthu
>>> <rnachimu(a)redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> I am not able to resume a VM which was paused because of
>>> gluster client quorum issue. Here is what happened in my
>>> setup.
>>>
>>> 1. Created a gluster storage domain which is backed by
>>> gluster volume with replica 3.
>>> 2. Killed one brick process. So only two bricks are running
>>> in replica 3 setup.
>>> 3. Created two VMs
>>> 4. Started some IO using fio on both of the VMs
>>> 5. After some time got the following error in gluster mount
>>> and VMs moved to paused state.
>>> " server 10.70.45.17:49217
>>> <http://10.70.45.17:49217/> has not responded in the last
>>> 42 seconds, disconnecting."
>>> "vmstore-replicate-0:
>>> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as
>>> quorum is not met"
>>> more gluster mount logs at http://pastebin.com/UmiUQq0F
>>> 6. After some time gluster quorum is active and I am able
>>> to write the the gluster file system.
>>> 7. When I try to resume the VM it doesn't work and I got
>>> following error in vdsm log.
>>> http://pastebin.com/aXiamY15
>>>
>>>
>>> Regards,
>>> Ramesh
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--------------050104050809050706040602
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 09/24/2015 02:38 AM, Darrell Budic
wrote:<br>
</div>
<blockquote
cite="mid:F5280320-1598-4721-A5D4-CE3035F1E7C3@onholyground.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
This is a known issue in overt 3.5.x and below. It’s been solved
in the upcoming ovirt 3.6.
<div class=""><br class="">
</div>
<div class="">Related to <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1172905"
class="">https://bugzilla.redhat.com/show_bug.cgi?id=1172905...;,
the fix involved setting up a special cgroup for the mount, but
i can’t find the exact details atm.</div>
<div class=""><br class="">
</div>
</blockquote>
<br>
I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I
am not sure above bug <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1172905"
class="">1172905</a> fixes this correctly.<br>
<br>
Regards,<br>
Ramesh<br>
<br>
<blockquote
cite="mid:F5280320-1598-4721-A5D4-CE3035F1E7C3@onholyground.com"
type="cite">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu
<<a moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
class="">rnachimu(a)redhat.com</a>&gt;
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF"
class=""> <br
class="">
<br class="">
<div class="moz-cite-prefix">On 09/22/2015 05:57 PM,
Alastair Neil wrote:<br class="">
</div>
<blockquote
cite="mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail.com"
type="cite" class="">
<div dir="ltr" class="">You need to set the
gluster.server-quorum-ratio to 51%</div>
<div class="gmail_extra"><br class="">
</div>
</blockquote>
<br class="">
I did that. But still I am facing the same issue. VM get
paused when I do some I/O using fio on some disks backed
by gluster. I am not able to resume the VM after this.
Now only way is to bring down the VM and run again. It
runs successfully on the same host without any issue.<br
class="">
<br class="">
Regards,<br class="">
Ramesh<br class="">
<br class="">
<blockquote
cite="mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail.com"
type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">On 22 September 2015 at
08:25, Ramesh Nachimuthu <span dir="ltr"
class=""><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank" class=""><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"
class=""><span
class=""> <br class="">
<br class="">
<div class="">On 09/22/2015 05:43 PM,
Alastair Neil wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">what are
the gluster-quorum-type
and gluster.server-quorum-ratio
settings on the volume?</div>
<div class="gmail_extra"><br
class="">
</div>
</blockquote>
<br class="">
</span>
<div style="outline-style:none"
class="">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"
class=""><b
class="">cluster.server-quorum-type</b>:server<br
class="">
<div title=""
style="outline-style:none"
class="">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"
class=""><b
class="">cluster.quorum-type</b>:auto<br
class="">
<b class="">gluster.server-quorum-ratio
is not set.</b><br class="">
<br class="">
</div>
</div>
One brick process is purposefully killed
but remaining two bricks are up and
running.<br class="">
<br class="">
Regards,<br class="">
Ramesh<br class="">
</div>
</div>
<span class=""> <br class="">
<blockquote type="cite" class="">
<div class="gmail_extra">
<div class="gmail_quote">On 22 September
2015 at 06:24, Ramesh Nachimuthu <span
dir="ltr"
class=""><<a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com"><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br class="">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF" class="">
Hi,<br
class="">
<br class="">
I am not able to resume a VM
which was paused because of
gluster client quorum issue. Here
is what happened in my setup. <br
class="">
<br class="">
1. Created a gluster storage
domain which is backed by gluster
volume with replica 3. <br
class="">
2. Killed one brick process. So
only two bricks are running in
replica 3 setup.<br class="">
3. Created two VMs<br class="">
4. Started some IO using fio on
both of the VMs<br class="">
5. After some time got the
following error in gluster mount
and VMs moved to paused state.<br
class="">
" <span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"
class="">server <a
moz-do-not-send="true"
href="http://10.70.45.17:49217/"
target="_blank"
class="">10.70.45.17:49217</a>
has not responded in the last 42
seconds, disconnecting."<br
class="">
"</span><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"
class=""><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"
class="">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE as quorum is not
met</span>"<br
class="">
more gluster mount logs at
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiU...
class="">
</span>6. After some time gluster
quorum is active and I am able to
write the the gluster file system.<br
class="">
7. When I try to resume the VM it
doesn't work and I got following
error in vdsm log.<br class="">
<a moz-do-not-send="true"
href="http://pastebin.com/aXiamY15"
target="_blank"
class="">http://pastebin.com/aXiamY15</a><br
class="">
<br class="">
<br class="">
Regards,<br class="">
Ramesh<br class="">
<br class="">
</div>
<br class="">
_______________________________________________<br class="">
Users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank"
class="">Users(a)ovirt.org</a><br
class="">
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank"
class="">http://lists.ovirt.org/mailman/listinfo/users</a...
class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</span></div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
_______________________________________________<br
class="">
Users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
class="">Users(a)ovirt.org</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br>
</body>
</html>
--------------050104050809050706040602--
8:58 a.m.
On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
On 09/24/2015 02:38 AM, Darrell Budic wrote:
This is a known issue in overt 3.5.x and below. It’s been solved in the
upcoming ovirt 3.6.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1172905, the fix
involved setting up a special cgroup for the mount, but i can’t find the
exact details atm.
I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am
not sure above bug 1172905
<https://bugzilla.redhat.com/show_bug.cgi?id=1172905> fixes this
correctly.
I think the root cause is the same - qemu cannot recover from glusterfs
unmount, and the only way to resume the vm is to restart it with a fresh
mount.
The mentioned bug handle the case where stopping vdsm kills the glusterfs
mount helper. This issue is fixed in 3.6.
The issue here seems different. I suggest you open a bug so gluster guys
can investigate this.
Nir
Regards,
Ramesh
On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%
I did that. But still I am facing the same issue. VM get paused when I do
some I/O using fio on some disks backed by gluster. I am not able to resume
the VM after this. Now only way is to bring down the VM and run again. It
runs successfully on the same host without any issue.
Regards,
Ramesh
On 22 September 2015 at 08:25, Ramesh Nachimuthu < <rnachimu(a)redhat.com>
rnachimu(a)redhat.com> wrote:
>
>
> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>
> what are the gluster-quorum-type and gluster.server-quorum-ratio
> settings on the volume?
>
>
> *cluster.server-quorum-type*:server
> *cluster.quorum-type*:auto
> *gluster.server-quorum-ratio is not set.*
>
> One brick process is purposefully killed but remaining two bricks are up
> and running.
>
> Regards,
> Ramesh
>
> On 22 September 2015 at 06:24, Ramesh Nachimuthu < <rnachimu(a)redhat.com>
> rnachimu(a)redhat.com> wrote:
>
>> Hi,
>>
>> I am not able to resume a VM which was paused because of gluster
>> client quorum issue. Here is what happened in my setup.
>>
>> 1. Created a gluster storage domain which is backed by gluster volume
>> with replica 3.
>> 2. Killed one brick process. So only two bricks are running in replica 3
>> setup.
>> 3. Created two VMs
>> 4. Started some IO using fio on both of the VMs
>> 5. After some time got the following error in gluster mount and VMs
>> moved to paused state.
>> " server 10.70.45.17:49217 has not responded in the last 42
>> seconds, disconnecting."
>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
>> Failing WRITE as quorum is not met"
>> more gluster mount logs at http://pastebin.com/UmiUQq0F
>> 6. After some time gluster quorum is active and I am able to write the
>> the gluster file system.
>> 7. When I try to resume the VM it doesn't work and I got following error
>> in vdsm log.
>> http://pastebin.com/aXiamY15
>>
>>
>> Regards,
>> Ramesh
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:06 a.m.
This is a multi-part message in MIME format.
--------------030503030503080006060300
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
On 09/24/2015 11:28 AM, Nir Soffer wrote:
On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu
<rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>> wrote:
On 09/24/2015 02:38 AM, Darrell Budic wrote:
> This is a known issue in overt 3.5.x and below. It’s been solved
> in the upcoming ovirt 3.6.
>
> Related to https://bugzilla.redhat.com/show_bug.cgi?id=1172905,
> the fix involved setting up a special cgroup for the mount, but i
> can’t find the exact details atm.
>
I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So
I am not sure above bug 1172905
<https://bugzilla.redhat.com/show_bug.cgi?id=1172905> fixes this
correctly.
I think the root cause is the same - qemu cannot recover from
glusterfs unmount, and the only way to resume the vm is to restart it
with a fresh mount.
The mentioned bug handle the case where stopping vdsm kills the
glusterfs mount helper. This issue is fixed in 3.6.
The issue here seems different. I suggest you open a bug so gluster
guys can investigate this.
Seems like I am hitting the issue reported in bz
https://bugzilla.redhat.com/show_bug.cgi?id=1171261.
Regards,
Ramesh
Nir
Regards,
Ramesh
>
>> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu
>> <rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>> wrote:
>>
>>
>>
>> On 09/22/2015 05:57 PM, Alastair Neil wrote:
>>> You need to set the gluster.server-quorum-ratio to 51%
>>>
>>
>> I did that. But still I am facing the same issue. VM get paused
>> when I do some I/O using fio on some disks backed by gluster. I
>> am not able to resume the VM after this. Now only way is to
>> bring down the VM and run again. It runs successfully on the
>> same host without any issue.
>>
>> Regards,
>> Ramesh
>>
>>> On 22 September 2015 at 08:25, Ramesh Nachimuthu
>>> <rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>> wrote:
>>>
>>>
>>>
>>> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>>>> what are the gluster-quorum-type
>>>> and gluster.server-quorum-ratio settings on the volume?
>>>>
>>>
>>> *cluster.server-quorum-type*:server
>>> *cluster.quorum-type*:auto
>>> *gluster.server-quorum-ratio is not set.*
>>>
>>> One brick process is purposefully killed but remaining two
>>> bricks are up and running.
>>>
>>> Regards,
>>> Ramesh
>>>
>>>> On 22 September 2015 at 06:24, Ramesh Nachimuthu
>>>> <rnachimu(a)redhat.com <mailto:rnachimu@redhat.com>>
wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am not able to resume a VM which was paused
>>>> because of gluster client quorum issue. Here is what
>>>> happened in my setup.
>>>>
>>>> 1. Created a gluster storage domain which is backed by
>>>> gluster volume with replica 3.
>>>> 2. Killed one brick process. So only two bricks are
>>>> running in replica 3 setup.
>>>> 3. Created two VMs
>>>> 4. Started some IO using fio on both of the VMs
>>>> 5. After some time got the following error in gluster
>>>> mount and VMs moved to paused state.
>>>> " server 10.70.45.17:49217
>>>> <http://10.70.45.17:49217/> has not responded in the
>>>> last 42 seconds, disconnecting."
>>>> "vmstore-replicate-0:
>>>> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as
>>>> quorum is not met"
>>>> more gluster mount logs at
>>>> http://pastebin.com/UmiUQq0F
>>>> 6. After some time gluster quorum is active and I am
>>>> able to write the the gluster file system.
>>>> 7. When I try to resume the VM it doesn't work and I
>>>> got following error in vdsm log.
>>>> http://pastebin.com/aXiamY15
>>>>
>>>>
>>>> Regards,
>>>> Ramesh
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--------------030503030503080006060300
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 09/24/2015 11:28 AM, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyyvi0mtEVyae4TJxq-67B7ZKwnEVwUu5Dp6DrChO1Y2_Yw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Thu, Sep 24, 2015 at 7:37 AM,
Ramesh Nachimuthu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span
class=""> <br>
<br>
<div>On 09/24/2015 02:38 AM, Darrell Budic wrote:<br>
</div>
<blockquote type="cite"> This is a known issue in
overt 3.5.x and below. It’s been solved in the
upcoming ovirt 3.6.
<div><br>
</div>
<div>Related to <a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1172905"
target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=...;,
the fix involved setting up a special cgroup for
the mount, but i can’t find the exact details atm.</div>
<div><br>
</div>
</blockquote>
<br>
</span> I have vdsm 4.17.6-0.el7.centos already
installed on the hosts. So I am not sure above bug <a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1172905"
target="_blank">1172905</a> fixes this
correctly.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I think the root cause is the same - qemu cannot
recover from glusterfs unmount, and the only way to resume
the vm is to restart it with a fresh mount.</div>
<div><br>
</div>
<div>The mentioned bug handle the case where stopping vdsm
kills the glusterfs mount helper. This issue is fixed in
3.6. </div>
<div><br>
</div>
<div>The issue here seems different. I suggest you open a
bug so gluster guys can investigate this.</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Seems like I am hitting the issue reported in bz
<a class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1171261">h...;.
<br>
<br>
Regards,<br>
Ramesh<br>
<br>
<blockquote
cite="mid:CAMRbyyvi0mtEVyae4TJxq-67B7ZKwnEVwUu5Dp6DrChO1Y2_Yw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Nir</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Regards,<br>
Ramesh
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div><br>
<div>
<blockquote type="cite">
<div>On Sep 23, 2015, at 7:38 AM, Ramesh
Nachimuthu <<a moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank">rnachimu(a)redhat.com</a>&gt;
wrote:</div>
<br>
<div>
<div text="#000000"
bgcolor="#FFFFFF"> <br>
<br>
<div>On 09/22/2015 05:57 PM, Alastair
Neil wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">You need to set the
gluster.server-quorum-ratio to 51%</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
<br>
I did that. But still I am facing the
same issue. VM get paused when I do some
I/O using fio on some disks backed by
gluster. I am not able to resume the VM
after this. Now only way is to bring
down the VM and run again. It runs
successfully on the same host without
any issue.<br>
<br>
Regards,<br>
Ramesh<br>
<br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 22
September 2015 at 08:25, Ramesh
Nachimuthu <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com"
target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF"><span>
<br>
<br>
<div>On 09/22/2015 05:43 PM,
Alastair Neil wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">what are
the gluster-quorum-type
and gluster.server-quorum-ratio
settings on the volume?</div>
<div
class="gmail_extra"><br>
</div>
</blockquote>
<br>
</span>
<div
style="outline-style:none">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"><b>cluster.server-quorum-type</b>:server<br>
<div title=""
style="outline-style:none">
<div
style="overflow:hidden;text-overflow:ellipsis;white-space:nowrap"><b>cluster.quorum-type</b>:auto<br>
<b>gluster.server-quorum-ratio
is not set.</b><br>
<br>
</div>
</div>
One brick process is
purposefully killed but
remaining two bricks are
up and running.<br>
<br>
Regards,<br>
Ramesh<br>
</div>
</div>
<span> <br>
<blockquote type="cite">
<div class="gmail_extra">
<div
class="gmail_quote">On
22 September 2015 at
06:24, Ramesh
Nachimuthu <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rnachimu@redhat.com" target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a></a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF">
Hi,<br>
<br>
I am not able
to resume a VM
which was paused
because of gluster
client quorum
issue. Here is
what happened in
my setup. <br>
<br>
1. Created a
gluster storage
domain which is
backed by gluster
volume with
replica 3. <br>
2. Killed one
brick process. So
only two bricks
are running in
replica 3 setup.<br>
3. Created two VMs<br>
4. Started some IO
using fio on both
of the VMs<br>
5. After some time
got the following
error in gluster
mount and VMs
moved to paused
state.<br>
" <span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">server
<a
moz-do-not-send="true"
href="http://10.70.45.17:49217/"
target="_blank">10.70.45.17:49217</a>
has not
responded in the
last 42 seconds,
disconnecting."<br>
"</span><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)"><span
style="color:rgb(51,51,51);font-family:monospace;font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none;background-color:rgb(255,255,255)">vmstore-replicate-0:
e16d1e40-2b6e-4f19-977d-e099f465dfc6:
Failing WRITE
as quorum is
not
met</span>"<br>
more
gluster mount
logs at <a
moz-do-not-send="true"
href="http://pastebin.com/UmiUQq0F" target="_blank"><a
class="moz-txt-link-freetext"
href="http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiU...
</span>6. After
some time gluster
quorum is active
and I am able to
write the the
gluster file
system.<br>
7. When I try to
resume the VM it
doesn't work and I
got following
error in vdsm log.<br>
<a
moz-do-not-send="true"
href="http://pastebin.com/aXiamY15" target="_blank"><a
class="moz-txt-link-freetext"
href="http://pastebin.com/aXiamY15">http://pastebin.com/aXia...
<br>
<br>
Regards,<br>
Ramesh<br>
<br>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a
moz-do-not-send="true"
href="mailto:Users@ovirt.org" target="_blank"><a
class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a></a><br>
<a
moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer"
target="_blank"><a
class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------030503030503080006060300--
9:27 a.m.
On Thu, Sep 24, 2015 at 9:06 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
On 09/24/2015 11:28 AM, Nir Soffer wrote:
On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu < <rnachimu(a)redhat.com>
rnachimu(a)redhat.com> wrote:
>
>
> On 09/24/2015 02:38 AM, Darrell Budic wrote:
>
> This is a known issue in overt 3.5.x and below. It’s been solved in the
> upcoming ovirt 3.6.
>
> Related to https://bugzilla.redhat.com/show_bug.cgi?id=1172905, the fix
> involved setting up a special cgroup for the mount, but i can’t find the
> exact details atm.
>
>
> I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am
> not sure above bug 1172905
> <https://bugzilla.redhat.com/show_bug.cgi?id=1172905> fixes this
> correctly.
>
I think the root cause is the same - qemu cannot recover from glusterfs
unmount, and the only way to resume the vm is to restart it with a fresh
mount.
The mentioned bug handle the case where stopping vdsm kills the glusterfs
mount helper. This issue is fixed in 3.6.
The issue here seems different. I suggest you open a bug so gluster guys
can investigate this.
Seems like I am hitting the issue reported in bz
https://bugzilla.redhat.com/show_bug.cgi?id=1171261.
Indeed.
I would open an ovirt bug anyway and make it depend on the glusterfs bug.
We need a way to track this issues, and having no ovirt/rhev hides this
issue.
Regards,
Ramesh
Nir
>
> Regards,
> Ramesh
>
>
>
> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com>
> wrote:
>
>
>
> On 09/22/2015 05:57 PM, Alastair Neil wrote:
>
> You need to set the gluster.server-quorum-ratio to 51%
>
>
> I did that. But still I am facing the same issue. VM get paused when I do
> some I/O using fio on some disks backed by gluster. I am not able to resume
> the VM after this. Now only way is to bring down the VM and run again. It
> runs successfully on the same host without any issue.
>
> Regards,
> Ramesh
>
> On 22 September 2015 at 08:25, Ramesh Nachimuthu < <rnachimu(a)redhat.com>
> rnachimu(a)redhat.com> wrote:
>
>>
>>
>> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>>
>> what are the gluster-quorum-type and gluster.server-quorum-ratio
>> settings on the volume?
>>
>>
>> *cluster.server-quorum-type*:server
>> *cluster.quorum-type*:auto
>> *gluster.server-quorum-ratio is not set.*
>>
>> One brick process is purposefully killed but remaining two bricks are
>> up and running.
>>
>> Regards,
>> Ramesh
>>
>> On 22 September 2015 at 06:24, Ramesh Nachimuthu <
<rnachimu(a)redhat.com>
>> rnachimu(a)redhat.com> wrote:
>>
>>> Hi,
>>>
>>> I am not able to resume a VM which was paused because of gluster
>>> client quorum issue. Here is what happened in my setup.
>>>
>>> 1. Created a gluster storage domain which is backed by gluster volume
>>> with replica 3.
>>> 2. Killed one brick process. So only two bricks are running in replica
>>> 3 setup.
>>> 3. Created two VMs
>>> 4. Started some IO using fio on both of the VMs
>>> 5. After some time got the following error in gluster mount and VMs
>>> moved to paused state.
>>> " server 10.70.45.17:49217 has not responded in the last 42
>>> seconds, disconnecting."
>>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
>>> Failing WRITE as quorum is not met"
>>> more gluster mount logs at <http://pastebin.com/UmiUQq0F>
>>> http://pastebin.com/UmiUQq0F
>>> 6. After some time gluster quorum is active and I am able to write the
>>> the gluster file system.
>>> 7. When I try to resume the VM it doesn't work and I got following
>>> error in vdsm log.
>>> <http://pastebin.com/aXiamY15>http://pastebin.com/aXiamY15
>>>
>>>
>>> Regards,
>>> Ramesh
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> <Users@ovirt.org>Users(a)ovirt.org
>>> <http://lists.ovirt.org/mailman/listinfo/users>
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
8:52 a.m.
The details are here:
https://gerrit.ovirt.org/#/c/40240
The link exist on the bug of course.
On Thu, Sep 24, 2015 at 12:08 AM, Darrell Budic <budic(a)onholyground.com>
wrote:
This is a known issue in overt 3.5.x and below. It’s been solved in
the
upcoming ovirt 3.6.
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1172905, the fix
involved setting up a special cgroup for the mount, but i can’t find the
exact details atm.
On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%
I did that. But still I am facing the same issue. VM get paused when I do
some I/O using fio on some disks backed by gluster. I am not able to resume
the VM after this. Now only way is to bring down the VM and run again. It
runs successfully on the same host without any issue.
Regards,
Ramesh
On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)redhat.com>
wrote:
>
>
> On 09/22/2015 05:43 PM, Alastair Neil wrote:
>
> what are the gluster-quorum-type and gluster.server-quorum-ratio
> settings on the volume?
>
>
> *cluster.server-quorum-type*:server
> *cluster.quorum-type*:auto
> *gluster.server-quorum-ratio is not set.*
>
> One brick process is purposefully killed but remaining two bricks are up
> and running.
>
> Regards,
> Ramesh
>
> On 22 September 2015 at 06:24, Ramesh Nachimuthu < <rnachimu(a)redhat.com>
> rnachimu(a)redhat.com> wrote:
>
>> Hi,
>>
>> I am not able to resume a VM which was paused because of gluster
>> client quorum issue. Here is what happened in my setup.
>>
>> 1. Created a gluster storage domain which is backed by gluster volume
>> with replica 3.
>> 2. Killed one brick process. So only two bricks are running in replica 3
>> setup.
>> 3. Created two VMs
>> 4. Started some IO using fio on both of the VMs
>> 5. After some time got the following error in gluster mount and VMs
>> moved to paused state.
>> " server 10.70.45.17:49217 has not responded in the last 42
>> seconds, disconnecting."
>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6:
>> Failing WRITE as quorum is not met"
>> more gluster mount logs at <http://pastebin.com/UmiUQq0F>
>> http://pastebin.com/UmiUQq0F
>> 6. After some time gluster quorum is active and I am able to write the
>> the gluster file system.
>> 7. When I try to resume the VM it doesn't work and I got following error
>> in vdsm log.
>> http://pastebin.com/aXiamY15
>>
>>
>> Regards,
>> Ramesh
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3347
days inactive
3349
days old
10 comments
4 participants
participants (4)
-
Alastair Neil -
Darrell Budic -
Nir Soffer -
Ramesh Nachimuthu