On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu@redhat.com> = wrote: =20 =20 =20 On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51% =20 =20 I did that. But still I am facing the same issue. VM get paused when I = do some I/O using fio on some disks backed by gluster. I am not able to = resume the VM after this. Now only way is to bring down the VM and run = again. It runs successfully on the same host without any issue. =20 Regards, Ramesh =20 On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu@redhat.com = <mailto:rnachimu@redhat.com>> wrote: =20 =20 On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the gluster-quorum-type and gluster.server-quorum-ratio = settings on the volume? =20 =20 cluster.server-quorum-type:server cluster.quorum-type:auto gluster.server-quorum-ratio is not set. =20 One brick process is purposefully killed but remaining two bricks = are up and running. =20 Regards, Ramesh =20 On 22 September 2015 at 06:24, Ramesh Nachimuthu < = <mailto:rnachimu@redhat.com>rnachimu@redhat.com = <mailto:rnachimu@redhat.com>> wrote: Hi, =20 I am not able to resume a VM which was paused because of gluster = client quorum issue. Here is what happened in my setup.=20 =20 1. Created a gluster storage domain which is backed by gluster = volume with replica 3.=20 2. Killed one brick process. So only two bricks are running in = replica 3 setup. 3. Created two VMs 4. Started some IO using fio on both of the VMs 5. After some time got the following error in gluster mount and VMs = moved to paused state. " server 10.70.45.17:49217 <http://10.70.45.17:49217/> has = not responded in the last 42 seconds, disconnecting." "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: = Failing WRITE as quorum is not met" more gluster mount logs at = <http://pastebin.com/UmiUQq0F>http://pastebin.com/UmiUQq0F = <http://pastebin.com/UmiUQq0F> 6. After some time gluster quorum is active and I am able to write =
--Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved = in the upcoming ovirt 3.6. Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the = fix involved setting up a special cgroup for the mount, but i can=E2=80=99= t find the exact details atm. the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following = error in vdsm log. http://pastebin.com/aXiamY15 <http://pastebin.com/aXiamY15> =20 =20 Regards, Ramesh =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 =20 =20 =20
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">This is a known issue in overt 3.5.x and below. It=E2=80=99s = been solved in the upcoming ovirt 3.6.<div class=3D""><br = class=3D""></div><div class=3D"">Related to <a = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905" = class=3D"">https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905</a>, = the fix involved setting up a special cgroup for the mount, but i = can=E2=80=99t find the exact details atm.</div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""><div><blockquote = type=3D"cite" class=3D""><div class=3D"">On Sep 23, 2015, at 7:38 AM, = Ramesh Nachimuthu <<a href=3D"mailto:rnachimu@redhat.com" = class=3D"">rnachimu@redhat.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""> =20 <meta content=3D"text/html; charset=3Dutf-8" = http-equiv=3D"Content-Type" class=3D""> =20 <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> <br class=3D""> <br class=3D""> <div class=3D"moz-cite-prefix">On 09/22/2015 05:57 PM, Alastair Neil wrote:<br class=3D""> </div> <blockquote = cite=3D"mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D"">You need to set the = gluster.server-quorum-ratio to 51%</div> <div class=3D"gmail_extra"><br class=3D""> </div> </blockquote> <br class=3D""> I did that. But still I am facing the same issue. VM get paused when I do some I/O using fio on some disks backed by gluster. I am not able to resume the VM after this. Now only way is to bring down the VM and run again. It runs successfully on the same host without any issue.<br class=3D""> <br class=3D""> Regards,<br class=3D""> Ramesh<br class=3D""> <br class=3D""> <blockquote = cite=3D"mid:CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail= .com" type=3D"cite" class=3D""> <div class=3D"gmail_extra"> <div class=3D"gmail_quote">On 22 September 2015 at 08:25, Ramesh Nachimuthu <span dir=3D"ltr" class=3D""><<a = moz-do-not-send=3D"true" href=3D"mailto:rnachimu@redhat.com" = target=3D"_blank" class=3D"">rnachimu@redhat.com</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""><span = class=3D""> <br class=3D""> <br class=3D""> <div class=3D"">On 09/22/2015 05:43 PM, Alastair Neil = wrote:<br class=3D""> </div> <blockquote type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D"">what are = the gluster-quorum-type and gluster.server-quorum-ratio settings = on the volume?</div> <div class=3D"gmail_extra"><br class=3D""> </div> </blockquote> <br class=3D""> </span> <div style=3D"outline-style:none" class=3D""> <div = style=3D"overflow:hidden;text-overflow:ellipsis;white-space:nowrap" = class=3D""><b class=3D"">cluster.server-quorum-type</b>:server<br = class=3D""> <div title=3D"" style=3D"outline-style:none" class=3D"">= <div = style=3D"overflow:hidden;text-overflow:ellipsis;white-space:nowrap" = class=3D""><b class=3D"">cluster.quorum-type</b>:auto<br class=3D""> <b class=3D"">gluster.server-quorum-ratio is not = set.</b><br class=3D""> <br class=3D""> </div> </div> One brick process is purposefully killed but remaining two bricks are up and running.<br class=3D""> <br class=3D""> Regards,<br class=3D""> Ramesh<br class=3D""> </div> </div> <span class=3D""> <br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"gmail_extra"> <div class=3D"gmail_quote">On 22 September 2015 at 06:24, Ramesh Nachimuthu <span dir=3D"ltr" = class=3D""><<a moz-do-not-send=3D"true" = href=3D"mailto:rnachimu@redhat.com" target=3D"_blank" class=3D""></a><a = class=3D"moz-txt-link-abbreviated" = href=3D"mailto:rnachimu@redhat.com">rnachimu@redhat.com</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text=3D"#000000" bgcolor=3D"#FFFFFF" = class=3D""> Hi,<br class=3D""> <br class=3D""> I am not able to resume a VM = which was paused because of gluster client quorum issue. Here is what happened in my setup. <br = class=3D""> <br class=3D""> 1. Created a gluster storage domain which is backed by gluster volume with replica 3. <br = class=3D""> 2. Killed one brick process. So only two bricks are running in replica 3 setup.<br = class=3D""> 3. Created two VMs<br class=3D""> 4. Started some IO using fio on both of the VMs<br class=3D""> 5. After some time got the following error in gluster mount and VMs moved to paused = state.<br class=3D""> = " <span = style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty= le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin= e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-= space:normal;word-spacing:0px;display:inline!important;float:none;backgrou= nd-color:rgb(255,255,255)" class=3D"">server <a moz-do-not-send=3D"true" = href=3D"http://10.70.45.17:49217/" target=3D"_blank" = class=3D"">10.70.45.17:49217</a> has not responded in the last 42 seconds, disconnecting."<br class=3D""> "</span><span = style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty= le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin= e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-= space:normal;word-spacing:0px;display:inline!important;float:none;backgrou= nd-color:rgb(255,255,255)" class=3D""><span = style=3D"color:rgb(51,51,51);font-family:monospace;font-size:11px;font-sty= le:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;lin= e-height:13.2px;text-align:left;text-indent:0px;text-transform:none;white-= space:normal;word-spacing:0px;display:inline!important;float:none;backgrou= nd-color:rgb(255,255,255)" class=3D"">vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not = met</span>"<br class=3D""> more gluster = mount logs at <a moz-do-not-send=3D"true" = href=3D"http://pastebin.com/UmiUQq0F" target=3D"_blank" class=3D""></a><a = class=3D"moz-txt-link-freetext" = href=3D"http://pastebin.com/UmiUQq0F">http://pastebin.com/UmiUQq0F</a><br = class=3D""> </span>6. After some time gluster quorum is active and I am able to write the the gluster file system.<br class=3D""> 7. When I try to resume the VM it doesn't work and I got following error in vdsm log.<br = class=3D""> <a = moz-do-not-send=3D"true" href=3D"http://pastebin.com/aXiamY15" = target=3D"_blank" class=3D"">http://pastebin.com/aXiamY15</a><br = class=3D""> <br class=3D""> <br class=3D""> Regards,<br class=3D""> Ramesh<br class=3D""> <br class=3D""> </div> <br class=3D""> = _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a moz-do-not-send=3D"true" = href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> <a moz-do-not-send=3D"true" = href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" = target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman/listinfo/users</a><br = class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </blockquote> <br class=3D""> </span></div> </blockquote> </div> <br class=3D""> </div> </blockquote> <br class=3D""> </div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50--