From rnachimu at redhat.com Tue Sep 22 06:24:58 2015 Content-Type: multipart/mixed; boundary="===============2293197443651235188==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Tue, 22 Sep 2015 15:54:54 +0530 Message-ID: <56012C76.9010408@redhat.com> --===============2293197443651235188== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050502080707090302020409 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hi, I am not able to resume a VM which was paused because of gluster = client quorum issue. Here is what happened in my setup. 1. Created a gluster storage domain which is backed by gluster volume = with replica 3. 2. Killed one brick process. So only two bricks are running in replica 3 = setup. 3. Created two VMs 4. Started some IO using fio on both of the VMs 5. After some time got the following error in gluster mount and VMs = moved to paused state. " server 10.70.45.17:49217 has not responded in the last 42 = seconds, disconnecting." "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: = Failing WRITE as quorum is not met" more gluster mount logs at http://pastebin.com/UmiUQq0F 6. After some time gluster quorum is active and I am able to write the = the gluster file system. 7. When I try to resume the VM it doesn't work and I got following error = in vdsm log. http://pastebin.com/aXiamY15 Regards, Ramesh --------------050502080707090302020409 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hi,

=C2=A0=C2=A0 I am not able to resume a VM which was paused because of g= luster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 more gluster mount logs at http://pasteb= in.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://pastebin.com/aXiamY15


Regards,
Ramesh

--------------050502080707090302020409-- --===============2293197443651235188== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTA1MDIwODA3MDcwOTAzMDIwMjA0MDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhpLAoKICAgIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIGEgVk0gd2hpY2ggd2FzIHBhdXNlZCBi ZWNhdXNlIG9mIGdsdXN0ZXIgCmNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBoYXBw ZW5lZCBpbiBteSBzZXR1cC4KCjEuIENyZWF0ZWQgYSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWluIHdo aWNoIGlzIGJhY2tlZCBieSBnbHVzdGVyIHZvbHVtZSAKd2l0aCByZXBsaWNhIDMuCjIuIEtpbGxl ZCBvbmUgYnJpY2sgcHJvY2Vzcy4gU28gb25seSB0d28gYnJpY2tzIGFyZSBydW5uaW5nIGluIHJl cGxpY2EgMyAKc2V0dXAuCjMuIENyZWF0ZWQgdHdvIFZNcwo0LiBTdGFydGVkIHNvbWUgSU8gdXNp bmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNcwo1LiBBZnRlciBzb21lIHRpbWUgZ290IHRoZSBmb2xs b3dpbmcgZXJyb3IgaW4gZ2x1c3RlciBtb3VudCBhbmQgVk1zIAptb3ZlZCB0byBwYXVzZWQgc3Rh dGUuCiAgICAgICAgICAiIHNlcnZlciAxMC43MC40NS4xNzo0OTIxNyBoYXMgbm90IHJlc3BvbmRl ZCBpbiB0aGUgbGFzdCA0MiAKc2Vjb25kcywgZGlzY29ubmVjdGluZy4iCiAgICAgICAidm1zdG9y ZS1yZXBsaWNhdGUtMDogZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OiAKRmFp bGluZyBXUklURSBhcyBxdW9ydW0gaXMgbm90IG1ldCIKICAgICAgIG1vcmUgZ2x1c3RlciBtb3Vu dCBsb2dzIGF0IGh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEYKNi4gQWZ0ZXIgc29tZSB0aW1l IGdsdXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvIHdyaXRlIHRoZSAKdGhl IGdsdXN0ZXIgZmlsZSBzeXN0ZW0uCjcuIFdoZW4gSSB0cnkgdG8gcmVzdW1lIHRoZSBWTSBpdCBk b2Vzbid0IHdvcmsgYW5kIEkgZ290IGZvbGxvd2luZyBlcnJvciAKaW4gdmRzbSBsb2cuCiAgICAg ICBodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1CgoKUmVnYXJkcywKUmFtZXNoCgoKLS0tLS0t LS0tLS0tLS0wNTA1MDIwODA3MDcwOTAzMDIwMjA0MDkKQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7 IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJpdAoKPGh0bWw+CiAg PGhlYWQ+CgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogIDwvaGVhZD4KICA8Ym9keSB0ZXh0PSIjMDAwMDAwIiBi Z2NvbG9yPSIjRkZGRkZGIj4KICAgIEhpLDxicj4KICAgIDxicj4KICAgIMKgwqAgSSBhbSBub3Qg YWJsZSB0byByZXN1bWUgYSBWTSB3aGljaCB3YXMgcGF1c2VkIGJlY2F1c2Ugb2YgZ2x1c3Rlcgog ICAgY2xpZW50IHF1b3J1bSBpc3N1ZS4gSGVyZSBpcyB3aGF0IGhhcHBlbmVkIGluIG15IHNldHVw LiA8YnI+CiAgICA8YnI+CiAgICAxLiBDcmVhdGVkIGEgZ2x1c3RlciBzdG9yYWdlIGRvbWFpbiB3 aGljaCBpcyBiYWNrZWQgYnkgZ2x1c3RlcgogICAgdm9sdW1lIHdpdGggcmVwbGljYSAzLiA8YnI+ CiAgICAyLiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkgdHdvIGJyaWNrcyBhcmUg cnVubmluZyBpbgogICAgcmVwbGljYSAzIHNldHVwLjxicj4KICAgIDMuIENyZWF0ZWQgdHdvIFZN czxicj4KICAgIDQuIFN0YXJ0ZWQgc29tZSBJTyB1c2luZyBmaW8gb24gYm90aCBvZiB0aGUgVk1z PGJyPgogICAgNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluIGds dXN0ZXIgbW91bnQgYW5kIFZNcwogICAgbW92ZWQgdG8gcGF1c2VkIHN0YXRlLjxicj4KICAgIMKg wqDCoMKgwqDCoMKgwqAgIgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250 ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogICAgPHNwYW4gc3R5bGU9ImNvbG9yOiBy Z2IoNTEsIDUxLCA1MSk7IGZvbnQtZmFtaWx5OiBtb25vc3BhY2U7CiAgICAgIGZvbnQtc2l6ZTog MTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsKICAgICAgZm9u dC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgbGluZS1oZWlnaHQ6IDEz LjJweDsKICAgICAgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogbGVmdDsgdGV4dC1pbmRlbnQ6 IDBweDsgdGV4dC10cmFuc2Zvcm06CiAgICAgIG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdp ZG93czogMTsgd29yZC1zcGFjaW5nOiAwcHg7CiAgICAgIC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OgogICAgICBub25lOyBi YWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ij5zZXJ2ZXIKICAgICAgMTAuNzAu NDUuMTc6NDkyMTcgaGFzIG5vdCByZXNwb25kZWQgaW4gdGhlIGxhc3QgNDIgc2Vjb25kcywKICAg ICAgZGlzY29ubmVjdGluZy4iPGJyPgogICAgICDCoMKgwqDCoMKgICI8L3NwYW4+PHNwYW4gc3R5 bGU9ImNvbG9yOiByZ2IoNTEsIDUxLCA1MSk7IGZvbnQtZmFtaWx5OgogICAgICBtb25vc3BhY2U7 IGZvbnQtc2l6ZTogMTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6CiAgICAg IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgbGlu ZS1oZWlnaHQ6CiAgICAgIDEzLjJweDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogbGVmdDsg dGV4dC1pbmRlbnQ6IDBweDsKICAgICAgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNl OiBub3JtYWw7IHdpZG93czogMTsKICAgICAgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgZGlzcGxheTogaW5saW5lCiAgICAgICFpbXBvcnRhbnQ7IGZs b2F0OiBub25lOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ij48c3Bhbgog ICAgICAgIHN0eWxlPSJjb2xvcjogcmdiKDUxLCA1MSwgNTEpOyBmb250LWZhbWlseTogbW9ub3Nw YWNlOwogICAgICAgIGZvbnQtc2l6ZTogMTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQ6IG5vcm1hbDsKICAgICAgICBmb250LXdlaWdodDogbm9ybWFsOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyBsaW5lLWhlaWdodDoKICAgICAgICAxMy4ycHg7IG9ycGhhbnM6IGF1dG87IHRl eHQtYWxpZ246IGxlZnQ7IHRleHQtaW5kZW50OiAwcHg7CiAgICAgICAgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdpZG93czogMTsKICAgICAgICB3b3JkLXNwYWNp bmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBkaXNwbGF5OgogICAgICAg IGlubGluZSAhaW1wb3J0YW50OyBmbG9hdDogbm9uZTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1 NSwgMjU1LAogICAgICAgIDI1NSk7Ij52bXN0b3JlLXJlcGxpY2F0ZS0wOgogICAgICAgIGUxNmQx ZTQwLTJiNmUtNGYxOS05NzdkLWUwOTlmNDY1ZGZjNjogRmFpbGluZyBXUklURSBhcyBxdW9ydW0g aXMKICAgICAgICBub3QgbWV0PC9zcGFuPiI8YnI+CiAgICAgIMKgwqDCoMKgwqAgbW9yZSBnbHVz dGVyIG1vdW50IGxvZ3MgYXQgPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0i aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRiI+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEw RjwvYT48YnI+CiAgICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGdsdXN0ZXIgcXVvcnVtIGlz IGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvCiAgICB3cml0ZSB0aGUgdGhlIGdsdXN0ZXIgZmlsZSBz eXN0ZW0uPGJyPgogICAgNy4gV2hlbiBJIHRyeSB0byByZXN1bWUgdGhlIFZNIGl0IGRvZXNuJ3Qg d29yayBhbmQgSSBnb3QgZm9sbG93aW5nCiAgICBlcnJvciBpbiB2ZHNtIGxvZy48YnI+CiAgICDC oMKgwqDCoMKgIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9w YXN0ZWJpbi5jb20vYVhpYW1ZMTUiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTU8L2E+PGJy PgogICAgPGJyPgogICAgPGJyPgogICAgUmVnYXJkcyw8YnI+CiAgICBSYW1lc2g8YnI+CiAgICA8 YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTA1MDUwMjA4MDcwNzA5MDMwMjAy MDQwOS0tCg== --===============2293197443651235188==-- From ajneil.tech at gmail.com Tue Sep 22 08:13:48 2015 Content-Type: multipart/mixed; boundary="===============6108044548865332619==" MIME-Version: 1.0 From: Alastair Neil To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Tue, 22 Sep 2015 08:13:47 -0400 Message-ID: In-Reply-To: 56012C76.9010408@redhat.com --===============6108044548865332619== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable what are the gluster-quorum-type and gluster.server-quorum-ratio settings on the volume? On 22 September 2015 at 06:24, Ramesh Nachimuthu wrote: > Hi, > > I am not able to resume a VM which was paused because of gluster client > quorum issue. Here is what happened in my setup. > > 1. Created a gluster storage domain which is backed by gluster volume with > replica 3. > 2. Killed one brick process. So only two bricks are running in replica 3 > setup. > 3. Created two VMs > 4. Started some IO using fio on both of the VMs > 5. After some time got the following error in gluster mount and VMs moved > to paused state. > " server 10.70.45.17:49217 has not responded in the last 42 > seconds, disconnecting." > "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing > WRITE as quorum is not met" > more gluster mount logs at http://pastebin.com/UmiUQq0F > 6. After some time gluster quorum is active and I am able to write the the > gluster file system. > 7. When I try to resume the VM it doesn't work and I got following error > in vdsm log. > http://pastebin.com/aXiamY15 > > > Regards, > Ramesh > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============6108044548865332619== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+d2hhdCBhcmUgdGhlwqBnbHVzdGVyLXF1b3J1bS10eXBlIGFuZMKgZ2x1 c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvIMKgc2V0dGluZ3Mgb24gdGhlIHZvbHVtZT88L2Rpdj48 ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiAy MiBTZXB0ZW1iZXIgMjAxNSBhdCAwNjoyNCwgUmFtZXNoIE5hY2hpbXV0aHUgPHNwYW4gZGlyPSJs dHIiPiZsdDs8YSBocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxh bmsiPnJuYWNoaW11QHJlZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1 b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1s ZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogIAoKICAgIAogIAogIDxkaXYg dGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICBIaSw8YnI+CiAgICA8YnI+CiAg ICDCoMKgIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIGEgVk0gd2hpY2ggd2FzIHBhdXNlZCBiZWNh dXNlIG9mIGdsdXN0ZXIKICAgIGNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBoYXBw ZW5lZCBpbiBteSBzZXR1cC4gPGJyPgogICAgPGJyPgogICAgMS4gQ3JlYXRlZCBhIGdsdXN0ZXIg c3RvcmFnZSBkb21haW4gd2hpY2ggaXMgYmFja2VkIGJ5IGdsdXN0ZXIKICAgIHZvbHVtZSB3aXRo IHJlcGxpY2EgMy4gPGJyPgogICAgMi4gS2lsbGVkIG9uZSBicmljayBwcm9jZXNzLiBTbyBvbmx5 IHR3byBicmlja3MgYXJlIHJ1bm5pbmcgaW4KICAgIHJlcGxpY2EgMyBzZXR1cC48YnI+CiAgICAz LiBDcmVhdGVkIHR3byBWTXM8YnI+CiAgICA0LiBTdGFydGVkIHNvbWUgSU8gdXNpbmcgZmlvIG9u IGJvdGggb2YgdGhlIFZNczxicj4KICAgIDUuIEFmdGVyIHNvbWUgdGltZSBnb3QgdGhlIGZvbGxv d2luZyBlcnJvciBpbiBnbHVzdGVyIG1vdW50IGFuZCBWTXMKICAgIG1vdmVkIHRvIHBhdXNlZCBz dGF0ZS48YnI+CiAgICDCoMKgwqDCoMKgwqDCoMKgICZxdW90OwogICAgCiAgICA8c3BhbiBzdHls ZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXplOjEx cHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3Jt YWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxl ZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFs O3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFj a2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj5zZXJ2ZXIKICAgICAgPGEgaHJlZj0iaHR0 cDovLzEwLjcwLjQ1LjE3OjQ5MjE3IiB0YXJnZXQ9Il9ibGFuayI+MTAuNzAuNDUuMTc6NDkyMTc8 L2E+IGhhcyBub3QgcmVzcG9uZGVkIGluIHRoZSBsYXN0IDQyIHNlY29uZHMsCiAgICAgIGRpc2Nv bm5lY3RpbmcuJnF1b3Q7PGJyPgogICAgICDCoMKgwqDCoMKgICZxdW90Ozwvc3Bhbj48c3BhbiBz dHlsZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXpl OjExcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpu b3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWdu OmxlZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9y bWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7 YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj48c3BhbiBzdHlsZT0iY29sb3I6cmdi KDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1zdHls ZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNw YWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4dC1pbmRl bnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQtc3BhY2lu ZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3VuZC1jb2xv cjpyZ2IoMjU1LDI1NSwyNTUpIj52bXN0b3JlLXJlcGxpY2F0ZS0wOgogICAgICAgIGUxNmQxZTQw LTJiNmUtNGYxOS05NzdkLWUwOTlmNDY1ZGZjNjogRmFpbGluZyBXUklURSBhcyBxdW9ydW0gaXMK ICAgICAgICBub3QgbWV0PC9zcGFuPiZxdW90Ozxicj4KICAgICAgwqDCoMKgwqDCoCBtb3JlIGds dXN0ZXIgbW91bnQgbG9ncyBhdCA8YSBocmVmPSJodHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBG IiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRjwvYT48YnI+CiAg ICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGdsdXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQg SSBhbSBhYmxlIHRvCiAgICB3cml0ZSB0aGUgdGhlIGdsdXN0ZXIgZmlsZSBzeXN0ZW0uPGJyPgog ICAgNy4gV2hlbiBJIHRyeSB0byByZXN1bWUgdGhlIFZNIGl0IGRvZXNuJiMzOTt0IHdvcmsgYW5k IEkgZ290IGZvbGxvd2luZwogICAgZXJyb3IgaW4gdmRzbSBsb2cuPGJyPgogICAgwqDCoMKgwqDC oCA8YSBocmVmPSJodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1IiB0YXJnZXQ9Il9ibGFuayI+ aHR0cDovL3Bhc3RlYmluLmNvbS9hWGlhbVkxNTwvYT48YnI+CiAgICA8YnI+CiAgICA8YnI+CiAg ICBSZWdhcmRzLDxicj4KICAgIFJhbWVzaDxicj4KICAgIDxicj4KICA8L2Rpdj4KCjxicj5fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFp bGluZyBsaXN0PGJyPgo8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0Bvdmly dC5vcmc8L2E+PGJyPgo8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0 cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVv dGU+PC9kaXY+PGJyPjwvZGl2Pgo= --===============6108044548865332619==-- From rnachimu at redhat.com Tue Sep 22 08:25:30 2015 Content-Type: multipart/mixed; boundary="===============2253974178033794051==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Tue, 22 Sep 2015 17:55:25 +0530 Message-ID: <560148B5.5010909@redhat.com> In-Reply-To: CA+SarwpUsiNz5MsUz66LYmEua-G=P5v_mbzOMkiErJZWwdE2MQ@mail.gmail.com --===============2253974178033794051== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------000009040208060202040809 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit On 09/22/2015 05:43 PM, Alastair Neil wrote: > what are the gluster-quorum-type and gluster.server-quorum-ratio = > settings on the volume? > *cluster.server-quorum-type*:server *cluster.quorum-type*:auto *gluster.server-quorum-ratio is not set.* One brick process is purposefully killed but remaining two bricks are = up and running. Regards, Ramesh > On 22 September 2015 at 06:24, Ramesh Nachimuthu > wrote: > > Hi, > > I am not able to resume a VM which was paused because of > gluster client quorum issue. Here is what happened in my setup. > > 1. Created a gluster storage domain which is backed by gluster > volume with replica 3. > 2. Killed one brick process. So only two bricks are running in > replica 3 setup. > 3. Created two VMs > 4. Started some IO using fio on both of the VMs > 5. After some time got the following error in gluster mount and > VMs moved to paused state. > " server 10.70.45.17:49217 has > not responded in the last 42 seconds, disconnecting." > "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: > Failing WRITE as quorum is not met" > more gluster mount logs at http://pastebin.com/UmiUQq0F > 6. After some time gluster quorum is active and I am able to write > the the gluster file system. > 7. When I try to resume the VM it doesn't work and I got following > error in vdsm log. > http://pastebin.com/aXiamY15 > > > Regards, > Ramesh > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --------------000009040208060202040809 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the=C2=A0gluster-quorum-type and=C2=A0gluster.server-quorum-ratio =C2=A0settings on the volume?<= /div>


cluster= .server-quorum-type:server
cl= uster.quorum-type:auto
gluster.server-quorum-ratio is not set.

One brick process is purposefully killed=C2=A0 but remaining two bricks are up and running.

Regards,
Ramesh

On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnach= imu(a)redhat.com> wrote:
Hi,

=C2=A0=C2=A0 I am not able to resume a VM which was paused be= cause of gluster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 " server 10.70= .45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 more gluster mount logs at <= a class=3D"moz-txt-link-freetext" href=3D"http://pastebin.com/UmiUQq0F">htt= p://pastebin.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 htt= p://pastebin.com/aXiamY15


Regards,
Ramesh


_______________________________________________
Users mailing list
U= sers(a)ovirt.org
http://lists.ovirt.org/m= ailman/listinfo/users



--------------000009040208060202040809-- --===============2253974178033794051== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wMDAwMDkwNDAyMDgwNjAyMDIwNDA4MDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CgoKT24gMDkvMjIvMjAxNSAwNTo0MyBQTSwgQWxhc3RhaXIgTmVpbCB3cm90ZToKPiB3aGF0IGFy ZSB0aGUgZ2x1c3Rlci1xdW9ydW0tdHlwZSBhbmQgZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlv IAo+ICBzZXR0aW5ncyBvbiB0aGUgdm9sdW1lPwo+CgoqY2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5 cGUqOnNlcnZlcgoqY2x1c3Rlci5xdW9ydW0tdHlwZSo6YXV0bwoqZ2x1c3Rlci5zZXJ2ZXItcXVv cnVtLXJhdGlvIGlzIG5vdCBzZXQuKgoKT25lIGJyaWNrIHByb2Nlc3MgaXMgcHVycG9zZWZ1bGx5 IGtpbGxlZCAgYnV0IHJlbWFpbmluZyB0d28gYnJpY2tzIGFyZSAKdXAgYW5kIHJ1bm5pbmcuCgpS ZWdhcmRzLApSYW1lc2gKCj4gT24gMjIgU2VwdGVtYmVyIDIwMTUgYXQgMDY6MjQsIFJhbWVzaCBO YWNoaW11dGh1IDxybmFjaGltdUByZWRoYXQuY29tIAo+IDxtYWlsdG86cm5hY2hpbXVAcmVkaGF0 LmNvbT4+IHdyb3RlOgo+Cj4gICAgIEhpLAo+Cj4gICAgICAgIEkgYW0gbm90IGFibGUgdG8gcmVz dW1lIGEgVk0gd2hpY2ggd2FzIHBhdXNlZCBiZWNhdXNlIG9mCj4gICAgIGdsdXN0ZXIgY2xpZW50 IHF1b3J1bSBpc3N1ZS4gSGVyZSBpcyB3aGF0IGhhcHBlbmVkIGluIG15IHNldHVwLgo+Cj4gICAg IDEuIENyZWF0ZWQgYSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWluIHdoaWNoIGlzIGJhY2tlZCBieSBn bHVzdGVyCj4gICAgIHZvbHVtZSB3aXRoIHJlcGxpY2EgMy4KPiAgICAgMi4gS2lsbGVkIG9uZSBi cmljayBwcm9jZXNzLiBTbyBvbmx5IHR3byBicmlja3MgYXJlIHJ1bm5pbmcgaW4KPiAgICAgcmVw bGljYSAzIHNldHVwLgo+ICAgICAzLiBDcmVhdGVkIHR3byBWTXMKPiAgICAgNC4gU3RhcnRlZCBz b21lIElPIHVzaW5nIGZpbyBvbiBib3RoIG9mIHRoZSBWTXMKPiAgICAgNS4gQWZ0ZXIgc29tZSB0 aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluIGdsdXN0ZXIgbW91bnQgYW5kCj4gICAgIFZN cyBtb3ZlZCB0byBwYXVzZWQgc3RhdGUuCj4gICAgICAgICAgICAgICIgc2VydmVyIDEwLjcwLjQ1 LjE3OjQ5MjE3IDxodHRwOi8vMTAuNzAuNDUuMTc6NDkyMTc+IGhhcwo+ICAgICBub3QgcmVzcG9u ZGVkIGluIHRoZSBsYXN0IDQyIHNlY29uZHMsIGRpc2Nvbm5lY3RpbmcuIgo+ICAgICAgICAgICAi dm1zdG9yZS1yZXBsaWNhdGUtMDogZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2 Ogo+ICAgICBGYWlsaW5nIFdSSVRFIGFzIHF1b3J1bSBpcyBub3QgbWV0Igo+ICAgICAgICAgICBt b3JlIGdsdXN0ZXIgbW91bnQgbG9ncyBhdCBodHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBGCj4g ICAgIDYuIEFmdGVyIHNvbWUgdGltZSBnbHVzdGVyIHF1b3J1bSBpcyBhY3RpdmUgYW5kIEkgYW0g YWJsZSB0byB3cml0ZQo+ICAgICB0aGUgdGhlIGdsdXN0ZXIgZmlsZSBzeXN0ZW0uCj4gICAgIDcu IFdoZW4gSSB0cnkgdG8gcmVzdW1lIHRoZSBWTSBpdCBkb2Vzbid0IHdvcmsgYW5kIEkgZ290IGZv bGxvd2luZwo+ICAgICBlcnJvciBpbiB2ZHNtIGxvZy4KPiAgICAgaHR0cDovL3Bhc3RlYmluLmNv bS9hWGlhbVkxNQo+Cj4KPiAgICAgUmVnYXJkcywKPiAgICAgUmFtZXNoCj4KPgo+ICAgICBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+ICAgICBVc2VycyBt YWlsaW5nIGxpc3QKPiAgICAgVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3Jn Pgo+ICAgICBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPgo+ CgoKLS0tLS0tLS0tLS0tLS0wMDAwMDkwNDAyMDgwNjAyMDIwNDA4MDkKQ29udGVudC1UeXBlOiB0 ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJpdAoK PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRm LTgiIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSI+CiAgPC9oZWFkPgogIDxib2R5IHRleHQ9IiMw MDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGJyPgogICAgPGJyPgogICAgPGRpdiBjbGFz cz0ibW96LWNpdGUtcHJlZml4Ij5PbiAwOS8yMi8yMDE1IDA1OjQzIFBNLCBBbGFzdGFpciBOZWls CiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUKY2l0ZT0ibWlkOkNB K1NhcndwVXNpTno1TXNVejY2TFltRXVhLUc9UDV2X21iek9Na2lFckpaV3dkRTJNUUBtYWlsLmdt YWlsLmNvbSIKICAgICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPndoYXQgYXJl IHRoZcKgZ2x1c3Rlci1xdW9ydW0tdHlwZQogICAgICAgIGFuZMKgZ2x1c3Rlci5zZXJ2ZXItcXVv cnVtLXJhdGlvIMKgc2V0dGluZ3Mgb24gdGhlIHZvbHVtZT88L2Rpdj4KICAgICAgPGRpdiBjbGFz cz0iZ21haWxfZXh0cmEiPjxicj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8 YnI+CiAgICA8ZGl2IHN0eWxlPSJvdXRsaW5lLXN0eWxlOm5vbmU7IiBfX2d3dF9jZWxsPSJjZWxs LWd3dC11aWQtODkwNDkiPgogICAgICA8ZGl2IGNsYXNzPSIiIHN0eWxlPSJvdmVyZmxvdzogaGlk ZGVuOyB0ZXh0LW92ZXJmbG93OiBlbGxpcHNpczsKICAgICAgICB3aGl0ZS1zcGFjZTogbm93cmFw OyIKICAgICAgICBpZD0iU3ViVGFiVm9sdW1lUGFyYW1ldGVyVmlld190YWJsZV9jb250ZW50X2Nv bDFfcm93OSI+PGI+Y2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+OnNlcnZlcjxicj4KICAg ICAgICA8ZGl2IHRpdGxlPSIiIHRhYmluZGV4PSIwIiBzdHlsZT0ib3V0bGluZS1zdHlsZTogbm9u ZTsiCiAgICAgICAgICBfX2d3dF9jZWxsPSJjZWxsLWd3dC11aWQtODkxODciPgogICAgICAgICAg PGRpdiBjbGFzcz0iIiBzdHlsZT0ib3ZlcmZsb3c6IGhpZGRlbjsgdGV4dC1vdmVyZmxvdzoKICAg ICAgICAgICAgZWxsaXBzaXM7IHdoaXRlLXNwYWNlOiBub3dyYXA7IgogICAgICAgICAgICBpZD0i U3ViVGFiVm9sdW1lUGFyYW1ldGVyVmlld190YWJsZV9jb250ZW50X2NvbDFfcm93MTMiPjxiPmNs dXN0ZXIucXVvcnVtLXR5cGU8L2I+OmF1dG88YnI+CiAgICAgICAgICAgIDxiPmdsdXN0ZXIuc2Vy dmVyLXF1b3J1bS1yYXRpbyBpcyBub3Qgc2V0LjwvYj48YnI+CiAgICAgICAgICAgIDxicj4KICAg ICAgICAgIDwvZGl2PgogICAgICAgIDwvZGl2PgogICAgICAgIE9uZSBicmljayBwcm9jZXNzIGlz IHB1cnBvc2VmdWxseSBraWxsZWTCoCBidXQgcmVtYWluaW5nIHR3bwogICAgICAgIGJyaWNrcyBh cmUgdXAgYW5kIHJ1bm5pbmcuPGJyPgogICAgICAgIDxicj4KICAgICAgICBSZWdhcmRzLDxicj4K ICAgICAgICBSYW1lc2g8YnI+CiAgICAgIDwvZGl2PgogICAgPC9kaXY+CiAgICA8YnI+CiAgICA8 YmxvY2txdW90ZQpjaXRlPSJtaWQ6Q0ErU2Fyd3BVc2lOejVNc1V6NjZMWW1FdWEtRz1QNXZfbWJ6 T01raUVySlpXd2RFMk1RQG1haWwuZ21haWwuY29tIgogICAgICB0eXBlPSJjaXRlIj4KICAgICAg PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3Rl Ij5PbiAyMiBTZXB0ZW1iZXIgMjAxNSBhdCAwNjoyNCwgUmFtZXNoCiAgICAgICAgICBOYWNoaW11 dGh1IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIgogICAgICAg ICAgICAgIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+ cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgd3JvdGU6PGJyPgog ICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAg MCAwCiAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1s ZWZ0OjFleCI+CiAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZG RiI+IEhpLDxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgwqDCoCBJIGFtIG5v dCBhYmxlIHRvIHJlc3VtZSBhIFZNIHdoaWNoIHdhcyBwYXVzZWQgYmVjYXVzZQogICAgICAgICAg ICAgIG9mIGdsdXN0ZXIgY2xpZW50IHF1b3J1bSBpc3N1ZS4gSGVyZSBpcyB3aGF0IGhhcHBlbmVk IGluCiAgICAgICAgICAgICAgbXkgc2V0dXAuIDxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAg ICAgICAgICAgMS4gQ3JlYXRlZCBhIGdsdXN0ZXIgc3RvcmFnZSBkb21haW4gd2hpY2ggaXMgYmFj a2VkIGJ5CiAgICAgICAgICAgICAgZ2x1c3RlciB2b2x1bWUgd2l0aCByZXBsaWNhIDMuIDxicj4K ICAgICAgICAgICAgICAyLiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkgdHdvIGJy aWNrcyBhcmUKICAgICAgICAgICAgICBydW5uaW5nIGluIHJlcGxpY2EgMyBzZXR1cC48YnI+CiAg ICAgICAgICAgICAgMy4gQ3JlYXRlZCB0d28gVk1zPGJyPgogICAgICAgICAgICAgIDQuIFN0YXJ0 ZWQgc29tZSBJTyB1c2luZyBmaW8gb24gYm90aCBvZiB0aGUgVk1zPGJyPgogICAgICAgICAgICAg IDUuIEFmdGVyIHNvbWUgdGltZSBnb3QgdGhlIGZvbGxvd2luZyBlcnJvciBpbiBnbHVzdGVyCiAg ICAgICAgICAgICAgbW91bnQgYW5kIFZNcyBtb3ZlZCB0byBwYXVzZWQgc3RhdGUuPGJyPgogICAg ICAgICAgICAgIMKgwqDCoMKgwqDCoMKgwqAgIiA8c3BhbgpzdHlsZT0iY29sb3I6cmdiKDUxLDUx LDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1zdHlsZTpub3Jt YWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6 bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4dC1pbmRlbnQ6MHB4 O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7 ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3VuZC1jb2xvcjpyZ2Io MjU1LDI1NSwyNTUpIj5zZXJ2ZXIKCiAgICAgICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9 InRydWUiCiAgICAgICAgICAgICAgICAgIGhyZWY9Imh0dHA6Ly8xMC43MC40NS4xNzo0OTIxNyIg dGFyZ2V0PSJfYmxhbmsiPjEwLjcwLjQ1LjE3OjQ5MjE3PC9hPgogICAgICAgICAgICAgICAgaGFz IG5vdCByZXNwb25kZWQgaW4gdGhlIGxhc3QgNDIgc2Vjb25kcywKICAgICAgICAgICAgICAgIGRp c2Nvbm5lY3RpbmcuIjxicj4KICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgIjwvc3Bhbj48c3Bh bgpzdHlsZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1z aXplOjExcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdo dDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFs aWduOmxlZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6 bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5v bmU7YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj48c3BhbgpzdHlsZT0iY29sb3I6 cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1z dHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVy LXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4dC1p bmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQtc3Bh Y2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3VuZC1j b2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj52bXN0b3JlLXJlcGxpY2F0ZS0wOgoKICAgICAgICAgICAg ICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OiBGYWlsaW5nIFdSSVRF IGFzCiAgICAgICAgICAgICAgICAgIHF1b3J1bSBpcyBub3QgbWV0PC9zcGFuPiI8YnI+CiAgICAg ICAgICAgICAgICDCoMKgwqDCoMKgIG1vcmUgZ2x1c3RlciBtb3VudCBsb2dzIGF0IDxhCiAgICAg ICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgaHJl Zj0iaHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRiIgdGFyZ2V0PSJfYmxhbmsiPjxhIGNsYXNz PSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFx MEYiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEY8L2E+PC9hPjxicj4KICAgICAgICAgICAg ICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGdsdXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQg SQogICAgICAgICAgICAgIGFtIGFibGUgdG8gd3JpdGUgdGhlIHRoZSBnbHVzdGVyIGZpbGUgc3lz dGVtLjxicj4KICAgICAgICAgICAgICA3LiBXaGVuIEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQg ZG9lc24ndCB3b3JrIGFuZCBJIGdvdAogICAgICAgICAgICAgIGZvbGxvd2luZyBlcnJvciBpbiB2 ZHNtIGxvZy48YnI+CiAgICAgICAgICAgICAgwqDCoMKgwqDCoCA8YSBtb3otZG8tbm90LXNlbmQ9 InRydWUiCiAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1 IiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Bhc3RlYmluLmNvbS9hWGlhbVkxNTwvYT48YnI+CiAg ICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICBSZWdhcmRz LDxicj4KICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICA8L2Rpdj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgICBfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAgICAgVXNlcnMgbWFp bGluZyBsaXN0PGJyPgogICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhyZWY9 Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAg ICAgIDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICBocmVmPSJodHRwOi8v bGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAgcmVs PSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWls bWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgPC9i bG9ja3F1b3RlPgogICAgICAgIDwvZGl2PgogICAgICAgIDxicj4KICAgICAgPC9kaXY+CiAgICA8 L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTAw MDAwOTA0MDIwODA2MDIwMjA0MDgwOS0tCg== --===============2253974178033794051==-- From ajneil.tech at gmail.com Tue Sep 22 08:27:31 2015 Content-Type: multipart/mixed; boundary="===============5088681294855979610==" MIME-Version: 1.0 From: Alastair Neil To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Tue, 22 Sep 2015 08:27:30 -0400 Message-ID: In-Reply-To: 560148B5.5010909@redhat.com --===============5088681294855979610== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable You need to set the gluster.server-quorum-ratio to 51% On 22 September 2015 at 08:25, Ramesh Nachimuthu wrote: > > > On 09/22/2015 05:43 PM, Alastair Neil wrote: > > what are the gluster-quorum-type and gluster.server-quorum-ratio settings > on the volume? > > > *cluster.server-quorum-type*:server > *cluster.quorum-type*:auto > *gluster.server-quorum-ratio is not set.* > > One brick process is purposefully killed but remaining two bricks are up > and running. > > Regards, > Ramesh > > On 22 September 2015 at 06:24, Ramesh Nachimuthu > wrote: > >> Hi, >> >> I am not able to resume a VM which was paused because of gluster >> client quorum issue. Here is what happened in my setup. >> >> 1. Created a gluster storage domain which is backed by gluster volume >> with replica 3. >> 2. Killed one brick process. So only two bricks are running in replica 3 >> setup. >> 3. Created two VMs >> 4. Started some IO using fio on both of the VMs >> 5. After some time got the following error in gluster mount and VMs moved >> to paused state. >> " server 10.70.45.17:49217 has not responded in the last 42 >> seconds, disconnecting." >> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: >> Failing WRITE as quorum is not met" >> more gluster mount logs at >> http://pastebin.com/UmiUQq0F >> 6. After some time gluster quorum is active and I am able to write the >> the gluster file system. >> 7. When I try to resume the VM it doesn't work and I got following error >> in vdsm log. >> http://pastebin.com/aXiamY15 >> >> >> Regards, >> Ramesh >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > --===============5088681294855979610== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+WW91IG5lZWQgdG8gc2V0IHRoZSBnbHVzdGVyLnNlcnZlci1xdW9ydW0t cmF0aW8gdG8gNTElPC9kaXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48ZGl2IGNsYXNz PSJnbWFpbF9xdW90ZSI+T24gMjIgU2VwdGVtYmVyIDIwMTUgYXQgMDg6MjUsIFJhbWVzaCBOYWNo aW11dGh1IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhh dC5jb20iIHRhcmdldD0iX2JsYW5rIj5ybmFjaGltdUByZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+ IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46 MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4K ICAKICAgIAogIAogIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+PHNwYW4g Y2xhc3M9IiI+CiAgICA8YnI+CiAgICA8YnI+CiAgICA8ZGl2Pk9uIDA5LzIyLzIwMTUgMDU6NDMg UE0sIEFsYXN0YWlyIE5laWwKICAgICAgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2tx dW90ZSB0eXBlPSJjaXRlIj4KICAgICAgPGRpdiBkaXI9Imx0ciI+d2hhdCBhcmUgdGhlwqBnbHVz dGVyLXF1b3J1bS10eXBlCiAgICAgICAgYW5kwqBnbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0aW8g wqBzZXR0aW5ncyBvbiB0aGUgdm9sdW1lPzwvZGl2PgogICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9l eHRyYSI+PGJyPgogICAgICA8L2Rpdj4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxicj4KICAgIDwv c3Bhbj48ZGl2IHN0eWxlPSJvdXRsaW5lLXN0eWxlOm5vbmUiPgogICAgICA8ZGl2IHN0eWxlPSJv dmVyZmxvdzpoaWRkZW47dGV4dC1vdmVyZmxvdzplbGxpcHNpczt3aGl0ZS1zcGFjZTpub3dyYXAi PjxiPmNsdXN0ZXIuc2VydmVyLXF1b3J1bS10eXBlPC9iPjpzZXJ2ZXI8YnI+CiAgICAgICAgPGRp diB0aXRsZT0iIiBzdHlsZT0ib3V0bGluZS1zdHlsZTpub25lIj4KICAgICAgICAgIDxkaXYgc3R5 bGU9Im92ZXJmbG93OmhpZGRlbjt0ZXh0LW92ZXJmbG93OmVsbGlwc2lzO3doaXRlLXNwYWNlOm5v d3JhcCI+PGI+Y2x1c3Rlci5xdW9ydW0tdHlwZTwvYj46YXV0bzxicj4KICAgICAgICAgICAgPGI+ Z2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvIGlzIG5vdCBzZXQuPC9iPjxicj4KICAgICAgICAg ICAgPGJyPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgPC9kaXY+CiAgICAgICAgT25lIGJyaWNr IHByb2Nlc3MgaXMgcHVycG9zZWZ1bGx5IGtpbGxlZMKgIGJ1dCByZW1haW5pbmcgdHdvCiAgICAg ICAgYnJpY2tzIGFyZSB1cCBhbmQgcnVubmluZy48YnI+CiAgICAgICAgPGJyPgogICAgICAgIFJl Z2FyZHMsPGJyPgogICAgICAgIFJhbWVzaDxicj4KICAgICAgPC9kaXY+CiAgICA8L2Rpdj48c3Bh biBjbGFzcz0iIj4KICAgIDxicj4KICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICA8 ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUi Pk9uIDIyIFNlcHRlbWJlciAyMDE1IGF0IDA2OjI0LCBSYW1lc2gKICAgICAgICAgIE5hY2hpbXV0 aHUgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNv bSIgdGFyZ2V0PSJfYmxhbmsiPnJuYWNoaW11QHJlZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj4KICAg ICAgICAgIHdyb3RlOjxicj4KICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90 ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3Bh ZGRpbmctbGVmdDoxZXgiPgogICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9 IiNGRkZGRkYiPiBIaSw8YnI+CiAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgIMKgwqAg SSBhbSBub3QgYWJsZSB0byByZXN1bWUgYSBWTSB3aGljaCB3YXMgcGF1c2VkIGJlY2F1c2UKICAg ICAgICAgICAgICBvZiBnbHVzdGVyIGNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBo YXBwZW5lZCBpbgogICAgICAgICAgICAgIG15IHNldHVwLiA8YnI+CiAgICAgICAgICAgICAgPGJy PgogICAgICAgICAgICAgIDEuIENyZWF0ZWQgYSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWluIHdoaWNo IGlzIGJhY2tlZCBieQogICAgICAgICAgICAgIGdsdXN0ZXIgdm9sdW1lIHdpdGggcmVwbGljYSAz LiA8YnI+CiAgICAgICAgICAgICAgMi4gS2lsbGVkIG9uZSBicmljayBwcm9jZXNzLiBTbyBvbmx5 IHR3byBicmlja3MgYXJlCiAgICAgICAgICAgICAgcnVubmluZyBpbiByZXBsaWNhIDMgc2V0dXAu PGJyPgogICAgICAgICAgICAgIDMuIENyZWF0ZWQgdHdvIFZNczxicj4KICAgICAgICAgICAgICA0 LiBTdGFydGVkIHNvbWUgSU8gdXNpbmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNczxicj4KICAgICAg ICAgICAgICA1LiBBZnRlciBzb21lIHRpbWUgZ290IHRoZSBmb2xsb3dpbmcgZXJyb3IgaW4gZ2x1 c3RlcgogICAgICAgICAgICAgIG1vdW50IGFuZCBWTXMgbW92ZWQgdG8gcGF1c2VkIHN0YXRlLjxi cj4KICAgICAgICAgICAgICDCoMKgwqDCoMKgwqDCoMKgICZxdW90OyA8c3BhbiBzdHlsZT0iY29s b3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9u dC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0 dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4 dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQt c3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3Vu ZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj5zZXJ2ZXIKCiAgICAgICAgICAgICAgICA8YSBocmVm PSJodHRwOi8vMTAuNzAuNDUuMTc6NDkyMTciIHRhcmdldD0iX2JsYW5rIj4xMC43MC40NS4xNzo0 OTIxNzwvYT4KICAgICAgICAgICAgICAgIGhhcyBub3QgcmVzcG9uZGVkIGluIHRoZSBsYXN0IDQy IHNlY29uZHMsCiAgICAgICAgICAgICAgICBkaXNjb25uZWN0aW5nLiZxdW90Ozxicj4KICAgICAg ICAgICAgICAgIMKgwqDCoMKgwqAgJnF1b3Q7PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpyZ2Io NTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250LXN0eWxl Om5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3Bh Y2luZzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0LWluZGVu dDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1zcGFjaW5n OjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5kLWNvbG9y OnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuIHN0eWxlPSJjb2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQt ZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZh cmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3JtYWw7bGlu ZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFu c2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1zcGFjaW5nOjBweDtkaXNwbGF5Omlu bGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1 NSkiPnZtc3RvcmUtcmVwbGljYXRlLTA6CgogICAgICAgICAgICAgICAgICBlMTZkMWU0MC0yYjZl LTRmMTktOTc3ZC1lMDk5ZjQ2NWRmYzY6IEZhaWxpbmcgV1JJVEUgYXMKICAgICAgICAgICAgICAg ICAgcXVvcnVtIGlzIG5vdCBtZXQ8L3NwYW4+JnF1b3Q7PGJyPgogICAgICAgICAgICAgICAgwqDC oMKgwqDCoCBtb3JlIGdsdXN0ZXIgbW91bnQgbG9ncyBhdCA8YSBocmVmPSJodHRwOi8vcGFzdGVi aW4uY29tL1VtaVVRcTBGIiB0YXJnZXQ9Il9ibGFuayI+PC9hPjxhIGhyZWY9Imh0dHA6Ly9wYXN0 ZWJpbi5jb20vVW1pVVFxMEYiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vcGFzdGViaW4uY29tL1Vt aVVRcTBGPC9hPjxicj4KICAgICAgICAgICAgICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGds dXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQgSQogICAgICAgICAgICAgIGFtIGFibGUgdG8gd3Jp dGUgdGhlIHRoZSBnbHVzdGVyIGZpbGUgc3lzdGVtLjxicj4KICAgICAgICAgICAgICA3LiBXaGVu IEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQgZG9lc24mIzM5O3Qgd29yayBhbmQgSSBnb3QKICAg ICAgICAgICAgICBmb2xsb3dpbmcgZXJyb3IgaW4gdmRzbSBsb2cuPGJyPgogICAgICAgICAgICAg IMKgwqDCoMKgwqAgPGEgaHJlZj0iaHR0cDovL3Bhc3RlYmluLmNvbS9hWGlhbVkxNSIgdGFyZ2V0 PSJfYmxhbmsiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTU8L2E+PGJyPgogICAgICAgICAg ICAgIDxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgUmVnYXJkcyw8YnI+CiAg ICAgICAgICAgICAgUmFtZXNoPGJyPgogICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgPC9k aXY+CiAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX188YnI+CiAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlz dDxicj4KICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0 PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAgICAgIDxhIGhyZWY9Imh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVy cmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgPC9ibG9ja3F1b3Rl PgogICAgICAgIDwvZGl2PgogICAgICAgIDxicj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVv dGU+CiAgICA8YnI+CiAgPC9zcGFuPjwvZGl2PgoKPC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rp dj4K --===============5088681294855979610==-- From rnachimu at redhat.com Wed Sep 23 08:38:06 2015 Content-Type: multipart/mixed; boundary="===============8675196981788065954==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Wed, 23 Sep 2015 18:08:02 +0530 Message-ID: <56029D2A.9050004@redhat.com> In-Reply-To: CA+SarwoorU3LWG6+sR-tJ1BEbQa1k4WRrRXkaXy3z-EPRrz7Uw@mail.gmail.com --===============8675196981788065954== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------040609060800000303090805 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit On 09/22/2015 05:57 PM, Alastair Neil wrote: > You need to set the gluster.server-quorum-ratio to 51% > I did that. But still I am facing the same issue. VM get paused when I = do some I/O using fio on some disks backed by gluster. I am not able to = resume the VM after this. Now only way is to bring down the VM and run = again. It runs successfully on the same host without any issue. Regards, Ramesh > On 22 September 2015 at 08:25, Ramesh Nachimuthu > wrote: > > > > On 09/22/2015 05:43 PM, Alastair Neil wrote: >> what are the gluster-quorum-type and gluster.server-quorum-ratio >> settings on the volume? >> > > *cluster.server-quorum-type*:server > *cluster.quorum-type*:auto > *gluster.server-quorum-ratio is not set.* > > One brick process is purposefully killed but remaining two bricks > are up and running. > > Regards, > Ramesh > >> On 22 September 2015 at 06:24, Ramesh Nachimuthu >> > wrote: >> >> Hi, >> >> I am not able to resume a VM which was paused because of >> gluster client quorum issue. Here is what happened in my setup. >> >> 1. Created a gluster storage domain which is backed by >> gluster volume with replica 3. >> 2. Killed one brick process. So only two bricks are running >> in replica 3 setup. >> 3. Created two VMs >> 4. Started some IO using fio on both of the VMs >> 5. After some time got the following error in gluster mount >> and VMs moved to paused state. >> " server 10.70.45.17:49217 >> has not responded in the last 42 >> seconds, disconnecting." >> "vmstore-replicate-0: >> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum >> is not met" >> more gluster mount logs at http://pastebin.com/UmiUQq0F >> 6. After some time gluster quorum is active and I am able to >> write the the gluster file system. >> 7. When I try to resume the VM it doesn't work and I got >> following error in vdsm log. >> http://pastebin.com/aXiamY15 >> >> >> Regards, >> Ramesh >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > --------------040609060800000303090805 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%


I did that. But still I am facing the same issue. VM get paused when I do some I/O using fio on some disks backed by gluster. I am not able to resume the VM after this. Now only way is to bring down the VM and run again. It runs successfully on the same host without any issue.

Regards,
Ramesh

On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnach= imu(a)redhat.com> wrote:


On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the=C2=A0gluster-quorum-type and=C2=A0gluster.server-quorum-ratio =C2=A0settings on = the volume?
cluster.server-quorum-type:server
cluster.quorum-type:auto
gluster.server-quorum-ratio is not set.

One brick process is purposefully killed=C2=A0 but remaining two bricks are up and running.

Regards,
Ramesh

On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.com&= gt; wrote:
Hi,

=C2=A0=C2=A0 I am not able to resume a VM which w= as paused because of gluster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 more gluster mou= nt logs at http://pastebin.com/UmiUQq0F<= /a>
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://pastebin.com/aXiamY15<= /a>


Regards,
Ramesh


_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists= .ovirt.org/mailman/listinfo/users





--------------040609060800000303090805-- --===============8675196981788065954== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNDA2MDkwNjA4MDAwMDAzMDMwOTA4MDUKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CgoKT24gMDkvMjIvMjAxNSAwNTo1NyBQTSwgQWxhc3RhaXIgTmVpbCB3cm90ZToKPiBZb3UgbmVl ZCB0byBzZXQgdGhlIGdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyB0byA1MSUKPgoKSSBkaWQg dGhhdC4gQnV0IHN0aWxsIEkgYW0gZmFjaW5nIHRoZSBzYW1lIGlzc3VlLiBWTSBnZXQgcGF1c2Vk IHdoZW4gSSAKZG8gc29tZSBJL08gdXNpbmcgZmlvIG9uIHNvbWUgZGlza3MgYmFja2VkIGJ5IGds dXN0ZXIuIEkgYW0gbm90IGFibGUgdG8gCnJlc3VtZSB0aGUgVk0gYWZ0ZXIgdGhpcy4gTm93IG9u bHkgd2F5IGlzIHRvIGJyaW5nIGRvd24gdGhlIFZNIGFuZCBydW4gCmFnYWluLiBJdCBydW5zIHN1 Y2Nlc3NmdWxseSBvbiB0aGUgc2FtZSBob3N0IHdpdGhvdXQgYW55IGlzc3VlLgoKUmVnYXJkcywK UmFtZXNoCgo+IE9uIDIyIFNlcHRlbWJlciAyMDE1IGF0IDA4OjI1LCBSYW1lc2ggTmFjaGltdXRo dSA8cm5hY2hpbXVAcmVkaGF0LmNvbSAKPiA8bWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20+PiB3 cm90ZToKPgo+Cj4KPiAgICAgT24gMDkvMjIvMjAxNSAwNTo0MyBQTSwgQWxhc3RhaXIgTmVpbCB3 cm90ZToKPj4gICAgIHdoYXQgYXJlIHRoZSBnbHVzdGVyLXF1b3J1bS10eXBlIGFuZCBnbHVzdGVy LnNlcnZlci1xdW9ydW0tcmF0aW8KPj4gICAgICBzZXR0aW5ncyBvbiB0aGUgdm9sdW1lPwo+Pgo+ Cj4gICAgICpjbHVzdGVyLnNlcnZlci1xdW9ydW0tdHlwZSo6c2VydmVyCj4gICAgICpjbHVzdGVy LnF1b3J1bS10eXBlKjphdXRvCj4gICAgICpnbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0aW8gaXMg bm90IHNldC4qCj4KPiAgICAgT25lIGJyaWNrIHByb2Nlc3MgaXMgcHVycG9zZWZ1bGx5IGtpbGxl ZCAgYnV0IHJlbWFpbmluZyB0d28gYnJpY2tzCj4gICAgIGFyZSB1cCBhbmQgcnVubmluZy4KPgo+ ICAgICBSZWdhcmRzLAo+ICAgICBSYW1lc2gKPgo+PiAgICAgT24gMjIgU2VwdGVtYmVyIDIwMTUg YXQgMDY6MjQsIFJhbWVzaCBOYWNoaW11dGh1Cj4+ICAgICA8cm5hY2hpbXVAcmVkaGF0LmNvbSA8 bWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20+PiB3cm90ZToKPj4KPj4gICAgICAgICBIaSwKPj4K Pj4gICAgICAgICAgICBJIGFtIG5vdCBhYmxlIHRvIHJlc3VtZSBhIFZNIHdoaWNoIHdhcyBwYXVz ZWQgYmVjYXVzZSBvZgo+PiAgICAgICAgIGdsdXN0ZXIgY2xpZW50IHF1b3J1bSBpc3N1ZS4gSGVy ZSBpcyB3aGF0IGhhcHBlbmVkIGluIG15IHNldHVwLgo+Pgo+PiAgICAgICAgIDEuIENyZWF0ZWQg YSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWluIHdoaWNoIGlzIGJhY2tlZCBieQo+PiAgICAgICAgIGds dXN0ZXIgdm9sdW1lIHdpdGggcmVwbGljYSAzLgo+PiAgICAgICAgIDIuIEtpbGxlZCBvbmUgYnJp Y2sgcHJvY2Vzcy4gU28gb25seSB0d28gYnJpY2tzIGFyZSBydW5uaW5nCj4+ICAgICAgICAgaW4g cmVwbGljYSAzIHNldHVwLgo+PiAgICAgICAgIDMuIENyZWF0ZWQgdHdvIFZNcwo+PiAgICAgICAg IDQuIFN0YXJ0ZWQgc29tZSBJTyB1c2luZyBmaW8gb24gYm90aCBvZiB0aGUgVk1zCj4+ICAgICAg ICAgNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluIGdsdXN0ZXIg bW91bnQKPj4gICAgICAgICBhbmQgVk1zIG1vdmVkIHRvIHBhdXNlZCBzdGF0ZS4KPj4gICAgICAg ICAgICAgICAgICAiIHNlcnZlciAxMC43MC40NS4xNzo0OTIxNwo+PiAgICAgICAgIDxodHRwOi8v MTAuNzAuNDUuMTc6NDkyMTc+IGhhcyBub3QgcmVzcG9uZGVkIGluIHRoZSBsYXN0IDQyCj4+ICAg ICAgICAgc2Vjb25kcywgZGlzY29ubmVjdGluZy4iCj4+ICAgICAgICAgICAgICAgInZtc3RvcmUt cmVwbGljYXRlLTA6Cj4+ICAgICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVk ZmM2OiBGYWlsaW5nIFdSSVRFIGFzIHF1b3J1bQo+PiAgICAgICAgIGlzIG5vdCBtZXQiCj4+ICAg ICAgICAgICAgICAgbW9yZSBnbHVzdGVyIG1vdW50IGxvZ3MgYXQgaHR0cDovL3Bhc3RlYmluLmNv bS9VbWlVUXEwRgo+PiAgICAgICAgIDYuIEFmdGVyIHNvbWUgdGltZSBnbHVzdGVyIHF1b3J1bSBp cyBhY3RpdmUgYW5kIEkgYW0gYWJsZSB0bwo+PiAgICAgICAgIHdyaXRlIHRoZSB0aGUgZ2x1c3Rl ciBmaWxlIHN5c3RlbS4KPj4gICAgICAgICA3LiBXaGVuIEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0g aXQgZG9lc24ndCB3b3JrIGFuZCBJIGdvdAo+PiAgICAgICAgIGZvbGxvd2luZyBlcnJvciBpbiB2 ZHNtIGxvZy4KPj4gICAgICAgICBodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1Cj4+Cj4+Cj4+ ICAgICAgICAgUmVnYXJkcywKPj4gICAgICAgICBSYW1lc2gKPj4KPj4KPj4gICAgICAgICBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+PiAgICAgICAgIFVz ZXJzIG1haWxpbmcgbGlzdAo+PiAgICAgICAgIFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRvOlVzZXJz QG92aXJ0Lm9yZz4KPj4gICAgICAgICBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnMKPj4KPj4KPgo+CgoKLS0tLS0tLS0tLS0tLS0wNDA2MDkwNjA4MDAwMDAzMDMw OTA4MDUKQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFu c2Zlci1FbmNvZGluZzogOGJpdAoKPGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBjb250ZW50PSJ0 ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSI+CiAgPC9o ZWFkPgogIDxib2R5IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGJyPgog ICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAwOS8yMi8yMDE1IDA1 OjU3IFBNLCBBbGFzdGFpciBOZWlsCiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJs b2NrcXVvdGUKY2l0ZT0ibWlkOkNBK1Nhcndvb3JVM0xXRzYrc1ItdEoxQkViUWExazRXUnJSWGth WHkzei1FUFJyejdVd0BtYWlsLmdtYWlsLmNvbSIKICAgICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxk aXYgZGlyPSJsdHIiPllvdSBuZWVkIHRvIHNldCB0aGUgZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJh dGlvIHRvCiAgICAgICAgNTElPC9kaXY+CiAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48 YnI+CiAgICAgIDwvZGl2PgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogICAgSSBkaWQgdGhh dC4gQnV0IHN0aWxsIEkgYW0gZmFjaW5nIHRoZSBzYW1lIGlzc3VlLiBWTSBnZXQgcGF1c2VkIHdo ZW4KICAgIEkgZG8gc29tZSBJL08gdXNpbmcgZmlvIG9uIHNvbWUgZGlza3MgYmFja2VkIGJ5IGds dXN0ZXIuIEkgYW0gbm90CiAgICBhYmxlIHRvIHJlc3VtZSB0aGUgVk0gYWZ0ZXIgdGhpcy4gTm93 IG9ubHkgd2F5IGlzIHRvIGJyaW5nIGRvd24gdGhlCiAgICBWTSBhbmQgcnVuIGFnYWluLiBJdCBy dW5zIHN1Y2Nlc3NmdWxseSBvbiB0aGUgc2FtZSBob3N0IHdpdGhvdXQgYW55CiAgICBpc3N1ZS48 YnI+CiAgICA8YnI+CiAgICBSZWdhcmRzLDxicj4KICAgIFJhbWVzaDxicj4KICAgIDxicj4KICAg IDxibG9ja3F1b3RlCmNpdGU9Im1pZDpDQStTYXJ3b29yVTNMV0c2K3NSLXRKMUJFYlFhMWs0V1Jy UlhrYVh5M3otRVBScno3VXdAbWFpbC5nbWFpbC5jb20iCiAgICAgIHR5cGU9ImNpdGUiPgogICAg ICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVv dGUiPk9uIDIyIFNlcHRlbWJlciAyMDE1IGF0IDA4OjI1LCBSYW1lc2gKICAgICAgICAgIE5hY2hp bXV0aHUgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAg ICAgICAgICAgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5r Ij5ybmFjaGltdUByZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+CiAgICAgICAgICB3cm90ZTo8YnI+ CiAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46 MCAwIDAKICAgICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5n LWxlZnQ6MWV4Ij4KICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZG RkZGIj48c3BhbiBjbGFzcz0iIj4gPGJyPgogICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAg ICAgICAgPGRpdj5PbiAwOS8yMi8yMDE1IDA1OjQzIFBNLCBBbGFzdGFpciBOZWlsIHdyb3RlOjxi cj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPndoYXQgYXJlIHRoZcKg Z2x1c3Rlci1xdW9ydW0tdHlwZQogICAgICAgICAgICAgICAgICAgIGFuZMKgZ2x1c3Rlci5zZXJ2 ZXItcXVvcnVtLXJhdGlvIMKgc2V0dGluZ3Mgb24gdGhlCiAgICAgICAgICAgICAgICAgICAgdm9s dW1lPzwvZGl2PgogICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJy PgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4K ICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICA8L3NwYW4+CiAgICAgICAgICAgICAg PGRpdiBzdHlsZT0ib3V0bGluZS1zdHlsZTpub25lIj4KICAgICAgICAgICAgICAgIDxkaXYKICAg ICAgICAgICAgICAgICAgc3R5bGU9Im92ZXJmbG93OmhpZGRlbjt0ZXh0LW92ZXJmbG93OmVsbGlw c2lzO3doaXRlLXNwYWNlOm5vd3JhcCI+PGI+Y2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+ OnNlcnZlcjxicj4KICAgICAgICAgICAgICAgICAgPGRpdiB0aXRsZT0iIiBzdHlsZT0ib3V0bGlu ZS1zdHlsZTpub25lIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAg ICAgICBzdHlsZT0ib3ZlcmZsb3c6aGlkZGVuO3RleHQtb3ZlcmZsb3c6ZWxsaXBzaXM7d2hpdGUt c3BhY2U6bm93cmFwIj48Yj5jbHVzdGVyLnF1b3J1bS10eXBlPC9iPjphdXRvPGJyPgogICAgICAg ICAgICAgICAgICAgICAgPGI+Z2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvIGlzIG5vdCBzZXQu PC9iPjxicj4KICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIE9uZSBicmlj ayBwcm9jZXNzIGlzIHB1cnBvc2VmdWxseSBraWxsZWTCoCBidXQKICAgICAgICAgICAgICAgICAg cmVtYWluaW5nIHR3byBicmlja3MgYXJlIHVwIGFuZCBydW5uaW5nLjxicj4KICAgICAgICAgICAg ICAgICAgPGJyPgogICAgICAgICAgICAgICAgICBSZWdhcmRzLDxicj4KICAgICAgICAgICAgICAg ICAgUmFtZXNoPGJyPgogICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+ CiAgICAgICAgICAgICAgPHNwYW4gY2xhc3M9IiI+IDxicj4KICAgICAgICAgICAgICAgIDxibG9j a3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9l eHRyYSI+CiAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9uIDIy IFNlcHRlbWJlciAyMDE1IGF0CiAgICAgICAgICAgICAgICAgICAgICAwNjoyNCwgUmFtZXNoIE5h Y2hpbXV0aHUgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YQogICAgICAgICAgICAgICAgICAgICAgICAg IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICBocmVmPSJt YWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSIKICAgICAgICAgICAgICAgICAgICAgICAgICB0YXJn ZXQ9Il9ibGFuayI+PGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIgaHJlZj0ibWFp bHRvOnJuYWNoaW11QHJlZGhhdC5jb20iPnJuYWNoaW11QHJlZGhhdC5jb208L2E+PC9hPiZndDs8 L3NwYW4+CiAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICAgICAgICAg ICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwCiAg ICAgICAgICAgICAgICAgICAgICAgIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYwogICAgICAg ICAgICAgICAgICAgICAgICBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAgICAg ICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4gSGksPGJyPgog ICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICDC oMKgIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIGEgVk0gd2hpY2ggd2FzCiAgICAgICAgICAgICAg ICAgICAgICAgICAgcGF1c2VkIGJlY2F1c2Ugb2YgZ2x1c3RlciBjbGllbnQgcXVvcnVtIGlzc3Vl LgogICAgICAgICAgICAgICAgICAgICAgICAgIEhlcmUgaXMgd2hhdCBoYXBwZW5lZCBpbiBteSBz ZXR1cC4gPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAxLiBDcmVhdGVkIGEgZ2x1c3RlciBzdG9yYWdlIGRvbWFpbiB3aGljaCBpcwog ICAgICAgICAgICAgICAgICAgICAgICAgIGJhY2tlZCBieSBnbHVzdGVyIHZvbHVtZSB3aXRoIHJl cGxpY2EgMy4gPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDIuIEtpbGxlZCBvbmUgYnJp Y2sgcHJvY2Vzcy4gU28gb25seSB0d28KICAgICAgICAgICAgICAgICAgICAgICAgICBicmlja3Mg YXJlIHJ1bm5pbmcgaW4gcmVwbGljYSAzIHNldHVwLjxicj4KICAgICAgICAgICAgICAgICAgICAg ICAgICAzLiBDcmVhdGVkIHR3byBWTXM8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgNC4g U3RhcnRlZCBzb21lIElPIHVzaW5nIGZpbyBvbiBib3RoIG9mIHRoZQogICAgICAgICAgICAgICAg ICAgICAgICAgIFZNczxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA1LiBBZnRlciBzb21l IHRpbWUgZ290IHRoZSBmb2xsb3dpbmcgZXJyb3IgaW4KICAgICAgICAgICAgICAgICAgICAgICAg ICBnbHVzdGVyIG1vdW50IGFuZCBWTXMgbW92ZWQgdG8gcGF1c2VkIHN0YXRlLjxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgwqDCoMKgICIgPHNwYW4Kc3R5bGU9ImNvbG9y OnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZToxMXB4O2ZvbnQt c3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2xldHRl ci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0O3RleHQt aW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1hbDt3b3JkLXNw YWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpub25lO2JhY2tncm91bmQt Y29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+c2VydmVyCgoKICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgaHJlZj0iaHR0cDovLzEwLjcwLjQ1LjE3OjQ5MjE3IgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB0YXJnZXQ9Il9ibGFuayI+MTAuNzAuNDUuMTc6NDkyMTc8L2E+IGhhcwogICAgICAg ICAgICAgICAgICAgICAgICAgICAgbm90IHJlc3BvbmRlZCBpbiB0aGUgbGFzdCA0MiBzZWNvbmRz LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgZGlzY29ubmVjdGluZy4iPGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgwqDCoMKgwqDCoCAiPC9zcGFuPjxzcGFuCnN0eWxlPSJjb2xv cjpyZ2IoNTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250 LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0 ZXItc3BhY2luZzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0 LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1z cGFjaW5nOjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5k LWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuCnN0eWxlPSJjb2xvcjpyZ2IoNTEsNTEsNTEp O2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250LXN0eWxlOm5vcm1hbDtm b250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3Jt YWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0LWluZGVudDowcHg7dGV4 dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1zcGFjaW5nOjBweDtkaXNw bGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUs MjU1LDI1NSkiPnZtc3RvcmUtcmVwbGljYXRlLTA6CgoKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBGYWlsaW5nIFdSSVRFIGFzIHF1b3J1bSBpcyBub3QgbWV0PC9zcGFu PiI8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgIG1vcmUgZ2x1c3Rl ciBtb3VudCBsb2dzIGF0IDxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1u b3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cDov L3Bhc3RlYmluLmNvbS9VbWlVUXEwRiIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFy Z2V0PSJfYmxhbmsiPjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6 Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEYiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEY8L2E+ PC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0 aW1lIGdsdXN0ZXIgcXVvcnVtIGlzCiAgICAgICAgICAgICAgICAgICAgICAgICAgYWN0aXZlIGFu ZCBJIGFtIGFibGUgdG8gd3JpdGUgdGhlIHRoZSBnbHVzdGVyCiAgICAgICAgICAgICAgICAgICAg ICAgICAgZmlsZSBzeXN0ZW0uPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDcuIFdoZW4g SSB0cnkgdG8gcmVzdW1lIHRoZSBWTSBpdCBkb2Vzbid0IHdvcmsKICAgICAgICAgICAgICAgICAg ICAgICAgICBhbmQgSSBnb3QgZm9sbG93aW5nIGVycm9yIGluIHZkc20gbG9nLjxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgIDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIK ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhp YW1ZMTUiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayI+aHR0cDov L3Bhc3RlYmluLmNvbS9hWGlhbVkxNTwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAg PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAg ICAgICBSZWdhcmRzLDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICA8L2Rp dj4KICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICBf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAgICAgICAg ICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAgICAgICAgICAgICAgICAgICAg ICAgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92 aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxhIG1vei1kby1ub3Qtc2Vu ZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAgICAgICAgICAgICAg cmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9t YWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPGJy PgogICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAg IDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+ CiAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICA8YnI+CiAgICAg ICAgICAgICAgPC9zcGFuPjwvZGl2PgogICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgIDwv ZGl2PgogICAgICAgIDxicj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+ CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTA0MDYwOTA2MDgwMDAwMDMwMzA5MDgw NS0tCg== --===============8675196981788065954==-- From budic at onholyground.com Wed Sep 23 17:08:58 2015 Content-Type: multipart/mixed; boundary="===============7468391435079651325==" MIME-Version: 1.0 From: Darrell Budic To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Wed, 23 Sep 2015 16:08:49 -0500 Message-ID: In-Reply-To: 56029D2A.9050004@redhat.com --===============7468391435079651325== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 This is a known issue in overt 3.5.x and below. It=3DE2=3D80=3D99s been sol= ved =3D in the upcoming ovirt 3.6. Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D3D1172905, the =3D fix involved setting up a special cgroup for the mount, but i can=3DE2=3D80= =3D99=3D t find the exact details atm. > On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu = =3D wrote: >=3D20 >=3D20 >=3D20 > On 09/22/2015 05:57 PM, Alastair Neil wrote: >> You need to set the gluster.server-quorum-ratio to 51% >>=3D20 >=3D20 > I did that. But still I am facing the same issue. VM get paused when I = =3D do some I/O using fio on some disks backed by gluster. I am not able to =3D resume the VM after this. Now only way is to bring down the VM and run =3D again. It runs successfully on the same host without any issue. >=3D20 > Regards, > Ramesh >=3D20 >> On 22 September 2015 at 08:25, Ramesh Nachimuthu > wrote: >>=3D20 >>=3D20 >> On 09/22/2015 05:43 PM, Alastair Neil wrote: >>> what are the gluster-quorum-type and gluster.server-quorum-ratio =3D settings on the volume? >>>=3D20 >>=3D20 >> cluster.server-quorum-type:server >> cluster.quorum-type:auto >> gluster.server-quorum-ratio is not set. >>=3D20 >> One brick process is purposefully killed but remaining two bricks =3D are up and running. >>=3D20 >> Regards, >> Ramesh >>=3D20 >>> On 22 September 2015 at 06:24, Ramesh Nachimuthu < =3D rnachimu(a)redhat.com =3D > wrote: >>> Hi, >>>=3D20 >>> I am not able to resume a VM which was paused because of gluster =3D client quorum issue. Here is what happened in my setup.=3D20 >>>=3D20 >>> 1. Created a gluster storage domain which is backed by gluster =3D volume with replica 3.=3D20 >>> 2. Killed one brick process. So only two bricks are running in =3D replica 3 setup. >>> 3. Created two VMs >>> 4. Started some IO using fio on both of the VMs >>> 5. After some time got the following error in gluster mount and VMs =3D moved to paused state. >>> " server 10.70.45.17:49217 has =3D not responded in the last 42 seconds, disconnecting." >>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: =3D Failing WRITE as quorum is not met" >>> more gluster mount logs at =3D http://pastebin.com/UmiUQq0F =3D >>> 6. After some time gluster quorum is active and I am able to write =3D the the gluster file system. >>> 7. When I try to resume the VM it doesn't work and I got following =3D error in vdsm log. >>> http://pastebin.com/aXiamY15 >>>=3D20 >>>=3D20 >>> Regards, >>> Ramesh >>>=3D20 >>>=3D20 >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users =3D >>>=3D20 >>>=3D20 >>=3D20 >>=3D20 >=3D20 > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --Apple-Mail=3D_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8 This is a known issue in overt 3.5.x and below. It=3DE2=3D80= =3D99s =3D been solved in the upcoming ovirt 3.6.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D3D1172905, = =3D the fix involved setting up a special cgroup for the mount, but i =3D can=3DE2=3D80=3D99t find the exact details atm.


On Sep 23, 2015, at 7:38 AM,= =3D Ramesh Nachimuthu <rnachimu(a)redhat.com> wrote:

=3D20 =3D20


On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the =3D gluster.server-quorum-ratio to 51%


I did that. But still I am facing the same issue. VM get paused when I do some I/O using fio on some disks backed by gluster. I am not able to resume the VM after this. Now only way is to bring down the VM and run again. It runs successfully on the same host without any issue.

Regards,
Ramesh

On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)redhat.com> wrote:


On 09/22/2015 05:43 PM, Alastair Neil =3D wrote:
what are =3D the gluster-quorum-type and gluster.server-quorum-ratio  settings =3D on the volume?


cluster.server-quorum-type:server
=3D
cluster.quorum-type:auto
gluster.server-quorum-ratio is not = =3D set.

One brick process is purposefully killed  but remaining two bricks are up and running.

Regards,
Ramesh

On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.com> wrote:
Hi,

   I am not able to resume a VM =3D which was paused because of gluster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused =3D state.
=3D          " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting."
      "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not =3D met"
      more gluster =3D mount logs at http://pastebin.com/UmiUQq0F 6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
      http://pastebin.com/aXiamY15


Regards,
Ramesh


=3D _______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users





_______________________________________________
Users =3D mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

= =3D --Apple-Mail=3D_6A5ECE5F-7775-4FAA-8073-61A6AFDB8E50-- --===============7468391435079651325== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fNkE1RUNFNUYtNzc3NS00RkFBLTgwNzMtNjFBNkFGREI4RTUwCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKVGhpcyBpcyBhIGtub3duIGlzc3VlIGluIG92ZXJ0IDMu NS54IGFuZCBiZWxvdy4gSXQ9RTI9ODA9OTlzIGJlZW4gc29sdmVkID0KaW4gdGhlIHVwY29taW5n IG92aXJ0IDMuNi4KClJlbGF0ZWQgdG8gaHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tL3Nob3df YnVnLmNnaT9pZD0zRDExNzI5MDUsIHRoZSA9CmZpeCBpbnZvbHZlZCBzZXR0aW5nIHVwIGEgc3Bl Y2lhbCBjZ3JvdXAgZm9yIHRoZSBtb3VudCwgYnV0IGkgY2FuPUUyPTgwPTk5PQp0IGZpbmQgdGhl IGV4YWN0IGRldGFpbHMgYXRtLgoKCj4gT24gU2VwIDIzLCAyMDE1LCBhdCA3OjM4IEFNLCBSYW1l c2ggTmFjaGltdXRodSA8cm5hY2hpbXVAcmVkaGF0LmNvbT4gPQp3cm90ZToKPj0yMAo+PTIwCj49 MjAKPiBPbiAwOS8yMi8yMDE1IDA1OjU3IFBNLCBBbGFzdGFpciBOZWlsIHdyb3RlOgo+PiBZb3Ug bmVlZCB0byBzZXQgdGhlIGdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyB0byA1MSUKPj49MjAK Pj0yMAo+IEkgZGlkIHRoYXQuIEJ1dCBzdGlsbCBJIGFtIGZhY2luZyB0aGUgc2FtZSBpc3N1ZS4g Vk0gZ2V0IHBhdXNlZCB3aGVuIEkgPQpkbyBzb21lIEkvTyB1c2luZyBmaW8gb24gc29tZSBkaXNr cyBiYWNrZWQgYnkgZ2x1c3Rlci4gSSBhbSBub3QgYWJsZSB0byA9CnJlc3VtZSB0aGUgVk0gYWZ0 ZXIgdGhpcy4gTm93IG9ubHkgd2F5IGlzIHRvIGJyaW5nIGRvd24gdGhlIFZNIGFuZCBydW4gPQph Z2Fpbi4gSXQgcnVucyBzdWNjZXNzZnVsbHkgb24gdGhlIHNhbWUgaG9zdCB3aXRob3V0IGFueSBp c3N1ZS4KPj0yMAo+IFJlZ2FyZHMsCj4gUmFtZXNoCj49MjAKPj4gT24gMjIgU2VwdGVtYmVyIDIw MTUgYXQgMDg6MjUsIFJhbWVzaCBOYWNoaW11dGh1IDxybmFjaGltdUByZWRoYXQuY29tID0KPG1h aWx0bzpybmFjaGltdUByZWRoYXQuY29tPj4gd3JvdGU6Cj4+PTIwCj4+PTIwCj4+IE9uIDA5LzIy LzIwMTUgMDU6NDMgUE0sIEFsYXN0YWlyIE5laWwgd3JvdGU6Cj4+PiB3aGF0IGFyZSB0aGUgZ2x1 c3Rlci1xdW9ydW0tdHlwZSBhbmQgZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvICA9CnNldHRp bmdzIG9uIHRoZSB2b2x1bWU/Cj4+Pj0yMAo+Pj0yMAo+PiBjbHVzdGVyLnNlcnZlci1xdW9ydW0t dHlwZTpzZXJ2ZXIKPj4gY2x1c3Rlci5xdW9ydW0tdHlwZTphdXRvCj4+IGdsdXN0ZXIuc2VydmVy LXF1b3J1bS1yYXRpbyBpcyBub3Qgc2V0Lgo+Pj0yMAo+PiBPbmUgYnJpY2sgcHJvY2VzcyBpcyBw dXJwb3NlZnVsbHkga2lsbGVkICBidXQgcmVtYWluaW5nIHR3byBicmlja3MgPQphcmUgdXAgYW5k IHJ1bm5pbmcuCj4+PTIwCj4+IFJlZ2FyZHMsCj4+IFJhbWVzaAo+Pj0yMAo+Pj4gT24gMjIgU2Vw dGVtYmVyIDIwMTUgYXQgMDY6MjQsIFJhbWVzaCBOYWNoaW11dGh1IDwgPQo8bWFpbHRvOnJuYWNo aW11QHJlZGhhdC5jb20+cm5hY2hpbXVAcmVkaGF0LmNvbSA9CjxtYWlsdG86cm5hY2hpbXVAcmVk aGF0LmNvbT4+IHdyb3RlOgo+Pj4gSGksCj4+Pj0yMAo+Pj4gICAgSSBhbSBub3QgYWJsZSB0byBy ZXN1bWUgYSBWTSB3aGljaCB3YXMgcGF1c2VkIGJlY2F1c2Ugb2YgZ2x1c3RlciA9CmNsaWVudCBx dW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBoYXBwZW5lZCBpbiBteSBzZXR1cC49MjAKPj4+PTIw Cj4+PiAxLiBDcmVhdGVkIGEgZ2x1c3RlciBzdG9yYWdlIGRvbWFpbiB3aGljaCBpcyBiYWNrZWQg YnkgZ2x1c3RlciA9CnZvbHVtZSB3aXRoIHJlcGxpY2EgMy49MjAKPj4+IDIuIEtpbGxlZCBvbmUg YnJpY2sgcHJvY2Vzcy4gU28gb25seSB0d28gYnJpY2tzIGFyZSBydW5uaW5nIGluID0KcmVwbGlj YSAzIHNldHVwLgo+Pj4gMy4gQ3JlYXRlZCB0d28gVk1zCj4+PiA0LiBTdGFydGVkIHNvbWUgSU8g dXNpbmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNcwo+Pj4gNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0 aGUgZm9sbG93aW5nIGVycm9yIGluIGdsdXN0ZXIgbW91bnQgYW5kIFZNcyA9Cm1vdmVkIHRvIHBh dXNlZCBzdGF0ZS4KPj4+ICAgICAgICAgICIgc2VydmVyIDEwLjcwLjQ1LjE3OjQ5MjE3IDxodHRw Oi8vMTAuNzAuNDUuMTc6NDkyMTcvPiBoYXMgPQpub3QgcmVzcG9uZGVkIGluIHRoZSBsYXN0IDQy IHNlY29uZHMsIGRpc2Nvbm5lY3RpbmcuIgo+Pj4gICAgICAgInZtc3RvcmUtcmVwbGljYXRlLTA6 IGUxNmQxZTQwLTJiNmUtNGYxOS05NzdkLWUwOTlmNDY1ZGZjNjogPQpGYWlsaW5nIFdSSVRFIGFz IHF1b3J1bSBpcyBub3QgbWV0Igo+Pj4gICAgICAgbW9yZSBnbHVzdGVyIG1vdW50IGxvZ3MgYXQg ID0KPGh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEY+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlV UXEwRiA9CjxodHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBGPgo+Pj4gNi4gQWZ0ZXIgc29tZSB0 aW1lIGdsdXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvIHdyaXRlID0KdGhl IHRoZSBnbHVzdGVyIGZpbGUgc3lzdGVtLgo+Pj4gNy4gV2hlbiBJIHRyeSB0byByZXN1bWUgdGhl IFZNIGl0IGRvZXNuJ3Qgd29yayBhbmQgSSBnb3QgZm9sbG93aW5nID0KZXJyb3IgaW4gdmRzbSBs b2cuCj4+PiAgICAgICBodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1IDxodHRwOi8vcGFzdGVi aW4uY29tL2FYaWFtWTE1Pgo+Pj49MjAKPj4+PTIwCj4+PiBSZWdhcmRzLAo+Pj4gUmFtZXNoCj4+ Pj0yMAo+Pj49MjAKPj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCj4+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4+IFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRv OlVzZXJzQG92aXJ0Lm9yZz4KPj4+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0 aW5mby91c2VycyA9CjxodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNl cnM+Cj4+Pj0yMAo+Pj49MjAKPj49MjAKPj49MjAKPj0yMAo+IF9fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gVXNlcnMgbWFpbGluZyBsaXN0Cj4gVXNlcnNA b3ZpcnQub3JnCj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJz CgoKLS1BcHBsZS1NYWlsPV82QTVFQ0U1Ri03Nzc1LTRGQUEtODA3My02MUE2QUZEQjhFNTAKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQpDb250ZW50LVR5cGU6IHRl eHQvaHRtbDsKCWNoYXJzZXQ9dXRmLTgKCjxodG1sPjxoZWFkPjxtZXRhIGh0dHAtZXF1aXY9M0Qi Q29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQvaHRtbCA9CmNoYXJzZXQ9M0R1dGYtOCI+PC9o ZWFkPjxib2R5IHN0eWxlPTNEIndvcmQtd3JhcDogYnJlYWstd29yZDsgPQotd2Via2l0LW5ic3At bW9kZTogc3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogYWZ0ZXItd2hpdGUtc3BhY2U7IiA9CmNs YXNzPTNEIiI+VGhpcyBpcyBhIGtub3duIGlzc3VlIGluIG92ZXJ0IDMuNS54IGFuZCBiZWxvdy4g SXQ9RTI9ODA9OTlzID0KYmVlbiBzb2x2ZWQgaW4gdGhlIHVwY29taW5nIG92aXJ0IDMuNi48ZGl2 IGNsYXNzPTNEIiI+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+UmVsYXRl ZCB0byZuYnNwOzxhID0KaHJlZj0zRCJodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19i dWcuY2dpP2lkPTNEMTE3MjkwNSIgPQpjbGFzcz0zRCIiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0 LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9M0QxMTcyOTA1PC9hPiwgPQp0aGUgZml4IGludm9sdmVkIHNl dHRpbmcgdXAgYSBzcGVjaWFsIGNncm91cCBmb3IgdGhlIG1vdW50LCBidXQgaSA9CmNhbj1FMj04 MD05OXQgZmluZCB0aGUgZXhhY3QgZGV0YWlscyBhdG0uPC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxi ciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxkaXY+ PGJsb2NrcXVvdGUgPQp0eXBlPTNEImNpdGUiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPk9u IFNlcCAyMywgMjAxNSwgYXQgNzozOCBBTSwgPQpSYW1lc2ggTmFjaGltdXRodSAmbHQ7PGEgaHJl Zj0zRCJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSIgPQpjbGFzcz0zRCIiPnJuYWNoaW11QHJl ZGhhdC5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj48YnIgPQpjbGFzcz0zRCJBcHBsZS1pbnRlcmNo YW5nZS1uZXdsaW5lIj48ZGl2IGNsYXNzPTNEIiI+CiA9MjAKICAgIDxtZXRhIGNvbnRlbnQ9M0Qi dGV4dC9odG1sOyBjaGFyc2V0PTNEdXRmLTgiID0KaHR0cC1lcXVpdj0zRCJDb250ZW50LVR5cGUi IGNsYXNzPTNEIiI+CiA9MjAKICA8ZGl2IHRleHQ9M0QiIzAwMDAwMCIgYmdjb2xvcj0zRCIjRkZG RkZGIiBjbGFzcz0zRCIiPgogICAgPGJyIGNsYXNzPTNEIiI+CiAgICA8YnIgY2xhc3M9M0QiIj4K ICAgIDxkaXYgY2xhc3M9M0QibW96LWNpdGUtcHJlZml4Ij5PbiAwOS8yMi8yMDE1IDA1OjU3IFBN LCBBbGFzdGFpciBOZWlsCiAgICAgIHdyb3RlOjxiciBjbGFzcz0zRCIiPgogICAgPC9kaXY+CiAg ICA8YmxvY2txdW90ZSA9CmNpdGU9M0QibWlkOkNBK1Nhcndvb3JVM0xXRzYrc1ItdEoxQkViUWEx azRXUnJSWGthWHkzei1FUFJyejdVd0BtYWlsLmdtYWlsPQouY29tIiB0eXBlPTNEImNpdGUiIGNs YXNzPTNEIiI+CiAgICAgIDxkaXYgZGlyPTNEImx0ciIgY2xhc3M9M0QiIj5Zb3UgbmVlZCB0byBz ZXQgdGhlID0KZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvIHRvCiAgICAgICAgNTElPC9kaXY+ CiAgICAgIDxkaXYgY2xhc3M9M0QiZ21haWxfZXh0cmEiPjxiciBjbGFzcz0zRCIiPgogICAgICA8 L2Rpdj4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxiciBjbGFzcz0zRCIiPgogICAgSSBkaWQgdGhh dC4gQnV0IHN0aWxsIEkgYW0gZmFjaW5nIHRoZSBzYW1lIGlzc3VlLiBWTSBnZXQgcGF1c2VkIHdo ZW4KICAgIEkgZG8gc29tZSBJL08gdXNpbmcgZmlvIG9uIHNvbWUgZGlza3MgYmFja2VkIGJ5IGds dXN0ZXIuIEkgYW0gbm90CiAgICBhYmxlIHRvIHJlc3VtZSB0aGUgVk0gYWZ0ZXIgdGhpcy4gTm93 IG9ubHkgd2F5IGlzIHRvIGJyaW5nIGRvd24gdGhlCiAgICBWTSBhbmQgcnVuIGFnYWluLiBJdCBy dW5zIHN1Y2Nlc3NmdWxseSBvbiB0aGUgc2FtZSBob3N0IHdpdGhvdXQgYW55CiAgICBpc3N1ZS48 YnIgY2xhc3M9M0QiIj4KICAgIDxiciBjbGFzcz0zRCIiPgogICAgUmVnYXJkcyw8YnIgY2xhc3M9 M0QiIj4KICAgIFJhbWVzaDxiciBjbGFzcz0zRCIiPgogICAgPGJyIGNsYXNzPTNEIiI+CiAgICA8 YmxvY2txdW90ZSA9CmNpdGU9M0QibWlkOkNBK1Nhcndvb3JVM0xXRzYrc1ItdEoxQkViUWExazRX UnJSWGthWHkzei1FUFJyejdVd0BtYWlsLmdtYWlsPQouY29tIiB0eXBlPTNEImNpdGUiIGNsYXNz PTNEIiI+CiAgICAgIDxkaXYgY2xhc3M9M0QiZ21haWxfZXh0cmEiPgogICAgICAgIDxkaXYgY2xh c3M9M0QiZ21haWxfcXVvdGUiPk9uIDIyIFNlcHRlbWJlciAyMDE1IGF0IDA4OjI1LCBSYW1lc2gK ICAgICAgICAgIE5hY2hpbXV0aHUgPHNwYW4gZGlyPTNEImx0ciIgY2xhc3M9M0QiIj4mbHQ7PGEg PQptb3otZG8tbm90LXNlbmQ9M0QidHJ1ZSIgaHJlZj0zRCJtYWlsdG86cm5hY2hpbXVAcmVkaGF0 LmNvbSIgPQp0YXJnZXQ9M0QiX2JsYW5rIiBjbGFzcz0zRCIiPnJuYWNoaW11QHJlZGhhdC5jb208 L2E+Jmd0Ozwvc3Bhbj4KICAgICAgICAgIHdyb3RlOjxiciBjbGFzcz0zRCIiPgogICAgICAgICAg PGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNEIm1hcmdpbjowIDAgMAog ICAgICAgICAgICAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDox ZXgiPgogICAgICAgICAgICA8ZGl2IHRleHQ9M0QiIzAwMDAwMCIgYmdjb2xvcj0zRCIjRkZGRkZG IiBjbGFzcz0zRCIiPjxzcGFuID0KY2xhc3M9M0QiIj4gPGJyIGNsYXNzPTNEIiI+CiAgICAgICAg ICAgICAgICA8YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9M0QiIj5P biAwOS8yMi8yMDE1IDA1OjQzIFBNLCBBbGFzdGFpciBOZWlsID0Kd3JvdGU6PGJyIGNsYXNzPTNE IiI+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5 cGU9M0QiY2l0ZSIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9M0QibHRy IiBjbGFzcz0zRCIiPndoYXQgYXJlID0KdGhlJm5ic3A7Z2x1c3Rlci1xdW9ydW0tdHlwZQogICAg ICAgICAgICAgICAgICAgIGFuZCZuYnNwO2dsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyAmbmJz cDtzZXR0aW5ncyA9Cm9uIHRoZQogICAgICAgICAgICAgICAgICAgIHZvbHVtZT88L2Rpdj4KICAg ICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyIGNsYXNzPTNEIiI+ CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgog ICAgICAgICAgICAgICAgPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgPC9zcGFuPgogICAg ICAgICAgICAgIDxkaXYgc3R5bGU9M0Qib3V0bGluZS1zdHlsZTpub25lIiBjbGFzcz0zRCIiPgog ICAgICAgICAgICAgICAgPGRpdiA9CnN0eWxlPTNEIm92ZXJmbG93OmhpZGRlbjt0ZXh0LW92ZXJm bG93OmVsbGlwc2lzO3doaXRlLXNwYWNlOm5vd3JhcCIgPQpjbGFzcz0zRCIiPjxiIGNsYXNzPTNE IiI+Y2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+OnNlcnZlcjxiciA9CmNsYXNzPTNEIiI+ CiAgICAgICAgICAgICAgICAgIDxkaXYgdGl0bGU9M0QiIiBzdHlsZT0zRCJvdXRsaW5lLXN0eWxl Om5vbmUiIGNsYXNzPTNEIiI+PQoKICAgICAgICAgICAgICAgICAgICA8ZGl2ID0Kc3R5bGU9M0Qi b3ZlcmZsb3c6aGlkZGVuO3RleHQtb3ZlcmZsb3c6ZWxsaXBzaXM7d2hpdGUtc3BhY2U6bm93cmFw IiA9CmNsYXNzPTNEIiI+PGIgY2xhc3M9M0QiIj5jbHVzdGVyLnF1b3J1bS10eXBlPC9iPjphdXRv PGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICA8YiBjbGFzcz0zRCIiPmdsdXN0 ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyBpcyBub3QgPQpzZXQuPC9iPjxiciBjbGFzcz0zRCIiPgog ICAgICAgICAgICAgICAgICAgICAgPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAg PC9kaXY+CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICBPbmUgYnJp Y2sgcHJvY2VzcyBpcyBwdXJwb3NlZnVsbHkga2lsbGVkJm5ic3A7IGJ1dAogICAgICAgICAgICAg ICAgICByZW1haW5pbmcgdHdvIGJyaWNrcyBhcmUgdXAgYW5kIHJ1bm5pbmcuPGJyIGNsYXNzPTNE IiI+CiAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICBS ZWdhcmRzLDxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICBSYW1lc2g8YnIgY2xhc3M9 M0QiIj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAg ICAgICAgIDxzcGFuIGNsYXNzPTNEIiI+IDxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAg PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICA8 ZGl2IGNsYXNzPTNEImdtYWlsX2V4dHJhIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNz PTNEImdtYWlsX3F1b3RlIj5PbiAyMiBTZXB0ZW1iZXIgMjAxNSBhdAogICAgICAgICAgICAgICAg ICAgICAgMDY6MjQsIFJhbWVzaCBOYWNoaW11dGh1IDxzcGFuIGRpcj0zRCJsdHIiID0KY2xhc3M9 M0QiIj4mbHQ7PGEgbW96LWRvLW5vdC1zZW5kPTNEInRydWUiID0KaHJlZj0zRCJtYWlsdG86cm5h Y2hpbXVAcmVkaGF0LmNvbSIgdGFyZ2V0PTNEIl9ibGFuayIgY2xhc3M9M0QiIj48L2E+PGEgPQpj bGFzcz0zRCJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiID0KaHJlZj0zRCJtYWlsdG86cm5hY2hp bXVAcmVkaGF0LmNvbSI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAg ICAgICAgICAgICAgICAgd3JvdGU6PGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAg ICA8YmxvY2txdW90ZSBjbGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgPQow CiAgICAgICAgICAgICAgICAgICAgICAgIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYwogICAg ICAgICAgICAgICAgICAgICAgICBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAg ICAgICAgICAgICAgPGRpdiB0ZXh0PTNEIiMwMDAwMDAiIGJnY29sb3I9M0QiI0ZGRkZGRiIgPQpj bGFzcz0zRCIiPiBIaSw8YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8 YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAmbmJzcDsmbmJzcDsgSSBh bSBub3QgYWJsZSB0byByZXN1bWUgYSBWTSA9CndoaWNoIHdhcwogICAgICAgICAgICAgICAgICAg ICAgICAgIHBhdXNlZCBiZWNhdXNlIG9mIGdsdXN0ZXIgY2xpZW50IHF1b3J1bSBpc3N1ZS4KICAg ICAgICAgICAgICAgICAgICAgICAgICBIZXJlIGlzIHdoYXQgaGFwcGVuZWQgaW4gbXkgc2V0dXAu IDxiciA9CmNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyIGNsYXNzPTNE IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgMS4gQ3JlYXRlZCBhIGdsdXN0ZXIgc3RvcmFn ZSBkb21haW4gd2hpY2ggaXMKICAgICAgICAgICAgICAgICAgICAgICAgICBiYWNrZWQgYnkgZ2x1 c3RlciB2b2x1bWUgd2l0aCByZXBsaWNhIDMuIDxiciA9CmNsYXNzPTNEIiI+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgMi4gS2lsbGVkIG9uZSBicmljayBwcm9jZXNzLiBTbyBvbmx5IHR3bwog ICAgICAgICAgICAgICAgICAgICAgICAgIGJyaWNrcyBhcmUgcnVubmluZyBpbiByZXBsaWNhIDMg c2V0dXAuPGJyID0KY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAzLiBDcmVh dGVkIHR3byBWTXM8YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICA0LiBT dGFydGVkIHNvbWUgSU8gdXNpbmcgZmlvIG9uIGJvdGggb2YgdGhlCiAgICAgICAgICAgICAgICAg ICAgICAgICAgVk1zPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgNS4g QWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluCiAgICAgICAgICAgICAg ICAgICAgICAgICAgZ2x1c3RlciBtb3VudCBhbmQgVk1zIG1vdmVkIHRvIHBhdXNlZCA9CnN0YXRl LjxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICAgICAgICAgID0KJm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7ICIgPHNwYW4gPQpzdHlsZT0zRCJj b2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtm b250LXN0eT0KbGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFs O2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW49CmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFsaWduOmxl ZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtPQpzcGFjZTpub3Jt YWw7d29yZC1zcGFjaW5nOjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTti YWNrZ3JvdT0KbmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSIgY2xhc3M9M0QiIj5zZXJ2ZXIKCgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEgbW96LWRvLW5vdC1zZW5kPTNEInRydWUiID0K aHJlZj0zRCJodHRwOi8vMTAuNzAuNDUuMTc6NDkyMTcvIiB0YXJnZXQ9M0QiX2JsYW5rIiA9CmNs YXNzPTNEIiI+MTAuNzAuNDUuMTc6NDkyMTc8L2E+IGhhcwogICAgICAgICAgICAgICAgICAgICAg ICAgICAgbm90IHJlc3BvbmRlZCBpbiB0aGUgbGFzdCA0MiBzZWNvbmRzLAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgZGlzY29ubmVjdGluZy4iPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgIjwvc3Bhbj48 c3BhbiA9CnN0eWxlPTNEImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNl O2ZvbnQtc2l6ZToxMXB4O2ZvbnQtc3R5PQpsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtm b250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbj0KZS1oZWlnaHQ6MTMu MnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3 aGl0ZS09CnNwYWNlOm5vcm1hbDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9y dGFudDtmbG9hdDpub25lO2JhY2tncm91PQpuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIiBjbGFz cz0zRCIiPjxzcGFuID0Kc3R5bGU9M0QiY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTpt b25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1zdHk9CmxlOm5vcm1hbDtmb250LXZhcmlhbnQ6 bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3JtYWw7bGluPQplLWhl aWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9y bTpub25lO3doaXRlLT0Kc3BhY2U6bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxp bmUhaW1wb3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3U9Cm5kLWNvbG9yOnJnYigyNTUsMjU1LDI1 NSkiIGNsYXNzPTNEIiI+dm1zdG9yZS1yZXBsaWNhdGUtMDoKCgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBlMTZkMWU0MC0yYjZlLTRmMTktOTc3ZC1lMDk5ZjQ2NWRmYzY6CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIEZhaWxpbmcgV1JJVEUgYXMgcXVvcnVtIGlzIG5vdCA9Cm1l dDwvc3Bhbj4iPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgbW9yZSBnbHVzdGVyID0KbW91bnQgbG9ncyBhdCA8 YSBtb3otZG8tbm90LXNlbmQ9M0QidHJ1ZSIgPQpocmVmPTNEImh0dHA6Ly9wYXN0ZWJpbi5jb20v VW1pVVFxMEYiIHRhcmdldD0zRCJfYmxhbmsiIGNsYXNzPTNEIiI+PC9hPjxhID0KY2xhc3M9M0Qi bW96LXR4dC1saW5rLWZyZWV0ZXh0IiA9CmhyZWY9M0QiaHR0cDovL3Bhc3RlYmluLmNvbS9VbWlV UXEwRiI+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRjwvYT48YnIgPQpjbGFzcz0zRCIiPgog ICAgICAgICAgICAgICAgICAgICAgICAgIDwvc3Bhbj42LiBBZnRlciBzb21lIHRpbWUgZ2x1c3Rl ciBxdW9ydW0gaXMKICAgICAgICAgICAgICAgICAgICAgICAgICBhY3RpdmUgYW5kIEkgYW0gYWJs ZSB0byB3cml0ZSB0aGUgdGhlIGdsdXN0ZXIKICAgICAgICAgICAgICAgICAgICAgICAgICBmaWxl IHN5c3RlbS48YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICA3LiBXaGVu IEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQgZG9lc24ndCB3b3JrCiAgICAgICAgICAgICAgICAg ICAgICAgICAgYW5kIEkgZ290IGZvbGxvd2luZyBlcnJvciBpbiB2ZHNtIGxvZy48YnIgPQpjbGFz cz0zRCIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyA8YSA9Cm1vei1kby1ub3Qtc2VuZD0zRCJ0cnVlIiBocmVmPTNEImh0dHA6Ly9wYXN0 ZWJpbi5jb20vYVhpYW1ZMTUiID0KdGFyZ2V0PTNEIl9ibGFuayIgY2xhc3M9M0QiIj5odHRwOi8v cGFzdGViaW4uY29tL2FYaWFtWTE1PC9hPjxiciA9CmNsYXNzPTNEIiI+CiAgICAgICAgICAgICAg ICAgICAgICAgICAgPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPGJy IGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgUmVnYXJkcyw8YnIgY2xhc3M9 M0QiIj4KICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnIgY2xhc3M9M0QiIj4KICAg ICAgICAgICAgICAgICAgICAgICAgICA8YnIgY2xhc3M9M0QiIj4KICAgICAgICAgICAgICAgICAg ICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0zRCIiPgogICAg ICAgICAgICAgICAgICAgICAgICA9Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fPGJyIGNsYXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgIFVzZXJz IG1haWxpbmcgbGlzdDxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICAgICAgICA8YSBt b3otZG8tbm90LXNlbmQ9M0QidHJ1ZSIgPQpocmVmPTNEIm1haWx0bzpVc2Vyc0BvdmlydC5vcmci IHRhcmdldD0zRCJfYmxhbmsiID0KY2xhc3M9M0QiIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNs YXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxhIG1vei1kby1ub3Qtc2VuZD0zRCJ0 cnVlIiA9CmhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Vz ZXJzIiByZWw9M0Qibm9yZWZlcnJlciIgPQp0YXJnZXQ9M0QiX2JsYW5rIiA9CmNsYXNzPTNEIiI+ aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxiciA9CmNs YXNzPTNEIiI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0zRCIiPgogICAgICAg ICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgIDwvZGl2Pgog ICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0zRCIiPgogICAgICAgICAgICAgICAgICA8L2Rp dj4KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAgIDxiciBjbGFz cz0zRCIiPgogICAgICAgICAgICAgIDwvc3Bhbj48L2Rpdj4KICAgICAgICAgIDwvYmxvY2txdW90 ZT4KICAgICAgICA8L2Rpdj4KICAgICAgICA8YnIgY2xhc3M9M0QiIj4KICAgICAgPC9kaXY+CiAg ICA8L2Jsb2NrcXVvdGU+CiAgICA8YnIgY2xhc3M9M0QiIj4KICA8L2Rpdj4KCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyIGNsYXNzPTNEIiI+VXNlcnMg PQptYWlsaW5nIGxpc3Q8YnIgY2xhc3M9M0QiIj48YSBocmVmPTNEIm1haWx0bzpVc2Vyc0Bvdmly dC5vcmciID0KY2xhc3M9M0QiIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyID0KY2xhc3M9M0QiIj5o dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8YnIgPQpjbGFzcz0z RCIiPjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48L2Rpdj48L2JvZHk+ PC9odG1sPj0KCi0tQXBwbGUtTWFpbD1fNkE1RUNFNUYtNzc3NS00RkFBLTgwNzMtNjFBNkFGREI4 RTUwLS0K --===============7468391435079651325==-- From rnachimu at redhat.com Thu Sep 24 00:37:41 2015 Content-Type: multipart/mixed; boundary="===============7093251404989740408==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Thu, 24 Sep 2015 10:07:35 +0530 Message-ID: <56037E0F.1090904@redhat.com> In-Reply-To: F5280320-1598-4721-A5D4-CE3035F1E7C3@onholyground.com --===============7093251404989740408== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050104050809050706040602 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit On 09/24/2015 02:38 AM, Darrell Budic wrote: > This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved = in = > the upcoming ovirt 3.6. > > Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the = > fix involved setting up a special cgroup for the mount, but i can=E2=80= =99t = > find the exact details atm. > I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am = not sure above bug 1172905 = fixes this correctl= y. Regards, Ramesh > >> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu > > wrote: >> >> >> >> On 09/22/2015 05:57 PM, Alastair Neil wrote: >>> You need to set the gluster.server-quorum-ratio to 51% >>> >> >> I did that. But still I am facing the same issue. VM get paused when = >> I do some I/O using fio on some disks backed by gluster. I am not = >> able to resume the VM after this. Now only way is to bring down the = >> VM and run again. It runs successfully on the same host without any = >> issue. >> >> Regards, >> Ramesh >> >>> On 22 September 2015 at 08:25, Ramesh Nachimuthu = >>> > wrote: >>> >>> >>> >>> On 09/22/2015 05:43 PM, Alastair Neil wrote: >>>> what are the gluster-quorum-type >>>> and gluster.server-quorum-ratio settings on the volume? >>>> >>> >>> *cluster.server-quorum-type*:server >>> *cluster.quorum-type*:auto >>> *gluster.server-quorum-ratio is not set.* >>> >>> One brick process is purposefully killed but remaining two >>> bricks are up and running. >>> >>> Regards, >>> Ramesh >>> >>>> On 22 September 2015 at 06:24, Ramesh Nachimuthu >>>> wrote: >>>> >>>> Hi, >>>> >>>> I am not able to resume a VM which was paused because of >>>> gluster client quorum issue. Here is what happened in my >>>> setup. >>>> >>>> 1. Created a gluster storage domain which is backed by >>>> gluster volume with replica 3. >>>> 2. Killed one brick process. So only two bricks are running >>>> in replica 3 setup. >>>> 3. Created two VMs >>>> 4. Started some IO using fio on both of the VMs >>>> 5. After some time got the following error in gluster mount >>>> and VMs moved to paused state. >>>> " server 10.70.45.17:49217 >>>> has not responded in the last >>>> 42 seconds, disconnecting." >>>> "vmstore-replicate-0: >>>> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as >>>> quorum is not met" >>>> more gluster mount logs at http://pastebin.com/UmiUQq0F >>>> 6. After some time gluster quorum is active and I am able >>>> to write the the gluster file system. >>>> 7. When I try to resume the VM it doesn't work and I got >>>> following error in vdsm log. >>>> http://pastebin.com/aXiamY15 >>>> >>>> >>>> Regards, >>>> Ramesh >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>> >>> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > --------------050104050809050706040602 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

On 09/24/2015 02:38 AM, Darrell Budic wrote:
This is a known issue in overt 3.5.x and below. It=E2=80=99s been sol= ved in the upcoming ovirt 3.6.

Related to=C2=A0https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905<= /a>, the fix involved setting up a special cgroup for the mount, but i can=E2=80=99t find the exact details atm.


I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am not sure above bug 1172905 fixes this correctly.

Regards,
Ramesh


On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a= )redhat.com> wrote:



On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%


I did that. But still I am facing the same issue. VM get paused when I do some I/O using fio on some disks backed by gluster. I am not able to resume the VM after this. Now only way is to bring down the VM and run again. It runs successfully on the same host without any issue.

Regards,
Ramesh

On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)redhat.c= om> wrote:


On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the=C2=A0gluster-quorum-type and=C2=A0gluster.server-quorum-ratio =C2=A0settings on the volume?
cluster.server-quoru= m-type:server
cluster.quorum-t= ype:auto
gluster.server-quorum-ratio is not set.

One brick process is purposefully killed=C2= =A0 but remaining two bricks are up and running.

Regards,
Ramesh

On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.com> wrote:
Hi,

=C2=A0=C2=A0 I am not able to resume = a VM which was paused because of gluster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 " server 10.7= 0.45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465d= fc6: Failing WRITE as quorum is not met"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 more= gluster mount logs at http://pastebin.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http:/= /pastebin.com/aXiamY15


Regards,
Ramesh


_______________________________________________
Users mailing list
Users(a)= ovirt.org
http://lists.ovirt.org/mai= lman/listinfo/users





_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users<= /a>


--------------050104050809050706040602-- --===============7093251404989740408== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTAxMDQwNTA4MDkwNTA3MDYwNDA2MDIKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CgoKT24gMDkvMjQvMjAxNSAwMjozOCBBTSwgRGFycmVsbCBCdWRpYyB3cm90ZToKPiBUaGlzIGlz IGEga25vd24gaXNzdWUgaW4gb3ZlcnQgMy41LnggYW5kIGJlbG93LiBJdOKAmXMgYmVlbiBzb2x2 ZWQgaW4gCj4gdGhlIHVwY29taW5nIG92aXJ0IDMuNi4KPgo+IFJlbGF0ZWQgdG8gaHR0cHM6Ly9i dWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD0xMTcyOTA1LCB0aGUgCj4gZml4IGlu dm9sdmVkIHNldHRpbmcgdXAgYSBzcGVjaWFsIGNncm91cCBmb3IgdGhlIG1vdW50LCBidXQgaSBj YW7igJl0IAo+IGZpbmQgdGhlIGV4YWN0IGRldGFpbHMgYXRtLgo+CgpJIGhhdmUgdmRzbSA0LjE3 LjYtMC5lbDcuY2VudG9zIGFscmVhZHkgaW5zdGFsbGVkIG9uIHRoZSBob3N0cy4gU28gSSBhbSAK bm90IHN1cmUgYWJvdmUgYnVnIDExNzI5MDUgCjxodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20v c2hvd19idWcuY2dpP2lkPTExNzI5MDU+IGZpeGVzIHRoaXMgY29ycmVjdGx5LgoKUmVnYXJkcywK UmFtZXNoCgo+Cj4+IE9uIFNlcCAyMywgMjAxNSwgYXQgNzozOCBBTSwgUmFtZXNoIE5hY2hpbXV0 aHUgPHJuYWNoaW11QHJlZGhhdC5jb20gCj4+IDxtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbT4+ IHdyb3RlOgo+Pgo+Pgo+Pgo+PiBPbiAwOS8yMi8yMDE1IDA1OjU3IFBNLCBBbGFzdGFpciBOZWls IHdyb3RlOgo+Pj4gWW91IG5lZWQgdG8gc2V0IHRoZSBnbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0 aW8gdG8gNTElCj4+Pgo+Pgo+PiBJIGRpZCB0aGF0LiBCdXQgc3RpbGwgSSBhbSBmYWNpbmcgdGhl IHNhbWUgaXNzdWUuIFZNIGdldCBwYXVzZWQgd2hlbiAKPj4gSSBkbyBzb21lIEkvTyB1c2luZyBm aW8gb24gc29tZSBkaXNrcyBiYWNrZWQgYnkgZ2x1c3Rlci4gSSBhbSBub3QgCj4+IGFibGUgdG8g cmVzdW1lIHRoZSBWTSBhZnRlciB0aGlzLiBOb3cgb25seSB3YXkgaXMgdG8gYnJpbmcgZG93biB0 aGUgCj4+IFZNIGFuZCBydW4gYWdhaW4uIEl0IHJ1bnMgc3VjY2Vzc2Z1bGx5IG9uIHRoZSBzYW1l IGhvc3Qgd2l0aG91dCBhbnkgCj4+IGlzc3VlLgo+Pgo+PiBSZWdhcmRzLAo+PiBSYW1lc2gKPj4K Pj4+IE9uIDIyIFNlcHRlbWJlciAyMDE1IGF0IDA4OjI1LCBSYW1lc2ggTmFjaGltdXRodSAKPj4+ IDxybmFjaGltdUByZWRoYXQuY29tIDxtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbT4+IHdyb3Rl Ogo+Pj4KPj4+Cj4+Pgo+Pj4gICAgIE9uIDA5LzIyLzIwMTUgMDU6NDMgUE0sIEFsYXN0YWlyIE5l aWwgd3JvdGU6Cj4+Pj4gICAgIHdoYXQgYXJlIHRoZSBnbHVzdGVyLXF1b3J1bS10eXBlCj4+Pj4g ICAgIGFuZCBnbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0aW8gIHNldHRpbmdzIG9uIHRoZSB2b2x1 bWU/Cj4+Pj4KPj4+Cj4+PiAgICAgKmNsdXN0ZXIuc2VydmVyLXF1b3J1bS10eXBlKjpzZXJ2ZXIK Pj4+ICAgICAqY2x1c3Rlci5xdW9ydW0tdHlwZSo6YXV0bwo+Pj4gICAgICpnbHVzdGVyLnNlcnZl ci1xdW9ydW0tcmF0aW8gaXMgbm90IHNldC4qCj4+Pgo+Pj4gICAgIE9uZSBicmljayBwcm9jZXNz IGlzIHB1cnBvc2VmdWxseSBraWxsZWQgYnV0IHJlbWFpbmluZyB0d28KPj4+ICAgICBicmlja3Mg YXJlIHVwIGFuZCBydW5uaW5nLgo+Pj4KPj4+ICAgICBSZWdhcmRzLAo+Pj4gICAgIFJhbWVzaAo+ Pj4KPj4+PiAgICAgT24gMjIgU2VwdGVtYmVyIDIwMTUgYXQgMDY6MjQsIFJhbWVzaCBOYWNoaW11 dGh1Cj4+Pj4gICAgIDxybmFjaGltdUByZWRoYXQuY29tPiB3cm90ZToKPj4+Pgo+Pj4+ICAgICAg ICAgSGksCj4+Pj4KPj4+PiAgICAgICAgICAgIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIGEgVk0g d2hpY2ggd2FzIHBhdXNlZCBiZWNhdXNlIG9mCj4+Pj4gICAgICAgICBnbHVzdGVyIGNsaWVudCBx dW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBoYXBwZW5lZCBpbiBteQo+Pj4+ICAgICAgICAgc2V0 dXAuCj4+Pj4KPj4+PiAgICAgICAgIDEuIENyZWF0ZWQgYSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWlu IHdoaWNoIGlzIGJhY2tlZCBieQo+Pj4+ICAgICAgICAgZ2x1c3RlciB2b2x1bWUgd2l0aCByZXBs aWNhIDMuCj4+Pj4gICAgICAgICAyLiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkg dHdvIGJyaWNrcyBhcmUgcnVubmluZwo+Pj4+ICAgICAgICAgaW4gcmVwbGljYSAzIHNldHVwLgo+ Pj4+ICAgICAgICAgMy4gQ3JlYXRlZCB0d28gVk1zCj4+Pj4gICAgICAgICA0LiBTdGFydGVkIHNv bWUgSU8gdXNpbmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNcwo+Pj4+ICAgICAgICAgNS4gQWZ0ZXIg c29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluIGdsdXN0ZXIgbW91bnQKPj4+PiAg ICAgICAgIGFuZCBWTXMgbW92ZWQgdG8gcGF1c2VkIHN0YXRlLgo+Pj4+ICAgICAgICAgICAgICAg ICAgIiBzZXJ2ZXIgMTAuNzAuNDUuMTc6NDkyMTcKPj4+PiAgICAgICAgIDxodHRwOi8vMTAuNzAu NDUuMTc6NDkyMTcvPiBoYXMgbm90IHJlc3BvbmRlZCBpbiB0aGUgbGFzdAo+Pj4+ICAgICAgICAg NDIgc2Vjb25kcywgZGlzY29ubmVjdGluZy4iCj4+Pj4gICAgICAgICAgICAgICAidm1zdG9yZS1y ZXBsaWNhdGUtMDoKPj4+PiAgICAgICAgIGUxNmQxZTQwLTJiNmUtNGYxOS05NzdkLWUwOTlmNDY1 ZGZjNjogRmFpbGluZyBXUklURSBhcwo+Pj4+ICAgICAgICAgcXVvcnVtIGlzIG5vdCBtZXQiCj4+ Pj4gICAgICAgICAgICAgICBtb3JlIGdsdXN0ZXIgbW91bnQgbG9ncyBhdCBodHRwOi8vcGFzdGVi aW4uY29tL1VtaVVRcTBGCj4+Pj4gICAgICAgICA2LiBBZnRlciBzb21lIHRpbWUgZ2x1c3RlciBx dW9ydW0gaXMgYWN0aXZlIGFuZCBJIGFtIGFibGUKPj4+PiAgICAgICAgIHRvIHdyaXRlIHRoZSB0 aGUgZ2x1c3RlciBmaWxlIHN5c3RlbS4KPj4+PiAgICAgICAgIDcuIFdoZW4gSSB0cnkgdG8gcmVz dW1lIHRoZSBWTSBpdCBkb2Vzbid0IHdvcmsgYW5kIEkgZ290Cj4+Pj4gICAgICAgICBmb2xsb3dp bmcgZXJyb3IgaW4gdmRzbSBsb2cuCj4+Pj4gICAgICAgICBodHRwOi8vcGFzdGViaW4uY29tL2FY aWFtWTE1Cj4+Pj4KPj4+Pgo+Pj4+ICAgICAgICAgUmVnYXJkcywKPj4+PiAgICAgICAgIFJhbWVz aAo+Pj4+Cj4+Pj4KPj4+PiAgICAgICAgIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCj4+Pj4gICAgICAgICBVc2VycyBtYWlsaW5nIGxpc3QKPj4+PiAgICAg ICAgIFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4KPj4+PiAgICAgICAg IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+Pj4+Cj4+Pj4K Pj4+Cj4+Pgo+Pgo+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwo+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4gVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNl cnNAb3ZpcnQub3JnPgo+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v dXNlcnMKPgoKCi0tLS0tLS0tLS0tLS0tMDUwMTA0MDUwODA5MDUwNzA2MDQwNjAyCkNvbnRlbnQt VHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6 IDhiaXQKCjxodG1sPgogIDxoZWFkPgogICAgPG1ldGEgY29udGVudD0idGV4dC9odG1sOyBjaGFy c2V0PXV0Zi04IiBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiPgogIDwvaGVhZD4KICA8Ym9keSB0 ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxicj4KICAgIDxicj4KICAgIDxk aXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+T24gMDkvMjQvMjAxNSAwMjozOCBBTSwgRGFycmVs bCBCdWRpYwogICAgICB3cm90ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlCiAgICAg IGNpdGU9Im1pZDpGNTI4MDMyMC0xNTk4LTQ3MjEtQTVENC1DRTMwMzVGMUU3QzNAb25ob2x5Z3Jv dW5kLmNvbSIKICAgICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRl bnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICAgICAgVGhpcyBp cyBhIGtub3duIGlzc3VlIGluIG92ZXJ0IDMuNS54IGFuZCBiZWxvdy4gSXTigJlzIGJlZW4gc29s dmVkCiAgICAgIGluIHRoZSB1cGNvbWluZyBvdmlydCAzLjYuCiAgICAgIDxkaXYgY2xhc3M9IiI+ PGJyIGNsYXNzPSIiPgogICAgICA8L2Rpdj4KICAgICAgPGRpdiBjbGFzcz0iIj5SZWxhdGVkIHRv wqA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAgICAgICBocmVmPSJodHRwczovL2J1Z3pp bGxhLnJlZGhhdC5jb20vc2hvd19idWcuY2dpP2lkPTExNzI5MDUiCiAgICAgICAgICBjbGFzcz0i Ij5odHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19idWcuY2dpP2lkPTExNzI5MDU8L2E+ LAogICAgICAgIHRoZSBmaXggaW52b2x2ZWQgc2V0dGluZyB1cCBhIHNwZWNpYWwgY2dyb3VwIGZv ciB0aGUgbW91bnQsIGJ1dAogICAgICAgIGkgY2Fu4oCZdCBmaW5kIHRoZSBleGFjdCBkZXRhaWxz IGF0bS48L2Rpdj4KICAgICAgPGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+CiAgICAgIDwvZGl2 PgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogICAgSSBoYXZlIHZkc20gNC4xNy42LTAuZWw3 LmNlbnRvcyBhbHJlYWR5IGluc3RhbGxlZCBvbiB0aGUgaG9zdHMuIFNvIEkKICAgIGFtIG5vdCBz dXJlIGFib3ZlIGJ1ZyA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAgIGhyZWY9Imh0dHBz Oi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3MjkwNSIKICAgICAgY2xh c3M9IiI+MTE3MjkwNTwvYT4gZml4ZXMgdGhpcyBjb3JyZWN0bHkuPGJyPgogICAgPGJyPgogICAg UmVnYXJkcyw8YnI+CiAgICBSYW1lc2g8YnI+CiAgICA8YnI+CiAgICA8YmxvY2txdW90ZQogICAg ICBjaXRlPSJtaWQ6RjUyODAzMjAtMTU5OC00NzIxLUE1RDQtQ0UzMDM1RjFFN0MzQG9uaG9seWdy b3VuZC5jb20iCiAgICAgIHR5cGU9ImNpdGUiPgogICAgICA8ZGl2IGNsYXNzPSIiPjxiciBjbGFz cz0iIj4KICAgICAgICA8ZGl2PgogICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xh c3M9IiI+CiAgICAgICAgICAgIDxkaXYgY2xhc3M9IiI+T24gU2VwIDIzLCAyMDE1LCBhdCA3OjM4 IEFNLCBSYW1lc2ggTmFjaGltdXRodQogICAgICAgICAgICAgICZsdDs8YSBtb3otZG8tbm90LXNl bmQ9InRydWUiCiAgICAgICAgICAgICAgICBocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNv bSIgY2xhc3M9IiI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7CiAgICAgICAgICAgICAgd3Jv dGU6PC9kaXY+CiAgICAgICAgICAgIDxiciBjbGFzcz0iQXBwbGUtaW50ZXJjaGFuZ2UtbmV3bGlu ZSI+CiAgICAgICAgICAgIDxkaXYgY2xhc3M9IiI+CiAgICAgICAgICAgICAgPG1ldGEgY29udGVu dD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04IgogICAgICAgICAgICAgICAgaHR0cC1lcXVpdj0i Q29udGVudC1UeXBlIiBjbGFzcz0iIj4KICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAi IGJnY29sb3I9IiNGRkZGRkYiIGNsYXNzPSIiPiA8YnIKICAgICAgICAgICAgICAgICAgY2xhc3M9 IiI+CiAgICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICA8ZGl2IGNs YXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA5LzIyLzIwMTUgMDU6NTcgUE0sCiAgICAgICAgICAg ICAgICAgIEFsYXN0YWlyIE5laWwgd3JvdGU6PGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAg PC9kaXY+CiAgICAgICAgICAgICAgICA8YmxvY2txdW90ZQpjaXRlPSJtaWQ6Q0ErU2Fyd29vclUz TFdHNitzUi10SjFCRWJRYTFrNFdSclJYa2FYeTN6LUVQUnJ6N1V3QG1haWwuZ21haWwuY29tIgog ICAgICAgICAgICAgICAgICB0eXBlPSJjaXRlIiBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAg PGRpdiBkaXI9Imx0ciIgY2xhc3M9IiI+WW91IG5lZWQgdG8gc2V0IHRoZQogICAgICAgICAgICAg ICAgICAgIGdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyB0byA1MSU8L2Rpdj4KICAgICAgICAg ICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxiciBjbGFzcz0iIj4KICAgICAgICAg ICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAg ICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICBJIGRpZCB0aGF0LiBCdXQgc3RpbGwg SSBhbSBmYWNpbmcgdGhlIHNhbWUgaXNzdWUuIFZNIGdldAogICAgICAgICAgICAgICAgcGF1c2Vk IHdoZW4gSSBkbyBzb21lIEkvTyB1c2luZyBmaW8gb24gc29tZSBkaXNrcyBiYWNrZWQKICAgICAg ICAgICAgICAgIGJ5IGdsdXN0ZXIuIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIHRoZSBWTSBhZnRl ciB0aGlzLgogICAgICAgICAgICAgICAgTm93IG9ubHkgd2F5IGlzIHRvIGJyaW5nIGRvd24gdGhl IFZNIGFuZCBydW4gYWdhaW4uIEl0CiAgICAgICAgICAgICAgICBydW5zIHN1Y2Nlc3NmdWxseSBv biB0aGUgc2FtZSBob3N0IHdpdGhvdXQgYW55IGlzc3VlLjxicgogICAgICAgICAgICAgICAgICBj bGFzcz0iIj4KICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgIFJl Z2FyZHMsPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgUmFtZXNoPGJyIGNsYXNzPSIiPgog ICAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUK Y2l0ZT0ibWlkOkNBK1Nhcndvb3JVM0xXRzYrc1ItdEoxQkViUWExazRXUnJSWGthWHkzei1FUFJy ejdVd0BtYWlsLmdtYWlsLmNvbSIKICAgICAgICAgICAgICAgICAgdHlwZT0iY2l0ZSIgY2xhc3M9 IiI+CiAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAg ICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gMjIgU2VwdGVtYmVyIDIwMTUg YXQKICAgICAgICAgICAgICAgICAgICAgIDA4OjI1LCBSYW1lc2ggTmFjaGltdXRodSA8c3BhbiBk aXI9Imx0ciIgY2xhc3M9IiI+Jmx0OzxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRv LW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Im1haWx0bzpy bmFjaGltdUByZWRoYXQuY29tIgogICAgICAgICAgICAgICAgICAgICAgICAgIHRhcmdldD0iX2Js YW5rIiBjbGFzcz0iIj48YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJt YWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT48L2E+Jmd0 Ozwvc3Bhbj4KICAgICAgICAgICAgICAgICAgICAgIHdyb3RlOjxiciBjbGFzcz0iIj4KICAgICAg ICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1h cmdpbjowIDAKICAgICAgICAgICAgICAgICAgICAgICAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAj Y2NjCiAgICAgICAgICAgICAgICAgICAgICAgIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogICAg ICAgICAgICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYi IGNsYXNzPSIiPjxzcGFuCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iIj4gPGJy IGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iIj5PbiAwOS8yMi8yMDE1IDA1OjQz IFBNLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBBbGFzdGFpciBOZWlsIHdyb3RlOjxi ciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9IiI+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiIGNsYXNzPSIiPndoYXQgYXJl CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGhlwqBnbHVzdGVyLXF1b3J1bS10eXBl CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYW5kwqBnbHVzdGVyLnNlcnZlci1xdW9y dW0tcmF0aW8KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoHNldHRpbmdzIG9uIHRo ZSB2b2x1bWU/PC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9 ImdtYWlsX2V4dHJhIj48YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAg ICAgIDwvc3Bhbj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IHN0eWxlPSJvdXRsaW5l LXN0eWxlOm5vbmUiIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0ib3ZlcmZsb3c6aGlkZGVuO3RleHQt b3ZlcmZsb3c6ZWxsaXBzaXM7d2hpdGUtc3BhY2U6bm93cmFwIgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBjbGFzcz0iIj48YiBjbGFzcz0iIj5jbHVzdGVyLnNlcnZlci1xdW9ydW0tdHlw ZTwvYj46c2VydmVyPGJyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9IiI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgdGl0bGU9IiIgc3R5bGU9Im91dGxp bmUtc3R5bGU6bm9uZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iIj4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBzdHlsZT0ib3ZlcmZsb3c6aGlkZGVuO3RleHQtb3ZlcmZsb3c6ZWxsaXBz aXM7d2hpdGUtc3BhY2U6bm93cmFwIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Y2xhc3M9IiI+PGIgY2xhc3M9IiI+Y2x1c3Rlci5xdW9ydW0tdHlwZTwvYj46YXV0bzxicgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iIj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIDxiIGNsYXNzPSIiPmdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRp bwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpcyBub3Qgc2V0LjwvYj48YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBPbmUgYnJp Y2sgcHJvY2VzcyBpcyBwdXJwb3NlZnVsbHkga2lsbGVkwqAKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgYnV0IHJlbWFpbmluZyB0d28gYnJpY2tzIGFyZSB1cCBhbmQKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgcnVubmluZy48YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg UmVnYXJkcyw8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFJhbWVz aDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAg ICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgIDxzcGFu IGNsYXNzPSIiPiA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8Ymxv Y2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiAyMiBTZXB0ZW1iZXIKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDIwMTUgYXQgMDY6MjQsIFJhbWVzaCBOYWNoaW11dGh1IDxz cGFuCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpcj0ibHRyIiBjbGFzcz0i Ij4mbHQ7PGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb3otZG8tbm90 LXNlbmQ9InRydWUiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9 Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSI+PGEgY2xhc3M9Im1vei10 eHQtbGluay1hYmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20iPnJu YWNoaW11QHJlZGhhdC5jb208L2E+PC9hPiZndDs8L3NwYW4+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB3cm90ZTo8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYwogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJnY29sb3I9IiNGRkZGRkYiIGNsYXNzPSIi PiBIaSw8YnIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsYXNzPSIi PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgIEkgYW0gbm90IGFibGUgdG8g cmVzdW1lIGEgVk0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3aGljaCB3 YXMgcGF1c2VkIGJlY2F1c2Ugb2YKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBnbHVzdGVyIGNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBpcyB3aGF0IGhhcHBlbmVkIGluIG15IHNldHVwLiA8YnIKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAxLiBDcmVhdGVkIGEgZ2x1c3RlciBzdG9yYWdlCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZG9tYWluIHdoaWNoIGlzIGJhY2tlZCBieSBn bHVzdGVyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdm9sdW1lIHdpdGgg cmVwbGljYSAzLiA8YnIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNs YXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDIuIEtpbGxlZCBv bmUgYnJpY2sgcHJvY2Vzcy4gU28KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBvbmx5IHR3byBicmlja3MgYXJlIHJ1bm5pbmcgaW4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICByZXBsaWNhIDMgc2V0dXAuPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDMuIENyZWF0ZWQgdHdvIFZNczxiciBjbGFzcz0iIj4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA0LiBTdGFydGVkIHNvbWUgSU8g dXNpbmcgZmlvIG9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYm90aCBv ZiB0aGUgVk1zPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDUuIEFmdGVyIHNvbWUgdGltZSBnb3QgdGhlCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgZm9sbG93aW5nIGVycm9yIGluIGdsdXN0ZXIgbW91bnQKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBhbmQgVk1zIG1vdmVkIHRvIHBhdXNlZCBzdGF0ZS48 YnIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsYXNzPSIiPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqDCoMKgwqAgIiA8c3Bh bgpzdHlsZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25vc3BhY2U7Zm9udC1z aXplOjExcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDtmb250LXdlaWdo dDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEzLjJweDt0ZXh0LWFs aWduOmxlZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7d2hpdGUtc3BhY2U6 bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0YW50O2Zsb2F0Om5v bmU7YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgY2xhc3M9IiI+c2VydmVyIDxhCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0iaHR0cDovLzEwLjcwLjQ1LjE3 OjQ5MjE3LyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0 PSJfYmxhbmsiIGNsYXNzPSIiPjEwLjcwLjQ1LjE3OjQ5MjE3PC9hPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgaGFzIG5vdCByZXNwb25kZWQgaW4gdGhlIGxhc3QgNDIK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNlY29uZHMsIGRpc2Nvbm5l Y3RpbmcuIjxicgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFz cz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAg Ijwvc3Bhbj48c3BhbgpzdHlsZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTptb25v c3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1h bDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0OjEz LjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5vbmU7 d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1wb3J0 YW50O2Zsb2F0Om5vbmU7YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9IiI+PHNwYW4Kc3R5bGU9ImNv bG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZToxMXB4O2Zv bnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2xl dHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0O3Rl eHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1hbDt3b3Jk LXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpub25lO2JhY2tncm91 bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgY2xhc3M9IiI+dm1zdG9yZS1yZXBsaWNhdGUtMDoKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0 NjVkZmM2OgoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRmFpbGlu ZyBXUklURSBhcyBxdW9ydW0gaXMgbm90CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIG1ldDwvc3Bhbj4iPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgwqDCoMKgwqDCoCBtb3JlIGdsdXN0ZXIgbW91bnQgbG9ncyBhdAog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEgbW96LWRvLW5vdC1zZW5k PSJ0cnVlIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0i bW96LXR4dC1saW5rLWZyZWV0ZXh0IgpocmVmPSJodHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBG Ij5odHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBGPC9hPjxicgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGdsdXN0ZXIKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBxdW9ydW0gaXMgYWN0aXZlIGFuZCBJIGFtIGFi bGUgdG8KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3cml0ZSB0aGUgdGhl IGdsdXN0ZXIgZmlsZSBzeXN0ZW0uPGJyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA3 LiBXaGVuIEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBkb2Vzbid0IHdvcmsgYW5kIEkgZ290IGZvbGxvd2luZwogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVycm9yIGluIHZkc20gbG9nLjxiciBjbGFzcz0i Ij4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgIDxhIG1v ei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTUiCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIgY2xhc3M9IiI+aHR0cDovL3Bh c3RlYmluLmNvbS9hWGlhbVkxNTwvYT48YnIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUmVnYXJkcyw8 YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUmFtZXNo PGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiciBj bGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnIgY2xhc3M9IiI+CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdDxiciBjbGFzcz0i Ij4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEgbW96LWRvLW5vdC1zZW5k PSJ0cnVlIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Im1haWx0 bzpVc2Vyc0BvdmlydC5vcmciCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg dGFyZ2V0PSJfYmxhbmsiIGNsYXNzPSIiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnIKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGEgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby91c2VycyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGNsYXNzPSIiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9s aXN0aW5mby91c2VyczwvYT48YnIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyIGNsYXNz PSIiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9k aXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAg PC9zcGFuPjwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAg ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAg ICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAg ICAgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnIKICAg ICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdDxi ciBjbGFzcz0iIj4KICAgICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhyZWY9 Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciCiAgICAgICAgICAgICAgICBjbGFzcz0iIj5Vc2Vyc0Bv dmlydC5vcmc8L2E+PGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgIDxhIGNsYXNzPSJtb3otdHh0 LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0 aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJz PC9hPjxiciBjbGFzcz0iIj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Jsb2NrcXVv dGU+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJyIGNsYXNzPSIiPgogICAgICA8L2Rpdj4KICAg IDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0t MDUwMTA0MDUwODA5MDUwNzA2MDQwNjAyLS0K --===============7093251404989740408==-- From nsoffer at redhat.com Thu Sep 24 01:52:54 2015 Content-Type: multipart/mixed; boundary="===============1995505348501880443==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Thu, 24 Sep 2015 08:52:52 +0300 Message-ID: In-Reply-To: F5280320-1598-4721-A5D4-CE3035F1E7C3@onholyground.com --===============1995505348501880443== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable The details are here: https://gerrit.ovirt.org/#/c/40240 The link exist on the bug of course. On Thu, Sep 24, 2015 at 12:08 AM, Darrell Budic wrote: > This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved = in the > upcoming ovirt 3.6. > > Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the fix > involved setting up a special cgroup for the mount, but i can=E2=80=99t f= ind the > exact details atm. > > > On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu > wrote: > > > > On 09/22/2015 05:57 PM, Alastair Neil wrote: > > You need to set the gluster.server-quorum-ratio to 51% > > > I did that. But still I am facing the same issue. VM get paused when I do > some I/O using fio on some disks backed by gluster. I am not able to resu= me > the VM after this. Now only way is to bring down the VM and run again. It > runs successfully on the same host without any issue. > > Regards, > Ramesh > > On 22 September 2015 at 08:25, Ramesh Nachimuthu > wrote: > >> >> >> On 09/22/2015 05:43 PM, Alastair Neil wrote: >> >> what are the gluster-quorum-type and gluster.server-quorum-ratio >> settings on the volume? >> >> >> *cluster.server-quorum-type*:server >> *cluster.quorum-type*:auto >> *gluster.server-quorum-ratio is not set.* >> >> One brick process is purposefully killed but remaining two bricks are up >> and running. >> >> Regards, >> Ramesh >> >> On 22 September 2015 at 06:24, Ramesh Nachimuthu < >> rnachimu(a)redhat.com> wrote: >> >>> Hi, >>> >>> I am not able to resume a VM which was paused because of gluster >>> client quorum issue. Here is what happened in my setup. >>> >>> 1. Created a gluster storage domain which is backed by gluster volume >>> with replica 3. >>> 2. Killed one brick process. So only two bricks are running in replica 3 >>> setup. >>> 3. Created two VMs >>> 4. Started some IO using fio on both of the VMs >>> 5. After some time got the following error in gluster mount and VMs >>> moved to paused state. >>> " server 10.70.45.17:49217 has not responded in the last 42 >>> seconds, disconnecting." >>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: >>> Failing WRITE as quorum is not met" >>> more gluster mount logs at >>> http://pastebin.com/UmiUQq0F >>> 6. After some time gluster quorum is active and I am able to write the >>> the gluster file system. >>> 7. When I try to resume the VM it doesn't work and I got following error >>> in vdsm log. >>> http://pastebin.com/aXiamY15 >>> >>> >>> Regards, >>> Ramesh >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============1995505348501880443== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+VGhlIGRldGFpbHMgYXJlIGhlcmU6PGRpdj48YSBocmVmPSJodHRwczov L2dlcnJpdC5vdmlydC5vcmcvIy9jLzQwMjQwIj5odHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9j LzQwMjQwPC9hPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhlIGxpbmsgZXhpc3Qgb24gdGhl IGJ1ZyBvZiBjb3Vyc2UuPC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48 ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1LCBTZXAgMjQsIDIwMTUgYXQgMTI6MDggQU0s IERhcnJlbGwgQnVkaWMgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86YnVkaWNA b25ob2x5Z3JvdW5kLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmJ1ZGljQG9uaG9seWdyb3VuZC5jb208 L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIg c3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRp bmctbGVmdDoxZXgiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj5UaGlzIGlzIGEg a25vd24gaXNzdWUgaW4gb3ZlcnQgMy41LnggYW5kIGJlbG93LiBJdOKAmXMgYmVlbiBzb2x2ZWQg aW4gdGhlIHVwY29taW5nIG92aXJ0IDMuNi48ZGl2Pjxicj48L2Rpdj48ZGl2PlJlbGF0ZWQgdG/C oDxhIGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3 MjkwNSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1 Zy5jZ2k/aWQ9MTE3MjkwNTwvYT4sIHRoZSBmaXggaW52b2x2ZWQgc2V0dGluZyB1cCBhIHNwZWNp YWwgY2dyb3VwIGZvciB0aGUgbW91bnQsIGJ1dCBpIGNhbuKAmXQgZmluZCB0aGUgZXhhY3QgZGV0 YWlscyBhdG0uPC9kaXY+PGRpdj48ZGl2IGNsYXNzPSJoNSI+PGRpdj48YnI+PC9kaXY+PGRpdj48 YnI+PGRpdj48YmxvY2txdW90ZSB0eXBlPSJjaXRlIj48ZGl2Pk9uIFNlcCAyMywgMjAxNSwgYXQg NzozOCBBTSwgUmFtZXNoIE5hY2hpbXV0aHUgJmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUBy ZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7IHdy b3RlOjwvZGl2Pjxicj48ZGl2PgogIAogICAgCiAgCiAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2Nv bG9yPSIjRkZGRkZGIj4KICAgIDxicj4KICAgIDxicj4KICAgIDxkaXY+T24gMDkvMjIvMjAxNSAw NTo1NyBQTSwgQWxhc3RhaXIgTmVpbAogICAgICB3cm90ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxi bG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICA8ZGl2IGRpcj0ibHRyIj5Zb3UgbmVlZCB0byBz ZXQgdGhlIGdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbyB0bwogICAgICAgIDUxJTwvZGl2Pgog ICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgogICAgICA8L2Rpdj4KICAgIDwvYmxv Y2txdW90ZT4KICAgIDxicj4KICAgIEkgZGlkIHRoYXQuIEJ1dCBzdGlsbCBJIGFtIGZhY2luZyB0 aGUgc2FtZSBpc3N1ZS4gVk0gZ2V0IHBhdXNlZCB3aGVuCiAgICBJIGRvIHNvbWUgSS9PIHVzaW5n IGZpbyBvbiBzb21lIGRpc2tzIGJhY2tlZCBieSBnbHVzdGVyLiBJIGFtIG5vdAogICAgYWJsZSB0 byByZXN1bWUgdGhlIFZNIGFmdGVyIHRoaXMuIE5vdyBvbmx5IHdheSBpcyB0byBicmluZyBkb3du IHRoZQogICAgVk0gYW5kIHJ1biBhZ2Fpbi4gSXQgcnVucyBzdWNjZXNzZnVsbHkgb24gdGhlIHNh bWUgaG9zdCB3aXRob3V0IGFueQogICAgaXNzdWUuPGJyPgogICAgPGJyPgogICAgUmVnYXJkcyw8 YnI+CiAgICBSYW1lc2g8YnI+CiAgICA8YnI+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4K ICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgIDxkaXYgY2xhc3M9ImdtYWls X3F1b3RlIj5PbiAyMiBTZXB0ZW1iZXIgMjAxNSBhdCAwODoyNSwgUmFtZXNoCiAgICAgICAgICBO YWNoaW11dGh1IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJl ZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5ybmFjaGltdUByZWRoYXQuY29tPC9hPiZndDs8L3Nw YW4+CiAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21h aWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBz b2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBi Z2NvbG9yPSIjRkZGRkZGIj48c3Bhbj4gPGJyPgogICAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICAgICAgPGRpdj5PbiAwOS8yMi8yMDE1IDA1OjQzIFBNLCBBbGFzdGFpciBOZWlsIHdyb3Rl Ojxicj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUg dHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPndoYXQgYXJlIHRo ZcKgZ2x1c3Rlci1xdW9ydW0tdHlwZQogICAgICAgICAgICAgICAgICAgIGFuZMKgZ2x1c3Rlci5z ZXJ2ZXItcXVvcnVtLXJhdGlvIMKgc2V0dGluZ3Mgb24gdGhlCiAgICAgICAgICAgICAgICAgICAg dm9sdW1lPzwvZGl2PgogICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+ PGJyPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90 ZT4KICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICA8L3NwYW4+CiAgICAgICAgICAg ICAgPGRpdiBzdHlsZT0ib3V0bGluZS1zdHlsZTpub25lIj4KICAgICAgICAgICAgICAgIDxkaXYg c3R5bGU9Im92ZXJmbG93OmhpZGRlbjt0ZXh0LW92ZXJmbG93OmVsbGlwc2lzO3doaXRlLXNwYWNl Om5vd3JhcCI+PGI+Y2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+OnNlcnZlcjxicj4KICAg ICAgICAgICAgICAgICAgPGRpdiB0aXRsZT0iIiBzdHlsZT0ib3V0bGluZS1zdHlsZTpub25lIj4K ICAgICAgICAgICAgICAgICAgICA8ZGl2IHN0eWxlPSJvdmVyZmxvdzpoaWRkZW47dGV4dC1vdmVy ZmxvdzplbGxpcHNpczt3aGl0ZS1zcGFjZTpub3dyYXAiPjxiPmNsdXN0ZXIucXVvcnVtLXR5cGU8 L2I+OmF1dG88YnI+CiAgICAgICAgICAgICAgICAgICAgICA8Yj5nbHVzdGVyLnNlcnZlci1xdW9y dW0tcmF0aW8gaXMgbm90IHNldC48L2I+PGJyPgogICAgICAgICAgICAgICAgICAgICAgPGJyPgog ICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAg ICAgICAgICAgICAgT25lIGJyaWNrIHByb2Nlc3MgaXMgcHVycG9zZWZ1bGx5IGtpbGxlZMKgIGJ1 dAogICAgICAgICAgICAgICAgICByZW1haW5pbmcgdHdvIGJyaWNrcyBhcmUgdXAgYW5kIHJ1bm5p bmcuPGJyPgogICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgIFJlZ2FyZHMs PGJyPgogICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAgICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICA8c3Bhbj4gPGJyPgogICAgICAgICAg ICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgIDxkaXYgY2xh c3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9x dW90ZSI+T24gMjIgU2VwdGVtYmVyIDIwMTUgYXQKICAgICAgICAgICAgICAgICAgICAgIDA2OjI0 LCBSYW1lc2ggTmFjaGltdXRodSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpy bmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+PC9hPjxhIGhyZWY9Im1haWx0bzpy bmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwv YT4mZ3Q7PC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAg ICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2lu OjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+ CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZG RkZGRiI+IEhpLDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgwqDCoCBJIGFtIG5vdCBhYmxlIHRvIHJlc3VtZSBhIFZNIHdoaWNoIHdh cwogICAgICAgICAgICAgICAgICAgICAgICAgIHBhdXNlZCBiZWNhdXNlIG9mIGdsdXN0ZXIgY2xp ZW50IHF1b3J1bSBpc3N1ZS4KICAgICAgICAgICAgICAgICAgICAgICAgICBIZXJlIGlzIHdoYXQg aGFwcGVuZWQgaW4gbXkgc2V0dXAuIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgMS4gQ3JlYXRlZCBhIGdsdXN0ZXIgc3RvcmFnZSBk b21haW4gd2hpY2ggaXMKICAgICAgICAgICAgICAgICAgICAgICAgICBiYWNrZWQgYnkgZ2x1c3Rl ciB2b2x1bWUgd2l0aCByZXBsaWNhIDMuIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAy LiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkgdHdvCiAgICAgICAgICAgICAgICAg ICAgICAgICAgYnJpY2tzIGFyZSBydW5uaW5nIGluIHJlcGxpY2EgMyBzZXR1cC48YnI+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgMy4gQ3JlYXRlZCB0d28gVk1zPGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgIDQuIFN0YXJ0ZWQgc29tZSBJTyB1c2luZyBmaW8gb24gYm90aCBvZiB0aGUK ICAgICAgICAgICAgICAgICAgICAgICAgICBWTXM8YnI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluCiAgICAgICAg ICAgICAgICAgICAgICAgICAgZ2x1c3RlciBtb3VudCBhbmQgVk1zIG1vdmVkIHRvIHBhdXNlZCBz dGF0ZS48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgwqDCoMKgwqDCoMKgwqDCoCAmcXVv dDsgPHNwYW4gc3R5bGU9ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNl O2ZvbnQtc2l6ZToxMXB4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9u dC13ZWlnaHQ6bm9ybWFsO2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7 dGV4dC1hbGlnbjpsZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRl LXNwYWNlOm5vcm1hbDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtm bG9hdDpub25lO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+c2VydmVyCgoKICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly8xMC43MC40NS4xNzo0OTIx Ny8iIHRhcmdldD0iX2JsYW5rIj4xMC43MC40NS4xNzo0OTIxNzwvYT4gaGFzCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICBub3QgcmVzcG9uZGVkIGluIHRoZSBsYXN0IDQyIHNlY29uZHMsCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICBkaXNjb25uZWN0aW5nLiZxdW90Ozxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgJnF1b3Q7PC9zcGFuPjxzcGFuIHN0eWxl PSJjb2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFw eDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1h bDtsZXR0ZXItc3BhY2luZzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVm dDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7 d29yZC1zcGFjaW5nOjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNr Z3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuIHN0eWxlPSJjb2xvcjpyZ2IoNTEs NTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250LXN0eWxlOm5v cm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0ZXItc3BhY2lu Zzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0LWluZGVudDow cHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1zcGFjaW5nOjBw eDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5kLWNvbG9yOnJn YigyNTUsMjU1LDI1NSkiPnZtc3RvcmUtcmVwbGljYXRlLTA6CgoKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBGYWlsaW5nIFdSSVRFIGFzIHF1b3J1bSBpcyBub3QgbWV0 PC9zcGFuPiZxdW90Ozxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAg bW9yZSBnbHVzdGVyIG1vdW50IGxvZ3MgYXQgPGEgaHJlZj0iaHR0cDovL3Bhc3RlYmluLmNvbS9V bWlVUXEwRiIgdGFyZ2V0PSJfYmxhbmsiPjwvYT48YSBocmVmPSJodHRwOi8vcGFzdGViaW4uY29t L1VtaVVRcTBGIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRjwv YT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPC9zcGFuPjYuIEFmdGVyIHNvbWUgdGlt ZSBnbHVzdGVyIHF1b3J1bSBpcwogICAgICAgICAgICAgICAgICAgICAgICAgIGFjdGl2ZSBhbmQg SSBhbSBhYmxlIHRvIHdyaXRlIHRoZSB0aGUgZ2x1c3RlcgogICAgICAgICAgICAgICAgICAgICAg ICAgIGZpbGUgc3lzdGVtLjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA3LiBXaGVuIEkg dHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQgZG9lc24mIzM5O3Qgd29yawogICAgICAgICAgICAgICAg ICAgICAgICAgIGFuZCBJIGdvdCBmb2xsb3dpbmcgZXJyb3IgaW4gdmRzbSBsb2cuPGJyPgogICAg ICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgPGEgaHJlZj0iaHR0cDovL3Bhc3RlYmlu LmNvbS9hWGlhbVkxNSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1Z MTU8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgUmVnYXJkcyw8YnI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgUmFtZXNoPGJyPgogICAgICAgICAgICAgICAgICAgICAg ICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAg ICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX188YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIFVz ZXJzIG1haWxpbmcgbGlzdDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPGEgaHJlZj0ibWFp bHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48 YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5v cmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFu ayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4K ICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+ CiAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgPC9zcGFuPjwvZGl2PgogICAgICAg ICAgPC9ibG9ja3F1b3RlPgogICAgICAgIDwvZGl2PgogICAgICAgIDxicj4KICAgICAgPC9kaXY+ CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgPC9kaXY+CgpfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnI+PGEg aHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0 Lm9yZzwvYT48YnI+PGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzPC9hPjxicj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2 PjwvZGl2PjwvZGl2PjwvZGl2Pjxicj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVmPSJtYWlsdG86 VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVmPSJodHRwOi8v bGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIg dGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91 c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pgo= --===============1995505348501880443==-- From nsoffer at redhat.com Thu Sep 24 01:58:12 2015 Content-Type: multipart/mixed; boundary="===============1248959830058144388==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Thu, 24 Sep 2015 08:58:11 +0300 Message-ID: In-Reply-To: 56037E0F.1090904@redhat.com --===============1248959830058144388== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu wrote: > > > On 09/24/2015 02:38 AM, Darrell Budic wrote: > > This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved = in the > upcoming ovirt 3.6. > > Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the fix > involved setting up a special cgroup for the mount, but i can=E2=80=99t f= ind the > exact details atm. > > > I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am > not sure above bug 1172905 > fixes this > correctly. > I think the root cause is the same - qemu cannot recover from glusterfs unmount, and the only way to resume the vm is to restart it with a fresh mount. The mentioned bug handle the case where stopping vdsm kills the glusterfs mount helper. This issue is fixed in 3.6. The issue here seems different. I suggest you open a bug so gluster guys can investigate this. Nir > > Regards, > Ramesh > > > > On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu > wrote: > > > > On 09/22/2015 05:57 PM, Alastair Neil wrote: > > You need to set the gluster.server-quorum-ratio to 51% > > > I did that. But still I am facing the same issue. VM get paused when I do > some I/O using fio on some disks backed by gluster. I am not able to resu= me > the VM after this. Now only way is to bring down the VM and run again. It > runs successfully on the same host without any issue. > > Regards, > Ramesh > > On 22 September 2015 at 08:25, Ramesh Nachimuthu < > rnachimu(a)redhat.com> wrote: > >> >> >> On 09/22/2015 05:43 PM, Alastair Neil wrote: >> >> what are the gluster-quorum-type and gluster.server-quorum-ratio >> settings on the volume? >> >> >> *cluster.server-quorum-type*:server >> *cluster.quorum-type*:auto >> *gluster.server-quorum-ratio is not set.* >> >> One brick process is purposefully killed but remaining two bricks are up >> and running. >> >> Regards, >> Ramesh >> >> On 22 September 2015 at 06:24, Ramesh Nachimuthu < >> rnachimu(a)redhat.com> wrote: >> >>> Hi, >>> >>> I am not able to resume a VM which was paused because of gluster >>> client quorum issue. Here is what happened in my setup. >>> >>> 1. Created a gluster storage domain which is backed by gluster volume >>> with replica 3. >>> 2. Killed one brick process. So only two bricks are running in replica 3 >>> setup. >>> 3. Created two VMs >>> 4. Started some IO using fio on both of the VMs >>> 5. After some time got the following error in gluster mount and VMs >>> moved to paused state. >>> " server 10.70.45.17:49217 has not responded in the last 42 >>> seconds, disconnecting." >>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: >>> Failing WRITE as quorum is not met" >>> more gluster mount logs at http://pastebin.com/UmiUQq0F >>> 6. After some time gluster quorum is active and I am able to write the >>> the gluster file system. >>> 7. When I try to resume the VM it doesn't work and I got following error >>> in vdsm log. >>> http://pastebin.com/aXiamY15 >>> >>> >>> Regards, >>> Ramesh >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============1248959830058144388== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxkaXYgY2xhc3M9ImdtYWls X3F1b3RlIj5PbiBUaHUsIFNlcCAyNCwgMjAxNSBhdCA3OjM3IEFNLCBSYW1lc2ggTmFjaGltdXRo dSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29t IiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90 ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAw IC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgCiAg ICAKICAKICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPjxzcGFuIGNsYXNz PSIiPgogICAgPGJyPgogICAgPGJyPgogICAgPGRpdj5PbiAwOS8yNC8yMDE1IDAyOjM4IEFNLCBE YXJyZWxsIEJ1ZGljCiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUg dHlwZT0iY2l0ZSI+CiAgICAgIAogICAgICBUaGlzIGlzIGEga25vd24gaXNzdWUgaW4gb3ZlcnQg My41LnggYW5kIGJlbG93LiBJdOKAmXMgYmVlbiBzb2x2ZWQKICAgICAgaW4gdGhlIHVwY29taW5n IG92aXJ0IDMuNi4KICAgICAgPGRpdj48YnI+CiAgICAgIDwvZGl2PgogICAgICA8ZGl2PlJlbGF0 ZWQgdG/CoDxhIGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/ aWQ9MTE3MjkwNSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9z aG93X2J1Zy5jZ2k/aWQ9MTE3MjkwNTwvYT4sCiAgICAgICAgdGhlIGZpeCBpbnZvbHZlZCBzZXR0 aW5nIHVwIGEgc3BlY2lhbCBjZ3JvdXAgZm9yIHRoZSBtb3VudCwgYnV0CiAgICAgICAgaSBjYW7i gJl0IGZpbmQgdGhlIGV4YWN0IGRldGFpbHMgYXRtLjwvZGl2PgogICAgICA8ZGl2Pjxicj4KICAg ICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+PC9zcGFuPgogICAgSSBoYXZlIHZk c20gNC4xNy42LTAuZWw3LmNlbnRvcyBhbHJlYWR5IGluc3RhbGxlZCBvbiB0aGUgaG9zdHMuIFNv IEkKICAgIGFtIG5vdCBzdXJlIGFib3ZlIGJ1ZyA8YSBocmVmPSJodHRwczovL2J1Z3ppbGxhLnJl ZGhhdC5jb20vc2hvd19idWcuY2dpP2lkPTExNzI5MDUiIHRhcmdldD0iX2JsYW5rIj4xMTcyOTA1 PC9hPiBmaXhlcyB0aGlzIGNvcnJlY3RseS48YnI+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJy PjwvZGl2PjxkaXY+SSB0aGluayB0aGUgcm9vdCBjYXVzZSBpcyB0aGUgc2FtZSAtIHFlbXUgY2Fu bm90IHJlY292ZXIgZnJvbSBnbHVzdGVyZnMgdW5tb3VudCwgYW5kIHRoZSBvbmx5IHdheSB0byBy ZXN1bWUgdGhlIHZtIGlzIHRvIHJlc3RhcnQgaXQgd2l0aCBhIGZyZXNoIG1vdW50LjwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+VGhlIG1lbnRpb25lZCBidWcgaGFuZGxlIHRoZSBjYXNlIHdoZXJl IHN0b3BwaW5nIHZkc20ga2lsbHMgdGhlIGdsdXN0ZXJmcyBtb3VudCBoZWxwZXIuIFRoaXMgaXNz dWUgaXMgZml4ZWQgaW4gMy42LsKgPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGUgaXNzdWUg aGVyZSBzZWVtcyBkaWZmZXJlbnQuIEkgc3VnZ2VzdCB5b3Ugb3BlbiBhIGJ1ZyBzbyBnbHVzdGVy IGd1eXMgY2FuIGludmVzdGlnYXRlIHRoaXMuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5OaXI8 L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PsKgPC9kaXY+PGJsb2NrcXVv dGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxl ZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiB0ZXh0PSIjMDAwMDAwIiBi Z2NvbG9yPSIjRkZGRkZGIj4KICAgIDxicj4KICAgIFJlZ2FyZHMsPGJyPgogICAgUmFtZXNoPGRp dj48ZGl2IGNsYXNzPSJoNSI+PGJyPgogICAgPGJyPgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0 ZSI+CiAgICAgIDxkaXY+PGJyPgogICAgICAgIDxkaXY+CiAgICAgICAgICA8YmxvY2txdW90ZSB0 eXBlPSJjaXRlIj4KICAgICAgICAgICAgPGRpdj5PbiBTZXAgMjMsIDIwMTUsIGF0IDc6MzggQU0s IFJhbWVzaCBOYWNoaW11dGh1CiAgICAgICAgICAgICAgJmx0OzxhIGhyZWY9Im1haWx0bzpybmFj aGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4m Z3Q7CiAgICAgICAgICAgICAgd3JvdGU6PC9kaXY+CiAgICAgICAgICAgIDxicj4KICAgICAgICAg ICAgPGRpdj4KICAgICAgICAgICAgICAKICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAi IGJnY29sb3I9IiNGRkZGRkYiPiA8YnI+CiAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAg ICAgICA8ZGl2Pk9uIDA5LzIyLzIwMTUgMDU6NTcgUE0sCiAgICAgICAgICAgICAgICAgIEFsYXN0 YWlyIE5laWwgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAg ICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0 ciI+WW91IG5lZWQgdG8gc2V0IHRoZQogICAgICAgICAgICAgICAgICAgIGdsdXN0ZXIuc2VydmVy LXF1b3J1bS1yYXRpbyB0byA1MSU8L2Rpdj4KICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0i Z21haWxfZXh0cmEiPjxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAg ICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICBJIGRp ZCB0aGF0LiBCdXQgc3RpbGwgSSBhbSBmYWNpbmcgdGhlIHNhbWUgaXNzdWUuIFZNIGdldAogICAg ICAgICAgICAgICAgcGF1c2VkIHdoZW4gSSBkbyBzb21lIEkvTyB1c2luZyBmaW8gb24gc29tZSBk aXNrcyBiYWNrZWQKICAgICAgICAgICAgICAgIGJ5IGdsdXN0ZXIuIEkgYW0gbm90IGFibGUgdG8g cmVzdW1lIHRoZSBWTSBhZnRlciB0aGlzLgogICAgICAgICAgICAgICAgTm93IG9ubHkgd2F5IGlz IHRvIGJyaW5nIGRvd24gdGhlIFZNIGFuZCBydW4gYWdhaW4uIEl0CiAgICAgICAgICAgICAgICBy dW5zIHN1Y2Nlc3NmdWxseSBvbiB0aGUgc2FtZSBob3N0IHdpdGhvdXQgYW55IGlzc3VlLjxicj4K ICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgIFJlZ2FyZHMsPGJyPgogICAgICAg ICAgICAgICAgUmFtZXNoPGJyPgogICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg PGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9Imdt YWlsX2V4dHJhIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+ T24gMjIgU2VwdGVtYmVyIDIwMTUgYXQKICAgICAgICAgICAgICAgICAgICAgIDA4OjI1LCBSYW1l c2ggTmFjaGltdXRodSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGlt dUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+PC9hPjxhIGhyZWY9Im1haWx0bzpybmFjaGlt dUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7 PC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAg ICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAw IC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgICAg ICAgICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+ PHNwYW4+IDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDxkaXY+T24gMDkvMjIvMjAxNSAwNTo0MyBQTSwKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgQWxhc3RhaXIgTmVpbCB3cm90ZTo8YnI+CiAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1 b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0i bHRyIj53aGF0IGFyZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZcKgZ2x1c3Rl ci1xdW9ydW0tdHlwZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFuZMKgZ2x1c3Rl ci5zZXJ2ZXItcXVvcnVtLXJhdGlvCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgwqBz ZXR0aW5ncyBvbiB0aGUgdm9sdW1lPzwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8 L3NwYW4+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBzdHlsZT0ib3V0bGluZS1zdHls ZTpub25lIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgc3R5bGU9Im92ZXJmbG93 OmhpZGRlbjt0ZXh0LW92ZXJmbG93OmVsbGlwc2lzO3doaXRlLXNwYWNlOm5vd3JhcCI+PGI+Y2x1 c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+OnNlcnZlcjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgPGRpdiB0aXRsZT0iIiBzdHlsZT0ib3V0bGluZS1zdHlsZTpub25lIj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IHN0eWxlPSJvdmVyZmxvdzpoaWRkZW47 dGV4dC1vdmVyZmxvdzplbGxpcHNpczt3aGl0ZS1zcGFjZTpub3dyYXAiPjxiPmNsdXN0ZXIucXVv cnVtLXR5cGU8L2I+OmF1dG88YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 Yj5nbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0aW8KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgaXMgbm90IHNldC48L2I+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgT25lIGJyaWNrIHByb2Nlc3MgaXMgcHVycG9zZWZ1bGx5IGtpbGxlZMKgCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIGJ1dCByZW1haW5pbmcgdHdvIGJyaWNrcyBhcmUgdXAgYW5kCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJ1bm5pbmcuPGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFJlZ2FyZHMs PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgICAgICAgICAgICAgICA8c3Bhbj4gPGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gMjIgU2VwdGVtYmVyCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAyMDE1IGF0IDA2OjI0LCBSYW1lc2ggTmFjaGltdXRo dSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29t IiB0YXJnZXQ9Il9ibGFuayI+PC9hPjxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29t IiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0i bWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0 OjFleCI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAw MDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+IEhpLDxicj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg wqDCoCBJIGFtIG5vdCBhYmxlIHRvIHJlc3VtZSBhIFZNCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgd2hpY2ggd2FzIHBhdXNlZCBiZWNhdXNlIG9mCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZ2x1c3RlciBjbGllbnQgcXVvcnVtIGlzc3VlLiBIZXJl CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaXMgd2hhdCBoYXBwZW5lZCBp biBteSBzZXR1cC4gPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAxLiBDcmVhdGVkIGEgZ2x1 c3RlciBzdG9yYWdlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZG9tYWlu IHdoaWNoIGlzIGJhY2tlZCBieSBnbHVzdGVyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgdm9sdW1lIHdpdGggcmVwbGljYSAzLiA8YnI+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgMi4gS2lsbGVkIG9uZSBicmljayBwcm9jZXNzLiBTbwogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG9ubHkgdHdvIGJyaWNrcyBhcmUgcnVubmlu ZyBpbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJlcGxpY2EgMyBzZXR1 cC48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMy4gQ3JlYXRlZCB0 d28gVk1zPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDQuIFN0YXJ0 ZWQgc29tZSBJTyB1c2luZyBmaW8gb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBib3RoIG9mIHRoZSBWTXM8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBmb2xsb3dpbmcgZXJyb3IgaW4gZ2x1c3RlciBtb3VudAogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFuZCBWTXMgbW92ZWQgdG8gcGF1c2VkIHN0YXRl Ljxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgwqDC oMKgICZxdW90OyA8c3BhbiBzdHlsZT0iY29sb3I6cmdiKDUxLDUxLDUxKTtmb250LWZhbWlseTpt b25vc3BhY2U7Zm9udC1zaXplOjExcHg7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5v cm1hbDtmb250LXdlaWdodDpub3JtYWw7bGV0dGVyLXNwYWNpbmc6bm9ybWFsO2xpbmUtaGVpZ2h0 OjEzLjJweDt0ZXh0LWFsaWduOmxlZnQ7dGV4dC1pbmRlbnQ6MHB4O3RleHQtdHJhbnNmb3JtOm5v bmU7d2hpdGUtc3BhY2U6bm9ybWFsO3dvcmQtc3BhY2luZzowcHg7ZGlzcGxheTppbmxpbmUhaW1w b3J0YW50O2Zsb2F0Om5vbmU7YmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj5zZXJ2 ZXIgPGEgaHJlZj0iaHR0cDovLzEwLjcwLjQ1LjE3OjQ5MjE3LyIgdGFyZ2V0PSJfYmxhbmsiPjEw LjcwLjQ1LjE3OjQ5MjE3PC9hPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgaGFzIG5vdCByZXNwb25kZWQgaW4gdGhlIGxhc3QgNDIKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHNlY29uZHMsIGRpc2Nvbm5lY3RpbmcuJnF1b3Q7PGJyPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgwqDCoMKgwqDCoCAmcXVvdDs8L3Nw YW4+PHNwYW4gc3R5bGU9ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNl O2ZvbnQtc2l6ZToxMXB4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9u dC13ZWlnaHQ6bm9ybWFsO2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7 dGV4dC1hbGlnbjpsZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRl LXNwYWNlOm5vcm1hbDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtm bG9hdDpub25lO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+PHNwYW4gc3R5bGU9 ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZToxMXB4 O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFs O2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0 O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1hbDt3 b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpub25lO2JhY2tn cm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+dm1zdG9yZS1yZXBsaWNhdGUtMDoKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3 N2QtZTA5OWY0NjVkZmM2OgoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgRmFpbGluZyBXUklURSBhcyBxdW9ydW0gaXMgbm90CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG1ldDwvc3Bhbj4mcXVvdDs8YnI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgIG1vcmUgZ2x1c3RlciBtb3VudCBsb2dz IGF0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YSBocmVmPSJodHRw Oi8vcGFzdGViaW4uY29tL1VtaVVRcTBGIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Bhc3RlYmlu LmNvbS9VbWlVUXEwRjwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPC9zcGFuPjYuIEFmdGVyIHNvbWUgdGltZSBnbHVzdGVyCiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgcXVvcnVtIGlzIGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd3JpdGUgdGhlIHRoZSBnbHVzdGVyIGZp bGUgc3lzdGVtLjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA3LiBX aGVuIEkgdHJ5IHRvIHJlc3VtZSB0aGUgVk0gaXQKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBkb2VzbiYjMzk7dCB3b3JrIGFuZCBJIGdvdCBmb2xsb3dpbmcKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlcnJvciBpbiB2ZHNtIGxvZy48YnI+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgwqDCoMKgwqDCoCA8YSBocmVmPSJodHRw Oi8vcGFzdGViaW4uY29tL2FYaWFtWTE1IiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL3Bhc3RlYmlu LmNvbS9hWGlhbVkxNTwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSZWdhcmRzLDxicj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgpfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3Jn IiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWls bWFuL2xpc3RpbmZvL3VzZXJzIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRw Oi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg PC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxv Y2txdW90ZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICA8L3NwYW4+PC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVv dGU+CiAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgPGJyPgog ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAg ICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAg ICAgICBVc2VycyBtYWlsaW5nIGxpc3Q8YnI+CiAgICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRv OlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+ CiAgICAgICAgICAgICAgPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp c3RpbmZvL3VzZXJzIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWls bWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8 L2Jsb2NrcXVvdGU+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJyPgogICAgICA8L2Rpdj4KICAg IDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2Rpdj48L2Rpdj48L2Rpdj4KCjxicj5fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGlu ZyBsaXN0PGJyPgo8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5v cmc8L2E+PGJyPgo8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu Zm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5v dmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVvdGU+ PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo= --===============1248959830058144388==-- From rnachimu at redhat.com Thu Sep 24 02:06:16 2015 Content-Type: multipart/mixed; boundary="===============4653285908978238859==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Thu, 24 Sep 2015 11:36:11 +0530 Message-ID: <560392D3.6030306@redhat.com> In-Reply-To: CAMRbyyvi0mtEVyae4TJxq-67B7ZKwnEVwUu5Dp6DrChO1Y2_Yw@mail.gmail.com --===============4653285908978238859== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------030503030503080006060300 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit On 09/24/2015 11:28 AM, Nir Soffer wrote: > On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu = > > wrote: > > > > On 09/24/2015 02:38 AM, Darrell Budic wrote: >> This is a known issue in overt 3.5.x and below. It=E2=80=99s been so= lved >> in the upcoming ovirt 3.6. >> >> Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, >> the fix involved setting up a special cgroup for the mount, but i >> can=E2=80=99t find the exact details atm. >> > > I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So > I am not sure above bug 1172905 > fixes this > correctly. > > > I think the root cause is the same - qemu cannot recover from = > glusterfs unmount, and the only way to resume the vm is to restart it = > with a fresh mount. > > The mentioned bug handle the case where stopping vdsm kills the = > glusterfs mount helper. This issue is fixed in 3.6. > > The issue here seems different. I suggest you open a bug so gluster = > guys can investigate this. > Seems like I am hitting the issue reported in bz = https://bugzilla.redhat.com/show_bug.cgi?id=3D1171261. Regards, Ramesh > Nir > > > > Regards, > Ramesh > > >> >>> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu >>> > wrote: >>> >>> >>> >>> On 09/22/2015 05:57 PM, Alastair Neil wrote: >>>> You need to set the gluster.server-quorum-ratio to 51% >>>> >>> >>> I did that. But still I am facing the same issue. VM get paused >>> when I do some I/O using fio on some disks backed by gluster. I >>> am not able to resume the VM after this. Now only way is to >>> bring down the VM and run again. It runs successfully on the >>> same host without any issue. >>> >>> Regards, >>> Ramesh >>> >>>> On 22 September 2015 at 08:25, Ramesh Nachimuthu >>>> > wrote: >>>> >>>> >>>> >>>> On 09/22/2015 05:43 PM, Alastair Neil wrote: >>>>> what are the gluster-quorum-type >>>>> and gluster.server-quorum-ratio settings on the volume? >>>>> >>>> >>>> *cluster.server-quorum-type*:server >>>> *cluster.quorum-type*:auto >>>> *gluster.server-quorum-ratio is not set.* >>>> >>>> One brick process is purposefully killed but remaining two >>>> bricks are up and running. >>>> >>>> Regards, >>>> Ramesh >>>> >>>>> On 22 September 2015 at 06:24, Ramesh Nachimuthu >>>>> > wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am not able to resume a VM which was paused >>>>> because of gluster client quorum issue. Here is what >>>>> happened in my setup. >>>>> >>>>> 1. Created a gluster storage domain which is backed by >>>>> gluster volume with replica 3. >>>>> 2. Killed one brick process. So only two bricks are >>>>> running in replica 3 setup. >>>>> 3. Created two VMs >>>>> 4. Started some IO using fio on both of the VMs >>>>> 5. After some time got the following error in gluster >>>>> mount and VMs moved to paused state. >>>>> " server 10.70.45.17:49217 >>>>> has not responded in the >>>>> last 42 seconds, disconnecting." >>>>> "vmstore-replicate-0: >>>>> e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as >>>>> quorum is not met" >>>>> more gluster mount logs at >>>>> http://pastebin.com/UmiUQq0F >>>>> 6. After some time gluster quorum is active and I am >>>>> able to write the the gluster file system. >>>>> 7. When I try to resume the VM it doesn't work and I >>>>> got following error in vdsm log. >>>>> http://pastebin.com/aXiamY15 >>>>> >>>>> >>>>> Regards, >>>>> Ramesh >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --------------030503030503080006060300 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

On 09/24/2015 11:28 AM, Nir Soffer wrote:
On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com> wrote:
<= br>
On 09/24/2015 02:38 AM, Darrell Budic wrote:
This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved in the upcoming ovirt 3.6.

Related to=C2=A0https://bugzilla.redhat.com/show_= bug.cgi?id=3D1172905, the fix involved setting up a special cgroup for the mount, but i can=E2=80=99t find the exact details= atm.


 I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am not sure above bug 1172905 fixes this correctly.

I think the root cause is the same - qemu cannot recover from glusterfs unmount, and the only way to resume the vm is to restart it with a fresh mount.

The mentioned bug handle the case where stopping vdsm kills the glusterfs mount helper. This issue is fixed in 3.6.=C2=A0

The issue here seems different. I suggest you open a bug so gluster guys can investigate this.


Seems like I am hitting the issue reported in bz https://bugzilla.redhat.com/show_bug.cgi?id=3D11= 71261.

Regards,
Ramesh

Nir


=C2=A0

Regards,
Ramesh



On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu <rnachimu(a)redhat.com= > wrote:


On 09/22/2015 05:57 PM, Alastair Neil wrote:
You need to set the gluster.server-quorum-ratio to 51%


I did that. But still I am facing the same issue. VM get paused when I do some I/O using fio on some disks backed by gluster. I am not able to resume the VM after this. Now only way is to bring down the VM and run again. It runs successfully on the same host without any issue.

Regards,
Ramesh

On 22 September 2015 at 08:25, Ramesh Nachimuthu <rnachimu(a)red= hat.com> wrote:


On 09/22/2015 05:43 PM, Alastair Neil wrote:
what are the=C2=A0gluster-quorum-type and=C2=A0gluster.server-quo= rum-ratio =C2=A0settings on the volum= e?
cluster.server-quorum-type:se= rver
cluster.quorum-type:auto<= br> gluster.server-quorum-= ratio is not set.

One brick process is purposefully killed=C2=A0 but remaining two bricks are up and running.

Regards,
Ramesh

= On 22 September 2015 at 06:24, Ramesh Nachimuthu <rnachimu(a)redhat.= com> wrote:
Hi,

=C2=A0=C2=A0 I am not= able to resume a VM which was paused because of gluster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f1= 9-977d-e099f465dfc6: Failing WRITE as quorum is not met" =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 more gluster mount logs at http://pastebin.com/U= miUQq0F
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 http://pastebin.com/a= XiamY15


Regards,
Ramesh


_______________________________________________
Users mailing list
Users(a)ovirt.org<= br> http://lists.ovirt.org/mailman/listinfo/users





_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/ma= ilman/listinfo/users



_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org= /mailman/listinfo/users



--------------030503030503080006060300-- --===============4653285908978238859== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wMzA1MDMwMzA1MDMwODAwMDYwNjAzMDAKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CgoKT24gMDkvMjQvMjAxNSAxMToyOCBBTSwgTmlyIFNvZmZlciB3cm90ZToKPiBPbiBUaHUsIFNl cCAyNCwgMjAxNSBhdCA3OjM3IEFNLCBSYW1lc2ggTmFjaGltdXRodSAKPiA8cm5hY2hpbXVAcmVk aGF0LmNvbSA8bWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20+PiB3cm90ZToKPgo+Cj4KPiAgICAg T24gMDkvMjQvMjAxNSAwMjozOCBBTSwgRGFycmVsbCBCdWRpYyB3cm90ZToKPj4gICAgIFRoaXMg aXMgYSBrbm93biBpc3N1ZSBpbiBvdmVydCAzLjUueCBhbmQgYmVsb3cuIEl04oCZcyBiZWVuIHNv bHZlZAo+PiAgICAgaW4gdGhlIHVwY29taW5nIG92aXJ0IDMuNi4KPj4KPj4gICAgIFJlbGF0ZWQg dG8gaHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD0xMTcyOTA1LAo+ PiAgICAgdGhlIGZpeCBpbnZvbHZlZCBzZXR0aW5nIHVwIGEgc3BlY2lhbCBjZ3JvdXAgZm9yIHRo ZSBtb3VudCwgYnV0IGkKPj4gICAgIGNhbuKAmXQgZmluZCB0aGUgZXhhY3QgZGV0YWlscyBhdG0u Cj4+Cj4KPiAgICAgSSBoYXZlIHZkc20gNC4xNy42LTAuZWw3LmNlbnRvcyBhbHJlYWR5IGluc3Rh bGxlZCBvbiB0aGUgaG9zdHMuIFNvCj4gICAgIEkgYW0gbm90IHN1cmUgYWJvdmUgYnVnIDExNzI5 MDUKPiAgICAgPGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3 MjkwNT4gZml4ZXMgdGhpcwo+ICAgICBjb3JyZWN0bHkuCj4KPgo+IEkgdGhpbmsgdGhlIHJvb3Qg Y2F1c2UgaXMgdGhlIHNhbWUgLSBxZW11IGNhbm5vdCByZWNvdmVyIGZyb20gCj4gZ2x1c3RlcmZz IHVubW91bnQsIGFuZCB0aGUgb25seSB3YXkgdG8gcmVzdW1lIHRoZSB2bSBpcyB0byByZXN0YXJ0 IGl0IAo+IHdpdGggYSBmcmVzaCBtb3VudC4KPgo+IFRoZSBtZW50aW9uZWQgYnVnIGhhbmRsZSB0 aGUgY2FzZSB3aGVyZSBzdG9wcGluZyB2ZHNtIGtpbGxzIHRoZSAKPiBnbHVzdGVyZnMgbW91bnQg aGVscGVyLiBUaGlzIGlzc3VlIGlzIGZpeGVkIGluIDMuNi4KPgo+IFRoZSBpc3N1ZSBoZXJlIHNl ZW1zIGRpZmZlcmVudC4gSSBzdWdnZXN0IHlvdSBvcGVuIGEgYnVnIHNvIGdsdXN0ZXIgCj4gZ3V5 cyBjYW4gaW52ZXN0aWdhdGUgdGhpcy4KPgoKU2VlbXMgbGlrZSBJIGFtIGhpdHRpbmcgdGhlIGlz c3VlIHJlcG9ydGVkIGluIGJ6IApodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19idWcu Y2dpP2lkPTExNzEyNjEuCgpSZWdhcmRzLApSYW1lc2gKCj4gTmlyCj4KPgo+Cj4gICAgIFJlZ2Fy ZHMsCj4gICAgIFJhbWVzaAo+Cj4KPj4KPj4+ICAgICBPbiBTZXAgMjMsIDIwMTUsIGF0IDc6Mzgg QU0sIFJhbWVzaCBOYWNoaW11dGh1Cj4+PiAgICAgPHJuYWNoaW11QHJlZGhhdC5jb20gPG1haWx0 bzpybmFjaGltdUByZWRoYXQuY29tPj4gd3JvdGU6Cj4+Pgo+Pj4KPj4+Cj4+PiAgICAgT24gMDkv MjIvMjAxNSAwNTo1NyBQTSwgQWxhc3RhaXIgTmVpbCB3cm90ZToKPj4+PiAgICAgWW91IG5lZWQg dG8gc2V0IHRoZSBnbHVzdGVyLnNlcnZlci1xdW9ydW0tcmF0aW8gdG8gNTElCj4+Pj4KPj4+Cj4+ PiAgICAgSSBkaWQgdGhhdC4gQnV0IHN0aWxsIEkgYW0gZmFjaW5nIHRoZSBzYW1lIGlzc3VlLiBW TSBnZXQgcGF1c2VkCj4+PiAgICAgd2hlbiBJIGRvIHNvbWUgSS9PIHVzaW5nIGZpbyBvbiBzb21l IGRpc2tzIGJhY2tlZCBieSBnbHVzdGVyLiBJCj4+PiAgICAgYW0gbm90IGFibGUgdG8gcmVzdW1l IHRoZSBWTSBhZnRlciB0aGlzLiBOb3cgb25seSB3YXkgaXMgdG8KPj4+ICAgICBicmluZyBkb3du IHRoZSBWTSBhbmQgcnVuIGFnYWluLiBJdCBydW5zIHN1Y2Nlc3NmdWxseSBvbiB0aGUKPj4+ICAg ICBzYW1lIGhvc3Qgd2l0aG91dCBhbnkgaXNzdWUuCj4+Pgo+Pj4gICAgIFJlZ2FyZHMsCj4+PiAg ICAgUmFtZXNoCj4+Pgo+Pj4+ICAgICBPbiAyMiBTZXB0ZW1iZXIgMjAxNSBhdCAwODoyNSwgUmFt ZXNoIE5hY2hpbXV0aHUKPj4+PiAgICAgPHJuYWNoaW11QHJlZGhhdC5jb20gPG1haWx0bzpybmFj aGltdUByZWRoYXQuY29tPj4gd3JvdGU6Cj4+Pj4KPj4+Pgo+Pj4+Cj4+Pj4gICAgICAgICBPbiAw OS8yMi8yMDE1IDA1OjQzIFBNLCBBbGFzdGFpciBOZWlsIHdyb3RlOgo+Pj4+PiAgICAgICAgIHdo YXQgYXJlIHRoZSBnbHVzdGVyLXF1b3J1bS10eXBlCj4+Pj4+ICAgICAgICAgYW5kIGdsdXN0ZXIu c2VydmVyLXF1b3J1bS1yYXRpbyAgc2V0dGluZ3Mgb24gdGhlIHZvbHVtZT8KPj4+Pj4KPj4+Pgo+ Pj4+ICAgICAgICAgKmNsdXN0ZXIuc2VydmVyLXF1b3J1bS10eXBlKjpzZXJ2ZXIKPj4+PiAgICAg ICAgICpjbHVzdGVyLnF1b3J1bS10eXBlKjphdXRvCj4+Pj4gICAgICAgICAqZ2x1c3Rlci5zZXJ2 ZXItcXVvcnVtLXJhdGlvIGlzIG5vdCBzZXQuKgo+Pj4+Cj4+Pj4gICAgICAgICBPbmUgYnJpY2sg cHJvY2VzcyBpcyBwdXJwb3NlZnVsbHkga2lsbGVkICBidXQgcmVtYWluaW5nIHR3bwo+Pj4+ICAg ICAgICAgYnJpY2tzIGFyZSB1cCBhbmQgcnVubmluZy4KPj4+Pgo+Pj4+ICAgICAgICAgUmVnYXJk cywKPj4+PiAgICAgICAgIFJhbWVzaAo+Pj4+Cj4+Pj4+ICAgICAgICAgT24gMjIgU2VwdGVtYmVy IDIwMTUgYXQgMDY6MjQsIFJhbWVzaCBOYWNoaW11dGh1Cj4+Pj4+ICAgICAgICAgPHJuYWNoaW11 QHJlZGhhdC5jb20gPG1haWx0bzpybmFjaGltdUByZWRoYXQuY29tPj4gd3JvdGU6Cj4+Pj4+Cj4+ Pj4+ICAgICAgICAgICAgIEhpLAo+Pj4+Pgo+Pj4+PiAgICAgICAgICAgICAgICBJIGFtIG5vdCBh YmxlIHRvIHJlc3VtZSBhIFZNIHdoaWNoIHdhcyBwYXVzZWQKPj4+Pj4gICAgICAgICAgICAgYmVj YXVzZSBvZiBnbHVzdGVyIGNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdAo+Pj4+PiAg ICAgICAgICAgICBoYXBwZW5lZCBpbiBteSBzZXR1cC4KPj4+Pj4KPj4+Pj4gICAgICAgICAgICAg MS4gQ3JlYXRlZCBhIGdsdXN0ZXIgc3RvcmFnZSBkb21haW4gd2hpY2ggaXMgYmFja2VkIGJ5Cj4+ Pj4+ICAgICAgICAgICAgIGdsdXN0ZXIgdm9sdW1lIHdpdGggcmVwbGljYSAzLgo+Pj4+PiAgICAg ICAgICAgICAyLiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkgdHdvIGJyaWNrcyBh cmUKPj4+Pj4gICAgICAgICAgICAgcnVubmluZyBpbiByZXBsaWNhIDMgc2V0dXAuCj4+Pj4+ICAg ICAgICAgICAgIDMuIENyZWF0ZWQgdHdvIFZNcwo+Pj4+PiAgICAgICAgICAgICA0LiBTdGFydGVk IHNvbWUgSU8gdXNpbmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNcwo+Pj4+PiAgICAgICAgICAgICA1 LiBBZnRlciBzb21lIHRpbWUgZ290IHRoZSBmb2xsb3dpbmcgZXJyb3IgaW4gZ2x1c3Rlcgo+Pj4+ PiAgICAgICAgICAgICBtb3VudCBhbmQgVk1zIG1vdmVkIHRvIHBhdXNlZCBzdGF0ZS4KPj4+Pj4g ICAgICAgICAgICAgICAgICAgICAgIiBzZXJ2ZXIgMTAuNzAuNDUuMTc6NDkyMTcKPj4+Pj4gICAg ICAgICAgICAgPGh0dHA6Ly8xMC43MC40NS4xNzo0OTIxNy8+IGhhcyBub3QgcmVzcG9uZGVkIGlu IHRoZQo+Pj4+PiAgICAgICAgICAgICBsYXN0IDQyIHNlY29uZHMsIGRpc2Nvbm5lY3RpbmcuIgo+ Pj4+PiAgICAgICAgICAgICAgICAgICAidm1zdG9yZS1yZXBsaWNhdGUtMDoKPj4+Pj4gICAgICAg ICAgICAgZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OiBGYWlsaW5nIFdSSVRF IGFzCj4+Pj4+ICAgICAgICAgICAgIHF1b3J1bSBpcyBub3QgbWV0Igo+Pj4+PiAgICAgICAgICAg ICAgICAgICBtb3JlIGdsdXN0ZXIgbW91bnQgbG9ncyBhdAo+Pj4+PiAgICAgICAgICAgICBodHRw Oi8vcGFzdGViaW4uY29tL1VtaVVRcTBGCj4+Pj4+ICAgICAgICAgICAgIDYuIEFmdGVyIHNvbWUg dGltZSBnbHVzdGVyIHF1b3J1bSBpcyBhY3RpdmUgYW5kIEkgYW0KPj4+Pj4gICAgICAgICAgICAg YWJsZSB0byB3cml0ZSB0aGUgdGhlIGdsdXN0ZXIgZmlsZSBzeXN0ZW0uCj4+Pj4+ICAgICAgICAg ICAgIDcuIFdoZW4gSSB0cnkgdG8gcmVzdW1lIHRoZSBWTSBpdCBkb2Vzbid0IHdvcmsgYW5kIEkK Pj4+Pj4gICAgICAgICAgICAgZ290IGZvbGxvd2luZyBlcnJvciBpbiB2ZHNtIGxvZy4KPj4+Pj4g ICAgICAgICAgICAgaHR0cDovL3Bhc3RlYmluLmNvbS9hWGlhbVkxNQo+Pj4+Pgo+Pj4+Pgo+Pj4+ PiAgICAgICAgICAgICBSZWdhcmRzLAo+Pj4+PiAgICAgICAgICAgICBSYW1lc2gKPj4+Pj4KPj4+ Pj4KPj4+Pj4gICAgICAgICAgICAgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18KPj4+Pj4gICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0Cj4+Pj4+ICAg ICAgICAgICAgIFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4KPj4+Pj4g ICAgICAgICAgICAgaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJz Cj4+Pj4+Cj4+Pj4+Cj4+Pj4KPj4+Pgo+Pj4KPj4+ICAgICBfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwo+Pj4gICAgIFVzZXJzIG1haWxpbmcgbGlzdAo+Pj4g ICAgIFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4KPj4+ICAgICBodHRw Oi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj4KPgo+Cj4gICAgIF9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gICAgIFVzZXJz IG1haWxpbmcgbGlzdAo+ICAgICBVc2Vyc0BvdmlydC5vcmcgPG1haWx0bzpVc2Vyc0BvdmlydC5v cmc+Cj4gICAgIGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+ Cj4KCgotLS0tLS0tLS0tLS0tLTAzMDUwMzAzMDUwMzA4MDAwNjA2MDMwMApDb250ZW50LVR5cGU6 IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0 Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11 dGYtOCIgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIj4KICA8L2hlYWQ+CiAgPGJvZHkgdGV4dD0i IzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICA8YnI+CiAgICA8YnI+CiAgICA8ZGl2IGNs YXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA5LzI0LzIwMTUgMTE6MjggQU0sIE5pciBTb2ZmZXIK ICAgICAgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZQpjaXRlPSJtaWQ6Q0FN UmJ5eXZpMG10RVZ5YWU0VEp4cS02N0I3Wkt3bkVWd1V1NURwNkRyQ2hPMVkyX1l3QG1haWwuZ21h aWwuY29tIgogICAgICB0eXBlPSJjaXRlIj4KICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAg PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVv dGUiPk9uIFRodSwgU2VwIDI0LCAyMDE1IGF0IDc6MzcgQU0sCiAgICAgICAgICAgIFJhbWVzaCBO YWNoaW11dGh1IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEKICAgICAgICAgICAgICAgIG1vei1kby1u b3Qtc2VuZD0idHJ1ZSIgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20iCiAgICAgICAg ICAgICAgICB0YXJnZXQ9Il9ibGFuayI+PGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRl ZCIgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20iPnJuYWNoaW11QHJlZGhhdC5jb208 L2E+PC9hPiZndDs8L3NwYW4+CiAgICAgICAgICAgIHdyb3RlOjxicj4KICAgICAgICAgICAgPGJs b2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAgICAg ICAgICAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4K ICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPjxzcGFu IGNsYXNzPSIiPiA8YnI+CiAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAg PGRpdj5PbiAwOS8yNC8yMDE1IDAyOjM4IEFNLCBEYXJyZWxsIEJ1ZGljIHdyb3RlOjxicj4KICAg ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9 ImNpdGUiPiBUaGlzIGlzIGEga25vd24gaXNzdWUgaW4KICAgICAgICAgICAgICAgICAgICBvdmVy dCAzLjUueCBhbmQgYmVsb3cuIEl04oCZcyBiZWVuIHNvbHZlZCBpbiB0aGUKICAgICAgICAgICAg ICAgICAgICB1cGNvbWluZyBvdmlydCAzLjYuCiAgICAgICAgICAgICAgICAgICAgPGRpdj48YnI+ CiAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgPGRpdj5SZWxh dGVkIHRvwqA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiCiAgICAgICAgICAgICAgICAgICAgICAg IGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3Mjkw NSIKICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVnemls bGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3MjkwNTwvYT4sCiAgICAgICAgICAgICAg ICAgICAgICB0aGUgZml4IGludm9sdmVkIHNldHRpbmcgdXAgYSBzcGVjaWFsIGNncm91cCBmb3IK ICAgICAgICAgICAgICAgICAgICAgIHRoZSBtb3VudCwgYnV0IGkgY2Fu4oCZdCBmaW5kIHRoZSBl eGFjdCBkZXRhaWxzIGF0bS48L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAg ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgog ICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICA8L3NwYW4+IEkgaGF2ZSB2ZHNt IDQuMTcuNi0wLmVsNy5jZW50b3MgYWxyZWFkeQogICAgICAgICAgICAgICAgaW5zdGFsbGVkIG9u IHRoZSBob3N0cy4gU28gSSBhbSBub3Qgc3VyZSBhYm92ZSBidWcgPGEKICAgICAgICAgICAgICAg ICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAgICAgICBocmVmPSJodHRwczov L2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19idWcuY2dpP2lkPTExNzI5MDUiCiAgICAgICAgICAg ICAgICAgIHRhcmdldD0iX2JsYW5rIj4xMTcyOTA1PC9hPiBmaXhlcyB0aGlzIGNvcnJlY3RseS48 YnI+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAg ICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8ZGl2PkkgdGhp bmsgdGhlIHJvb3QgY2F1c2UgaXMgdGhlIHNhbWUgLSBxZW11IGNhbm5vdAogICAgICAgICAgICAg IHJlY292ZXIgZnJvbSBnbHVzdGVyZnMgdW5tb3VudCwgYW5kIHRoZSBvbmx5IHdheSB0byByZXN1 bWUKICAgICAgICAgICAgICB0aGUgdm0gaXMgdG8gcmVzdGFydCBpdCB3aXRoIGEgZnJlc2ggbW91 bnQuPC9kaXY+CiAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAg ICAgICAgPGRpdj5UaGUgbWVudGlvbmVkIGJ1ZyBoYW5kbGUgdGhlIGNhc2Ugd2hlcmUgc3RvcHBp bmcgdmRzbQogICAgICAgICAgICAgIGtpbGxzIHRoZSBnbHVzdGVyZnMgbW91bnQgaGVscGVyLiBU aGlzIGlzc3VlIGlzIGZpeGVkIGluCiAgICAgICAgICAgICAgMy42LsKgPC9kaXY+CiAgICAgICAg ICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgPGRpdj5UaGUgaXNz dWUgaGVyZSBzZWVtcyBkaWZmZXJlbnQuIEkgc3VnZ2VzdCB5b3Ugb3BlbiBhCiAgICAgICAgICAg ICAgYnVnIHNvIGdsdXN0ZXIgZ3V5cyBjYW4gaW52ZXN0aWdhdGUgdGhpcy48L2Rpdj4KICAgICAg ICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPC9kaXY+CiAgICAg ICAgPC9kaXY+CiAgICAgIDwvZGl2PgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogICAgU2Vl bXMgbGlrZSBJIGFtIGhpdHRpbmcgdGhlIGlzc3VlIHJlcG9ydGVkIGluIGJ6CiAgICA8YSBjbGFz cz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5j b20vc2hvd19idWcuY2dpP2lkPTExNzEyNjEiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9z aG93X2J1Zy5jZ2k/aWQ9MTE3MTI2MTwvYT4uIDxicj4KICAgIDxicj4KICAgIFJlZ2FyZHMsPGJy PgogICAgUmFtZXNoPGJyPgogICAgPGJyPgogICAgPGJsb2NrcXVvdGUKY2l0ZT0ibWlkOkNBTVJi eXl2aTBtdEVWeWFlNFRKeHEtNjdCN1pLd25FVndVdTVEcDZEckNoTzFZMl9Zd0BtYWlsLmdtYWls LmNvbSIKICAgICAgdHlwZT0iY2l0ZSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgIDxk aXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3Rl Ij4KICAgICAgICAgICAgPGRpdj5OaXI8L2Rpdj4KICAgICAgICAgICAgPGRpdj48YnI+CiAgICAg ICAgICAgIDwvZGl2PgogICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgIDxkaXY+wqA8L2Rpdj4KICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9Imdt YWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAgICAgICAgICAgLjhleDtib3JkZXIt bGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAgICA8ZGl2 IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPiA8YnI+CiAgICAgICAgICAgICAgICBS ZWdhcmRzLDxicj4KICAgICAgICAgICAgICAgIFJhbWVzaAogICAgICAgICAgICAgICAgPGRpdj4K ICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iaDUiPjxicj4KICAgICAgICAgICAgICAgICAg ICA8YnI+CiAgICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAg ICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj4K ICAgICAgICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDxkaXY+T24gU2VwIDIzLCAyMDE1LCBhdCA3OjM4IEFNLCBS YW1lc2gKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgTmFjaGltdXRodSAmbHQ7PGEgbW96 LWRvLW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9 Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHRhcmdldD0iX2JsYW5rIj5ybmFjaGltdUByZWRoYXQuY29tPC9hPiZndDsKCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHdyb3RlOjwvZGl2PgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4g PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8ZGl2Pk9uIDA5LzIyLzIwMTUgMDU6NTcgUE0sIEFsYXN0YWlyCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBOZWlsIHdyb3RlOjxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxkaXYgZGlyPSJsdHIiPllvdSBuZWVkIHRvIHNldCB0aGUKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvIHRvIDUxJTwv ZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxf ZXh0cmEiPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSSBk aWQgdGhhdC4gQnV0IHN0aWxsIEkgYW0gZmFjaW5nIHRoZQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHNhbWUgaXNzdWUuIFZNIGdldCBwYXVzZWQgd2hlbiBJIGRvIHNvbWUKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBJL08gdXNpbmcgZmlvIG9uIHNvbWUgZGlza3MgYmFj a2VkIGJ5CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2x1c3Rlci4gSSBhbSBub3Qg YWJsZSB0byByZXN1bWUgdGhlIFZNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWZ0 ZXIgdGhpcy4gTm93IG9ubHkgd2F5IGlzIHRvIGJyaW5nCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgZG93biB0aGUgVk0gYW5kIHJ1biBhZ2Fpbi4gSXQgcnVucwogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHN1Y2Nlc3NmdWxseSBvbiB0aGUgc2FtZSBob3N0IHdpdGhvdXQK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhbnkgaXNzdWUuPGJyPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBSZWdhcmRzLDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gMjIKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBTZXB0ZW1iZXIgMjAxNSBhdCAwODoyNSwg UmFtZXNoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgTmFjaGltdXRodSA8 c3BhbiBkaXI9Imx0ciI+Jmx0OzxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOnJuYWNoaW11QHJlZGhhdC5jb20iCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhcmdldD0iX2JsYW5rIj48YSBjbGFz cz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0 LmNvbSI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT48L2E+Jmd0Ozwvc3Bhbj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3R5bGU9Im1hcmdpbjowIDAgMAogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHgg I2NjYwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc29saWQ7cGFkZGlu Zy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2 IHRleHQ9IiMwMDAwMDAiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IGJnY29sb3I9IiNGRkZGRkYiPjxzcGFuPiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxkaXY+T24gMDkvMjIvMjAxNSAwNTo0MyBQTSwKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIEFsYXN0YWlyIE5laWwgd3JvdGU6PGJyPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUi PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9 Imx0ciI+d2hhdCBhcmUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgdGhlwqBnbHVzdGVyLXF1b3J1bS10eXBlCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGFuZMKgZ2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvCgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoHNldHRpbmdz IG9uIHRoZSB2b2x1bWU/PC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg c3R5bGU9Im91dGxpbmUtc3R5bGU6bm9uZSI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPGRpdgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgc3R5bGU9Im92ZXJmbG93OmhpZGRlbjt0ZXh0LW92ZXJmbG93OmVsbGlwc2lzO3do aXRlLXNwYWNlOm5vd3JhcCI+PGI+Y2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU8L2I+OnNlcnZl cjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYg dGl0bGU9IiIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg c3R5bGU9Im91dGxpbmUtc3R5bGU6bm9uZSI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIDxkaXYKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBzdHlsZT0ib3ZlcmZsb3c6aGlkZGVuO3RleHQtb3ZlcmZsb3c6ZWxs aXBzaXM7d2hpdGUtc3BhY2U6bm93cmFwIj48Yj5jbHVzdGVyLnF1b3J1bS10eXBlPC9iPjphdXRv PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxi PmdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbwoKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGlzIG5vdCBzZXQuPC9iPjxicj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBPbmUgYnJpY2sgcHJvY2VzcyBpcwogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHVycG9zZWZ1bGx5IGtpbGxlZMKg IGJ1dAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVtYWlu aW5nIHR3byBicmlja3MgYXJlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB1cCBhbmQgcnVubmluZy48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBSZWdhcmRzLDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIFJhbWVzaDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxzcGFu PiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJsb2Nr cXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDIyIFNlcHRlbWJl ciAyMDE1IGF0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgMDY6MjQsIFJhbWVzaAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIE5hY2hpbXV0aHUgPHNwYW4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGRpcj0ibHRyIj4mbHQ7PGEKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVl IgpocmVmPSJtYWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPjxhIGNs YXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRo YXQuY29tIj5ybmFjaGltdUByZWRoYXQuY29tPC9hPjwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdyb3RlOjxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3M9 ImdtYWlsX3F1b3RlIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgc3R5bGU9Im1hcmdpbjowIDAgMAogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgLjhleDtib3JkZXItbGVmdDoxcHgKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICNjY2MKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNvbGlkO3BhZGRpbmctbGVm dDoxZXgiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGRpdiB0ZXh0PSIjMDAwMDAwIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSGksPGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqAgSSBhbSBub3Qg YWJsZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB0byByZXN1bWUgYSBWTQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB3aGljaCB3YXMgcGF1c2VkCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJlY2F1c2Ugb2YgZ2x1c3RlcgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGllbnQgcXVvcnVt CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlz c3VlLiBIZXJlIGlzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHdoYXQgaGFwcGVuZWQgaW4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgbXkgc2V0dXAuIDxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAxLiBDcmVhdGVkIGEKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2x1c3RlciBz dG9yYWdlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIGRvbWFpbiB3aGljaCBpcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBiYWNrZWQgYnkgZ2x1c3RlcgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB2b2x1bWUgd2l0aAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXBsaWNhIDMuIDxicj4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMi4g S2lsbGVkIG9uZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBicmljayBwcm9jZXNzLiBTbwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBvbmx5IHR3byBicmlja3MKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXJlIHJ1bm5pbmcgaW4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVwbGljYSAz IHNldHVwLjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgMy4gQ3JlYXRlZCB0d28gVk1zPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA0LiBTdGFydGVkIHNvbWUgSU8KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdXNpbmcgZmlvIG9u IGJvdGgKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgb2YgdGhlIFZNczxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgNS4gQWZ0ZXIgc29tZSB0aW1lCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdvdCB0aGUgZm9sbG93aW5nCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVycm9yIGluIGds dXN0ZXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgbW91bnQgYW5kIFZNcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBtb3ZlZCB0byBwYXVzZWQKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgc3RhdGUuPGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgwqDCoMKgICIgPHNw YW4Kc3R5bGU9ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQt c2l6ZToxMXB4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWln aHQ6bm9ybWFsO2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1h bGlnbjpsZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNl Om5vcm1hbDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpu b25lO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+c2VydmVyCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1u b3Qtc2VuZD0idHJ1ZSIKaHJlZj0iaHR0cDovLzEwLjcwLjQ1LjE3OjQ5MjE3LyIgdGFyZ2V0PSJf YmxhbmsiPjEwLjcwLjQ1LjE3OjQ5MjE3PC9hPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhhcyBub3QKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXNwb25kZWQgaW4gdGhlCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGFzdCA0 MiBzZWNvbmRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGRpc2Nvbm5lY3RpbmcuIjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgICI8L3NwYW4+PHNwYW4Kc3R5 bGU9ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZTox MXB4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9y bWFsO2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjps ZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1h bDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpub25lO2Jh Y2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1KSI+PHNwYW4Kc3R5bGU9ImNvbG9yOnJnYig1 MSw1MSw1MSk7Zm9udC1mYW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtc3R5bGU6 bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2xldHRlci1zcGFj aW5nOm5vcm1hbDtsaW5lLWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0O3RleHQtaW5kZW50 OjBweDt0ZXh0LXRyYW5zZm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1hbDt3b3JkLXNwYWNpbmc6 MHB4O2Rpc3BsYXk6aW5saW5lIWltcG9ydGFudDtmbG9hdDpub25lO2JhY2tncm91bmQtY29sb3I6 cmdiKDI1NSwyNTUsMjU1KSI+dm1zdG9yZS1yZXBsaWNhdGUtMDoKCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlMTZkMWU0MC0yYjZlLTRm MTktOTc3ZC1lMDk5ZjQ2NWRmYzY6CgoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIEZhaWxpbmcgV1JJVEUKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFzIHF1b3J1bSBpcwogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbm90 IG1ldDwvc3Bhbj4iPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgbW9yZQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdsdXN0ZXIgbW91bnQKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsb2dzIGF0IDxhCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBt b3otZG8tbm90LXNlbmQ9InRydWUiCmhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEYi IHRhcmdldD0iX2JsYW5rIj48YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJo dHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBGIj5odHRwOi8vcGFzdGViaW4uY29tL1VtaVVRcTBG PC9hPjwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDwvc3Bhbj42LiBBZnRlcgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBzb21lIHRpbWUgZ2x1c3RlcgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBxdW9ydW0gaXMgYWN0aXZlCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFuZCBJ IGFtIGFibGUgdG8KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgd3JpdGUgdGhlIHRoZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBnbHVzdGVyIGZpbGUKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3lzdGVtLjxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgNy4gV2hlbiBJIHRyeSB0bwog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXN1 bWUgdGhlIFZNIGl0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIGRvZXNuJ3Qgd29yayBhbmQgSQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBnb3QgZm9sbG93aW5nCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVycm9yIGluIHZkc20gbG9nLjxi cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg wqDCoMKgwqDCoCA8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKaHJlZj0iaHR0cDovL3Bhc3RlYmlu LmNvbS9hWGlhbVkxNSIgdGFyZ2V0PSJfYmxhbmsiPjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJl ZXRleHQiIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTUiPmh0dHA6Ly9wYXN0ZWJp bi5jb20vYVhpYW1ZMTU8L2E+PC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFJlZ2FyZHMsPGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSYW1lc2g8YnI+CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcg bGlzdDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIg dGFyZ2V0PSJfYmxhbmsiPjxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9 Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48L2E+PGJyPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5v dC1zZW5kPSJ0cnVlIgpocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu Zm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiPjxhIGNsYXNzPSJtb3otdHh0 LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0 aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJz PC9hPjwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9i bG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9zcGFuPjwvZGl2 PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDwvZGl2PgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXzxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0 PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRy dWUiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOlVzZXJzQG92 aXJ0Lm9yZyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayI+ VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEg bW96LWRvLW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhy ZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92 aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4K ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwvYmxv Y2txdW90ZT4KICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgIDwvZGl2 PgogICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAg ICAgPGJyPgogICAgICAgICAgICAgIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fPGJyPgogICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdDxicj4KICAg ICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhyZWY9Im1haWx0bzpVc2Vyc0Bv dmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAgICAgICAgPGEgbW96LWRv LW5vdC1zZW5kPSJ0cnVlIgogICAgICAgICAgICAgICAgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0 Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIgogICAgICAgICAgICAgICAgcmVsPSJub3JlZmVy cmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgIDwvYmxvY2tx dW90ZT4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPGJyPgogICAgICAgIDwvZGl2PgogICAg ICA8L2Rpdj4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0t LS0tLS0tLS0tLS0tMDMwNTAzMDMwNTAzMDgwMDA2MDYwMzAwLS0K --===============4653285908978238859==-- From nsoffer at redhat.com Thu Sep 24 02:27:37 2015 Content-Type: multipart/mixed; boundary="===============0958694004859629063==" MIME-Version: 1.0 From: Nir Soffer To: users at ovirt.org Subject: Re: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Thu, 24 Sep 2015 09:27:35 +0300 Message-ID: In-Reply-To: 560392D3.6030306@redhat.com --===============0958694004859629063== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 24, 2015 at 9:06 AM, Ramesh Nachimuthu wrote: > > > On 09/24/2015 11:28 AM, Nir Soffer wrote: > > On Thu, Sep 24, 2015 at 7:37 AM, Ramesh Nachimuthu < > rnachimu(a)redhat.com> wrote: > >> >> >> On 09/24/2015 02:38 AM, Darrell Budic wrote: >> >> This is a known issue in overt 3.5.x and below. It=E2=80=99s been solved= in the >> upcoming ovirt 3.6. >> >> Related to https://bugzilla.redhat.com/show_bug.cgi?id=3D1172905, the fix >> involved setting up a special cgroup for the mount, but i can=E2=80=99t = find the >> exact details atm. >> >> >> I have vdsm 4.17.6-0.el7.centos already installed on the hosts. So I am >> not sure above bug 1172905 >> fixes this >> correctly. >> > > I think the root cause is the same - qemu cannot recover from glusterfs > unmount, and the only way to resume the vm is to restart it with a fresh > mount. > > The mentioned bug handle the case where stopping vdsm kills the glusterfs > mount helper. This issue is fixed in 3.6. > > The issue here seems different. I suggest you open a bug so gluster guys > can investigate this. > > > Seems like I am hitting the issue reported in bz > https://bugzilla.redhat.com/show_bug.cgi?id=3D1171261. > Indeed. I would open an ovirt bug anyway and make it depend on the glusterfs bug. We need a way to track this issues, and having no ovirt/rhev hides this issue. > > Regards, > Ramesh > > > Nir > > > > >> >> Regards, >> Ramesh >> >> >> >> On Sep 23, 2015, at 7:38 AM, Ramesh Nachimuthu >> wrote: >> >> >> >> On 09/22/2015 05:57 PM, Alastair Neil wrote: >> >> You need to set the gluster.server-quorum-ratio to 51% >> >> >> I did that. But still I am facing the same issue. VM get paused when I do >> some I/O using fio on some disks backed by gluster. I am not able to res= ume >> the VM after this. Now only way is to bring down the VM and run again. It >> runs successfully on the same host without any issue. >> >> Regards, >> Ramesh >> >> On 22 September 2015 at 08:25, Ramesh Nachimuthu < >> rnachimu(a)redhat.com> wrote: >> >>> >>> >>> On 09/22/2015 05:43 PM, Alastair Neil wrote: >>> >>> what are the gluster-quorum-type and gluster.server-quorum-ratio >>> settings on the volume? >>> >>> >>> *cluster.server-quorum-type*:server >>> *cluster.quorum-type*:auto >>> *gluster.server-quorum-ratio is not set.* >>> >>> One brick process is purposefully killed but remaining two bricks are >>> up and running. >>> >>> Regards, >>> Ramesh >>> >>> On 22 September 2015 at 06:24, Ramesh Nachimuthu < >>> rnachimu(a)redhat.com> wrote: >>> >>>> Hi, >>>> >>>> I am not able to resume a VM which was paused because of gluster >>>> client quorum issue. Here is what happened in my setup. >>>> >>>> 1. Created a gluster storage domain which is backed by gluster volume >>>> with replica 3. >>>> 2. Killed one brick process. So only two bricks are running in replica >>>> 3 setup. >>>> 3. Created two VMs >>>> 4. Started some IO using fio on both of the VMs >>>> 5. After some time got the following error in gluster mount and VMs >>>> moved to paused state. >>>> " server 10.70.45.17:49217 has not responded in the last 42 >>>> seconds, disconnecting." >>>> "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: >>>> Failing WRITE as quorum is not met" >>>> more gluster mount logs at >>>> http://pastebin.com/UmiUQq0F >>>> 6. After some time gluster quorum is active and I am able to write the >>>> the gluster file system. >>>> 7. When I try to resume the VM it doesn't work and I got following >>>> error in vdsm log. >>>> http://pastebin.com/aXiamY15 >>>> >>>> >>>> Regards, >>>> Ramesh >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>> >>> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > --===============0958694004859629063== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxkaXYgY2xhc3M9ImdtYWls X3F1b3RlIj5PbiBUaHUsIFNlcCAyNCwgMjAxNSBhdCA5OjA2IEFNLCBSYW1lc2ggTmFjaGltdXRo dSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29t IiB0YXJnZXQ9Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90 ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAw IC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgCiAg ICAKICAKICA8ZGl2IHRleHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPjxzcGFuIGNsYXNz PSIiPgogICAgPGJyPgogICAgPGJyPgogICAgPGRpdj5PbiAwOS8yNC8yMDE1IDExOjI4IEFNLCBO aXIgU29mZmVyCiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSI+CiAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgIDxkaXYgY2xhc3M9ImdtYWls X2V4dHJhIj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBUaHUsIFNlcCAy NCwgMjAxNSBhdCA3OjM3IEFNLAogICAgICAgICAgICBSYW1lc2ggTmFjaGltdXRodSA8c3BhbiBk aXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9 Il9ibGFuayI+PC9hPjxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9 Il9ibGFuayI+cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgICB3 cm90ZTo8YnI+CiAgICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5 bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmct bGVmdDoxZXgiPgogICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZG RkZGRiI+PHNwYW4+IDxicj4KICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg ICA8ZGl2Pk9uIDA5LzI0LzIwMTUgMDI6MzggQU0sIERhcnJlbGwgQnVkaWMgd3JvdGU6PGJyPgog ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSI+IFRoaXMgaXMgYSBrbm93biBpc3N1ZSBpbgogICAgICAgICAgICAgICAgICAgIG92 ZXJ0IDMuNS54IGFuZCBiZWxvdy4gSXTigJlzIGJlZW4gc29sdmVkIGluIHRoZQogICAgICAgICAg ICAgICAgICAgIHVwY29taW5nIG92aXJ0IDMuNi4KICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxi cj4KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8ZGl2PlJl bGF0ZWQgdG/CoDxhIGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5j Z2k/aWQ9MTE3MjkwNSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNv bS9zaG93X2J1Zy5jZ2k/aWQ9MTE3MjkwNTwvYT4sCiAgICAgICAgICAgICAgICAgICAgICB0aGUg Zml4IGludm9sdmVkIHNldHRpbmcgdXAgYSBzcGVjaWFsIGNncm91cCBmb3IKICAgICAgICAgICAg ICAgICAgICAgIHRoZSBtb3VudCwgYnV0IGkgY2Fu4oCZdCBmaW5kIHRoZSBleGFjdCBkZXRhaWxz IGF0bS48L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAg ICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAg ICAgICA8YnI+CiAgICAgICAgICAgICAgICA8L3NwYW4+IEkgaGF2ZSB2ZHNtIDQuMTcuNi0wLmVs Ny5jZW50b3MgYWxyZWFkeQogICAgICAgICAgICAgICAgaW5zdGFsbGVkIG9uIHRoZSBob3N0cy4g U28gSSBhbSBub3Qgc3VyZSBhYm92ZSBidWcgPGEgaHJlZj0iaHR0cHM6Ly9idWd6aWxsYS5yZWRo YXQuY29tL3Nob3dfYnVnLmNnaT9pZD0xMTcyOTA1IiB0YXJnZXQ9Il9ibGFuayI+MTE3MjkwNTwv YT4gZml4ZXMgdGhpcyBjb3JyZWN0bHkuPGJyPgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAg ICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rp dj4KICAgICAgICAgICAgPGRpdj5JIHRoaW5rIHRoZSByb290IGNhdXNlIGlzIHRoZSBzYW1lIC0g cWVtdSBjYW5ub3QKICAgICAgICAgICAgICByZWNvdmVyIGZyb20gZ2x1c3RlcmZzIHVubW91bnQs IGFuZCB0aGUgb25seSB3YXkgdG8gcmVzdW1lCiAgICAgICAgICAgICAgdGhlIHZtIGlzIHRvIHJl c3RhcnQgaXQgd2l0aCBhIGZyZXNoIG1vdW50LjwvZGl2PgogICAgICAgICAgICA8ZGl2Pjxicj4K ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXY+VGhlIG1lbnRpb25lZCBidWcgaGFu ZGxlIHRoZSBjYXNlIHdoZXJlIHN0b3BwaW5nIHZkc20KICAgICAgICAgICAgICBraWxscyB0aGUg Z2x1c3RlcmZzIG1vdW50IGhlbHBlci4gVGhpcyBpc3N1ZSBpcyBmaXhlZCBpbgogICAgICAgICAg ICAgIDMuNi7CoDwvZGl2PgogICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgPC9kaXY+ CiAgICAgICAgICAgIDxkaXY+VGhlIGlzc3VlIGhlcmUgc2VlbXMgZGlmZmVyZW50LiBJIHN1Z2dl c3QgeW91IG9wZW4gYQogICAgICAgICAgICAgIGJ1ZyBzbyBnbHVzdGVyIGd1eXMgY2FuIGludmVz dGlnYXRlIHRoaXMuPC9kaXY+CiAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rp dj4KICAgICAgICAgIDwvZGl2PgogICAgICAgIDwvZGl2PgogICAgICA8L2Rpdj4KICAgIDwvYmxv Y2txdW90ZT4KICAgIDxicj48L3NwYW4+CiAgICBTZWVtcyBsaWtlIEkgYW0gaGl0dGluZyB0aGUg aXNzdWUgcmVwb3J0ZWQgaW4gYnoKICAgIDxhIGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0 LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3MTI2MSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vYnVn emlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTE3MTI2MTwvYT4uIDxicj48L2Rpdj48 L2Jsb2NrcXVvdGU+PGRpdj48YnI+PC9kaXY+PGRpdj5JbmRlZWQuPC9kaXY+PGRpdj48YnI+PC9k aXY+PGRpdj5JIHdvdWxkIG9wZW4gYW4gb3ZpcnQgYnVnIGFueXdheSBhbmQgbWFrZSBpdCBkZXBl bmQgb24gdGhlIGdsdXN0ZXJmcyBidWcuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5XZSBuZWVk IGEgd2F5IHRvIHRyYWNrIHRoaXMgaXNzdWVzLCBhbmQgaGF2aW5nIG5vIG92aXJ0L3JoZXYgaGlk ZXMgdGhpcyBpc3N1ZS48L2Rpdj48ZGl2PsKgPC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWls X3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29s aWQ7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZG Ij4KICAgIDxicj4KICAgIFJlZ2FyZHMsPGJyPgogICAgUmFtZXNoPGRpdj48ZGl2IGNsYXNzPSJo NSI+PGJyPgogICAgPGJyPgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAgICAgIDxkaXYg ZGlyPSJsdHIiPgogICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAgIDxk aXYgY2xhc3M9ImdtYWlsX3F1b3RlIj4KICAgICAgICAgICAgPGRpdj5OaXI8L2Rpdj4KICAgICAg ICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8ZGl2Pjxicj4K ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXY+wqA8L2Rpdj4KICAgICAgICAgICAg PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7 Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAg ICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4gPGJyPgogICAgICAgICAg ICAgICAgUmVnYXJkcyw8YnI+CiAgICAgICAgICAgICAgICBSYW1lc2gKICAgICAgICAgICAgICAg IDxkaXY+CiAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICAg ICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGRpdj5PbiBTZXAgMjMsIDIwMTUsIGF0IDc6MzggQU0sIFJhbWVz aAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBOYWNoaW11dGh1ICZsdDs8YSBocmVmPSJt YWlsdG86cm5hY2hpbXVAcmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPnJuYWNoaW11QHJlZGhh dC5jb208L2E+Jmd0OwoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd3JvdGU6PC9kaXY+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8ZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IHRleHQ9IiMwMDAw MDAiIGJnY29sb3I9IiNGRkZGRkYiPiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+T24gMDkvMjIvMjAx NSAwNTo1NyBQTSwgQWxhc3RhaXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE5l aWwgd3JvdGU6PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+WW91IG5lZWQgdG8g c2V0IHRoZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnbHVzdGVyLnNlcnZl ci1xdW9ydW0tcmF0aW8gdG8gNTElPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBJIGRpZCB0aGF0LiBCdXQgc3RpbGwgSSBhbSBmYWNpbmcgdGhl CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc2FtZSBpc3N1ZS4gVk0gZ2V0IHBhdXNl ZCB3aGVuIEkgZG8gc29tZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEkvTyB1c2lu ZyBmaW8gb24gc29tZSBkaXNrcyBiYWNrZWQgYnkKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBnbHVzdGVyLiBJIGFtIG5vdCBhYmxlIHRvIHJlc3VtZSB0aGUgVk0KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBhZnRlciB0aGlzLiBOb3cgb25seSB3YXkgaXMgdG8gYnJpbmcK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkb3duIHRoZSBWTSBhbmQgcnVuIGFnYWlu LiBJdCBydW5zCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3VjY2Vzc2Z1bGx5IG9u IHRoZSBzYW1lIGhvc3Qgd2l0aG91dAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFu eSBpc3N1ZS48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIFJlZ2FyZHMsPGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIFJhbWVzaDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0 ZSI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9l eHRyYSI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9Imdt YWlsX3F1b3RlIj5PbiAyMgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFNl cHRlbWJlciAyMDE1IGF0IDA4OjI1LCBSYW1lc2gKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBOYWNoaW11dGh1IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRv OnJuYWNoaW11QHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj48L2E+PGEgaHJlZj0ibWFpbHRv OnJuYWNoaW11QHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5ybmFjaGltdUByZWRoYXQuY29t PC9hPiZndDs8L3NwYW4+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd3Jv dGU6PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3Rl IGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0 OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj48 c3Bhbj4gPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2Pk9uIDA5 LzIyLzIwMTUgMDU6NDMgUE0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBBbGFzdGFpciBOZWlsIHdyb3RlOjxicj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPndoYXQgYXJlCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZcKgZ2x1c3Rlci1x dW9ydW0tdHlwZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBhbmTCoGdsdXN0ZXIuc2VydmVyLXF1b3J1bS1yYXRpbwoKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgwqBzZXR0aW5ncyBvbiB0aGUgdm9sdW1lPzwvZGl2 PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFz cz0iZ21haWxfZXh0cmEiPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvc3Bh bj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBzdHlsZT0i b3V0bGluZS1zdHlsZTpub25lIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2IHN0eWxlPSJvdmVyZmxvdzpoaWRkZW47dGV4dC1vdmVyZmxvdzplbGxpcHNp czt3aGl0ZS1zcGFjZTpub3dyYXAiPjxiPmNsdXN0ZXIuc2VydmVyLXF1b3J1bS10eXBlPC9iPjpz ZXJ2ZXI8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 ZGl2IHRpdGxlPSIiIHN0eWxlPSJvdXRsaW5lLXN0eWxlOm5vbmUiPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IHN0eWxlPSJvdmVyZmxvdzpoaWRk ZW47dGV4dC1vdmVyZmxvdzplbGxpcHNpczt3aGl0ZS1zcGFjZTpub3dyYXAiPjxiPmNsdXN0ZXIu cXVvcnVtLXR5cGU8L2I+OmF1dG88YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPGI+Z2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXJhdGlvCgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaXMgbm90IHNldC48 L2I+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9k aXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE9uZSBicmljayBw cm9jZXNzIGlzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBw dXJwb3NlZnVsbHkga2lsbGVkwqAgYnV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICByZW1haW5pbmcgdHdvIGJyaWNrcyBhcmUKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVwIGFuZCBydW5uaW5nLjxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFJlZ2FyZHMsPGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUmFtZXNoPGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPHNwYW4+IDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0i Z21haWxfcXVvdGUiPk9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgMjIgU2VwdGVtYmVyIDIwMTUgYXQKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAwNjoyNCwgUmFtZXNoCiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgTmFjaGltdXRodSA8c3BhbiBkaXI9Imx0ciI+ Jmx0OzxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+ PC9hPjxhIGhyZWY9Im1haWx0bzpybmFjaGltdUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+ cm5hY2hpbXVAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdyb3RlOjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxf cXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xp ZDtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEhpLDxi cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICDCoMKgIEkgYW0gbm90IGFibGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgdG8gcmVzdW1lIGEgVk0KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggd2FzIHBhdXNlZAogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiZWNhdXNlIG9mIGds dXN0ZXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgY2xpZW50IHF1b3J1bQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBpc3N1ZS4gSGVyZSBpcwogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB3aGF0IGhhcHBlbmVkIGluCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG15IHNldHVwLiA8YnI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMS4g Q3JlYXRlZCBhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIGdsdXN0ZXIgc3RvcmFnZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBkb21haW4gd2hpY2ggaXMKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYmFja2VkIGJ5IGdsdXN0ZXIKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdm9sdW1lIHdp dGgKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg cmVwbGljYSAzLiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDIuIEtpbGxlZCBvbmUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgYnJpY2sgcHJvY2Vzcy4gU28KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb25seSB0d28gYnJpY2tzCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFyZSBy dW5uaW5nIGluCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHJlcGxpY2EgMyBzZXR1cC48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDMuIENyZWF0ZWQgdHdvIFZNczxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgNC4gU3RhcnRlZCBz b21lIElPCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHVzaW5nIGZpbyBvbiBib3RoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG9mIHRoZSBWTXM8YnI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDUuIEFmdGVyIHNvbWUgdGltZQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBnb3QgdGhlIGZv bGxvd2luZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBlcnJvciBpbiBnbHVzdGVyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG1vdW50IGFuZCBWTXMKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW92ZWQgdG8gcGF1c2VkCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0YXRlLjxicj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgwqDCoMKg wqDCoMKgwqDCoCAmcXVvdDsgPHNwYW4gc3R5bGU9ImNvbG9yOnJnYig1MSw1MSw1MSk7Zm9udC1m YW1pbHk6bW9ub3NwYWNlO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFy aWFudDpub3JtYWw7Zm9udC13ZWlnaHQ6bm9ybWFsO2xldHRlci1zcGFjaW5nOm5vcm1hbDtsaW5l LWhlaWdodDoxMy4ycHg7dGV4dC1hbGlnbjpsZWZ0O3RleHQtaW5kZW50OjBweDt0ZXh0LXRyYW5z Zm9ybTpub25lO3doaXRlLXNwYWNlOm5vcm1hbDt3b3JkLXNwYWNpbmc6MHB4O2Rpc3BsYXk6aW5s aW5lIWltcG9ydGFudDtmbG9hdDpub25lO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUsMjU1 KSI+c2VydmVyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPGEgaHJlZj0iaHR0cDovLzEwLjcwLjQ1LjE3OjQ5MjE3LyIgdGFyZ2V0PSJfYmxh bmsiPjEwLjcwLjQ1LjE3OjQ5MjE3PC9hPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIGhhcyBub3QKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXNwb25kZWQgaW4gdGhlCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGFzdCA0MiBz ZWNvbmRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIGRpc2Nvbm5lY3RpbmcuJnF1b3Q7PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgJnF1b3Q7PC9zcGFuPjxz cGFuIHN0eWxlPSJjb2xvcjpyZ2IoNTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250 LXNpemU6MTFweDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2Vp Z2h0Om5vcm1hbDtsZXR0ZXItc3BhY2luZzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQt YWxpZ246bGVmdDt0ZXh0LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFj ZTpub3JtYWw7d29yZC1zcGFjaW5nOjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6 bm9uZTtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPjxzcGFuIHN0eWxlPSJjb2xv cjpyZ2IoNTEsNTEsNTEpO2ZvbnQtZmFtaWx5Om1vbm9zcGFjZTtmb250LXNpemU6MTFweDtmb250 LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO2ZvbnQtd2VpZ2h0Om5vcm1hbDtsZXR0 ZXItc3BhY2luZzpub3JtYWw7bGluZS1oZWlnaHQ6MTMuMnB4O3RleHQtYWxpZ246bGVmdDt0ZXh0 LWluZGVudDowcHg7dGV4dC10cmFuc2Zvcm06bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7d29yZC1z cGFjaW5nOjBweDtkaXNwbGF5OmlubGluZSFpbXBvcnRhbnQ7ZmxvYXQ6bm9uZTtiYWNrZ3JvdW5k LWNvbG9yOnJnYigyNTUsMjU1LDI1NSkiPnZtc3RvcmUtcmVwbGljYXRlLTA6CgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZTE2ZDFlNDAt MmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OgoKCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBGYWlsaW5nIFdSSVRFCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhcyBxdW9ydW0g aXMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIG5vdCBtZXQ8L3NwYW4+JnF1b3Q7PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIMKgwqDCoMKgwqAgbW9yZQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdsdXN0ZXIgbW91bnQK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBs b2dzIGF0IDxhIGhyZWY9Imh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEYiIHRhcmdldD0iX2Js YW5rIj48L2E+PGEgaHJlZj0iaHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRiIgdGFyZ2V0PSJf YmxhbmsiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEY8L2E+PGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L3NwYW4+Ni4gQWZ0ZXIK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc29t ZSB0aW1lIGdsdXN0ZXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgcXVvcnVtIGlzIGFjdGl2ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBhbmQgSSBhbSBhYmxlIHRvCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdyaXRlIHRoZSB0aGUKICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2x1c3Rl ciBmaWxlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIHN5c3RlbS48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDcuIFdoZW4gSSB0cnkgdG8KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgcmVzdW1lIHRoZSBWTSBpdAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkb2VzbiYjMzk7dCB3b3Jr IGFuZCBJCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIGdvdCBmb2xsb3dpbmcKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgZXJyb3IgaW4gdmRzbSBsb2cuPGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDCoMKgwqDCoMKgIDxhIGhyZWY9Imh0 dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTUiIHRhcmdldD0iX2JsYW5rIj48L2E+PGEgaHJlZj0i aHR0cDovL3Bhc3RlYmluLmNvbS9hWGlhbVkxNSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9wYXN0 ZWJpbi5jb20vYVhpYW1ZMTU8L2E+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgUmVnYXJkcyw8YnI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFJhbWVzaDxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+Cl9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBs aXN0PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPjwvYT48 YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3Zp cnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m by91c2VycyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+PC9hPjxhIGhyZWY9Imh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgdGFyZ2V0PSJfYmxh bmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJy PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9zcGFuPjwvZGl2PgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2 PgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0PGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0 YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2 PgogICAgICAgICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgICAgICAg ICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAg ICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICAgIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJy PgogICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdDxicj4KICAgICAgICAgICAgICA8YSBo cmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQu b3JnPC9hPjxicj4KICAgICAgICAgICAgICA8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsi Pmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+CiAg ICAgICAgICAgICAgPGJyPgogICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICA8L2Rp dj4KICAgICAgICAgIDxicj4KICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2Nr cXVvdGU+CiAgICA8YnI+CiAgPC9kaXY+PC9kaXY+PC9kaXY+Cgo8L2Jsb2NrcXVvdGU+PC9kaXY+ PGJyPjwvZGl2PjwvZGl2Pgo= --===============0958694004859629063==--