From rnachimu at redhat.com Tue Sep 22 06:24:58 2015 Content-Type: multipart/mixed; boundary="===============3356979838059583077==" MIME-Version: 1.0 From: Ramesh Nachimuthu To: users at ovirt.org Subject: [ovirt-users] Not able to resume a VM which was paused because of gluster quorum issue Date: Tue, 22 Sep 2015 15:54:54 +0530 Message-ID: <56012C76.9010408@redhat.com> --===============3356979838059583077== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050502080707090302020409 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hi, I am not able to resume a VM which was paused because of gluster = client quorum issue. Here is what happened in my setup. 1. Created a gluster storage domain which is backed by gluster volume = with replica 3. 2. Killed one brick process. So only two bricks are running in replica 3 = setup. 3. Created two VMs 4. Started some IO using fio on both of the VMs 5. After some time got the following error in gluster mount and VMs = moved to paused state. " server 10.70.45.17:49217 has not responded in the last 42 = seconds, disconnecting." "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: = Failing WRITE as quorum is not met" more gluster mount logs at http://pastebin.com/UmiUQq0F 6. After some time gluster quorum is active and I am able to write the = the gluster file system. 7. When I try to resume the VM it doesn't work and I got following error = in vdsm log. http://pastebin.com/aXiamY15 Regards, Ramesh --------------050502080707090302020409 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hi,

=C2=A0=C2=A0 I am not able to resume a VM which was paused because of g= luster client quorum issue. Here is what happened in my setup.

1. Created a gluster storage domain which is backed by gluster volume with replica 3.
2. Killed one brick process. So only two bricks are running in replica 3 setup.
3. Created two VMs
4. Started some IO using fio on both of the VMs
5. After some time got the following error in gluster mount and VMs moved to paused state.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 " server 10.70.45.17:49217 has not responded in the last 42 seconds, disconnecting."
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "vmstore-replicate-0: e16d1e40-2b6e-4f19-977d-e099f465dfc6: Failing WRITE as quorum is not met"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 more gluster mount logs at http://pasteb= in.com/UmiUQq0F
6. After some time gluster quorum is active and I am able to write the the gluster file system.
7. When I try to resume the VM it doesn't work and I got following error in vdsm log.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://pastebin.com/aXiamY15


Regards,
Ramesh

--------------050502080707090302020409-- --===============3356979838059583077== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTA1MDIwODA3MDcwOTAzMDIwMjA0MDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhpLAoKICAgIEkgYW0gbm90IGFibGUgdG8gcmVzdW1lIGEgVk0gd2hpY2ggd2FzIHBhdXNlZCBi ZWNhdXNlIG9mIGdsdXN0ZXIgCmNsaWVudCBxdW9ydW0gaXNzdWUuIEhlcmUgaXMgd2hhdCBoYXBw ZW5lZCBpbiBteSBzZXR1cC4KCjEuIENyZWF0ZWQgYSBnbHVzdGVyIHN0b3JhZ2UgZG9tYWluIHdo aWNoIGlzIGJhY2tlZCBieSBnbHVzdGVyIHZvbHVtZSAKd2l0aCByZXBsaWNhIDMuCjIuIEtpbGxl ZCBvbmUgYnJpY2sgcHJvY2Vzcy4gU28gb25seSB0d28gYnJpY2tzIGFyZSBydW5uaW5nIGluIHJl cGxpY2EgMyAKc2V0dXAuCjMuIENyZWF0ZWQgdHdvIFZNcwo0LiBTdGFydGVkIHNvbWUgSU8gdXNp bmcgZmlvIG9uIGJvdGggb2YgdGhlIFZNcwo1LiBBZnRlciBzb21lIHRpbWUgZ290IHRoZSBmb2xs b3dpbmcgZXJyb3IgaW4gZ2x1c3RlciBtb3VudCBhbmQgVk1zIAptb3ZlZCB0byBwYXVzZWQgc3Rh dGUuCiAgICAgICAgICAiIHNlcnZlciAxMC43MC40NS4xNzo0OTIxNyBoYXMgbm90IHJlc3BvbmRl ZCBpbiB0aGUgbGFzdCA0MiAKc2Vjb25kcywgZGlzY29ubmVjdGluZy4iCiAgICAgICAidm1zdG9y ZS1yZXBsaWNhdGUtMDogZTE2ZDFlNDAtMmI2ZS00ZjE5LTk3N2QtZTA5OWY0NjVkZmM2OiAKRmFp bGluZyBXUklURSBhcyBxdW9ydW0gaXMgbm90IG1ldCIKICAgICAgIG1vcmUgZ2x1c3RlciBtb3Vu dCBsb2dzIGF0IGh0dHA6Ly9wYXN0ZWJpbi5jb20vVW1pVVFxMEYKNi4gQWZ0ZXIgc29tZSB0aW1l IGdsdXN0ZXIgcXVvcnVtIGlzIGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvIHdyaXRlIHRoZSAKdGhl IGdsdXN0ZXIgZmlsZSBzeXN0ZW0uCjcuIFdoZW4gSSB0cnkgdG8gcmVzdW1lIHRoZSBWTSBpdCBk b2Vzbid0IHdvcmsgYW5kIEkgZ290IGZvbGxvd2luZyBlcnJvciAKaW4gdmRzbSBsb2cuCiAgICAg ICBodHRwOi8vcGFzdGViaW4uY29tL2FYaWFtWTE1CgoKUmVnYXJkcywKUmFtZXNoCgoKLS0tLS0t LS0tLS0tLS0wNTA1MDIwODA3MDcwOTAzMDIwMjA0MDkKQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7 IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJpdAoKPGh0bWw+CiAg PGhlYWQ+CgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogIDwvaGVhZD4KICA8Ym9keSB0ZXh0PSIjMDAwMDAwIiBi Z2NvbG9yPSIjRkZGRkZGIj4KICAgIEhpLDxicj4KICAgIDxicj4KICAgIMKgwqAgSSBhbSBub3Qg YWJsZSB0byByZXN1bWUgYSBWTSB3aGljaCB3YXMgcGF1c2VkIGJlY2F1c2Ugb2YgZ2x1c3Rlcgog ICAgY2xpZW50IHF1b3J1bSBpc3N1ZS4gSGVyZSBpcyB3aGF0IGhhcHBlbmVkIGluIG15IHNldHVw LiA8YnI+CiAgICA8YnI+CiAgICAxLiBDcmVhdGVkIGEgZ2x1c3RlciBzdG9yYWdlIGRvbWFpbiB3 aGljaCBpcyBiYWNrZWQgYnkgZ2x1c3RlcgogICAgdm9sdW1lIHdpdGggcmVwbGljYSAzLiA8YnI+ CiAgICAyLiBLaWxsZWQgb25lIGJyaWNrIHByb2Nlc3MuIFNvIG9ubHkgdHdvIGJyaWNrcyBhcmUg cnVubmluZyBpbgogICAgcmVwbGljYSAzIHNldHVwLjxicj4KICAgIDMuIENyZWF0ZWQgdHdvIFZN czxicj4KICAgIDQuIFN0YXJ0ZWQgc29tZSBJTyB1c2luZyBmaW8gb24gYm90aCBvZiB0aGUgVk1z PGJyPgogICAgNS4gQWZ0ZXIgc29tZSB0aW1lIGdvdCB0aGUgZm9sbG93aW5nIGVycm9yIGluIGds dXN0ZXIgbW91bnQgYW5kIFZNcwogICAgbW92ZWQgdG8gcGF1c2VkIHN0YXRlLjxicj4KICAgIMKg wqDCoMKgwqDCoMKgwqAgIgogICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250 ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogICAgPHNwYW4gc3R5bGU9ImNvbG9yOiBy Z2IoNTEsIDUxLCA1MSk7IGZvbnQtZmFtaWx5OiBtb25vc3BhY2U7CiAgICAgIGZvbnQtc2l6ZTog MTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsKICAgICAgZm9u dC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgbGluZS1oZWlnaHQ6IDEz LjJweDsKICAgICAgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogbGVmdDsgdGV4dC1pbmRlbnQ6 IDBweDsgdGV4dC10cmFuc2Zvcm06CiAgICAgIG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdp ZG93czogMTsgd29yZC1zcGFjaW5nOiAwcHg7CiAgICAgIC13ZWJraXQtdGV4dC1zdHJva2Utd2lk dGg6IDBweDsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OgogICAgICBub25lOyBi YWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ij5zZXJ2ZXIKICAgICAgMTAuNzAu NDUuMTc6NDkyMTcgaGFzIG5vdCByZXNwb25kZWQgaW4gdGhlIGxhc3QgNDIgc2Vjb25kcywKICAg ICAgZGlzY29ubmVjdGluZy4iPGJyPgogICAgICDCoMKgwqDCoMKgICI8L3NwYW4+PHNwYW4gc3R5 bGU9ImNvbG9yOiByZ2IoNTEsIDUxLCA1MSk7IGZvbnQtZmFtaWx5OgogICAgICBtb25vc3BhY2U7 IGZvbnQtc2l6ZTogMTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6CiAgICAg IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgbGlu ZS1oZWlnaHQ6CiAgICAgIDEzLjJweDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogbGVmdDsg dGV4dC1pbmRlbnQ6IDBweDsKICAgICAgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNl OiBub3JtYWw7IHdpZG93czogMTsKICAgICAgd29yZC1zcGFjaW5nOiAwcHg7IC13ZWJraXQtdGV4 dC1zdHJva2Utd2lkdGg6IDBweDsgZGlzcGxheTogaW5saW5lCiAgICAgICFpbXBvcnRhbnQ7IGZs b2F0OiBub25lOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMjU1LCAyNTUsIDI1NSk7Ij48c3Bhbgog ICAgICAgIHN0eWxlPSJjb2xvcjogcmdiKDUxLCA1MSwgNTEpOyBmb250LWZhbWlseTogbW9ub3Nw YWNlOwogICAgICAgIGZvbnQtc2l6ZTogMTFweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZh cmlhbnQ6IG5vcm1hbDsKICAgICAgICBmb250LXdlaWdodDogbm9ybWFsOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyBsaW5lLWhlaWdodDoKICAgICAgICAxMy4ycHg7IG9ycGhhbnM6IGF1dG87IHRl eHQtYWxpZ246IGxlZnQ7IHRleHQtaW5kZW50OiAwcHg7CiAgICAgICAgdGV4dC10cmFuc2Zvcm06 IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdpZG93czogMTsKICAgICAgICB3b3JkLXNwYWNp bmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBkaXNwbGF5OgogICAgICAg IGlubGluZSAhaW1wb3J0YW50OyBmbG9hdDogbm9uZTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1 NSwgMjU1LAogICAgICAgIDI1NSk7Ij52bXN0b3JlLXJlcGxpY2F0ZS0wOgogICAgICAgIGUxNmQx ZTQwLTJiNmUtNGYxOS05NzdkLWUwOTlmNDY1ZGZjNjogRmFpbGluZyBXUklURSBhcyBxdW9ydW0g aXMKICAgICAgICBub3QgbWV0PC9zcGFuPiI8YnI+CiAgICAgIMKgwqDCoMKgwqAgbW9yZSBnbHVz dGVyIG1vdW50IGxvZ3MgYXQgPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0i aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEwRiI+aHR0cDovL3Bhc3RlYmluLmNvbS9VbWlVUXEw RjwvYT48YnI+CiAgICA8L3NwYW4+Ni4gQWZ0ZXIgc29tZSB0aW1lIGdsdXN0ZXIgcXVvcnVtIGlz IGFjdGl2ZSBhbmQgSSBhbSBhYmxlIHRvCiAgICB3cml0ZSB0aGUgdGhlIGdsdXN0ZXIgZmlsZSBz eXN0ZW0uPGJyPgogICAgNy4gV2hlbiBJIHRyeSB0byByZXN1bWUgdGhlIFZNIGl0IGRvZXNuJ3Qg d29yayBhbmQgSSBnb3QgZm9sbG93aW5nCiAgICBlcnJvciBpbiB2ZHNtIGxvZy48YnI+CiAgICDC oMKgwqDCoMKgIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9w YXN0ZWJpbi5jb20vYVhpYW1ZMTUiPmh0dHA6Ly9wYXN0ZWJpbi5jb20vYVhpYW1ZMTU8L2E+PGJy PgogICAgPGJyPgogICAgPGJyPgogICAgUmVnYXJkcyw8YnI+CiAgICBSYW1lc2g8YnI+CiAgICA8 YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTA1MDUwMjA4MDcwNzA5MDMwMjAy MDQwOS0tCg== --===============3356979838059583077==--