From ebenahar at redhat.com Sun Aug 9 10:24:56 2015 Content-Type: multipart/mixed; boundary="===============4189641226383560458==" MIME-Version: 1.0 From: Elad Ben Aharon To: users at ovirt.org Subject: Re: [ovirt-users] Failed to Reconstruct Master Domain for Data Center Date: Sun, 09 Aug 2015 10:24:54 -0400 Message-ID: <1835645202.5650249.1439130294446.JavaMail.zimbra@redhat.com> In-Reply-To: 1703626980.1293890.1439071641698.JavaMail.yahoo@mail.yahoo.com --===============4189641226383560458== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ------=3D_Part_5650248_1998353805.1439130294445 Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit Hi Will, = Which oVirt version are you using? (rpm -q ovirt-engine) = Is the GlusterFS storage domain the master? = Also, please provide: = - /var/log/ovirt-engine/engine.log = - /var/log/vdsm/vdsm.log from both nodes = - /var/log/gluster/* = And the output of 'rpm -q glusterfs' from both hosts. = Thanks, = Elad = ----- Original Message ----- From: "Will K" = To: users(a)ovirt.org = Sent: Sunday, August 9, 2015 1:07:21 AM = Subject: [ovirt-users] Failed to Reconstruct Master Domain for Data Center = Hi all, = Yes, I got this failed to active master storage domain issue. I'm sort of n= ot sure where to start after a few attempts and many google results. Is a g= ood lead on how to troubleshoot this master storage failed to start issue? = I have a 2 node ovirt setup. It is a simple setup with GlusterFS with one v= olume only. Both nodes were put to "maintenance mode" before reboot. After = reboot, I checked the followings were all up: = vdsmd, ovirt-engine, ovirt-websocket-proxy, nfs, nfslock, glusterd, gluster= fsd = I can reach to both nodes and I can activate them in overt. however, the ma= ster storage can't be activated. = I tried to start with just node1. It worked fine and storage domain was up.= But once I booted up and activated node2, the NFS export domain just fails= followed by the Master domain failure. = Any suggestion where to start? = Will = _______________________________________________ = Users mailing list = Users(a)ovirt.org = http://lists.ovirt.org/mailman/listinfo/users = ------=3D_Part_5650248_1998353805.1439130294445 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: quoted-printable

Hi Will,

Which oVirt version are you using? (rpm -q ovirt-engine)

Is the GlusterFS storage domain the master?

Also, please p= =3D rovide:

- /var/log/ovirt-engine/engine.log

- /var/log/vdsm/vdsm.log from both nodes

- /var/l= =3D og/gluster/*

And the output of 'rpm -q gluster= =3D fs' from both hosts.

Thanks,

Elad

From: "Will K" <y= =3D etanotherwill(a)yahoo.com>
To: users(a)ovirt.org
Sent: <= /b>S=3D unday, August 9, 2015 1:07:21 AM
Subject: [ovirt-users] Failed to= =3D Reconstruct Master Domain for Data Center

Hi all,

Yes, I got this failed to active master storage domain issue. I'= =3D m sort of not sure where to start after a few attempts and many google resu= =3D lts. Is a good lead on how to troubleshoot this master storage failed to st= =3D art issue?
I have a= 2 n=3D ode ovirt setup. It is a simple setup with GlusterFS with one volume = =3D only. Both nodes were put to "maintenance mode" before reboot. After = =3D reboot, I checked the followings were all up:
vdsmd, ovirt-engine, ov= irt-=3D websocket-proxy, nfs, nfslock, glusterd, glusterfsd

I can reach to = both=3D nodes and I can activate them in overt. however, the master storage can't = =3D be activated.

I tried to start with just node1. It worked fine and sto= ra=3D ge domain was up. But once I booted up and activated node2, the NFS e= =3D xport domain just fails followed by the Master domain failure.
<= =3D div id=3D3D"yui_3_16_0_1_1439066928785_9026" dir=3D3D"ltr" class=3D3D"">
A= ny sug=3D gestion where to start?

Will

=
=3D

_______________________________________________
Users ma= =3D iling list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/= us=3D ers

------=3D_Part_5650248_1998353805.1439130294445-- --===============4189641226383560458== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS0tLS0tPV9QYXJ0XzU2NTAyNDhfMTk5ODM1MzgwNS4xNDM5MTMwMjk0NDQ1CkNvbnRlbnQtVHlw ZTogdGV4dC9wbGFpbjsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA3 Yml0CgpIaSBXaWxsLCAKCldoaWNoIG9WaXJ0IHZlcnNpb24gYXJlIHlvdSB1c2luZz8gKHJwbSAt cSBvdmlydC1lbmdpbmUpIApJcyB0aGUgR2x1c3RlckZTIHN0b3JhZ2UgZG9tYWluIHRoZSBtYXN0 ZXI/IApBbHNvLCBwbGVhc2UgcHJvdmlkZTogCgotIC92YXIvbG9nL292aXJ0LWVuZ2luZS9lbmdp bmUubG9nIAotIC92YXIvbG9nL3Zkc20vdmRzbS5sb2cgZnJvbSBib3RoIG5vZGVzIAotIC92YXIv bG9nL2dsdXN0ZXIvKiAKCkFuZCB0aGUgb3V0cHV0IG9mICdycG0gLXEgZ2x1c3RlcmZzJyBmcm9t IGJvdGggaG9zdHMuIAoKClRoYW5rcywgCkVsYWQgCgotLS0tLSBPcmlnaW5hbCBNZXNzYWdlIC0t LS0tCgpGcm9tOiAiV2lsbCBLIiA8eWV0YW5vdGhlcndpbGxAeWFob28uY29tPiAKVG86IHVzZXJz QG92aXJ0Lm9yZyAKU2VudDogU3VuZGF5LCBBdWd1c3QgOSwgMjAxNSAxOjA3OjIxIEFNIApTdWJq ZWN0OiBbb3ZpcnQtdXNlcnNdIEZhaWxlZCB0byBSZWNvbnN0cnVjdCBNYXN0ZXIgRG9tYWluIGZv ciBEYXRhIENlbnRlciAKCkhpIGFsbCwgCgpZZXMsIEkgZ290IHRoaXMgZmFpbGVkIHRvIGFjdGl2 ZSBtYXN0ZXIgc3RvcmFnZSBkb21haW4gaXNzdWUuIEknbSBzb3J0IG9mIG5vdCBzdXJlIHdoZXJl IHRvIHN0YXJ0IGFmdGVyIGEgZmV3IGF0dGVtcHRzIGFuZCBtYW55IGdvb2dsZSByZXN1bHRzLiBJ cyBhIGdvb2QgbGVhZCBvbiBob3cgdG8gdHJvdWJsZXNob290IHRoaXMgbWFzdGVyIHN0b3JhZ2Ug ZmFpbGVkIHRvIHN0YXJ0IGlzc3VlPyAKCkkgaGF2ZSBhIDIgbm9kZSBvdmlydCBzZXR1cC4gSXQg aXMgYSBzaW1wbGUgc2V0dXAgd2l0aCBHbHVzdGVyRlMgd2l0aCBvbmUgdm9sdW1lIG9ubHkuIEJv dGggbm9kZXMgd2VyZSBwdXQgdG8gIm1haW50ZW5hbmNlIG1vZGUiIGJlZm9yZSByZWJvb3QuIEFm dGVyIHJlYm9vdCwgSSBjaGVja2VkIHRoZSBmb2xsb3dpbmdzIHdlcmUgYWxsIHVwOiAKdmRzbWQs IG92aXJ0LWVuZ2luZSwgb3ZpcnQtd2Vic29ja2V0LXByb3h5LCBuZnMsIG5mc2xvY2ssIGdsdXN0 ZXJkLCBnbHVzdGVyZnNkIAoKSSBjYW4gcmVhY2ggdG8gYm90aCBub2RlcyBhbmQgSSBjYW4gYWN0 aXZhdGUgdGhlbSBpbiBvdmVydC4gaG93ZXZlciwgdGhlIG1hc3RlciBzdG9yYWdlIGNhbid0IGJl IGFjdGl2YXRlZC4gCgpJIHRyaWVkIHRvIHN0YXJ0IHdpdGgganVzdCBub2RlMS4gSXQgd29ya2Vk IGZpbmUgYW5kIHN0b3JhZ2UgZG9tYWluIHdhcyB1cC4gQnV0IG9uY2UgSSBib290ZWQgdXAgYW5k IGFjdGl2YXRlZCBub2RlMiwgdGhlIE5GUyBleHBvcnQgZG9tYWluIGp1c3QgZmFpbHMgZm9sbG93 ZWQgYnkgdGhlIE1hc3RlciBkb21haW4gZmFpbHVyZS4gCgpBbnkgc3VnZ2VzdGlvbiB3aGVyZSB0 byBzdGFydD8gCgpXaWxsIAoKCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18gClVzZXJzIG1haWxpbmcgbGlzdCAKVXNlcnNAb3ZpcnQub3JnIApodHRwOi8v bGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMgCgoKLS0tLS0tPV9QYXJ0XzU2 NTAyNDhfMTk5ODM1MzgwNS4xNDM5MTMwMjk0NDQ1CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBj aGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUK CjxodG1sPjxib2R5PjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IHRpbWVzIG5ldyByb21hbiwg bmV3IHlvcmssIHRpbWVzLCBzZT0KcmlmOyBmb250LXNpemU6IDEycHQ7IGNvbG9yOiAjMDAwMDAw Ij48ZGl2PkhpIFdpbGwsPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PQo+PGRpdj5XaGljaCBvVmly dCB2ZXJzaW9uIGFyZSB5b3UgdXNpbmc/IChycG0gLXEgb3ZpcnQtZW5naW5lKTxicj48L2Rpdj48 ZGk9CnY+SXMgdGhlIEdsdXN0ZXJGUyBzdG9yYWdlIGRvbWFpbiB0aGUgbWFzdGVyPyA8YnI+PC9k aXY+PGRpdj5BbHNvLCBwbGVhc2UgcD0Kcm92aWRlOjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48 ZGl2Pi0gL3Zhci9sb2cvb3ZpcnQtZW5naW5lL2VuZ2luZS5sb2cgPGJyPQo+PC9kaXY+PGRpdj4t IC92YXIvbG9nL3Zkc20vdmRzbS5sb2cgZnJvbSBib3RoIG5vZGVzPGJyPjwvZGl2PjxkaXY+LSAv dmFyL2w9Cm9nL2dsdXN0ZXIvKjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkFuZCB0aGUg b3V0cHV0IG9mICdycG0gLXEgZ2x1c3Rlcj0KZnMnIGZyb20gYm90aCBob3N0cy4gPGJyPjwvZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhhbmtzLDxiPQpyPjwvZGl2Pjxk aXY+RWxhZDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48aHIgaWQ9M0QiendjaHIiPjxkaXYgc3R5 bGU9M0QiY289CmxvcjojMDAwO2ZvbnQtd2VpZ2h0Om5vcm1hbDtmb250LXN0eWxlOm5vcm1hbDt0 ZXh0LWRlY29yYXRpb246bm9uZTtmb250LWZhbT0KaWx5OkhlbHZldGljYSxBcmlhbCxzYW5zLXNl cmlmO2ZvbnQtc2l6ZToxMnB0OyI+PGI+RnJvbTogPC9iPiJXaWxsIEsiICZsdDt5PQpldGFub3Ro ZXJ3aWxsQHlhaG9vLmNvbSZndDs8YnI+PGI+VG86IDwvYj51c2Vyc0BvdmlydC5vcmc8YnI+PGI+ U2VudDogPC9iPlM9CnVuZGF5LCBBdWd1c3QgOSwgMjAxNSAxOjA3OjIxIEFNPGJyPjxiPlN1Ympl Y3Q6IDwvYj5bb3ZpcnQtdXNlcnNdIEZhaWxlZCB0bz0KIFJlY29uc3RydWN0IE1hc3RlciBEb21h aW4gZm9yIERhdGEgQ2VudGVyPGJyPjxkaXY+PGJyPjwvZGl2PjxkaXYgc3R5bGU9M0QiPQpjb2xv cjojMDAwOyBiYWNrZ3JvdW5kLWNvbG9yOiNmZmY7IGZvbnQtZmFtaWx5OkhlbHZldGljYU5ldWUs IEhlbHZldGljYSBOZXU9CmUsIEhlbHZldGljYSwgQXJpYWwsIEx1Y2lkYSBHcmFuZGUsIHNhbnMt c2VyaWY7Zm9udC1zaXplOjE2cHgiPjxkaXYgaWQ9M0QieT0KdWlfM18xNl8wXzFfMTQzOTA2Njky ODc4NV85MTcwIj5IaSBhbGwsPC9kaXY+PGRpdiBpZD0zRCJ5dWlfM18xNl8wXzFfMTQzOTA2PQo2 OTI4Nzg1XzkwMzciPjxicj48L2Rpdj48ZGl2IGlkPTNEInl1aV8zXzE2XzBfMV8xNDM5MDY2OTI4 Nzg1XzkwMjYiIGRpcj0zRCI9Cmx0ciI+WWVzLCBJIGdvdCB0aGlzIGZhaWxlZCB0byBhY3RpdmUg bWFzdGVyIHN0b3JhZ2UgZG9tYWluIGlzc3VlLiAmbmJzcDtJJz0KbSBzb3J0IG9mIG5vdCBzdXJl IHdoZXJlIHRvIHN0YXJ0IGFmdGVyIGEgZmV3IGF0dGVtcHRzIGFuZCBtYW55IGdvb2dsZSByZXN1 PQpsdHMuIElzIGEgZ29vZCBsZWFkIG9uIGhvdyB0byB0cm91Ymxlc2hvb3QgdGhpcyBtYXN0ZXIg c3RvcmFnZSBmYWlsZWQgdG8gc3Q9CmFydCBpc3N1ZT88L2Rpdj48ZGl2IGlkPTNEInl1aV8zXzE2 XzBfMV8xNDM5MDY2OTI4Nzg1XzkwMjYiIGRpcj0zRCJsdHIiPjxicj0KPjwvZGl2PjxkaXYgaWQ9 M0QieXVpXzNfMTZfMF8xXzE0MzkwNjY5Mjg3ODVfOTAyNiIgZGlyPTNEImx0ciI+SSBoYXZlIGEg MiBuPQpvZGUgb3ZpcnQgc2V0dXAuIEl0IGlzIGEgc2ltcGxlIHNldHVwIHdpdGggR2x1c3RlckZT IHdpdGggJm5ic3A7b25lIHZvbHVtZSA9Cm9ubHkuIEJvdGggbm9kZXMgd2VyZSBwdXQgdG8gIm1h aW50ZW5hbmNlIG1vZGUiIGJlZm9yZSByZWJvb3QuICZuYnNwO0FmdGVyID0KcmVib290LCBJIGNo ZWNrZWQgdGhlIGZvbGxvd2luZ3Mgd2VyZSBhbGwgdXA6PC9kaXY+PGRpdiBpZD0zRCJ5dWlfM18x Nl8wXzFfPQoxNDM5MDY2OTI4Nzg1XzkwMjYiIGRpcj0zRCJsdHIiIGNsYXNzPTNEIiI+dmRzbWQs IG92aXJ0LWVuZ2luZSwmbmJzcDtvdmlydC09CndlYnNvY2tldC1wcm94eSwgbmZzLCBuZnNsb2Nr LCBnbHVzdGVyZCwmbmJzcDtnbHVzdGVyZnNkPC9kaXY+PGRpdiBpZD0zRCJ5dT0KaV8zXzE2XzBf MV8xNDM5MDY2OTI4Nzg1XzkwMjYiIGRpcj0zRCJsdHIiIGNsYXNzPTNEIiI+PGJyPjwvZGl2Pjxk aXYgaWQ9M0QiPQp5dWlfM18xNl8wXzFfMTQzOTA2NjkyODc4NV85MDI2IiBkaXI9M0QibHRyIiBj bGFzcz0zRCIiPkkgY2FuIHJlYWNoIHRvIGJvdGg9CiBub2RlcyBhbmQgSSBjYW4gYWN0aXZhdGUg dGhlbSBpbiBvdmVydC4gaG93ZXZlciwgdGhlIG1hc3RlciBzdG9yYWdlIGNhbid0ID0KYmUgYWN0 aXZhdGVkLjxicj48L2Rpdj48ZGl2IGlkPTNEInl1aV8zXzE2XzBfMV8xNDM5MDY2OTI4Nzg1Xzkw MjYiIGRpcj0zRCJsPQp0ciIgY2xhc3M9M0QiIj48YnI+PC9kaXY+PGRpdiBpZD0zRCJ5dWlfM18x Nl8wXzFfMTQzOTA2NjkyODc4NV85MDI2IiBkaXI9M0Q9CiJsdHIiIGNsYXNzPTNEIiI+SSB0cmll ZCB0byBzdGFydCB3aXRoIGp1c3Qgbm9kZTEuIEl0IHdvcmtlZCBmaW5lIGFuZCBzdG9yYT0KZ2Ug ZG9tYWluIHdhcyB1cC4gJm5ic3A7QnV0IG9uY2UgSSBib290ZWQgdXAgYW5kIGFjdGl2YXRlZCBu b2RlMiwgdGhlIE5GUyBlPQp4cG9ydCBkb21haW4ganVzdCBmYWlscyBmb2xsb3dlZCBieSB0aGUg TWFzdGVyIGRvbWFpbiBmYWlsdXJlLiZuYnNwOzwvZGl2Pjw9CmRpdiBpZD0zRCJ5dWlfM18xNl8w XzFfMTQzOTA2NjkyODc4NV85MDI2IiBkaXI9M0QibHRyIiBjbGFzcz0zRCIiPjxicj48L2Rpdj0K PjxkaXYgaWQ9M0QieXVpXzNfMTZfMF8xXzE0MzkwNjY5Mjg3ODVfOTAyNiIgZGlyPTNEImx0ciIg Y2xhc3M9M0QiIj5Bbnkgc3VnPQpnZXN0aW9uIHdoZXJlIHRvIHN0YXJ0PzwvZGl2PjxkaXYgaWQ9 M0QieXVpXzNfMTZfMF8xXzE0MzkwNjY5Mjg3ODVfOTAyNiIgZGk9CnI9M0QibHRyIiBjbGFzcz0z RCIiPjxicj48L2Rpdj48ZGl2IGlkPTNEInl1aV8zXzE2XzBfMV8xNDM5MDY2OTI4Nzg1XzkwMjYi ID0KZGlyPTNEImx0ciIgY2xhc3M9M0QiIj5XaWxsPC9kaXY+PGRpdiBpZD0zRCJ5dWlfM18xNl8w XzFfMTQzOTA2NjkyODc4NV85MDI2PQoiIGRpcj0zRCJsdHIiPjxicj48L2Rpdj48ZGl2IGlkPTNE Inl1aV8zXzE2XzBfMV8xNDM5MDY2OTI4Nzg1XzkwMjYiIGRpcj0zRCI9Cmx0ciI+PGJyPjwvZGl2 PjxkaXYgaWQ9M0QieXVpXzNfMTZfMF8xXzE0MzkwNjY5Mjg3ODVfOTAyNiIgZGlyPTNEImx0ciI+ PGJyPj0KPC9kaXY+PC9kaXY+PGJyPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fPGJyPlVzZXJzIG1hPQppbGluZyBsaXN0PGJyPlVzZXJzQG92aXJ0Lm9yZzxi cj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXM9CmVyczxicj48L2Rp dj48ZGl2Pjxicj48L2Rpdj48L2Rpdj48L2JvZHk+PC9odG1sPgotLS0tLS09X1BhcnRfNTY1MDI0 OF8xOTk4MzUzODA1LjE0MzkxMzAyOTQ0NDUtLQo= --===============4189641226383560458==--