From cameron.christensen at uk2group.com Fri Feb 19 17:47:22 2016 Content-Type: multipart/mixed; boundary="===============2411066407225894143==" MIME-Version: 1.0 From: Cameron Christensen To: users at ovirt.org Subject: Re: [ovirt-users] Missing dom_md/ids file Date: Fri, 19 Feb 2016 15:47:19 -0700 Message-ID: <1455922039.16201.17.camel@uk2group.com> In-Reply-To: CAMRbyyuffy83XQANdcDxb+s90m=PQEtPWa2OavHbk_YbGmAqFQ@mail.gmail.com --===============2411066407225894143== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --=3D-Y64HBS9PyTFj3Id7Uol1 Content-Type: multipart/alternative; boundary=3D"=3D-VOwXxmsV44gjJMcDeOV9" --=3D-VOwXxmsV44gjJMcDeOV9 Content-Type: text/plain; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable I was able to recreate the dom_md/ids file using the provided instructions. The data center and gluster storage are active. Thank you. Cameron On Sat, 2016-02-20 at 00:24 +0200, Nir Soffer wrote: > On Fri, Feb 19, 2016 at 11:54 PM, Cameron Christensen > wrote: > > I cannot put the gluster domain into maintenance. I'm believe this > > is > > because the data center has a status of Non responsive (because a > > host > > cannot connect to storage or start SPM). The only option available > > on the > > gluster storage is activate. I have put all the hosts into > > maintenance. Is > > this enough to continue with the initialize lockspace step? >=3D20 > Yes, if all hosts are in maintenance, no host will access the gluster > storage domain, and you can repair is safely. >=3D20 > >=3D20 > > On Fri, 2016-02-19 at 23:34 +0200, Nir Soffer wrote: > >=3D20 > > On Fri, Feb 19, 2016 at 10:58 PM, Cameron Christensen > > wrote: > >=3D20 > > Hello, > >=3D20 > > I am using glusterfs storage and ran into a split-brain issue. One > > of the > > file affected by split-brain was dom_md/ids. In attempts to fix the > > split-brain issue I deleted the dom_md/ids file. Is there a method > > to > > recreate or reconstruct this file? > >=3D20 > >=3D20 > > You can do this: > >=3D20 > > 1. Put the gluster domain to maintenance (via engine) > >=3D20 > > No host should access it while you reconstruct the ids file > >=3D20 > > 2. Mount the gluster volume manually > >=3D20 > > mkdir repair > > mount -t glusterfs :/ repair/ > >=3D20 > > 3. Create the file: > >=3D20 > > touch repair//dom_md/ids > >=3D20 > > 4. Initialize the lockspace > >=3D20 > > sanlock direct init -s :0:repair//dom_md/ids:0 > >=3D20 > > 5. Unmount the gluster volume > >=3D20 > > umount repair > >=3D20 > > 6. Activate the gluster domain (via engine) > >=3D20 > > The domain should become active after a while. > >=3D20 > >=3D20 > > David: can you confirm this is the best way to reconstruct the ids > > file? > >=3D20 > > Nir --=3D-VOwXxmsV44gjJMcDeOV9 Content-Type: text/html; charset=3D"utf-8" Content-Transfer-Encoding: quoted-printable

I was able to recreate the dom_md/ids file us= =3D ing the provided instructions. The data center and gluster storage are acti= =3D ve. Thank you.

Cameron

On= =3D Sat, 2016-02-20 at 00:24 +0200, Nir Soffer wrote:

On Fri, Feb 19, 2016 at 11:54 PM, Cameron Christensen
<cameron.christe=
nsen=3D
@uk2group.com> wrote:

I cannot put the gluster domain into maintenance. I'm believe this is
because the data center has a status of Non responsive (because a host
cannot connect to storage or start SPM). The only option available on the
gluster storage is activate. I have put all the hosts into maintenance. Is
this enough to continue with the initialize lockspace step?


Yes, if all hosts are in maintenance, no host will access the gluster
storage domain, and you can repair is safely.



On Fri, 2016-02-19 at 23:34 +0200, Nir Soffer wrote:

On Fri, Feb 19, 2016 at 10:58 PM, Cameron Christensen
<cameron.christe=
nsen=3D
@uk2group.com> wrote:

Hello,

I am using glusterfs storage and ran into a split-brain issue. One of the
file affected by split-brain was dom_md/ids. In attempts to fix the
split-brain issue I deleted the dom_md/ids file. Is there a method to
recreate or reconstruct this file?


You can do this:

1. Put the gluster domain to maintenance (via engine)

No host should access it while you reconstruct the ids file

2. Mount the gluster volume manually

mkdir repair
mount -t glusterfs <server>:/<path> repair/

3. Create the file:

touch repair/<sd_uuid>/dom_md/ids

4. Initialize the lockspace

sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:=
=3D
0

5. Unmount the gluster volume

umount repair

6. Activate the gluster domain (via engine)

The domain should become active after a while.


David: can you confirm this is the best way to reconstruct the ids file?

Nir

--=3D-VOwXxmsV44gjJMcDeOV9-- --=3D-Y64HBS9PyTFj3Id7Uol1 Content-Type: application/pgp-signature; name=3D"signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJWx5t3AAoJEM1PCzopIAOt3FQH/jN7Db9O1r83C4up1XaEZpvn Iq1cex1iH0TtOtwWmFElA6iW/9wUL3IfD3rgxHFwk+NOMYvidH1towCKww6UucVO nqF/9mvAiKjS1KD7JG7VOxKrcww2Q7KQMFYSTiaqSOFQGrTlD4Zt2767aYc4Xeln H9F4tnk21A3ceU9NAWCVRwQi2xiNaRHJCqzCRK1M/wM1EBzCQDRhLBtJka5mbOd9 RrvAuor7zEu64i9bMpnKZc0BwzFFDahNxlmUqT2rs4cU3lwqpgRx66UgoSuQUppx 3Zt/CObzMFqLrqSlDDf39JDyTBwKaGSkJDtHKgXLnPZLArECbMmyPsr+Oted8S8=3D =3DpWig -----END PGP SIGNATURE----- --=3D-Y64HBS9PyTFj3Id7Uol1-- --===============2411066407225894143== Content-Type: multipart/signed MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tPS1ZNjRIQlM5UHlURmozSWQ3VW9sMQpDb250ZW50LVR5cGU6IG11bHRpcGFydC9hbHRlcm5h dGl2ZTsgYm91bmRhcnk9Ij0tVk93WHhtc1Y0NGdqSk1jRGVPVjkiCgoKLS09LVZPd1h4bXNWNDRn akpNY0RlT1Y5CkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iVVRGLTgiCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCkkgd2FzIGFibGUgdG8gcmVj cmVhdGUgdGhlIGRvbV9tZC9pZHMgZmlsZSB1c2luZyB0aGUgcHJvdmlkZWQKaW5zdHJ1Y3Rpb25z LiBUaGUgZGF0YSBjZW50ZXIgYW5kIGdsdXN0ZXIgc3RvcmFnZSBhcmUgYWN0aXZlLiBUaGFuawp5 b3UuCkNhbWVyb24KT24gU2F0LCAyMDE2LTAyLTIwIGF0IDAwOjI0ICswMjAwLCBOaXIgU29mZmVy IHdyb3RlOgo+IE9uIEZyaSwgRmViIDE5LCAyMDE2IGF0IDExOjU0IFBNLCBDYW1lcm9uIENocmlz dGVuc2VuCj4gPGNhbWVyb24uY2hyaXN0ZW5zZW5AdWsyZ3JvdXAuY29tPiB3cm90ZToKPiA+IEkg Y2Fubm90IHB1dCB0aGUgZ2x1c3RlciBkb21haW4gaW50byBtYWludGVuYW5jZS4gSSdtIGJlbGll dmUgdGhpcwo+ID4gaXMKPiA+IGJlY2F1c2UgdGhlIGRhdGEgY2VudGVyIGhhcyBhIHN0YXR1cyBv ZiBOb24gcmVzcG9uc2l2ZSAoYmVjYXVzZSBhCj4gPiBob3N0Cj4gPiBjYW5ub3QgY29ubmVjdCB0 byBzdG9yYWdlIG9yIHN0YXJ0IFNQTSkuIFRoZSBvbmx5IG9wdGlvbiBhdmFpbGFibGUKPiA+IG9u IHRoZQo+ID4gZ2x1c3RlciBzdG9yYWdlIGlzIGFjdGl2YXRlLiBJIGhhdmUgcHV0IGFsbCB0aGUg aG9zdHMgaW50bwo+ID4gbWFpbnRlbmFuY2UuIElzCj4gPiB0aGlzIGVub3VnaCB0byBjb250aW51 ZSB3aXRoIHRoZSBpbml0aWFsaXplIGxvY2tzcGFjZSBzdGVwPwo+PTIwCj4gWWVzLCBpZiBhbGwg aG9zdHMgYXJlIGluIG1haW50ZW5hbmNlLCBubyBob3N0IHdpbGwgYWNjZXNzIHRoZSBnbHVzdGVy Cj4gc3RvcmFnZSBkb21haW4sIGFuZCB5b3UgY2FuIHJlcGFpciBpcyBzYWZlbHkuCj49MjAKPiA+ PTIwCj4gPiBPbiBGcmksIDIwMTYtMDItMTkgYXQgMjM6MzQgKzAyMDAsIE5pciBTb2ZmZXIgd3Jv dGU6Cj4gPj0yMAo+ID4gT24gRnJpLCBGZWIgMTksIDIwMTYgYXQgMTA6NTggUE0sIENhbWVyb24g Q2hyaXN0ZW5zZW4KPiA+IDxjYW1lcm9uLmNocmlzdGVuc2VuQHVrMmdyb3VwLmNvbT4gd3JvdGU6 Cj4gPj0yMAo+ID4gSGVsbG8sCj4gPj0yMAo+ID4gSSBhbSB1c2luZyBnbHVzdGVyZnMgc3RvcmFn ZSBhbmQgcmFuIGludG8gYSBzcGxpdC1icmFpbiBpc3N1ZS4gT25lCj4gPiBvZiB0aGUKPiA+IGZp bGUgYWZmZWN0ZWQgYnkgc3BsaXQtYnJhaW4gd2FzIGRvbV9tZC9pZHMuIEluIGF0dGVtcHRzIHRv IGZpeCB0aGUKPiA+IHNwbGl0LWJyYWluIGlzc3VlIEkgZGVsZXRlZCB0aGUgZG9tX21kL2lkcyBm aWxlLiBJcyB0aGVyZSBhIG1ldGhvZAo+ID4gdG8KPiA+IHJlY3JlYXRlIG9yIHJlY29uc3RydWN0 IHRoaXMgZmlsZT8KPiA+PTIwCj4gPj0yMAo+ID4gWW91IGNhbiBkbyB0aGlzOgo+ID49MjAKPiA+ IDEuIFB1dCB0aGUgZ2x1c3RlciBkb21haW4gdG8gbWFpbnRlbmFuY2UgKHZpYSBlbmdpbmUpCj4g Pj0yMAo+ID4gTm8gaG9zdCBzaG91bGQgYWNjZXNzIGl0IHdoaWxlIHlvdSByZWNvbnN0cnVjdCB0 aGUgaWRzIGZpbGUKPiA+PTIwCj4gPiAyLiBNb3VudCB0aGUgZ2x1c3RlciB2b2x1bWUgbWFudWFs bHkKPiA+PTIwCj4gPiBta2RpciByZXBhaXIKPiA+IG1vdW50IC10IGdsdXN0ZXJmcyA8c2VydmVy PjovPHBhdGg+IHJlcGFpci8KPiA+PTIwCj4gPiAzLiBDcmVhdGUgdGhlIGZpbGU6Cj4gPj0yMAo+ ID4gdG91Y2ggcmVwYWlyLzxzZF91dWlkPi9kb21fbWQvaWRzCj4gPj0yMAo+ID4gNC4gSW5pdGlh bGl6ZSB0aGUgbG9ja3NwYWNlCj4gPj0yMAo+ID4gc2FubG9jayBkaXJlY3QgaW5pdCAtcyA8c2Rf dXVpZD46MDpyZXBhaXIvPHNkX3V1aWQ+L2RvbV9tZC9pZHM6MAo+ID49MjAKPiA+IDUuIFVubW91 bnQgdGhlIGdsdXN0ZXIgdm9sdW1lCj4gPj0yMAo+ID4gdW1vdW50IHJlcGFpcgo+ID49MjAKPiA+ IDYuIEFjdGl2YXRlIHRoZSBnbHVzdGVyIGRvbWFpbiAodmlhIGVuZ2luZSkKPiA+PTIwCj4gPiBU aGUgZG9tYWluIHNob3VsZCBiZWNvbWUgYWN0aXZlIGFmdGVyIGEgd2hpbGUuCj4gPj0yMAo+ID49 MjAKPiA+IERhdmlkOiBjYW4geW91IGNvbmZpcm0gdGhpcyBpcyB0aGUgYmVzdCB3YXkgdG8gcmVj b25zdHJ1Y3QgdGhlIGlkcwo+ID4gZmlsZT8KPiA+PTIwCj4gPiBOaXIKLS09LVZPd1h4bXNWNDRn akpNY0RlT1Y5CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PSJ1dGYtOCIKQ29udGVu dC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+PGhlYWQ+PC9oZWFk Pjxib2R5PjxkaXY+SSB3YXMgYWJsZSB0byByZWNyZWF0ZSB0aGUgZG9tX21kL2lkcyBmaWxlIHVz PQppbmcgdGhlIHByb3ZpZGVkIGluc3RydWN0aW9ucy4gVGhlIGRhdGEgY2VudGVyIGFuZCBnbHVz dGVyIHN0b3JhZ2UgYXJlIGFjdGk9CnZlLiBUaGFuayB5b3UuPC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj5DYW1lcm9uPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Pbj0KIFNhdCwgMjAxNi0wMi0y MCBhdCAwMDoyNCArMDIwMCwgTmlyIFNvZmZlciB3cm90ZTo8L2Rpdj48YmxvY2txdW90ZSB0eXBl PTNEPQoiY2l0ZSI+PHByZT5PbiBGcmksIEZlYiAxOSwgMjAxNiBhdCAxMTo1NCBQTSwgQ2FtZXJv biBDaHJpc3RlbnNlbgombHQ7PGEgaHJlZj0zRCJtYWlsdG86Y2FtZXJvbi5jaHJpc3RlbnNlbkB1 azJncm91cC5jb20iPmNhbWVyb24uY2hyaXN0ZW5zZW49CkB1azJncm91cC5jb208L2E+Jmd0OyB3 cm90ZToKPGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIj4KSSBjYW5ub3QgcHV0IHRoZSBnbHVzdGVy IGRvbWFpbiBpbnRvIG1haW50ZW5hbmNlLiBJJ20gYmVsaWV2ZSB0aGlzIGlzCmJlY2F1c2UgdGhl IGRhdGEgY2VudGVyIGhhcyBhIHN0YXR1cyBvZiBOb24gcmVzcG9uc2l2ZSAoYmVjYXVzZSBhIGhv c3QKY2Fubm90IGNvbm5lY3QgdG8gc3RvcmFnZSBvciBzdGFydCBTUE0pLiBUaGUgb25seSBvcHRp b24gYXZhaWxhYmxlIG9uIHRoZQpnbHVzdGVyIHN0b3JhZ2UgaXMgYWN0aXZhdGUuIEkgaGF2ZSBw dXQgYWxsIHRoZSBob3N0cyBpbnRvIG1haW50ZW5hbmNlLiBJcwp0aGlzIGVub3VnaCB0byBjb250 aW51ZSB3aXRoIHRoZSBpbml0aWFsaXplIGxvY2tzcGFjZSBzdGVwPwo8L2Jsb2NrcXVvdGU+CgpZ ZXMsIGlmIGFsbCBob3N0cyBhcmUgaW4gbWFpbnRlbmFuY2UsIG5vIGhvc3Qgd2lsbCBhY2Nlc3Mg dGhlIGdsdXN0ZXIKc3RvcmFnZSBkb21haW4sIGFuZCB5b3UgY2FuIHJlcGFpciBpcyBzYWZlbHku Cgo8YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiPgoKT24gRnJpLCAyMDE2LTAyLTE5IGF0IDIzOjM0 ICswMjAwLCBOaXIgU29mZmVyIHdyb3RlOgoKT24gRnJpLCBGZWIgMTksIDIwMTYgYXQgMTA6NTgg UE0sIENhbWVyb24gQ2hyaXN0ZW5zZW4KJmx0OzxhIGhyZWY9M0QibWFpbHRvOmNhbWVyb24uY2hy aXN0ZW5zZW5AdWsyZ3JvdXAuY29tIj5jYW1lcm9uLmNocmlzdGVuc2VuPQpAdWsyZ3JvdXAuY29t PC9hPiZndDsgd3JvdGU6CgpIZWxsbywKCkkgYW0gdXNpbmcgZ2x1c3RlcmZzIHN0b3JhZ2UgYW5k IHJhbiBpbnRvIGEgc3BsaXQtYnJhaW4gaXNzdWUuIE9uZSBvZiB0aGUKZmlsZSBhZmZlY3RlZCBi eSBzcGxpdC1icmFpbiB3YXMgZG9tX21kL2lkcy4gSW4gYXR0ZW1wdHMgdG8gZml4IHRoZQpzcGxp dC1icmFpbiBpc3N1ZSBJIGRlbGV0ZWQgdGhlIGRvbV9tZC9pZHMgZmlsZS4gSXMgdGhlcmUgYSBt ZXRob2QgdG8KcmVjcmVhdGUgb3IgcmVjb25zdHJ1Y3QgdGhpcyBmaWxlPwoKCllvdSBjYW4gZG8g dGhpczoKCjEuIFB1dCB0aGUgZ2x1c3RlciBkb21haW4gdG8gbWFpbnRlbmFuY2UgKHZpYSBlbmdp bmUpCgpObyBob3N0IHNob3VsZCBhY2Nlc3MgaXQgd2hpbGUgeW91IHJlY29uc3RydWN0IHRoZSBp ZHMgZmlsZQoKMi4gTW91bnQgdGhlIGdsdXN0ZXIgdm9sdW1lIG1hbnVhbGx5Cgpta2RpciByZXBh aXIKbW91bnQgLXQgZ2x1c3RlcmZzICZsdDtzZXJ2ZXImZ3Q7Oi8mbHQ7cGF0aCZndDsgcmVwYWly LwoKMy4gQ3JlYXRlIHRoZSBmaWxlOgoKdG91Y2ggcmVwYWlyLyZsdDtzZF91dWlkJmd0Oy9kb21f bWQvaWRzCgo0LiBJbml0aWFsaXplIHRoZSBsb2Nrc3BhY2UKCnNhbmxvY2sgZGlyZWN0IGluaXQg LXMgJmx0O3NkX3V1aWQmZ3Q7OjA6cmVwYWlyLyZsdDtzZF91dWlkJmd0Oy9kb21fbWQvaWRzOj0K MAoKNS4gVW5tb3VudCB0aGUgZ2x1c3RlciB2b2x1bWUKCnVtb3VudCByZXBhaXIKCjYuIEFjdGl2 YXRlIHRoZSBnbHVzdGVyIGRvbWFpbiAodmlhIGVuZ2luZSkKClRoZSBkb21haW4gc2hvdWxkIGJl Y29tZSBhY3RpdmUgYWZ0ZXIgYSB3aGlsZS4KCgpEYXZpZDogY2FuIHlvdSBjb25maXJtIHRoaXMg aXMgdGhlIGJlc3Qgd2F5IHRvIHJlY29uc3RydWN0IHRoZSBpZHMgZmlsZT8KCk5pcgo8L2Jsb2Nr cXVvdGU+PC9wcmU+PC9ibG9ja3F1b3RlPjwvYm9keT48L2h0bWw+Ci0tPS1WT3dYeG1zVjQ0Z2pK TWNEZU9WOS0tCgotLT0tWTY0SEJTOVB5VEZqM0lkN1VvbDEKQ29udGVudC1UeXBlOiBhcHBsaWNh dGlvbi9wZ3Atc2lnbmF0dXJlOyBuYW1lPSJzaWduYXR1cmUuYXNjIgpDb250ZW50LURlc2NyaXB0 aW9uOiBUaGlzIGlzIGEgZGlnaXRhbGx5IHNpZ25lZCBtZXNzYWdlIHBhcnQKQ29udGVudC1UcmFu c2Zlci1FbmNvZGluZzogN2JpdAoKLS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KVmVyc2lv bjogR251UEcgdjEKCmlRRWNCQUFCQWdBR0JRSld4NXQzQUFvSkVNMVBDem9wSUFPdDNGUUgvak43 RGI5TzFyODNDNHVwMVhhRVpwdm4KSXExY2V4MWlIMFR0T3R3V21GRWxBNmlXLzl3VUwzSWZEM3Jn eEhGd2srTk9NWXZpZEgxdG93Q0t3dzZVdWNWTwpucUYvOW12QWlLalMxS0Q3Skc3Vk94S3Jjd3cy UTdLUU1GWVNUaWFxU09GUUdyVGxENFp0Mjc2N2FZYzRYZWxuCkg5RjR0bmsyMUEzY2VVOU5BV0NW UndRaTJ4aU5hUkhKQ3F6Q1JLMU0vd00xRUJ6Q1FEUmhMQnRKa2E1bWJPZDkKUnJ2QXVvcjd6RXU2 NGk5Yk1wbktaYzBCd3pGRkRhaE54bG1VcVQycnM0Y1UzbHdxcGdSeDY2VWdvU3VRVXBweAozWnQv Q09iek1GcUxycVNsRERmMzlKRHlUQndLYUdTa0pEdEhLZ1hMblBaTEFyRUNiTW15UHNyK090ZWQ4 Uzg9Cj1wV2lnCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQoKLS09LVk2NEhCUzlQeVRGajNJ ZDdVb2wxLS0KCg== --===============2411066407225894143==--