From mahdi.adnan at outlook.com Fri Aug 25 12:42:15 2017 Content-Type: multipart/mixed; boundary="===============3845808860793306259==" MIME-Version: 1.0 From: Mahdi Adnan To: users at ovirt.org Subject: [ovirt-users] No SPM after network issue Date: Thu, 24 Aug 2017 13:06:38 +0000 Message-ID: --===============3845808860793306259== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DM5PR01MB250631DCB527862BADB99861FF9A0DM5PR01MB2506prod_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, Our oVirt DC became unresponsive after networking issue between "Engine, Ho= =3D sts, and Gluster storage" after around 50 seconds network issue resolved bu= =3D t i lost SPM. sanlock log: 2017-08-24 16:00:05+0300 73290 [1127]: s14191 lockspace 1b34ff4c-5d9d-44f5-= =3D a22e-6ca411865833:1:/rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt_= =3D _imgs/1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids:0 2017-08-24 16:00:05+0300 73290 [1249]: 1b34ff4c aio collect RD 0x7fa6f40008= =3D c0:0x7fa6f40008d0:0x7fa6f4101000 result -5:0 match res 2017-08-24 16:00:05+0300 73290 [1249]: read_sectors delta_leader offset 0 r= =3D v -5 /rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt__imgs/1b34ff4c-= =3D 5d9d-44f5-a22e-6ca411865833/dom_md/ids 2017-08-24 16:00:06+0300 73291 [1127]: s14191 add_lockspace fail result -5 2017-08-24 16:00:08+0300 73293 [12039]: s14192 lockspace 1b34ff4c-5d9d-44f5= =3D -a22e-6ca411865833:1:/rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt= =3D __imgs/1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids:0 2017-08-24 16:00:08+0300 73293 [1367]: 1b34ff4c aio collect RD 0x7fa6f40008= =3D c0:0x7fa6f40008d0:0x7fa6f4101000 result -5:0 match res 2017-08-24 16:00:08+0300 73293 [1367]: read_sectors delta_leader offset 0 r= =3D v -5 /rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt__imgs/1b34ff4c-= =3D 5d9d-44f5-a22e-6ca411865833/dom_md/ids 2017-08-24 16:00:09+0300 73294 [12039]: s14192 add_lockspace fail result -5 --- i cant read anything from ids, it gives mr read io error. how can i recreate the ids file or reset sanlock without losing the whole D= =3D C ? Thanks. -- Respectfully Mahdi A. Mahdi --_000_DM5PR01MB250631DCB527862BADB99861FF9A0DM5PR01MB2506prod_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi,

Our oVirt DC became unresponsive after networking issue between "Engin= =3D e, Hosts, and Gluster storage" after around 50 seconds network issue r= =3D esolved but i lost SPM.
sanlock log:

2017-08-24 16:00:05+0300 73290 [1127]: s14191 lockspace 1b34ff4c-5= =3D d9d-44f5-a22e-6ca411865833:1:/rhev/data-center/mnt/glusterSD/192.168.209.19= =3D 5:_ovirt__imgs/1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids:0
2017-08-24 16:00:05+0300 73290 [1249]: 1b34ff4c aio collect RD 0x7= =3D fa6f40008c0:0x7fa6f40008d0:0x7fa6f4101000 result -5:0 match res
2017-08-24 16:00:05+0300 73290 [1249]: read_sectors delta_leader o= =3D ffset 0 rv -5 /rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt__imgs/= =3D 1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids
2017-08-24 16:00:06+0300 73291 [1127]: s14191 add_lockspace fail r= =3D esult -5
2017-08-24 16:00:08+0300 73293 [12039]: s14192 lockspace 1b34ff4c-= =3D 5d9d-44f5-a22e-6ca411865833:1:/rhev/data-center/mnt/glusterSD/192.168.209.1= =3D 95:_ovirt__imgs/1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids:0
2017-08-24 16:00:08+0300 73293 [1367]: 1b34ff4c aio collect RD 0x7= =3D fa6f40008c0:0x7fa6f40008d0:0x7fa6f4101000 result -5:0 match res
2017-08-24 16:00:08+0300 73293 [1367]: read_sectors delta_leader o= =3D ffset 0 rv -5 /rhev/data-center/mnt/glusterSD/192.168.209.195:_ovirt__imgs/= =3D 1b34ff4c-5d9d-44f5-a22e-6ca411865833/dom_md/ids
2017-08-24 16:00:09+0300 73294 [12039]: s14192 add_lockspace fail = =3D result -5


---

i cant read anything from ids, it gives mr read io error.
how can i recreate the ids file or reset sanlock without losing the whole D= =3D C ?

Thanks.


--

Respectfully
Mahdi A. Mahdi

--_000_DM5PR01MB250631DCB527862BADB99861FF9A0DM5PR01MB2506prod_-- --===============3845808860793306259== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RNNVBSMDFNQjI1MDYzMURDQjUyNzg2MkJBREI5OTg2MUZGOUEwRE01UFIwMU1CMjUw NnByb2RfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKSGksCgpPdXIgb1ZpcnQg REMgYmVjYW1lIHVucmVzcG9uc2l2ZSBhZnRlciBuZXR3b3JraW5nIGlzc3VlIGJldHdlZW4gIkVu Z2luZSwgSG89CnN0cywgYW5kIEdsdXN0ZXIgc3RvcmFnZSIgYWZ0ZXIgYXJvdW5kIDUwIHNlY29u ZHMgbmV0d29yayBpc3N1ZSByZXNvbHZlZCBidT0KdCBpIGxvc3QgU1BNLgpzYW5sb2NrIGxvZzoK CjIwMTctMDgtMjQgMTY6MDA6MDUrMDMwMCA3MzI5MCBbMTEyN106IHMxNDE5MSBsb2Nrc3BhY2Ug MWIzNGZmNGMtNWQ5ZC00NGY1LT0KYTIyZS02Y2E0MTE4NjU4MzM6MTovcmhldi9kYXRhLWNlbnRl ci9tbnQvZ2x1c3RlclNELzE5Mi4xNjguMjA5LjE5NTpfb3ZpcnRfPQpfaW1ncy8xYjM0ZmY0Yy01 ZDlkLTQ0ZjUtYTIyZS02Y2E0MTE4NjU4MzMvZG9tX21kL2lkczowCjIwMTctMDgtMjQgMTY6MDA6 MDUrMDMwMCA3MzI5MCBbMTI0OV06IDFiMzRmZjRjIGFpbyBjb2xsZWN0IFJEIDB4N2ZhNmY0MDAw OD0KYzA6MHg3ZmE2ZjQwMDA4ZDA6MHg3ZmE2ZjQxMDEwMDAgcmVzdWx0IC01OjAgbWF0Y2ggcmVz CjIwMTctMDgtMjQgMTY6MDA6MDUrMDMwMCA3MzI5MCBbMTI0OV06IHJlYWRfc2VjdG9ycyBkZWx0 YV9sZWFkZXIgb2Zmc2V0IDAgcj0KdiAtNSAvcmhldi9kYXRhLWNlbnRlci9tbnQvZ2x1c3RlclNE LzE5Mi4xNjguMjA5LjE5NTpfb3ZpcnRfX2ltZ3MvMWIzNGZmNGMtPQo1ZDlkLTQ0ZjUtYTIyZS02 Y2E0MTE4NjU4MzMvZG9tX21kL2lkcwoyMDE3LTA4LTI0IDE2OjAwOjA2KzAzMDAgNzMyOTEgWzEx MjddOiBzMTQxOTEgYWRkX2xvY2tzcGFjZSBmYWlsIHJlc3VsdCAtNQoyMDE3LTA4LTI0IDE2OjAw OjA4KzAzMDAgNzMyOTMgWzEyMDM5XTogczE0MTkyIGxvY2tzcGFjZSAxYjM0ZmY0Yy01ZDlkLTQ0 ZjU9Ci1hMjJlLTZjYTQxMTg2NTgzMzoxOi9yaGV2L2RhdGEtY2VudGVyL21udC9nbHVzdGVyU0Qv MTkyLjE2OC4yMDkuMTk1Ol9vdmlydD0KX19pbWdzLzFiMzRmZjRjLTVkOWQtNDRmNS1hMjJlLTZj YTQxMTg2NTgzMy9kb21fbWQvaWRzOjAKMjAxNy0wOC0yNCAxNjowMDowOCswMzAwIDczMjkzIFsx MzY3XTogMWIzNGZmNGMgYWlvIGNvbGxlY3QgUkQgMHg3ZmE2ZjQwMDA4PQpjMDoweDdmYTZmNDAw MDhkMDoweDdmYTZmNDEwMTAwMCByZXN1bHQgLTU6MCBtYXRjaCByZXMKMjAxNy0wOC0yNCAxNjow MDowOCswMzAwIDczMjkzIFsxMzY3XTogcmVhZF9zZWN0b3JzIGRlbHRhX2xlYWRlciBvZmZzZXQg MCByPQp2IC01IC9yaGV2L2RhdGEtY2VudGVyL21udC9nbHVzdGVyU0QvMTkyLjE2OC4yMDkuMTk1 Ol9vdmlydF9faW1ncy8xYjM0ZmY0Yy09CjVkOWQtNDRmNS1hMjJlLTZjYTQxMTg2NTgzMy9kb21f bWQvaWRzCjIwMTctMDgtMjQgMTY6MDA6MDkrMDMwMCA3MzI5NCBbMTIwMzldOiBzMTQxOTIgYWRk X2xvY2tzcGFjZSBmYWlsIHJlc3VsdCAtNQoKCi0tLQoKaSBjYW50IHJlYWQgYW55dGhpbmcgZnJv bSBpZHMsIGl0IGdpdmVzIG1yIHJlYWQgaW8gZXJyb3IuCmhvdyBjYW4gaSByZWNyZWF0ZSB0aGUg aWRzIGZpbGUgb3IgcmVzZXQgc2FubG9jayB3aXRob3V0IGxvc2luZyB0aGUgd2hvbGUgRD0KQyA/ CgpUaGFua3MuCgoKLS0KClJlc3BlY3RmdWxseQpNYWhkaSBBLiBNYWhkaQoKCi0tXzAwMF9ETTVQ UjAxTUIyNTA2MzFEQ0I1Mjc4NjJCQURCOTk4NjFGRjlBMERNNVBSMDFNQjI1MDZwcm9kXwpDb250 ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29udGVudC1UcmFuc2Zl ci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVx dWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWw7IGNoYXJzZXQ9M0Rpc28t ODg1OS09CjEiPgo8c3R5bGUgdHlwZT0zRCJ0ZXh0L2NzcyIgc3R5bGU9M0QiZGlzcGxheTpub25l OyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2k9Cm4tYm90dG9tOjA7fSAtLT48L3N0eWxlPgo8 L2hlYWQ+Cjxib2R5IGRpcj0zRCJsdHIiPgo8ZGl2IHN0eWxlPTNEImZvbnQtZmFtaWx5OiBDYWxp YnJpLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTJwdDs9CiBjb2xvcjogcmdi KDAsIDAsIDApOyI+CkhpLDwvZGl2Pgo8ZGl2IHN0eWxlPTNEImZvbnQtZmFtaWx5OiBDYWxpYnJp LCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTJwdDs9CiBjb2xvcjogcmdiKDAs IDAsIDApOyI+Cjxicj4KPC9kaXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENhbGlicmks IEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiByZ2IoMCwg MCwgMCk7Ij4KT3VyIG9WaXJ0IERDIGJlY2FtZSB1bnJlc3BvbnNpdmUgYWZ0ZXIgbmV0d29ya2lu ZyBpc3N1ZSBiZXR3ZWVuICZxdW90O0VuZ2luPQplLCBIb3N0cywgYW5kIEdsdXN0ZXIgc3RvcmFn ZSZxdW90OyBhZnRlciBhcm91bmQgNTAgc2Vjb25kcyBuZXR3b3JrIGlzc3VlIHI9CmVzb2x2ZWQg YnV0IGkgbG9zdCBTUE0uPC9kaXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENhbGlicmks IEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiByZ2IoMCwg MCwgMCk7Ij4Kc2FubG9jayBsb2c6PC9kaXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENh bGlicmksIEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiBy Z2IoMCwgMCwgMCk7Ij4KPGJyPgo8L2Rpdj4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogQ2Fs aWJyaSwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7PQogY29sb3I6IHJn YigwLCAwLCAwKTsiPgo8ZGl2PjIwMTctMDgtMjQgMTY6MDA6MDUmIzQzOzAzMDAgNzMyOTAgWzEx MjddOiBzMTQxOTEgbG9ja3NwYWNlIDFiMzRmZjRjLTU9CmQ5ZC00NGY1LWEyMmUtNmNhNDExODY1 ODMzOjE6L3JoZXYvZGF0YS1jZW50ZXIvbW50L2dsdXN0ZXJTRC8xOTIuMTY4LjIwOS4xOT0KNTpf b3ZpcnRfX2ltZ3MvMWIzNGZmNGMtNWQ5ZC00NGY1LWEyMmUtNmNhNDExODY1ODMzL2RvbV9tZC9p ZHM6MDwvZGl2Pgo8ZGl2PjIwMTctMDgtMjQgMTY6MDA6MDUmIzQzOzAzMDAgNzMyOTAgWzEyNDld OiAxYjM0ZmY0YyBhaW8gY29sbGVjdCBSRCAweDc9CmZhNmY0MDAwOGMwOjB4N2ZhNmY0MDAwOGQw OjB4N2ZhNmY0MTAxMDAwIHJlc3VsdCAtNTowIG1hdGNoIHJlczwvZGl2Pgo8ZGl2PjIwMTctMDgt MjQgMTY6MDA6MDUmIzQzOzAzMDAgNzMyOTAgWzEyNDldOiByZWFkX3NlY3RvcnMgZGVsdGFfbGVh ZGVyIG89CmZmc2V0IDAgcnYgLTUgL3JoZXYvZGF0YS1jZW50ZXIvbW50L2dsdXN0ZXJTRC8xOTIu MTY4LjIwOS4xOTU6X292aXJ0X19pbWdzLz0KMWIzNGZmNGMtNWQ5ZC00NGY1LWEyMmUtNmNhNDEx ODY1ODMzL2RvbV9tZC9pZHM8L2Rpdj4KPGRpdj4yMDE3LTA4LTI0IDE2OjAwOjA2JiM0MzswMzAw IDczMjkxIFsxMTI3XTogczE0MTkxIGFkZF9sb2Nrc3BhY2UgZmFpbCByPQplc3VsdCAtNTwvZGl2 Pgo8ZGl2PjIwMTctMDgtMjQgMTY6MDA6MDgmIzQzOzAzMDAgNzMyOTMgWzEyMDM5XTogczE0MTky IGxvY2tzcGFjZSAxYjM0ZmY0Yy09CjVkOWQtNDRmNS1hMjJlLTZjYTQxMTg2NTgzMzoxOi9yaGV2 L2RhdGEtY2VudGVyL21udC9nbHVzdGVyU0QvMTkyLjE2OC4yMDkuMT0KOTU6X292aXJ0X19pbWdz LzFiMzRmZjRjLTVkOWQtNDRmNS1hMjJlLTZjYTQxMTg2NTgzMy9kb21fbWQvaWRzOjA8L2Rpdj4K PGRpdj4yMDE3LTA4LTI0IDE2OjAwOjA4JiM0MzswMzAwIDczMjkzIFsxMzY3XTogMWIzNGZmNGMg YWlvIGNvbGxlY3QgUkQgMHg3PQpmYTZmNDAwMDhjMDoweDdmYTZmNDAwMDhkMDoweDdmYTZmNDEw MTAwMCByZXN1bHQgLTU6MCBtYXRjaCByZXM8L2Rpdj4KPGRpdj4yMDE3LTA4LTI0IDE2OjAwOjA4 JiM0MzswMzAwIDczMjkzIFsxMzY3XTogcmVhZF9zZWN0b3JzIGRlbHRhX2xlYWRlciBvPQpmZnNl dCAwIHJ2IC01IC9yaGV2L2RhdGEtY2VudGVyL21udC9nbHVzdGVyU0QvMTkyLjE2OC4yMDkuMTk1 Ol9vdmlydF9faW1ncy89CjFiMzRmZjRjLTVkOWQtNDRmNS1hMjJlLTZjYTQxMTg2NTgzMy9kb21f bWQvaWRzPC9kaXY+CjxkaXY+MjAxNy0wOC0yNCAxNjowMDowOSYjNDM7MDMwMCA3MzI5NCBbMTIw MzldOiBzMTQxOTIgYWRkX2xvY2tzcGFjZSBmYWlsID0KcmVzdWx0IC01PC9kaXY+Cjxicj4KPC9k aXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENhbGlicmksIEhlbHZldGljYSwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiByZ2IoMCwgMCwgMCk7Ij4KPGJyPgo8L2Rp dj4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogQ2FsaWJyaSwgSGVsdmV0aWNhLCBzYW5zLXNl cmlmOyBmb250LXNpemU6IDEycHQ7PQogY29sb3I6IHJnYigwLCAwLCAwKTsiPgotLS08L2Rpdj4K PGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogQ2FsaWJyaSwgSGVsdmV0aWNhLCBzYW5zLXNlcmlm OyBmb250LXNpemU6IDEycHQ7PQogY29sb3I6IHJnYigwLCAwLCAwKTsiPgo8YnI+CjwvZGl2Pgo8 ZGl2IHN0eWxlPTNEImZvbnQtZmFtaWx5OiBDYWxpYnJpLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWY7 IGZvbnQtc2l6ZTogMTJwdDs9CiBjb2xvcjogcmdiKDAsIDAsIDApOyI+CmkgY2FudCByZWFkIGFu eXRoaW5nIGZyb20gaWRzLCBpdCBnaXZlcyBtciByZWFkIGlvIGVycm9yLjwvZGl2Pgo8ZGl2IHN0 eWxlPTNEImZvbnQtZmFtaWx5OiBDYWxpYnJpLCBIZWx2ZXRpY2EsIHNhbnMtc2VyaWY7IGZvbnQt c2l6ZTogMTJwdDs9CiBjb2xvcjogcmdiKDAsIDAsIDApOyI+CmhvdyBjYW4gaSByZWNyZWF0ZSB0 aGUgaWRzIGZpbGUgb3IgcmVzZXQgc2FubG9jayB3aXRob3V0IGxvc2luZyB0aGUgd2hvbGUgRD0K QyA/PC9kaXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENhbGlicmksIEhlbHZldGljYSwg c2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiByZ2IoMCwgMCwgMCk7Ij4KPGJy Pgo8L2Rpdj4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogQ2FsaWJyaSwgSGVsdmV0aWNhLCBz YW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7PQogY29sb3I6IHJnYigwLCAwLCAwKTsiPgpUaGFu a3MuPC9kaXY+CjxkaXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6IENhbGlicmksIEhlbHZldGljYSwg c2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0Oz0KIGNvbG9yOiByZ2IoMCwgMCwgMCk7Ij4KPGJy Pgo8L2Rpdj4KPGRpdiBpZD0zRCJzaWduYXR1cmUiPjxicj4KPGRpdiBjbGFzcz0zRCJlY3htb3ot c2lnbmF0dXJlIj4tLSA8YnI+Cjxicj4KPGZvbnQgY29sb3I9M0QiIzMzNjZmZiI+PGZvbnQgY29s b3I9M0QiIzAwMDAwMCI+UmVzcGVjdGZ1bGx5PGI+PGJyPgo8L2I+PGI+TWFoZGkgQS4gTWFoZGk8 L2I+PC9mb250PjwvZm9udD48Zm9udCBjb2xvcj0zRCIjMzM2NmZmIj48YnI+Cjxicj4KPC9mb250 Pjxmb250IGNvbG9yPTNEIiMzMzY2ZmYiPjwvZm9udD48L2Rpdj4KPC9kaXY+CjwvYm9keT4KPC9o dG1sPgoKLS1fMDAwX0RNNVBSMDFNQjI1MDYzMURDQjUyNzg2MkJBREI5OTg2MUZGOUEwRE01UFIw MU1CMjUwNnByb2RfLS0K --===============3845808860793306259==--