--_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Tm8gY2x1ZSBob3cgdG8gZ2V0IHRoaXMgb3V0LiBJIGNhbiBtb3VudCBhbGwgc3RvcmFnZSBtYW51 YWxseSBvbiB0aGUgaHlwZXJ2aXNvcnMuIEl0IHNlZW1zIGxpa2UgYWZ0ZXIgYSByZWJvb3Qgb1Zp cnQgaXMgbm93IGhhdmluZyBzb21lIGlzc3VlIGFuZCB0aGUgc3RvcmFnZSBkb21haW4gaXMgc3R1 Y2sgaW4gbG9ja2VkIHN0YXRlLiBCZWNhdXNlIG9mIHRoaXMsIGNhbuKAmXQgYWN0aXZhdGUgYW55 IG90aGVyIHN0b3JhZ2UgZWl0aGVyLCBzbyB0aGUgb3RoZXIgZG9tYWlucyBhcmUgaW4gbWFpbnRl bmFuY2UgYW5kIHRoZSBtYXN0ZXIgc2l0cyBpbiBsb2NrZWQgc3RhdGUsIGhhcyBiZWVuIGZvciBo b3Vycy4NCg0KVGhpcyBzdGlja3Mgb3V0IG9uIGEgaHlwZXJ2aXNvcjoNCg0KU3RvcmFnZVBvb2xX cm9uZ01hc3RlcjogV3JvbmcgTWFzdGVyIGRvbWFpbiBvciBpdHMgdmVyc2lvbjogdSdTRD1kOGEw MTcyZS04MzdmLTQ1NTItOTJjNy01NjZkYzRlNTQ4ZTQsIHBvb2w9M2ZkMmFkOTItZTFlYi00OWMy LTkwNmQtMDBlYzIzM2Y2MTBhJw0KDQpOb3Qgc3VyZSwgbm90aGluZyBjaGFuZ2VkIG90aGVyIHRo YW4gYSByZWJvb3Qgb2YgdGhlIHN0b3JhZ2UuDQoNCkVuZ2luZSBsb2cgc2hvd3M6DQoNCltvcmcu b3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLlNldFZkc1N0YXR1c1ZEU0NvbW1hbmRdIChEZWZh dWx0UXVhcnR6U2NoZWR1bGVyOCkgWzU2OTY3MzJiXSBTVEFSVCwgU2V0VmRzU3RhdHVzVkRTQ29t bWFuZChIb3N0TmFtZSA9IFUzMVUzMk5vZGVBLCBTZXRWZHNTdGF0dXNWRFNDb21tYW5kUGFyYW1l dGVyczp7cnVuQXN5bmM9J3RydWUnLCBob3N0SWQ9JzcwZTJiOGU0LTA3NTItNDdhOC04ODRjLTgz N2EwMDAxM2U3OScsIHN0YXR1cz0nTm9uT3BlcmF0aW9uYWwnLCBub25PcGVyYXRpb25hbFJlYXNv bj0nU1RPUkFHRV9ET01BSU5fVU5SRUFDSEFCTEUnLCBzdG9wU3BtRmFpbHVyZUxvZ2dlZD0nZmFs c2UnLCBtYWludGVuYW5jZVJlYXNvbj0nbnVsbCd9KSwgbG9nIGlkOiA2ZGI5ODIwYQ0KDQpObyBp ZGVhIHdoeSBpdCBzYXlzIHVucmVhY2hhYmxlLCBpdCBjZXJ0YWlubHkgaXMgYmVjYXVzZSBJIGNh biBtYW51YWxseSBtb3VudCBBTEwgc3RvcmFnZSB0byB0aGUgaHlwZXJ2aXNvci4NCg0KU2VudCBm cm9tIE1haWw8aHR0cHM6Ly9nby5taWNyb3NvZnQuY29tL2Z3bGluay8/TGlua0lkPTU1MDk4Nj4g Zm9yIFdpbmRvd3MgMTANCg0K --_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_ Content-Type: text/html; charset="utf-8" Content-ID: <77EDC59EF3DBB0418BCEF6D897F65EE0@sct-15-1-789-14-msonline-outlook-8b04f.templateTenant> Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj ZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2 IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToy IDE1IDUgMiAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3Jt YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1i b3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp IixzYW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXBy aW9yaXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQph OnZpc2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9sbG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5 Ojk5Ow0KCWNvbG9yOiM5NTRGNzI7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQouTXNv Q2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTt9DQpAcGFnZSBXb3JkU2Vj dGlvbjENCgl7c2l6ZTo4LjVpbiAxMS4waW47DQoJbWFyZ2luOjEuMGluIDEuMGluIDEuMGluIDEu MGluO30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHls ZT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVOLVVTIiBsaW5rPSJibHVlIiB2bGluaz0iIzk1NEY3 MiI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Tm8g Y2x1ZSBob3cgdG8gZ2V0IHRoaXMgb3V0LiBJIGNhbiBtb3VudCBhbGwgc3RvcmFnZSBtYW51YWxs eSBvbiB0aGUgaHlwZXJ2aXNvcnMuIEl0IHNlZW1zIGxpa2UgYWZ0ZXIgYSByZWJvb3Qgb1ZpcnQg aXMgbm93IGhhdmluZyBzb21lIGlzc3VlIGFuZCB0aGUgc3RvcmFnZSBkb21haW4gaXMgc3R1Y2sg aW4gbG9ja2VkIHN0YXRlLiBCZWNhdXNlIG9mIHRoaXMsIGNhbuKAmXQgYWN0aXZhdGUgYW55IG90 aGVyIHN0b3JhZ2UNCiBlaXRoZXIsIHNvIHRoZSBvdGhlciBkb21haW5zIGFyZSBpbiBtYWludGVu YW5jZSBhbmQgdGhlIG1hc3RlciBzaXRzIGluIGxvY2tlZCBzdGF0ZSwgaGFzIGJlZW4gZm9yIGhv dXJzLjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+VGhpcyBzdGlja3Mgb3V0IG9uIGEgaHlwZXJ2aXNvcjo8L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPlN0b3JhZ2VQb29sV3JvbmdNYXN0ZXI6IFdyb25nIE1hc3RlciBkb21haW4gb3IgaXRz IHZlcnNpb246IHUnU0Q9ZDhhMDE3MmUtODM3Zi00NTUyLTkyYzctNTY2ZGM0ZTU0OGU0LCBwb29s PTNmZDJhZDkyLWUxZWItNDljMi05MDZkLTAwZWMyMzNmNjEwYSc8L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk5vdCBz dXJlLCBub3RoaW5nIGNoYW5nZWQgb3RoZXIgdGhhbiBhIHJlYm9vdCBvZiB0aGUgc3RvcmFnZS48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPkVuZ2luZSBsb2cgc2hvd3M6PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5bb3JnLm92aXJ0LmVu Z2luZS5jb3JlLnZkc2Jyb2tlci5TZXRWZHNTdGF0dXNWRFNDb21tYW5kXSAoRGVmYXVsdFF1YXJ0 elNjaGVkdWxlcjgpIFs1Njk2NzMyYl0gU1RBUlQsIFNldFZkc1N0YXR1c1ZEU0NvbW1hbmQoSG9z dE5hbWUgPSBVMzFVMzJOb2RlQSwgU2V0VmRzU3RhdHVzVkRTQ29tbWFuZFBhcmFtZXRlcnM6e3J1 bkFzeW5jPSd0cnVlJywgaG9zdElkPSc3MGUyYjhlNC0wNzUyLTQ3YTgtODg0Yy04MzdhMDAwMTNl NzknLA0KIHN0YXR1cz0nTm9uT3BlcmF0aW9uYWwnLCBub25PcGVyYXRpb25hbFJlYXNvbj0nU1RP UkFHRV9ET01BSU5fVU5SRUFDSEFCTEUnLCBzdG9wU3BtRmFpbHVyZUxvZ2dlZD0nZmFsc2UnLCBt YWludGVuYW5jZVJlYXNvbj0nbnVsbCd9KSwgbG9nIGlkOiA2ZGI5ODIwYTwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ Tm8gaWRlYSB3aHkgaXQgc2F5cyB1bnJlYWNoYWJsZSwgaXQgY2VydGFpbmx5IGlzIGJlY2F1c2Ug SSBjYW4gbWFudWFsbHkgbW91bnQgQUxMIHN0b3JhZ2UgdG8gdGhlIGh5cGVydmlzb3IuPC9wPg0K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj5TZW50IGZyb20gPGEgaHJlZj0iaHR0cHM6Ly9nby5taWNyb3NvZnQuY29tL2Z3bGlu ay8/TGlua0lkPTU1MDk4NiI+DQpNYWlsPC9hPiBmb3IgV2luZG93cyAxMDwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2JvZHk+DQo8L2h0 bWw+DQo= --_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_--
--_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable So apparently something didn=92t change the metadata to master before conne= ction was lost. I changed the metadata role to master and it came backup. S= eems emailing in helped because every time I can=92t figure something out, = email in a find it shortly after. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:43 PM To: users<mailto:users@ovirt.org> Subject: master storage domain stuck in locked state No clue how to get this out. I can mount all storage manually on the hyperv= isors. It seems like after a reboot oVirt is now having some issue and the = storage domain is stuck in locked state. Because of this, can=92t activate = any other storage either, so the other domains are in maintenance and the m= aster sits in locked state, has been for hours. This sticks out on a hypervisor: StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e= -837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a' Not sure, nothing changed other than a reboot of the storage. Engine log shows: [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche= duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075= 2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason= =3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false', maintenanc= eReason=3D'null'}), log id: 6db9820a No idea why it says unreachable, it certainly is because I can manually mou= nt ALL storage to the hypervisor. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window= s 10 --_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">So apparently something didn=92t change the metadata= to master before connection was lost. I changed the metadata role to maste= r and it came backup. Seems emailing in helped because every time I can=92t= figure something out, email in a find it shortly after.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:43 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <div class=3D"WordSection1"> <p class=3D"MsoNormal">No clue how to get this out. I can mount all storage= manually on the hypervisors. It seems like after a reboot oVirt is now hav= ing some issue and the storage domain is stuck in locked state. Because of = this, can=92t activate any other storage either, so the other domains are in maintenance and the master sits in loc= ked state, has been for hours.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">This sticks out on a hypervisor:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its v= ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4= 9c2-906d-00ec233f610a'</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of the= storage.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Engine log shows:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm= and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos= tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',= hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA= BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id:= 6db9820a</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">No idea why it says unreachable, it certainly is bec= ause I can manually mount ALL storage to the hypervisor.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Sent from <a href=3D"https://go.microsoft.com/fwlink= /?LinkId=3D550986"> Mail</a> for Windows 10</p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </div> </body> </html> --_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_--
--_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Spoke too soon. Some hosts came back up but the storage domain is still loc= ked so no vm=92s can be started. What is the proper way to force this to be= unlocked? Each time we look to move into production after successful testi= ng, something like this always seems to pop up at the last minute rending o= Virt questionable in terms of reliability for some unknown issue. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:54 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state So apparently something didn=92t change the metadata to master before conne= ction was lost. I changed the metadata role to master and it came backup. S= eems emailing in helped because every time I can=92t figure something out, = email in a find it shortly after. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:43 PM To: users<mailto:users@ovirt.org> Subject: master storage domain stuck in locked state No clue how to get this out. I can mount all storage manually on the hyperv= isors. It seems like after a reboot oVirt is now having some issue and the = storage domain is stuck in locked state. Because of this, can=92t activate = any other storage either, so the other domains are in maintenance and the m= aster sits in locked state, has been for hours. This sticks out on a hypervisor: StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e= -837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a' Not sure, nothing changed other than a reboot of the storage. Engine log shows: [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche= duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075= 2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason= =3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false', maintenanc= eReason=3D'null'}), log id: 6db9820a No idea why it says unreachable, it certainly is because I can manually mou= nt ALL storage to the hypervisor. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window= s 10 --_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the stor= age domain is still locked so no vm=92s can be started. What is the proper = way to force this to be unlocked? Each time we look to move into production= after successful testing, something like this always seems to pop up at the last minute rending oVirt question= able in terms of reliability for some unknown issue.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:54 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>RE: master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">So apparently something didn=92t change the metadata= to master before connection was lost. I changed the metadata role to maste= r and it came backup. Seems emailing in helped because every time I can=92t= figure something out, email in a find it shortly after.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:43 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <div class=3D"WordSection1"> <p class=3D"MsoNormal">No clue how to get this out. I can mount all storage= manually on the hypervisors. It seems like after a reboot oVirt is now hav= ing some issue and the storage domain is stuck in locked state. Because of = this, can=92t activate any other storage either, so the other domains are in maintenance and the master sits in loc= ked state, has been for hours.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">This sticks out on a hypervisor:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its v= ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4= 9c2-906d-00ec233f610a'</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of the= storage.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Engine log shows:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm= and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos= tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',= hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA= BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id:= 6db9820a</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">No idea why it says unreachable, it certainly is bec= ause I can manually mount ALL storage to the hypervisor.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Sent from <a href=3D"https://go.microsoft.com/fwlink= /?LinkId=3D550986"> Mail</a> for Windows 10</p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </div> </div> </body> </html> --_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_--
--_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Also cannot reinitialize the datacenter because the storage domain is locke= d. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 8:08 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state Spoke too soon. Some hosts came back up but the storage domain is still loc= ked so no vm=92s can be started. What is the proper way to force this to be= unlocked? Each time we look to move into production after successful testi= ng, something like this always seems to pop up at the last minute rending o= Virt questionable in terms of reliability for some unknown issue. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:54 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state So apparently something didn=92t change the metadata to master before conne= ction was lost. I changed the metadata role to master and it came backup. S= eems emailing in helped because every time I can=92t figure something out, = email in a find it shortly after. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:43 PM To: users<mailto:users@ovirt.org> Subject: master storage domain stuck in locked state No clue how to get this out. I can mount all storage manually on the hyperv= isors. It seems like after a reboot oVirt is now having some issue and the = storage domain is stuck in locked state. Because of this, can=92t activate = any other storage either, so the other domains are in maintenance and the m= aster sits in locked state, has been for hours. This sticks out on a hypervisor: StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e= -837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a' Not sure, nothing changed other than a reboot of the storage. Engine log shows: [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche= duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075= 2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason= =3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false', maintenanc= eReason=3D'null'}), log id: 6db9820a No idea why it says unreachable, it certainly is because I can manually mou= nt ALL storage to the hypervisor. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window= s 10 --_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Also cannot reinitialize the datacenter because the = storage domain is locked.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 8:08 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>RE: master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the stor= age domain is still locked so no vm=92s can be started. What is the proper = way to force this to be unlocked? Each time we look to move into production= after successful testing, something like this always seems to pop up at the last minute rending oVirt question= able in terms of reliability for some unknown issue.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:54 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>RE: master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">So apparently something didn=92t change the metadata= to master before connection was lost. I changed the metadata role to maste= r and it came backup. Seems emailing in helped because every time I can=92t= figure something out, email in a find it shortly after.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:43 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <div class=3D"WordSection1"> <p class=3D"MsoNormal">No clue how to get this out. I can mount all storage= manually on the hypervisors. It seems like after a reboot oVirt is now hav= ing some issue and the storage domain is stuck in locked state. Because of = this, can=92t activate any other storage either, so the other domains are in maintenance and the master sits in loc= ked state, has been for hours.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">This sticks out on a hypervisor:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its v= ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4= 9c2-906d-00ec233f610a'</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of the= storage.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Engine log shows:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm= and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos= tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',= hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA= BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), log id:= 6db9820a</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">No idea why it says unreachable, it certainly is bec= ause I can manually mount ALL storage to the hypervisor.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Sent from <a href=3D"https://go.microsoft.com/fwlink= /?LinkId=3D550986"> Mail</a> for Windows 10</p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </div> </div> </div> </body> </html> --_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_--
Hi Bill, Can you please attach the engine and VDSM logs. Does the storage domain still stuck? Regards, Maor On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com> wrote:
Also cannot reinitialize the datacenter because the storage domain is locked.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 8:08 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still locked so no vm’s can be started. What is the proper way to force this to be unlocked? Each time we look to move into production after successful testing, something like this always seems to pop up at the last minute rending oVirt questionable in terms of reliability for some unknown issue.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:54 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
So apparently something didn’t change the metadata to master before connection was lost. I changed the metadata role to master and it came backup. Seems emailing in helped because every time I can’t figure something out, email in a find it shortly after.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:43 PM *To: *users <users@ovirt.org> *Subject: *master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hypervisors. It seems like after a reboot oVirt is now having some issue and the storage domain is stuck in locked state. Because of this, can’t activate any other storage either, so the other domains are in maintenance and the master sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d- 00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName = U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='70e2b8e4-0752-47a8-884c-837a00013e79', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mount ALL storage to the hypervisor.
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs. Does the storage domain still stuck?
Also which oVirt version are you using?
Regards, Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com> wrote:
Also cannot reinitialize the datacenter because the storage domain is locked.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 8:08 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still locked so no vm’s can be started. What is the proper way to force this to be unlocked? Each time we look to move into production after successful testing, something like this always seems to pop up at the last minute rending oVirt questionable in terms of reliability for some unknown issue.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:54 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
So apparently something didn’t change the metadata to master before connection was lost. I changed the metadata role to master and it came backup. Seems emailing in helped because every time I can’t figure something out, email in a find it shortly after.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:43 PM *To: *users <users@ovirt.org> *Subject: *master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hypervisors. It seems like after a reboot oVirt is now having some issue and the storage domain is stuck in locked state. Because of this, can’t activate any other storage either, so the other domains are in maintenance and the master sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d-0 0ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName = U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='70e2b8e4-0752-47a8-884c-837a00013e79', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mount ALL storage to the hypervisor.
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hello, It was 4.0.5 however, we=92ve decided to pull the plug on oVirt for now as = it=92s too risky in taking down possibly a large number or servers due to t= his issue. I think oVirt should be a little less =93picky=94 if you will, o= n storage connections. For example, this specific issue prevented anything = storage related from being done. Because the =93master=94 was locked you ca= nnot: Add other storage Activate hosts Start VM=92s Reinitialize the datacenter Remove storage These points above a huge =96 while oVirt is indeed open source, upstream o= f RHEV and doesn=92t cost anything, I feel that in scenarios like this it c= ould be the downfall of oVirt itself being too risky. The logging with oVirt seems to be crazy though =96 we=92ve been testing it= now for about 2.5 years, maybe 3 years? Once oVirt gets in a state where i= t cannot connect to something, it just goes haywire =96 many likely don=92t= see this however, every time these things happened it when we=92re testing= failover scenarios to see how oVirt responds. A few recommendations I would make are: Drop the whole =93master=94 storage thing =96 it complicates setting storag= e up. Either connect, or don=92t connect. If there=92s connectivity issues,= oVirt gets hung up on switching to this =93master=94 storage. If you have = a single storage domain, you=92ll likely have problems as we=92ve experienc= ed because once oVirt cannot find the =93master=94 it begins to go berserk,= then spirals out of control there. It might not on small setups with a few= hypervisors, but on an install with a few hundred VM=92s, large number of = hypervisors etc, it seems to get ugly real quick. Stop trying to reconnect things, I think that=92s what I=92m looking for. W= hen something fails, oVirt just goes in a loop over and over which eventual= ly causes dashboard issues, crazy amounts of logs etc. It would be better i= f oVirt would just stop, make a log entry and then quit, maybe after a few = times. In my case, I could mount the storage manually to ALL hosts, I could even f= orce start the VM=92s with virsh. The oVirt dashboard just kept saying it w= as locked, and wouldn=92t let you do anything at all with the entire datace= nter. At this time, we=92ve pushed these servers back into production using our c= urrent hypervisor software which is stable but does not have the benefits o= f oVirt. It=92ll be revisited later on and is still in use for non-producti= on things. From: Maor Lipchuk<mailto:mlipchuk@redhat.com> Sent: Sunday, January 22, 2017 7:33 AM To: Bill Bill<mailto:jax2568@outlook.com> Cc: users<mailto:users@ovirt.org> Subject: Re: [ovirt-users] master storage domain stuck in locked state On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com<mailto:m= lipchuk@redhat.com>> wrote: Hi Bill, Can you please attach the engine and VDSM logs. Does the storage domain still stuck? Also which oVirt version are you using? Regards, Maor On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com<mailto:jax2= 568@outlook.com>> wrote: Also cannot reinitialize the datacenter because the storage domain is locke= d. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 8:08 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state Spoke too soon. Some hosts came back up but the storage domain is still loc= ked so no vm=92s can be started. What is the proper way to force this to be= unlocked? Each time we look to move into production after successful testi= ng, something like this always seems to pop up at the last minute rending o= Virt questionable in terms of reliability for some unknown issue. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:54 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state So apparently something didn=92t change the metadata to master before conne= ction was lost. I changed the metadata role to master and it came backup. S= eems emailing in helped because every time I can=92t figure something out, = email in a find it shortly after. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:43 PM To: users<mailto:users@ovirt.org> Subject: master storage domain stuck in locked state No clue how to get this out. I can mount all storage manually on the hyperv= isors. It seems like after a reboot oVirt is now having some issue and the = storage domain is stuck in locked state. Because of this, can=92t activate = any other storage either, so the other domains are in maintenance and the m= aster sits in locked state, has been for hours. This sticks out on a hypervisor: StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e= -837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a' Not sure, nothing changed other than a reboot of the storage. Engine log shows: [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche= duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075= 2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason= =3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false', maintenanc= eReason=3D'null'}), log id: 6db9820a No idea why it says unreachable, it certainly is because I can manually mou= nt ALL storage to the hypervisor. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window= s 10 _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hello,</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">It was 4.0.5 however, we=92ve decided to pull the pl= ug on oVirt for now as it=92s too risky in taking down possibly a large num= ber or servers due to this issue. I think oVirt should be a little less =93= picky=94 if you will, on storage connections. For example, this specific issue prevented anything storage related from b= eing done. Because the =93master=94 was locked you cannot:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Add other storage</p> <p class=3D"MsoNormal">Activate hosts</p> <p class=3D"MsoNormal">Start VM=92s</p> <p class=3D"MsoNormal">Reinitialize the datacenter</p> <p class=3D"MsoNormal">Remove storage</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">These points above a huge =96 while oVirt is indeed = open source, upstream of RHEV and doesn=92t cost anything, I feel that in s= cenarios like this it could be the downfall of oVirt itself being too risky= .</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">The logging with oVirt seems to be crazy though =96 = we=92ve been testing it now for about 2.5 years, maybe 3 years? Once oVirt = gets in a state where it cannot connect to something, it just goes haywire = =96 many likely don=92t see this however, every time these things happened it when we=92re testing failover scenario= s to see how oVirt responds.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">A few recommendations I would make are:</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Drop the whole =93master=94 storage thing =96 it com= plicates setting storage up. Either connect, or don=92t connect. If there= =92s connectivity issues, oVirt gets hung up on switching to this =93master= =94 storage. If you have a single storage domain, you=92ll likely have problems as we=92ve experienced because once oVirt ca= nnot find the =93master=94 it begins to go berserk, then spirals out of con= trol there. It might not on small setups with a few hypervisors, but on an = install with a few hundred VM=92s, large number of hypervisors etc, it seems to get ugly real quick.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Stop trying to reconnect things, I think that=92s wh= at I=92m looking for. When something fails, oVirt just goes in a loop over = and over which eventually causes dashboard issues, crazy amounts of logs et= c. It would be better if oVirt would just stop, make a log entry and then quit, maybe after a few times.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">In my case, I could mount the storage manually to AL= L hosts, I could even force start the VM=92s with virsh. The oVirt dashboar= d just kept saying it was locked, and wouldn=92t let you do anything at all= with the entire datacenter.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">At this time, we=92ve pushed these servers back into= production using our current hypervisor software which is stable but does = not have the benefits of oVirt. It=92ll be revisited later on and is still = in use for non-production things.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:mlipchuk@redhat.com">Maor Lipchuk</a><br> <b>Sent: </b>Sunday, January 22, 2017 7:33 AM<br> <b>To: </b><a href=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Cc: </b><a href=3D"mailto:users@ovirt.org">users</a><br> <b>Subject: </b>Re: [ovirt-users] master storage domain stuck in locked sta= te</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <div dir=3D"ltr"><br> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <s= pan dir=3D"ltr"> <<a href=3D"mailto:mlipchuk@redhat.com" target=3D"_blank">mlipchuk@redha= t.com</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div dir=3D"ltr">Hi Bill, <div><br> </div> <div>Can you please attach the engine and VDSM logs.</div> <div>Does the storage domain still stuck?</div> </div> </blockquote> <div><br> </div> <div>Also which oVirt version are you using?</div> <div> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div><br> </div> <div>Regards,</div> <div>Maor</div> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote"> <div> <div class=3D"h5">On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <span dir=3D"l= tr"><<a href=3D"mailto:jax2568@outlook.com" target=3D"_blank">jax2568@ou= tlook.com</a>></span> wrote:<br> </div> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div> <div class=3D"h5"> <div lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72"> <div class=3D"m_723276570252969701m_-2288590467547947270WordSection1"> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Also cannot reinitialize the datacenter because the = storage domain is locked.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 8:08 PM<span><br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>RE: master storage domain stuck in locked state</span></p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <span> <div> <div class=3D"m_723276570252969701m_-2288590467547947270WordSection1"> <p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the stor= age domain is still locked so no vm=92s can be started. What is the proper = way to force this to be unlocked? Each time we look to move into production= after successful testing, something like this always seems to pop up at the last minute rending oVirt question= able in terms of reliability for some unknown issue.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:54 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>RE: master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <div> <div class=3D"m_723276570252969701m_-2288590467547947270WordSection1"> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">So apparently something didn=92t change the metadata= to master before connection was lost. I changed the metadata role to maste= r and it came backup. Seems emailing in helped because every time I can=92t= figure something out, email in a find it shortly after.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:43 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <div> <div class=3D"m_723276570252969701m_-2288590467547947270WordSection1"> <p class=3D"MsoNormal">No clue how to get this out. I can mount all storage= manually on the hypervisors. It seems like after a reboot oVirt is now hav= ing some issue and the storage domain is stuck in locked state. Because of = this, can=92t activate any other storage either, so the other domains are in maintenance and the master sits in loc= ked state, has been for hours.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">This sticks out on a hypervisor:</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its v= ersion: u'SD=3Dd8a0172e-837f-4552-92c7-5<wbr>66dc4e548e4, pool=3D3fd2ad92-e= 1eb-49c2-906d-0<wbr>0ec233f610a'</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of the= storage.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Engine log shows:</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">[org.ovirt.engine.core.vdsbrok<wbr>er.SetVdsStatusVD= SCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSComman= d(HostNam<wbr>e =3D U31U32NodeA, SetVdsStatusVDSCommandParamete<wbr>rs:{run= Async=3D'true', hostId=3D'70e2b8e4-0752-47a8-884<wbr>c-837a00013e79', status=3D'NonOperational', nonOperationalReason=3D'STORAGE_<wbr>DOMAIN_UNR= EACHABLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), lo= g id: 6db9820a</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">No idea why it says unreachable, it certainly is bec= ause I can manually mount ALL storage to the hypervisor.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Sent from <a href=3D"https://go.microsoft.com/fwlink= /?LinkId=3D550986" target=3D"_blank"> Mail</a> for Windows 10</p> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> </div> </div> </div> </span></div> <br> </div> </div> ______________________________<wbr>_________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br=
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><= br> <br> </blockquote> </div> <br> </div> </blockquote> </div> <br> </div> </div> </div> </body> </html> --_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_--
On Jan 22, 2017 10:13 PM, "Bill Bill" <jax2568@outlook.com> wrote: Hello, It was 4.0.5 however, we’ve decided to pull the plug on oVirt for now as it’s too risky in taking down possibly a large number or servers due to this issue. I think oVirt should be a little less “picky” if you will, on storage connections. For example, this specific issue prevented anything storage related from being done. Because the “master” was locked you cannot: Add other storage Activate hosts Start VM’s Reinitialize the datacenter Remove storage These points above a huge – while oVirt is indeed open source, upstream of RHEV and doesn’t cost anything, I feel that in scenarios like this it could be the downfall of oVirt itself being too risky. The logging with oVirt seems to be crazy though – we’ve been testing it now for about 2.5 years, maybe 3 years? Once oVirt gets in a state where it cannot connect to something, it just goes haywire – many likely don’t see this however, every time these things happened it when we’re testing failover scenarios to see how oVirt responds. A few recommendations I would make are: Thank you for your recommendations. I agree with some, wholly disagree with others. I'd still appreciate if you could send us the requested logs. TIA, Y. Drop the whole “master” storage thing – it complicates setting storage up. Either connect, or don’t connect. If there’s connectivity issues, oVirt gets hung up on switching to this “master” storage. If you have a single storage domain, you’ll likely have problems as we’ve experienced because once oVirt cannot find the “master” it begins to go berserk, then spirals out of control there. It might not on small setups with a few hypervisors, but on an install with a few hundred VM’s, large number of hypervisors etc, it seems to get ugly real quick. Stop trying to reconnect things, I think that’s what I’m looking for. When something fails, oVirt just goes in a loop over and over which eventually causes dashboard issues, crazy amounts of logs etc. It would be better if oVirt would just stop, make a log entry and then quit, maybe after a few times. In my case, I could mount the storage manually to ALL hosts, I could even force start the VM’s with virsh. The oVirt dashboard just kept saying it was locked, and wouldn’t let you do anything at all with the entire datacenter. At this time, we’ve pushed these servers back into production using our current hypervisor software which is stable but does not have the benefits of oVirt. It’ll be revisited later on and is still in use for non-production things. *From: *Maor Lipchuk <mlipchuk@redhat.com> *Sent: *Sunday, January 22, 2017 7:33 AM *To: *Bill Bill <jax2568@outlook.com> *Cc: *users <users@ovirt.org> *Subject: *Re: [ovirt-users] master storage domain stuck in locked state On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs. Does the storage domain still stuck?
Also which oVirt version are you using?
Regards, Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com> wrote:
Also cannot reinitialize the datacenter because the storage domain is locked.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 8:08 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still locked so no vm’s can be started. What is the proper way to force this to be unlocked? Each time we look to move into production after successful testing, something like this always seems to pop up at the last minute rending oVirt questionable in terms of reliability for some unknown issue.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:54 PM *To: *users <users@ovirt.org> *Subject: *RE: master storage domain stuck in locked state
So apparently something didn’t change the metadata to master before connection was lost. I changed the metadata role to master and it came backup. Seems emailing in helped because every time I can’t figure something out, email in a find it shortly after.
*From: *Bill Bill <jax2568@outlook.com> *Sent: *Friday, January 20, 2017 7:43 PM *To: *users <users@ovirt.org> *Subject: *master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hypervisors. It seems like after a reboot oVirt is now having some issue and the storage domain is stuck in locked state. Because of this, can’t activate any other storage either, so the other domains are in maintenance and the master sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d-0 0ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName = U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='70e2b8e4-0752-47a8-884c-837a00013e79', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mount ALL storage to the hypervisor.
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi, Thanks for the reply back. Unfortunately, I don=92t have the logs anymore a= s we=92ve already pulled those servers out of the rack to be used for other= services. From: Yaniv Kaul<mailto:ykaul@redhat.com> Sent: Sunday, January 22, 2017 4:45 PM To: Bill Bill<mailto:jax2568@outlook.com> Cc: Ovirt Users<mailto:users@ovirt.org>; Maor Lipchuk<mailto:mlipchuk@redha= t.com> Subject: Re: [ovirt-users] master storage domain stuck in locked state On Jan 22, 2017 10:13 PM, "Bill Bill" <jax2568@outlook.com<mailto:jax2568@o= utlook.com>> wrote: Hello, It was 4.0.5 however, we=92ve decided to pull the plug on oVirt for now as = it=92s too risky in taking down possibly a large number or servers due to t= his issue. I think oVirt should be a little less =93picky=94 if you will, o= n storage connections. For example, this specific issue prevented anything = storage related from being done. Because the =93master=94 was locked you ca= nnot: Add other storage Activate hosts Start VM=92s Reinitialize the datacenter Remove storage These points above a huge =96 while oVirt is indeed open source, upstream o= f RHEV and doesn=92t cost anything, I feel that in scenarios like this it c= ould be the downfall of oVirt itself being too risky. The logging with oVirt seems to be crazy though =96 we=92ve been testing it= now for about 2.5 years, maybe 3 years? Once oVirt gets in a state where i= t cannot connect to something, it just goes haywire =96 many likely don=92t= see this however, every time these things happened it when we=92re testing= failover scenarios to see how oVirt responds. A few recommendations I would make are: Thank you for your recommendations. I agree with some, wholly disagree with= others. I'd still appreciate if you could send us the requested logs. TIA, Y. Drop the whole =93master=94 storage thing =96 it complicates setting storag= e up. Either connect, or don=92t connect. If there=92s connectivity issues,= oVirt gets hung up on switching to this =93master=94 storage. If you have = a single storage domain, you=92ll likely have problems as we=92ve experienc= ed because once oVirt cannot find the =93master=94 it begins to go berserk,= then spirals out of control there. It might not on small setups with a few= hypervisors, but on an install with a few hundred VM=92s, large number of = hypervisors etc, it seems to get ugly real quick. Stop trying to reconnect things, I think that=92s what I=92m looking for. W= hen something fails, oVirt just goes in a loop over and over which eventual= ly causes dashboard issues, crazy amounts of logs etc. It would be better i= f oVirt would just stop, make a log entry and then quit, maybe after a few = times. In my case, I could mount the storage manually to ALL hosts, I could even f= orce start the VM=92s with virsh. The oVirt dashboard just kept saying it w= as locked, and wouldn=92t let you do anything at all with the entire datace= nter. At this time, we=92ve pushed these servers back into production using our c= urrent hypervisor software which is stable but does not have the benefits o= f oVirt. It=92ll be revisited later on and is still in use for non-producti= on things. From: Maor Lipchuk<mailto:mlipchuk@redhat.com> Sent: Sunday, January 22, 2017 7:33 AM To: Bill Bill<mailto:jax2568@outlook.com> Cc: users<mailto:users@ovirt.org> Subject: Re: [ovirt-users] master storage domain stuck in locked state On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com<mailto:m= lipchuk@redhat.com>> wrote: Hi Bill, Can you please attach the engine and VDSM logs. Does the storage domain still stuck? Also which oVirt version are you using? Regards, Maor On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com<mailto:jax2= 568@outlook.com>> wrote: Also cannot reinitialize the datacenter because the storage domain is locke= d. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 8:08 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state Spoke too soon. Some hosts came back up but the storage domain is still loc= ked so no vm=92s can be started. What is the proper way to force this to be= unlocked? Each time we look to move into production after successful testi= ng, something like this always seems to pop up at the last minute rending o= Virt questionable in terms of reliability for some unknown issue. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:54 PM To: users<mailto:users@ovirt.org> Subject: RE: master storage domain stuck in locked state So apparently something didn=92t change the metadata to master before conne= ction was lost. I changed the metadata role to master and it came backup. S= eems emailing in helped because every time I can=92t figure something out, = email in a find it shortly after. From: Bill Bill<mailto:jax2568@outlook.com> Sent: Friday, January 20, 2017 7:43 PM To: users<mailto:users@ovirt.org> Subject: master storage domain stuck in locked state No clue how to get this out. I can mount all storage manually on the hyperv= isors. It seems like after a reboot oVirt is now having some issue and the = storage domain is stuck in locked state. Because of this, can=92t activate = any other storage either, so the other domains are in maintenance and the m= aster sits in locked state, has been for hours. This sticks out on a hypervisor: StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e= -837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a' Not sure, nothing changed other than a reboot of the storage. Engine log shows: [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche= duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, = SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075= 2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason= =3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false', maintenanc= eReason=3D'null'}), log id: 6db9820a No idea why it says unreachable, it certainly is because I can manually mou= nt ALL storage to the hypervisor. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window= s 10 _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hi,</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Thanks for the reply back. Unfortunately, I don=92t = have the logs anymore as we=92ve already pulled those servers out of the ra= ck to be used for other services.</p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E= 1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:ykaul@redhat.com">Yaniv Kaul</a><br> <b>Sent: </b>Sunday, January 22, 2017 4:45 PM<br> <b>To: </b><a href=3D"mailto:jax2568@outlook.com">Bill Bill</a><br> <b>Cc: </b><a href=3D"mailto:users@ovirt.org">Ovirt Users</a>; <a href=3D"m= ailto:mlipchuk@redhat.com"> Maor Lipchuk</a><br> <b>Subject: </b>Re: [ovirt-users] master storage domain stuck in locked sta= te</p> </div> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> <div> <div dir=3D"auto"> <div><br> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Jan 22, 2017 10:13 PM, "Bill Bill" = <<a href=3D"mailto:jax2568@outlook.com">jax2568@outlook.com</a>> wrot= e:<br type=3D"attribution"> <blockquote class=3D"quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc= solid;padding-left:1ex"> <div> <div class=3D"m_-823963203978568812WordSection1"> <p class=3D"MsoNormal">Hello,</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">It was 4.0.5 however, we=92ve decided to pull the pl= ug on oVirt for now as it=92s too risky in taking down possibly a large num= ber or servers due to this issue. I think oVirt should be a little less =93= picky=94 if you will, on storage connections. For example, this specific issue prevented anything storage related from b= eing done. Because the =93master=94 was locked you cannot:</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Add other storage</p> <p class=3D"MsoNormal">Activate hosts</p> <p class=3D"MsoNormal">Start VM=92s</p> <p class=3D"MsoNormal">Reinitialize the datacenter</p> <p class=3D"MsoNormal">Remove storage</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">These points above a huge =96 while oVirt is indeed = open source, upstream of RHEV and doesn=92t cost anything, I feel that in s= cenarios like this it could be the downfall of oVirt itself being too risky= .</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">The logging with oVirt seems to be crazy though =96 = we=92ve been testing it now for about 2.5 years, maybe 3 years? Once oVirt = gets in a state where it cannot connect to something, it just goes haywire = =96 many likely don=92t see this however, every time these things happened it when we=92re testing failover scenario= s to see how oVirt responds.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">A few recommendations I would make are:</p> </div> </div> </blockquote> </div> </div> </div> <div dir=3D"auto"><br> </div> <div dir=3D"auto">Thank you for your recommendations. I agree with some, wh= olly disagree with others. </div> <div dir=3D"auto">I'd still appreciate if you could send us the requested l= ogs. </div> <div dir=3D"auto"><br> </div> <div dir=3D"auto">TIA, </div> <div dir=3D"auto">Y. </div> <div dir=3D"auto"><br> </div> <div dir=3D"auto"> <div class=3D"gmail_extra"> <div class=3D"gmail_quote"> <blockquote class=3D"quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc= solid;padding-left:1ex"> <div> <div class=3D"m_-823963203978568812WordSection1"> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Drop the whole =93master=94 storage thing =96 it com= plicates setting storage up. Either connect, or don=92t connect. If there= =92s connectivity issues, oVirt gets hung up on switching to this =93master= =94 storage. If you have a single storage domain, you=92ll likely have problems as we=92ve experienced because once oVirt ca= nnot find the =93master=94 it begins to go berserk, then spirals out of con= trol there. It might not on small setups with a few hypervisors, but on an = install with a few hundred VM=92s, large number of hypervisors etc, it seems to get ugly real quick.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Stop trying to reconnect things, I think that=92s wh= at I=92m looking for. When something fails, oVirt just goes in a loop over = and over which eventually causes dashboard issues, crazy amounts of logs et= c. It would be better if oVirt would just stop, make a log entry and then quit, maybe after a few times.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">In my case, I could mount the storage manually to AL= L hosts, I could even force start the VM=92s with virsh. The oVirt dashboar= d just kept saying it was locked, and wouldn=92t let you do anything at all= with the entire datacenter.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">At this time, we=92ve pushed these servers back into= production using our current hypervisor software which is stable but does = not have the benefits of oVirt. It=92ll be revisited later on and is still = in use for non-production things.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:mlipchuk@redhat.com" target=3D"_blank">Maor Lipchuk</a><br> <b>Sent: </b>Sunday, January 22, 2017 7:33 AM<br> <b>To: </b><a href=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bi= ll</a><br> <b>Cc: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>Re: [ovirt-users] master storage domain stuck in locked sta= te</p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <div class=3D"elided-text"> <div> <div dir=3D"ltr"><br> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <s= pan dir=3D"ltr"> <<a href=3D"mailto:mlipchuk@redhat.com" target=3D"_blank">mlipchuk@redha= t.com</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div dir=3D"ltr">Hi Bill, <div><br> </div> <div>Can you please attach the engine and VDSM logs.</div> <div>Does the storage domain still stuck?</div> </div> </blockquote> <div><br> </div> <div>Also which oVirt version are you using?</div> <div> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div><br> </div> <div>Regards,</div> <div>Maor</div> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote"> <div> <div class=3D"m_-823963203978568812h5">On Sat, Jan 21, 2017 at 3:11 AM, Bil= l Bill <span dir=3D"ltr"> <<a href=3D"mailto:jax2568@outlook.com" target=3D"_blank">jax2568@outloo= k.com</a>></span> wrote:<br> </div> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div> <div class=3D"m_-823963203978568812h5"> <div lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72"> <div class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472= 70WordSection1"> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Also cannot reinitialize the datacenter because the = storage domain is locked.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 8:08 PM<span><br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>RE: master storage domain stuck in locked state</span></p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <span> <div> <div class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472= 70WordSection1"> <p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the stor= age domain is still locked so no vm=92s can be started. What is the proper = way to force this to be unlocked? Each time we look to move into production= after successful testing, something like this always seems to pop up at the last minute rending oVirt question= able in terms of reliability for some unknown issue.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:54 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>RE: master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <div> <div class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472= 70WordSection1"> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">So apparently something didn=92t change the metadata= to master before connection was lost. I changed the metadata role to maste= r and it came backup. Seems emailing in helped because every time I can=92t= figure something out, email in a find it shortly after.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal"><u></u> <u></u></p> <div style=3D"border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in = 0in 0in"> <p class=3D"MsoNormal" style=3D"border:none;padding:0in"><b>From: </b><a hr= ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bill</a><br> <b>Sent: </b>Friday, January 20, 2017 7:43 PM<br> <b>To: </b><a href=3D"mailto:users@ovirt.org" target=3D"_blank">users</a><b= r> <b>Subject: </b>master storage domain stuck in locked state</p> </div> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> <div> <div class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472= 70WordSection1"> <p class=3D"MsoNormal">No clue how to get this out. I can mount all storage= manually on the hypervisors. It seems like after a reboot oVirt is now hav= ing some issue and the storage domain is stuck in locked state. Because of = this, can=92t activate any other storage either, so the other domains are in maintenance and the master sits in loc= ked state, has been for hours.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">This sticks out on a hypervisor:</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its v= ersion: u'SD=3Dd8a0172e-837f-4552-92c7-5<wbr>66dc4e548e4, pool=3D3fd2ad92-e= 1eb-49c2-906d-0<wbr>0ec233f610a'</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of the= storage.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Engine log shows:</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">[org.ovirt.engine.core.vdsbrok<wbr>er.SetVdsStatusVD= SCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSComman= d(HostNam<wbr>e =3D U31U32NodeA, SetVdsStatusVDSCommandParamete<wbr>rs:{run= Async=3D'true', hostId=3D'70e2b8e4-0752-47a8-884<wbr>c-837a00013e79', status=3D'NonOperational', nonOperationalReason=3D'STORAGE_<wbr>DOMAIN_UNR= EACHABLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}), lo= g id: 6db9820a</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">No idea why it says unreachable, it certainly is bec= ause I can manually mount ALL storage to the hypervisor.</p> <p class=3D"MsoNormal"><u></u> <u></u></p> <p class=3D"MsoNormal">Sent from <a href=3D"https://go.microsoft.com/fwlink= /?LinkId=3D550986" target=3D"_blank"> Mail</a> for Windows 10</p> <p class=3D"MsoNormal"><u></u> <u></u></p> </div> </div> </div> </div> </span></div> <br> </div> </div> ______________________________<wbr>_________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br=
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><= br> <br> </blockquote> </div> <br> </div> </blockquote> </div> <br> </div> </div> </div> </div> </div> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><= br> <br> </blockquote> </div> <br> </div> </div> </div> </div> </body> </html> --_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_--
participants (3)
-
Bill Bill -
Maor Lipchuk -
Yaniv Kaul