3:43 a.m.
--_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Tm8gY2x1ZSBob3cgdG8gZ2V0IHRoaXMgb3V0LiBJIGNhbiBtb3VudCBhbGwgc3RvcmFnZSBtYW51
YWxseSBvbiB0aGUgaHlwZXJ2aXNvcnMuIEl0IHNlZW1zIGxpa2UgYWZ0ZXIgYSByZWJvb3Qgb1Zp
cnQgaXMgbm93IGhhdmluZyBzb21lIGlzc3VlIGFuZCB0aGUgc3RvcmFnZSBkb21haW4gaXMgc3R1
Y2sgaW4gbG9ja2VkIHN0YXRlLiBCZWNhdXNlIG9mIHRoaXMsIGNhbuKAmXQgYWN0aXZhdGUgYW55
IG90aGVyIHN0b3JhZ2UgZWl0aGVyLCBzbyB0aGUgb3RoZXIgZG9tYWlucyBhcmUgaW4gbWFpbnRl
bmFuY2UgYW5kIHRoZSBtYXN0ZXIgc2l0cyBpbiBsb2NrZWQgc3RhdGUsIGhhcyBiZWVuIGZvciBo
b3Vycy4NCg0KVGhpcyBzdGlja3Mgb3V0IG9uIGEgaHlwZXJ2aXNvcjoNCg0KU3RvcmFnZVBvb2xX
cm9uZ01hc3RlcjogV3JvbmcgTWFzdGVyIGRvbWFpbiBvciBpdHMgdmVyc2lvbjogdSdTRD1kOGEw
MTcyZS04MzdmLTQ1NTItOTJjNy01NjZkYzRlNTQ4ZTQsIHBvb2w9M2ZkMmFkOTItZTFlYi00OWMy
LTkwNmQtMDBlYzIzM2Y2MTBhJw0KDQpOb3Qgc3VyZSwgbm90aGluZyBjaGFuZ2VkIG90aGVyIHRo
YW4gYSByZWJvb3Qgb2YgdGhlIHN0b3JhZ2UuDQoNCkVuZ2luZSBsb2cgc2hvd3M6DQoNCltvcmcu
b3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLlNldFZkc1N0YXR1c1ZEU0NvbW1hbmRdIChEZWZh
dWx0UXVhcnR6U2NoZWR1bGVyOCkgWzU2OTY3MzJiXSBTVEFSVCwgU2V0VmRzU3RhdHVzVkRTQ29t
bWFuZChIb3N0TmFtZSA9IFUzMVUzMk5vZGVBLCBTZXRWZHNTdGF0dXNWRFNDb21tYW5kUGFyYW1l
dGVyczp7cnVuQXN5bmM9J3RydWUnLCBob3N0SWQ9JzcwZTJiOGU0LTA3NTItNDdhOC04ODRjLTgz
N2EwMDAxM2U3OScsIHN0YXR1cz0nTm9uT3BlcmF0aW9uYWwnLCBub25PcGVyYXRpb25hbFJlYXNv
bj0nU1RPUkFHRV9ET01BSU5fVU5SRUFDSEFCTEUnLCBzdG9wU3BtRmFpbHVyZUxvZ2dlZD0nZmFs
c2UnLCBtYWludGVuYW5jZVJlYXNvbj0nbnVsbCd9KSwgbG9nIGlkOiA2ZGI5ODIwYQ0KDQpObyBp
ZGVhIHdoeSBpdCBzYXlzIHVucmVhY2hhYmxlLCBpdCBjZXJ0YWlubHkgaXMgYmVjYXVzZSBJIGNh
biBtYW51YWxseSBtb3VudCBBTEwgc3RvcmFnZSB0byB0aGUgaHlwZXJ2aXNvci4NCg0KU2VudCBm
cm9tIE1haWw8aHR0cHM6Ly9nby5taWNyb3NvZnQuY29tL2Z3bGluay8/TGlua0lkPTU1MDk4Nj4g
Zm9yIFdpbmRvd3MgMTANCg0K
--_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_
Content-Type: text/html; charset="utf-8"
Content-ID:
<77EDC59EF3DBB0418BCEF6D897F65EE0(a)sct-15-1-789-14-msonline-outlook-8b04f.templateTenant>
Content-Transfer-Encoding: base64
PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4
bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo
dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo
dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp
dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l
dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg
bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj
ZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2
IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToy
IDE1IDUgMiAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3Jt
YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1i
b3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp
IixzYW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXBy
aW9yaXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQph
OnZpc2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9sbG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5
Ojk5Ow0KCWNvbG9yOiM5NTRGNzI7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQouTXNv
Q2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTt9DQpAcGFnZSBXb3JkU2Vj
dGlvbjENCgl7c2l6ZTo4LjVpbiAxMS4waW47DQoJbWFyZ2luOjEuMGluIDEuMGluIDEuMGluIDEu
MGluO30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHls
ZT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVOLVVTIiBsaW5rPSJibHVlIiB2bGluaz0iIzk1NEY3
MiI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Tm8g
Y2x1ZSBob3cgdG8gZ2V0IHRoaXMgb3V0LiBJIGNhbiBtb3VudCBhbGwgc3RvcmFnZSBtYW51YWxs
eSBvbiB0aGUgaHlwZXJ2aXNvcnMuIEl0IHNlZW1zIGxpa2UgYWZ0ZXIgYSByZWJvb3Qgb1ZpcnQg
aXMgbm93IGhhdmluZyBzb21lIGlzc3VlIGFuZCB0aGUgc3RvcmFnZSBkb21haW4gaXMgc3R1Y2sg
aW4gbG9ja2VkIHN0YXRlLiBCZWNhdXNlIG9mIHRoaXMsIGNhbuKAmXQgYWN0aXZhdGUgYW55IG90
aGVyIHN0b3JhZ2UNCiBlaXRoZXIsIHNvIHRoZSBvdGhlciBkb21haW5zIGFyZSBpbiBtYWludGVu
YW5jZSBhbmQgdGhlIG1hc3RlciBzaXRzIGluIGxvY2tlZCBzdGF0ZSwgaGFzIGJlZW4gZm9yIGhv
dXJzLjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+VGhpcyBzdGlja3Mgb3V0IG9uIGEgaHlwZXJ2aXNvcjo8L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPlN0b3JhZ2VQb29sV3JvbmdNYXN0ZXI6IFdyb25nIE1hc3RlciBkb21haW4gb3IgaXRz
IHZlcnNpb246IHUnU0Q9ZDhhMDE3MmUtODM3Zi00NTUyLTkyYzctNTY2ZGM0ZTU0OGU0LCBwb29s
PTNmZDJhZDkyLWUxZWItNDljMi05MDZkLTAwZWMyMzNmNjEwYSc8L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk5vdCBz
dXJlLCBub3RoaW5nIGNoYW5nZWQgb3RoZXIgdGhhbiBhIHJlYm9vdCBvZiB0aGUgc3RvcmFnZS48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPkVuZ2luZSBsb2cgc2hvd3M6PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5bb3JnLm92aXJ0LmVu
Z2luZS5jb3JlLnZkc2Jyb2tlci5TZXRWZHNTdGF0dXNWRFNDb21tYW5kXSAoRGVmYXVsdFF1YXJ0
elNjaGVkdWxlcjgpIFs1Njk2NzMyYl0gU1RBUlQsIFNldFZkc1N0YXR1c1ZEU0NvbW1hbmQoSG9z
dE5hbWUgPSBVMzFVMzJOb2RlQSwgU2V0VmRzU3RhdHVzVkRTQ29tbWFuZFBhcmFtZXRlcnM6e3J1
bkFzeW5jPSd0cnVlJywgaG9zdElkPSc3MGUyYjhlNC0wNzUyLTQ3YTgtODg0Yy04MzdhMDAwMTNl
NzknLA0KIHN0YXR1cz0nTm9uT3BlcmF0aW9uYWwnLCBub25PcGVyYXRpb25hbFJlYXNvbj0nU1RP
UkFHRV9ET01BSU5fVU5SRUFDSEFCTEUnLCBzdG9wU3BtRmFpbHVyZUxvZ2dlZD0nZmFsc2UnLCBt
YWludGVuYW5jZVJlYXNvbj0nbnVsbCd9KSwgbG9nIGlkOiA2ZGI5ODIwYTwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
Tm8gaWRlYSB3aHkgaXQgc2F5cyB1bnJlYWNoYWJsZSwgaXQgY2VydGFpbmx5IGlzIGJlY2F1c2Ug
SSBjYW4gbWFudWFsbHkgbW91bnQgQUxMIHN0b3JhZ2UgdG8gdGhlIGh5cGVydmlzb3IuPC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj5TZW50IGZyb20gPGEgaHJlZj0iaHR0cHM6Ly9nby5taWNyb3NvZnQuY29tL2Z3bGlu
ay8/TGlua0lkPTU1MDk4NiI+DQpNYWlsPC9hPiBmb3IgV2luZG93cyAxMDwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2JvZHk+DQo8L2h0
bWw+DQo=
--_000_CO2PR0801MB0743D9C32980648B61F582D5A6700CO2PR0801MB0743_--
3:54 a.m.
--_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
So apparently something didn=92t change the metadata to master before conne=
ction was lost. I changed the metadata role to master and it came backup. S=
eems emailing in helped because every time I can=92t figure something out, =
email in a find it shortly after.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 7:43 PM
To: users<mailto:users@ovirt.org>
Subject: master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hyperv=
isors. It seems like after a reboot oVirt is now having some issue and the =
storage domain is stuck in locked state. Because of this, can=92t activate =
any other storage either, so the other domains are in maintenance and the m=
aster sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e=
-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche=
duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075=
2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=
=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false',
maintenanc=
eReason=3D'null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mou=
nt ALL storage to the hypervisor.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10
--_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office"
xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3DWindows-1=
252">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">So apparently something didn=92t change the
metadata=
to master before connection was lost. I changed the metadata role to maste=
r and it came backup. Seems emailing in helped because every time I can=92t=
figure something out, email in a find
it shortly after.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 7:43 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div class=3D"WordSection1">
<p class=3D"MsoNormal">No clue how to get this out. I can mount all
storage=
manually on the hypervisors. It seems like after a reboot oVirt is now hav=
ing some issue and the storage domain is stuck in locked state. Because of =
this, can=92t activate any other storage
either, so the other domains are in maintenance and the master sits in loc=
ked state, has been for hours.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">This sticks out on a hypervisor:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its
v=
ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4=
9c2-906d-00ec233f610a'</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of
the=
storage.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Engine log shows:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p
class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm=
and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos=
tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',=
hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79',
status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA=
BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}),
log id:=
6db9820a</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">No idea why it says unreachable, it certainly is
bec=
ause I can manually mount ALL storage to the hypervisor.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Sent from <a
href=3D"https://go.microsoft.com/fwlink=
/?LinkId=3D550986">
Mail</a> for Windows 10</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
--_000_CO2PR0801MB07432067AF7CE4B288AB0420A6700CO2PR0801MB0743_--
4:08 a.m.
--_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Spoke too soon. Some hosts came back up but the storage domain is still loc=
ked so no vm=92s can be started. What is the proper way to force this to be=
unlocked? Each time we look to move into production after successful testi=
ng, something like this always seems to pop up at the last minute rending o=
Virt questionable in terms of reliability for some unknown issue.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 7:54 PM
To: users<mailto:users@ovirt.org>
Subject: RE: master storage domain stuck in locked state
So apparently something didn=92t change the metadata to master before conne=
ction was lost. I changed the metadata role to master and it came backup. S=
eems emailing in helped because every time I can=92t figure something out, =
email in a find it shortly after.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 7:43 PM
To: users<mailto:users@ovirt.org>
Subject: master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hyperv=
isors. It seems like after a reboot oVirt is now having some issue and the =
storage domain is stuck in locked state. Because of this, can=92t activate =
any other storage either, so the other domains are in maintenance and the m=
aster sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e=
-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche=
duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075=
2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=
=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false',
maintenanc=
eReason=3D'null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mou=
nt ALL storage to the hypervisor.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10
--_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office"
xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3DWindows-1=
252">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the
stor=
age domain is still locked so no vm=92s can be started. What is the proper =
way to force this to be unlocked? Each time we look to move into production=
after successful testing, something
like this always seems to pop up at the last minute rending oVirt question=
able in terms of reliability for some unknown issue.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 7:54 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>RE: master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">So apparently something didn=92t change the
metadata=
to master before connection was lost. I changed the metadata role to maste=
r and it came backup. Seems emailing in helped because every time I can=92t=
figure something out, email in a find
it shortly after.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 7:43 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div class=3D"WordSection1">
<p class=3D"MsoNormal">No clue how to get this out. I can mount all
storage=
manually on the hypervisors. It seems like after a reboot oVirt is now hav=
ing some issue and the storage domain is stuck in locked state. Because of =
this, can=92t activate any other storage
either, so the other domains are in maintenance and the master sits in loc=
ked state, has been for hours.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">This sticks out on a hypervisor:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its
v=
ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4=
9c2-906d-00ec233f610a'</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of
the=
storage.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Engine log shows:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p
class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm=
and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos=
tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',=
hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79',
status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA=
BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}),
log id:=
6db9820a</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">No idea why it says unreachable, it certainly is
bec=
ause I can manually mount ALL storage to the hypervisor.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Sent from <a
href=3D"https://go.microsoft.com/fwlink=
/?LinkId=3D550986">
Mail</a> for Windows 10</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>
--_000_CO2PR0801MB0743D5BED04E03821D47364FA6700CO2PR0801MB0743_--
4:11 a.m.
--_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Also cannot reinitialize the datacenter because the storage domain is locke=
d.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 8:08 PM
To: users<mailto:users@ovirt.org>
Subject: RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still loc=
ked so no vm=92s can be started. What is the proper way to force this to be=
unlocked? Each time we look to move into production after successful testi=
ng, something like this always seems to pop up at the last minute rending o=
Virt questionable in terms of reliability for some unknown issue.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 7:54 PM
To: users<mailto:users@ovirt.org>
Subject: RE: master storage domain stuck in locked state
So apparently something didn=92t change the metadata to master before conne=
ction was lost. I changed the metadata role to master and it came backup. S=
eems emailing in helped because every time I can=92t figure something out, =
email in a find it shortly after.
From: Bill Bill<mailto:jax2568@outlook.com>
Sent: Friday, January 20, 2017 7:43 PM
To: users<mailto:users@ovirt.org>
Subject: master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hyperv=
isors. It seems like after a reboot oVirt is now having some issue and the =
storage domain is stuck in locked state. Because of this, can=92t activate =
any other storage either, so the other domains are in maintenance and the m=
aster sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e=
-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche=
duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075=
2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=
=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false',
maintenanc=
eReason=3D'null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mou=
nt ALL storage to the hypervisor.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10
--_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office"
xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3DWindows-1=
252">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Also cannot reinitialize the datacenter because the
=
storage domain is locked.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 8:08 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>RE: master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the
stor=
age domain is still locked so no vm=92s can be started. What is the proper =
way to force this to be unlocked? Each time we look to move into production=
after successful testing, something
like this always seems to pop up at the last minute rending oVirt question=
able in terms of reliability for some unknown issue.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 7:54 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>RE: master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">So apparently something didn=92t change the
metadata=
to master before connection was lost. I changed the metadata role to maste=
r and it came backup. Seems emailing in helped because every time I can=92t=
figure something out, email in a find
it shortly after.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com">Bill Bill</a><br>
<b>Sent: </b>Friday, January 20, 2017 7:43 PM<br>
<b>To: </b><a
href=3D"mailto:users@ovirt.org">users</a><br>
<b>Subject: </b>master storage domain stuck in locked state</p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div class=3D"WordSection1">
<p class=3D"MsoNormal">No clue how to get this out. I can mount all
storage=
manually on the hypervisors. It seems like after a reboot oVirt is now hav=
ing some issue and the storage domain is stuck in locked state. Because of =
this, can=92t activate any other storage
either, so the other domains are in maintenance and the master sits in loc=
ked state, has been for hours.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">This sticks out on a hypervisor:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">StoragePoolWrongMaster: Wrong Master domain or its
v=
ersion: u'SD=3Dd8a0172e-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-4=
9c2-906d-00ec233f610a'</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Not sure, nothing changed other than a reboot of
the=
storage.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Engine log shows:</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p
class=3D"MsoNormal">[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSComm=
and] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(Hos=
tName =3D U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync=3D'true',=
hostId=3D'70e2b8e4-0752-47a8-884c-837a00013e79',
status=3D'NonOperational', nonOperationalReason=3D'STORAGE_DOMAIN_UNREACHA=
BLE', stopSpmFailureLogged=3D'false', maintenanceReason=3D'null'}),
log id:=
6db9820a</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">No idea why it says unreachable, it certainly is
bec=
ause I can manually mount ALL storage to the hypervisor.</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Sent from <a
href=3D"https://go.microsoft.com/fwlink=
/?LinkId=3D550986">
Mail</a> for Windows 10</p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>
--_000_CO2PR0801MB0743373790E89C7528C20BE4A6700CO2PR0801MB0743_--
3:31 p.m.
Hi Bill,
Can you please attach the engine and VDSM logs.
Does the storage domain still stuck?
Regards,
Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568(a)outlook.com> wrote:
Also cannot reinitialize the datacenter because the storage domain is
locked.
*From: *Bill Bill <jax2568(a)outlook.com>
*Sent: *Friday, January 20, 2017 8:08 PM
*To: *users <users(a)ovirt.org>
*Subject: *RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still
locked so no vm’s can be started. What is the proper way to force this to
be unlocked? Each time we look to move into production after successful
testing, something like this always seems to pop up at the last minute
rending oVirt questionable in terms of reliability for some unknown issue.
*From: *Bill Bill <jax2568(a)outlook.com>
*Sent: *Friday, January 20, 2017 7:54 PM
*To: *users <users(a)ovirt.org>
*Subject: *RE: master storage domain stuck in locked state
So apparently something didn’t change the metadata to master before
connection was lost. I changed the metadata role to master and it came
backup. Seems emailing in helped because every time I can’t figure
something out, email in a find it shortly after.
*From: *Bill Bill <jax2568(a)outlook.com>
*Sent: *Friday, January 20, 2017 7:43 PM
*To: *users <users(a)ovirt.org>
*Subject: *master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the
hypervisors. It seems like after a reboot oVirt is now having some issue
and the storage domain is stuck in locked state. Because of this, can’t
activate any other storage either, so the other domains are in maintenance
and the master sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version:
u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d-
00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName
= U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true',
hostId='70e2b8e4-0752-47a8-884c-837a00013e79', status='NonOperational',
nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
6db9820a
No idea why it says unreachable, it certainly is because I can manually
mount ALL storage to the hypervisor.
Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:33 p.m.
On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk(a)redhat.com> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs.
Does the storage domain still stuck?
Also which oVirt version are you using?
Regards,
Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568(a)outlook.com> wrote:
>
>
> Also cannot reinitialize the datacenter because the storage domain is
> locked.
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 8:08 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *RE: master storage domain stuck in locked state
>
>
>
> Spoke too soon. Some hosts came back up but the storage domain is still
> locked so no vm’s can be started. What is the proper way to force this to
> be unlocked? Each time we look to move into production after successful
> testing, something like this always seems to pop up at the last minute
> rending oVirt questionable in terms of reliability for some unknown issue.
>
>
>
>
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 7:54 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *RE: master storage domain stuck in locked state
>
>
>
>
>
> So apparently something didn’t change the metadata to master before
> connection was lost. I changed the metadata role to master and it came
> backup. Seems emailing in helped because every time I can’t figure
> something out, email in a find it shortly after.
>
>
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 7:43 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *master storage domain stuck in locked state
>
>
>
> No clue how to get this out. I can mount all storage manually on the
> hypervisors. It seems like after a reboot oVirt is now having some issue
> and the storage domain is stuck in locked state. Because of this, can’t
> activate any other storage either, so the other domains are in maintenance
> and the master sits in locked state, has been for hours.
>
>
>
> This sticks out on a hypervisor:
>
>
>
> StoragePoolWrongMaster: Wrong Master domain or its version:
> u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d-0
> 0ec233f610a'
>
>
>
> Not sure, nothing changed other than a reboot of the storage.
>
>
>
> Engine log shows:
>
>
>
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName
> = U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true',
> hostId='70e2b8e4-0752-47a8-884c-837a00013e79',
status='NonOperational',
> nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE',
> stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
6db9820a
>
>
>
> No idea why it says unreachable, it certainly is because I can manually
> mount ALL storage to the hypervisor.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
11:07 p.m.
--_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hello,
It was 4.0.5 however, we=92ve decided to pull the plug on oVirt for now as =
it=92s too risky in taking down possibly a large number or servers due to t=
his issue. I think oVirt should be a little less =93picky=94 if you will, o=
n storage connections. For example, this specific issue prevented anything =
storage related from being done. Because the =93master=94 was locked you ca=
nnot:
Add other storage
Activate hosts
Start VM=92s
Reinitialize the datacenter
Remove storage
These points above a huge =96 while oVirt is indeed open source, upstream o=
f RHEV and doesn=92t cost anything, I feel that in scenarios like this it c=
ould be the downfall of oVirt itself being too risky.
The logging with oVirt seems to be crazy though =96 we=92ve been testing it=
now for about 2.5 years, maybe 3 years? Once oVirt gets in a state where i=
t cannot connect to something, it just goes haywire =96 many likely don=92t=
see this however, every time these things happened it when we=92re testing=
failover scenarios to see how oVirt responds.
A few recommendations I would make are:
Drop the whole =93master=94 storage thing =96 it complicates setting storag=
e up. Either connect, or don=92t connect. If there=92s connectivity issues,=
oVirt gets hung up on switching to this =93master=94 storage. If you have =
a single storage domain, you=92ll likely have problems as we=92ve experienc=
ed because once oVirt cannot find the =93master=94 it begins to go berserk,=
then spirals out of control there. It might not on small setups with a few=
hypervisors, but on an install with a few hundred VM=92s, large number of =
hypervisors etc, it seems to get ugly real quick.
Stop trying to reconnect things, I think that=92s what I=92m looking for. W=
hen something fails, oVirt just goes in a loop over and over which eventual=
ly causes dashboard issues, crazy amounts of logs etc. It would be better i=
f oVirt would just stop, make a log entry and then quit, maybe after a few =
times.
In my case, I could mount the storage manually to ALL hosts, I could even f=
orce start the VM=92s with virsh. The oVirt dashboard just kept saying it w=
as locked, and wouldn=92t let you do anything at all with the entire datace=
nter.
At this time, we=92ve pushed these servers back into production using our c=
urrent hypervisor software which is stable but does not have the benefits o=
f oVirt. It=92ll be revisited later on and is still in use for non-producti=
on things.
From: Maor Lipchuk<mailto:mlipchuk@redhat.com Sent:
Sunday, January 22, 2017 7:33 AM
To: Bill Bill<mailto:jax2568@outlook.com Cc:
users<mailto:users@ovirt.org Subject: Re: [ovirt-users] master
storage domain stuck in locked state
On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com<mailto:m=
lipchuk(a)redhat.com>> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs.
Does the storage domain still stuck?
Also which oVirt version are you using?
Regards,
Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com<mailto:jax2=
568(a)outlook.com>> wrote:
Also cannot reinitialize the datacenter because the storage domain is locke=
d.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 8:08 PM
To: users<mailto:users@ovirt.org Subject:
RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still loc=
ked so no vm=92s can be started. What is the proper way to force this to be=
unlocked? Each time we look to move into production after successful testi=
ng, something like this always seems to pop up at the last minute rending o=
Virt questionable in terms of reliability for some unknown issue.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 7:54 PM
To: users<mailto:users@ovirt.org Subject:
RE: master storage domain stuck in locked state
So apparently something didn=92t change the metadata to master before conne=
ction was lost. I changed the metadata role to master and it came backup. S=
eems emailing in helped because every time I can=92t figure something out, =
email in a find it shortly after.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 7:43 PM
To: users<mailto:users@ovirt.org Subject:
master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hyperv=
isors. It seems like after a reboot oVirt is now having some issue and the =
storage domain is stuck in locked state. Because of this, can=92t activate =
any other storage either, so the other domains are in maintenance and the m=
aster sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e=
-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche=
duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075=
2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=
=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false',
maintenanc=
eReason=3D'null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mou=
nt ALL storage to the hypervisor.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" </head <body <meta
name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)" <style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style <div
class=3D"WordSection1" <p
class=3D"MsoNormal">Hello,</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">It was 4.0.5 however,
we=92ve decided to pull the pl=
ug on oVirt for now as it=92s too risky in taking down possibly a large num=
ber or servers due to this issue. I think oVirt should be a little less =93=
picky=94 if you will, on storage connections.
For example, this specific issue prevented anything storage related from b=
eing done. Because the =93master=94 was locked you cannot:</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Add other
storage</p <p
class=3D"MsoNormal">Activate hosts</p <p
class=3D"MsoNormal">Start VM=92s</p <p
class=3D"MsoNormal">Reinitialize the datacenter</p <p class=3D"MsoNormal">Remove storage</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">These points above a
huge =96 while oVirt is indeed =
open source, upstream of RHEV and doesn=92t cost anything, I feel that in s=
cenarios like this it could be the downfall of oVirt itself being too risky=
.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">The logging with oVirt
seems to be crazy though =96 =
we=92ve been testing it now for about 2.5 years, maybe 3 years? Once oVirt =
gets in a state where it cannot connect to something, it just goes haywire =
=96 many likely don=92t see this however,
every time these things happened it when we=92re testing failover scenario=
s to see how oVirt responds.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">A few recommendations I
would make are:</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Drop the whole
=93master=94 storage thing =96 it com=
plicates setting storage up. Either connect, or don=92t connect. If there=
=92s connectivity issues, oVirt gets hung up on switching to this =93master=
=94 storage. If you have a single storage domain,
you=92ll likely have problems as we=92ve experienced because once oVirt ca=
nnot find the =93master=94 it begins to go berserk, then spirals out of con=
trol there. It might not on small setups with a few hypervisors, but on an =
install with a few hundred VM=92s, large number
of hypervisors etc, it seems to get ugly real quick.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Stop trying to reconnect
things, I think that=92s wh=
at I=92m looking for. When something fails, oVirt just goes in a loop over =
and over which eventually causes dashboard issues, crazy amounts of logs et=
c. It would be better if oVirt would just
stop, make a log entry and then quit, maybe after a few times.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">In my case, I could
mount the storage manually to AL=
L hosts, I could even force start the VM=92s with virsh. The oVirt dashboar=
d just kept saying it was locked, and wouldn=92t let you do anything at all=
with the entire datacenter.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">At this time, we=92ve
pushed these servers back into=
production using our current hypervisor software which is stable but does =
not have the benefits of oVirt. It=92ll be revisited later on and is still =
in use for non-production things.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <div
style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in" <p
class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:mlipchuk@redhat.com">Maor Lipchuk</a><br <b>Sent: </b>Sunday, January 22, 2017 7:33
AM<br <b>To: </b><a
href=3D"mailto:jax2568@outlook.com">Bill Bill</a><br <b>Cc: </b><a
href=3D"mailto:users@ovirt.org">users</a><br <b>Subject: </b>Re: [ovirt-users] master storage
domain stuck in locked sta=
te</p </div <p
class=3D"MsoNormal"><o:p> </o:p></p </div
<div <div
dir=3D"ltr"><br <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <s=
pan dir=3D"ltr" <<a
href=3D"mailto:mlipchuk@redhat.com"
target=3D"_blank">mlipchuk@redha=
t.com</a>></span> wrote:<br
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1p=
x #ccc solid;padding-left:1ex" <div
dir=3D"ltr">Hi Bill,
<div><br </div <div>Can you please attach the engine and VDSM
logs.</div <div>Does the storage domain
still stuck?</div </div </blockquote
<div><br </div <div>Also which oVirt version are you using?</div <div> </div
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1p=
x #ccc solid;padding-left:1ex" <div
dir=3D"ltr" <div><br </div
<div>Regards,</div <div>Maor</div </div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote" <div <div
class=3D"h5">On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <span
dir=3D"l=
tr"><<a href=3D"mailto:jax2568@outlook.com"
target=3D"_blank">jax2568@ou=
tlook.com</a>></span> wrote:<br
</div </div <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"
<div <div class=3D"h5" <div lang=3D"EN-US" link=3D"blue"
vlink=3D"#954F72" <div
class=3D"m_723276570252969701m_-2288590467547947270WordSection1" <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Also cannot reinitialize
the datacenter because the =
storage domain is locked.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 8:08 PM<span><br
<b>To: </b><a href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>RE: master storage domain stuck in
locked state</span></p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<span <div <div
class=3D"m_723276570252969701m_-2288590467547947270WordSection1" <p class=3D"MsoNormal">Spoke too soon. Some
hosts came back up but the stor=
age domain is still locked so no vm=92s can be started. What is the proper =
way to force this to be unlocked? Each time we look to move into production=
after successful testing, something
like this always seems to pop up at the last minute rending oVirt question=
able in terms of reliability for some unknown issue.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 7:54 PM<br <b>To: </b><a
href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>RE: master storage domain stuck in
locked state</p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<div <div
class=3D"m_723276570252969701m_-2288590467547947270WordSection1" <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">So apparently something
didn=92t change the metadata=
to master before connection was lost. I changed the metadata role to maste=
r and it came backup. Seems emailing in helped because every time I can=92t=
figure something out, email in a find
it shortly after.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 7:43 PM<br <b>To: </b><a
href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>master storage domain stuck in
locked state</p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<div <div
class=3D"m_723276570252969701m_-2288590467547947270WordSection1" <p class=3D"MsoNormal">No clue how to get this
out. I can mount all storage=
manually on the hypervisors. It seems like after a reboot oVirt is now hav=
ing some issue and the storage domain is stuck in locked state. Because of =
this, can=92t activate any other storage
either, so the other domains are in maintenance and the master sits in loc=
ked state, has been for hours.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">This sticks out on a
hypervisor:</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">StoragePoolWrongMaster:
Wrong Master domain or its v=
ersion: u'SD=3Dd8a0172e-837f-4552-92c7-5<wbr>66dc4e548e4, pool=3D3fd2ad92-e=
1eb-49c2-906d-0<wbr>0ec233f610a'</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Not sure, nothing
changed other than a reboot of the=
storage.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Engine log
shows:</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal">[org.ovirt.engine.core.vdsbrok<wbr>er.SetVdsStatusVD=
SCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSComman=
d(HostNam<wbr>e =3D U31U32NodeA, SetVdsStatusVDSCommandParamete<wbr>rs:{run=
Async=3D'true',
hostId=3D'70e2b8e4-0752-47a8-884<wbr>c-837a00013e79',
status=3D'NonOperational',
nonOperationalReason=3D'STORAGE_<wbr>DOMAIN_UNR=
EACHABLE', stopSpmFailureLogged=3D'false',
maintenanceReason=3D'null'}), lo=
g id: 6db9820a</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">No idea why it says
unreachable, it certainly is bec=
ause I can manually mount ALL storage to the hypervisor.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Sent from <a
href=3D"https://go.microsoft.com/fwlink=
/?LinkId=3D550986" target=3D"_blank"
Mail</a> for Windows 10</p <p
class=3D"MsoNormal"><u></u> <u></u></p </div
</div </div </div
</span></div <br
</div </div
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org"
target=3D"_blank">Users(a)ovirt.org</a><br=
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/...
br <br </blockquote </div
<br </div </blockquote
</div <br
</div </div </div
</body </html
--_000_CO2PR0801MB0743662893F6DBFE28BB3C39A6730CO2PR0801MB0743_--
12:45 a.m.
On Jan 22, 2017 10:13 PM, "Bill Bill" <jax2568(a)outlook.com> wrote:
Hello,
It was 4.0.5 however, we’ve decided to pull the plug on oVirt for now as
it’s too risky in taking down possibly a large number or servers due to
this issue. I think oVirt should be a little less “picky” if you will, on
storage connections. For example, this specific issue prevented anything
storage related from being done. Because the “master” was locked you cannot:
Add other storage
Activate hosts
Start VM’s
Reinitialize the datacenter
Remove storage
These points above a huge – while oVirt is indeed open source, upstream of
RHEV and doesn’t cost anything, I feel that in scenarios like this it could
be the downfall of oVirt itself being too risky.
The logging with oVirt seems to be crazy though – we’ve been testing it now
for about 2.5 years, maybe 3 years? Once oVirt gets in a state where it
cannot connect to something, it just goes haywire – many likely don’t see
this however, every time these things happened it when we’re testing
failover scenarios to see how oVirt responds.
A few recommendations I would make are:
Thank you for your recommendations. I agree with some, wholly disagree with
others.
I'd still appreciate if you could send us the requested logs.
TIA,
Y.
Drop the whole “master” storage thing – it complicates setting storage up.
Either connect, or don’t connect. If there’s connectivity issues, oVirt
gets hung up on switching to this “master” storage. If you have a single
storage domain, you’ll likely have problems as we’ve experienced because
once oVirt cannot find the “master” it begins to go berserk, then spirals
out of control there. It might not on small setups with a few hypervisors,
but on an install with a few hundred VM’s, large number of hypervisors etc,
it seems to get ugly real quick.
Stop trying to reconnect things, I think that’s what I’m looking for. When
something fails, oVirt just goes in a loop over and over which eventually
causes dashboard issues, crazy amounts of logs etc. It would be better if
oVirt would just stop, make a log entry and then quit, maybe after a few
times.
In my case, I could mount the storage manually to ALL hosts, I could even
force start the VM’s with virsh. The oVirt dashboard just kept saying it
was locked, and wouldn’t let you do anything at all with the entire
datacenter.
At this time, we’ve pushed these servers back into production using our
current hypervisor software which is stable but does not have the benefits
of oVirt. It’ll be revisited later on and is still in use for
non-production things.
*From: *Maor Lipchuk <mlipchuk(a)redhat.com>
*Sent: *Sunday, January 22, 2017 7:33 AM
*To: *Bill Bill <jax2568(a)outlook.com>
*Cc: *users <users(a)ovirt.org>
*Subject: *Re: [ovirt-users] master storage domain stuck in locked state
On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk(a)redhat.com> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs.
Does the storage domain still stuck?
Also which oVirt version are you using?
Regards,
Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568(a)outlook.com> wrote:
>
>
> Also cannot reinitialize the datacenter because the storage domain is
> locked.
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 8:08 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *RE: master storage domain stuck in locked state
>
>
>
> Spoke too soon. Some hosts came back up but the storage domain is still
> locked so no vm’s can be started. What is the proper way to force this to
> be unlocked? Each time we look to move into production after successful
> testing, something like this always seems to pop up at the last minute
> rending oVirt questionable in terms of reliability for some unknown issue.
>
>
>
>
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 7:54 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *RE: master storage domain stuck in locked state
>
>
>
>
>
> So apparently something didn’t change the metadata to master before
> connection was lost. I changed the metadata role to master and it came
> backup. Seems emailing in helped because every time I can’t figure
> something out, email in a find it shortly after.
>
>
>
>
>
> *From: *Bill Bill <jax2568(a)outlook.com>
> *Sent: *Friday, January 20, 2017 7:43 PM
> *To: *users <users(a)ovirt.org>
> *Subject: *master storage domain stuck in locked state
>
>
>
> No clue how to get this out. I can mount all storage manually on the
> hypervisors. It seems like after a reboot oVirt is now having some issue
> and the storage domain is stuck in locked state. Because of this, can’t
> activate any other storage either, so the other domains are in maintenance
> and the master sits in locked state, has been for hours.
>
>
>
> This sticks out on a hypervisor:
>
>
>
> StoragePoolWrongMaster: Wrong Master domain or its version:
> u'SD=d8a0172e-837f-4552-92c7-566dc4e548e4, pool=3fd2ad92-e1eb-49c2-906d-0
> 0ec233f610a'
>
>
>
> Not sure, nothing changed other than a reboot of the storage.
>
>
>
> Engine log shows:
>
>
>
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSCommand(HostName
> = U31U32NodeA, SetVdsStatusVDSCommandParameters:{runAsync='true',
> hostId='70e2b8e4-0752-47a8-884c-837a00013e79',
status='NonOperational',
> nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE',
> stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
6db9820a
>
>
>
> No idea why it says unreachable, it certainly is because I can manually
> mount ALL storage to the hypervisor.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:49 a.m.
--_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi,
Thanks for the reply back. Unfortunately, I don=92t have the logs anymore a=
s we=92ve already pulled those servers out of the rack to be used for other=
services.
From: Yaniv Kaul<mailto:ykaul@redhat.com Sent:
Sunday, January 22, 2017 4:45 PM
To: Bill Bill<mailto:jax2568@outlook.com Cc: Ovirt
Users<mailto:users@ovirt.org>; Maor Lipchuk<mailto:mlipchuk@redha=
t.com Subject: Re: [ovirt-users] master storage domain stuck in
locked state
On Jan 22, 2017 10:13 PM, "Bill Bill"
<jax2568@outlook.com<mailto:jax2568@o=
utlook.com>> wrote:
Hello,
It was 4.0.5 however, we=92ve decided to pull the plug on oVirt for now as =
it=92s too risky in taking down possibly a large number or servers due to t=
his issue. I think oVirt should be a little less =93picky=94 if you will, o=
n storage connections. For example, this specific issue prevented anything =
storage related from being done. Because the =93master=94 was locked you ca=
nnot:
Add other storage
Activate hosts
Start VM=92s
Reinitialize the datacenter
Remove storage
These points above a huge =96 while oVirt is indeed open source, upstream o=
f RHEV and doesn=92t cost anything, I feel that in scenarios like this it c=
ould be the downfall of oVirt itself being too risky.
The logging with oVirt seems to be crazy though =96 we=92ve been testing it=
now for about 2.5 years, maybe 3 years? Once oVirt gets in a state where i=
t cannot connect to something, it just goes haywire =96 many likely don=92t=
see this however, every time these things happened it when we=92re testing=
failover scenarios to see how oVirt responds.
A few recommendations I would make are:
Thank you for your recommendations. I agree with some, wholly disagree with=
others.
I'd still appreciate if you could send us the requested logs.
TIA,
Y.
Drop the whole =93master=94 storage thing =96 it complicates setting storag=
e up. Either connect, or don=92t connect. If there=92s connectivity issues,=
oVirt gets hung up on switching to this =93master=94 storage. If you have =
a single storage domain, you=92ll likely have problems as we=92ve experienc=
ed because once oVirt cannot find the =93master=94 it begins to go berserk,=
then spirals out of control there. It might not on small setups with a few=
hypervisors, but on an install with a few hundred VM=92s, large number of =
hypervisors etc, it seems to get ugly real quick.
Stop trying to reconnect things, I think that=92s what I=92m looking for. W=
hen something fails, oVirt just goes in a loop over and over which eventual=
ly causes dashboard issues, crazy amounts of logs etc. It would be better i=
f oVirt would just stop, make a log entry and then quit, maybe after a few =
times.
In my case, I could mount the storage manually to ALL hosts, I could even f=
orce start the VM=92s with virsh. The oVirt dashboard just kept saying it w=
as locked, and wouldn=92t let you do anything at all with the entire datace=
nter.
At this time, we=92ve pushed these servers back into production using our c=
urrent hypervisor software which is stable but does not have the benefits o=
f oVirt. It=92ll be revisited later on and is still in use for non-producti=
on things.
From: Maor Lipchuk<mailto:mlipchuk@redhat.com Sent:
Sunday, January 22, 2017 7:33 AM
To: Bill Bill<mailto:jax2568@outlook.com Cc:
users<mailto:users@ovirt.org Subject: Re: [ovirt-users] master
storage domain stuck in locked state
On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <mlipchuk@redhat.com<mailto:m=
lipchuk(a)redhat.com>> wrote:
Hi Bill,
Can you please attach the engine and VDSM logs.
Does the storage domain still stuck?
Also which oVirt version are you using?
Regards,
Maor
On Sat, Jan 21, 2017 at 3:11 AM, Bill Bill <jax2568@outlook.com<mailto:jax2=
568(a)outlook.com>> wrote:
Also cannot reinitialize the datacenter because the storage domain is locke=
d.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 8:08 PM
To: users<mailto:users@ovirt.org Subject:
RE: master storage domain stuck in locked state
Spoke too soon. Some hosts came back up but the storage domain is still loc=
ked so no vm=92s can be started. What is the proper way to force this to be=
unlocked? Each time we look to move into production after successful testi=
ng, something like this always seems to pop up at the last minute rending o=
Virt questionable in terms of reliability for some unknown issue.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 7:54 PM
To: users<mailto:users@ovirt.org Subject:
RE: master storage domain stuck in locked state
So apparently something didn=92t change the metadata to master before conne=
ction was lost. I changed the metadata role to master and it came backup. S=
eems emailing in helped because every time I can=92t figure something out, =
email in a find it shortly after.
From: Bill Bill<mailto:jax2568@outlook.com Sent:
Friday, January 20, 2017 7:43 PM
To: users<mailto:users@ovirt.org Subject:
master storage domain stuck in locked state
No clue how to get this out. I can mount all storage manually on the hyperv=
isors. It seems like after a reboot oVirt is now having some issue and the =
storage domain is stuck in locked state. Because of this, can=92t activate =
any other storage either, so the other domains are in maintenance and the m=
aster sits in locked state, has been for hours.
This sticks out on a hypervisor:
StoragePoolWrongMaster: Wrong Master domain or its version: u'SD=3Dd8a0172e=
-837f-4552-92c7-566dc4e548e4, pool=3D3fd2ad92-e1eb-49c2-906d-00ec233f610a'
Not sure, nothing changed other than a reboot of the storage.
Engine log shows:
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzSche=
duler8) [5696732b] START, SetVdsStatusVDSCommand(HostName =3D U31U32NodeA, =
SetVdsStatusVDSCommandParameters:{runAsync=3D'true', hostId=3D'70e2b8e4-075=
2-47a8-884c-837a00013e79', status=3D'NonOperational', nonOperationalReason=
=3D'STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged=3D'false',
maintenanc=
eReason=3D'null'}), log id: 6db9820a
No idea why it says unreachable, it certainly is because I can manually mou=
nt ALL storage to the hypervisor.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=3D550986> for Window=
s 10
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" </head <body <meta
name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)" <style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style <div
class=3D"WordSection1" <p
class=3D"MsoNormal">Hi,</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Thanks for the reply
back. Unfortunately, I don=92t =
have the logs anymore as we=92ve already pulled those servers out of the ra=
ck to be used for other services.</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <div
style=3D"mso-element:para-border-div;border:none;border-top:solid #E1E=
1E1 1.0pt;padding:3.0pt 0in 0in 0in" <p
class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:ykaul@redhat.com">Yaniv Kaul</a><br <b>Sent: </b>Sunday, January 22, 2017 4:45
PM<br <b>To: </b><a
href=3D"mailto:jax2568@outlook.com">Bill Bill</a><br <b>Cc: </b><a
href=3D"mailto:users@ovirt.org">Ovirt Users</a>; <a href=3D"m=
ailto:mlipchuk@redhat.com" Maor Lipchuk</a><br <b>Subject: </b>Re: [ovirt-users] master storage
domain stuck in locked sta=
te</p </div <p
class=3D"MsoNormal"><o:p> </o:p></p </div
<div <div dir=3D"auto" <div><br <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Jan 22, 2017 10:13 PM, "Bill Bill"
=
<<a
href=3D"mailto:jax2568@outlook.com">jax2568@outlook.com</a>>
wrot=
e:<br type=3D"attribution"
<blockquote class=3D"quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc=
solid;padding-left:1ex" <div <div
class=3D"m_-823963203978568812WordSection1" <p
class=3D"MsoNormal">Hello,</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">It was 4.0.5 however,
we=92ve decided to pull the pl=
ug on oVirt for now as it=92s too risky in taking down possibly a large num=
ber or servers due to this issue. I think oVirt should be a little less =93=
picky=94 if you will, on storage connections.
For example, this specific issue prevented anything storage related from b=
eing done. Because the =93master=94 was locked you cannot:</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Add other
storage</p <p
class=3D"MsoNormal">Activate hosts</p <p
class=3D"MsoNormal">Start VM=92s</p <p
class=3D"MsoNormal">Reinitialize the datacenter</p <p class=3D"MsoNormal">Remove storage</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">These points above a
huge =96 while oVirt is indeed =
open source, upstream of RHEV and doesn=92t cost anything, I feel that in s=
cenarios like this it could be the downfall of oVirt itself being too risky=
.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">The logging with oVirt
seems to be crazy though =96 =
we=92ve been testing it now for about 2.5 years, maybe 3 years? Once oVirt =
gets in a state where it cannot connect to something, it just goes haywire =
=96 many likely don=92t see this however,
every time these things happened it when we=92re testing failover scenario=
s to see how oVirt responds.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">A few recommendations I
would make are:</p </div </div
</blockquote </div </div
</div <div
dir=3D"auto"><br </div <div dir=3D"auto">Thank you for your
recommendations. I agree with some, wh=
olly disagree with others. </div <div
dir=3D"auto">I'd still appreciate if you could send us the requested l=
ogs. </div <div
dir=3D"auto"><br </div <div dir=3D"auto">TIA, </div <div dir=3D"auto">Y. </div <div dir=3D"auto"><br </div <div
dir=3D"auto" <div
class=3D"gmail_extra" <div
class=3D"gmail_quote" <blockquote
class=3D"quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
solid;padding-left:1ex" <div <div
class=3D"m_-823963203978568812WordSection1" <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Drop the whole
=93master=94 storage thing =96 it com=
plicates setting storage up. Either connect, or don=92t connect. If there=
=92s connectivity issues, oVirt gets hung up on switching to this =93master=
=94 storage. If you have a single storage domain,
you=92ll likely have problems as we=92ve experienced because once oVirt ca=
nnot find the =93master=94 it begins to go berserk, then spirals out of con=
trol there. It might not on small setups with a few hypervisors, but on an =
install with a few hundred VM=92s, large number
of hypervisors etc, it seems to get ugly real quick.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Stop trying to reconnect
things, I think that=92s wh=
at I=92m looking for. When something fails, oVirt just goes in a loop over =
and over which eventually causes dashboard issues, crazy amounts of logs et=
c. It would be better if oVirt would just
stop, make a log entry and then quit, maybe after a few times.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">In my case, I could
mount the storage manually to AL=
L hosts, I could even force start the VM=92s with virsh. The oVirt dashboar=
d just kept saying it was locked, and wouldn=92t let you do anything at all=
with the entire datacenter.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">At this time, we=92ve
pushed these servers back into=
production using our current hypervisor software which is stable but does =
not have the benefits of oVirt. It=92ll be revisited later on and is still =
in use for non-production things.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:mlipchuk@redhat.com" target=3D"_blank">Maor
Lipchuk</a><br <b>Sent: </b>Sunday,
January 22, 2017 7:33 AM<br <b>To: </b><a
href=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill Bi=
ll</a><br <b>Cc: </b><a
href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>Re: [ovirt-users] master storage
domain stuck in locked sta=
te</p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div <div
class=3D"elided-text" <div <div
dir=3D"ltr"><br <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Sun, Jan 22, 2017 at 2:31 PM, Maor Lipchuk <s=
pan dir=3D"ltr" <<a
href=3D"mailto:mlipchuk@redhat.com"
target=3D"_blank">mlipchuk@redha=
t.com</a>></span> wrote:<br
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1p=
x #ccc solid;padding-left:1ex" <div
dir=3D"ltr">Hi Bill,
<div><br </div <div>Can you please attach the engine and VDSM
logs.</div <div>Does the storage domain
still stuck?</div </div </blockquote
<div><br </div <div>Also which oVirt version are you using?</div <div> </div
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1p=
x #ccc solid;padding-left:1ex" <div
dir=3D"ltr" <div><br </div
<div>Regards,</div <div>Maor</div </div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote" <div <div
class=3D"m_-823963203978568812h5">On Sat, Jan 21, 2017 at 3:11 AM, Bil=
l Bill <span dir=3D"ltr"
<<a href=3D"mailto:jax2568@outlook.com"
target=3D"_blank">jax2568@outloo=
k.com</a>></span> wrote:<br
</div </div <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"
<div <div
class=3D"m_-823963203978568812h5" <div
lang=3D"EN-US" link=3D"blue" vlink=3D"#954F72" <div
class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472=
70WordSection1" <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Also cannot reinitialize
the datacenter because the =
storage domain is locked.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 8:08 PM<span><br
<b>To: </b><a href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>RE: master storage domain stuck in
locked state</span></p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<span <div <div
class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472=
70WordSection1" <p
class=3D"MsoNormal">Spoke too soon. Some hosts came back up but the stor=
age domain is still locked so no vm=92s can be started. What is the proper =
way to force this to be unlocked? Each time we look to move into production=
after successful testing, something
like this always seems to pop up at the last minute rending oVirt question=
able in terms of reliability for some unknown issue.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 7:54 PM<br <b>To: </b><a
href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>RE: master storage domain stuck in
locked state</p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<div <div
class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472=
70WordSection1" <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">So apparently something
didn=92t change the metadata=
to master before connection was lost. I changed the metadata role to maste=
r and it came backup. Seems emailing in helped because every time I can=92t=
figure something out, email in a find
it shortly after.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal"><u></u> <u></u></p <div style=3D"border:none;border-top:solid #e1e1e1
1.0pt;padding:3.0pt 0in =
0in 0in" <p class=3D"MsoNormal"
style=3D"border:none;padding:0in"><b>From: </b><a hr=
ef=3D"mailto:jax2568@outlook.com" target=3D"_blank">Bill
Bill</a><br <b>Sent: </b>Friday,
January 20, 2017 7:43 PM<br <b>To: </b><a
href=3D"mailto:users@ovirt.org"
target=3D"_blank">users</a><b=
r <b>Subject: </b>master storage domain stuck in
locked state</p </div <p
class=3D"MsoNormal"><u></u> <u></u></p </div
<div <div
class=3D"m_-823963203978568812m_723276570252969701m_-22885904675479472=
70WordSection1" <p
class=3D"MsoNormal">No clue how to get this out. I can mount all storage=
manually on the hypervisors. It seems like after a reboot oVirt is now hav=
ing some issue and the storage domain is stuck in locked state. Because of =
this, can=92t activate any other storage
either, so the other domains are in maintenance and the master sits in loc=
ked state, has been for hours.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">This sticks out on a
hypervisor:</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">StoragePoolWrongMaster:
Wrong Master domain or its v=
ersion: u'SD=3Dd8a0172e-837f-4552-92c7-5<wbr>66dc4e548e4, pool=3D3fd2ad92-e=
1eb-49c2-906d-0<wbr>0ec233f610a'</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Not sure, nothing
changed other than a reboot of the=
storage.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Engine log
shows:</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p
class=3D"MsoNormal">[org.ovirt.engine.core.vdsbrok<wbr>er.SetVdsStatusVD=
SCommand] (DefaultQuartzScheduler8) [5696732b] START, SetVdsStatusVDSComman=
d(HostNam<wbr>e =3D U31U32NodeA, SetVdsStatusVDSCommandParamete<wbr>rs:{run=
Async=3D'true',
hostId=3D'70e2b8e4-0752-47a8-884<wbr>c-837a00013e79',
status=3D'NonOperational',
nonOperationalReason=3D'STORAGE_<wbr>DOMAIN_UNR=
EACHABLE', stopSpmFailureLogged=3D'false',
maintenanceReason=3D'null'}), lo=
g id: 6db9820a</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">No idea why it says
unreachable, it certainly is bec=
ause I can manually mount ALL storage to the hypervisor.</p <p
class=3D"MsoNormal"><u></u> <u></u></p <p class=3D"MsoNormal">Sent from <a
href=3D"https://go.microsoft.com/fwlink=
/?LinkId=3D550986" target=3D"_blank"
Mail</a> for Windows 10</p <p
class=3D"MsoNormal"><u></u> <u></u></p </div
</div </div </div
</span></div <br
</div </div
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org"
target=3D"_blank">Users(a)ovirt.org</a><br=
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/...
br <br </blockquote </div
<br </div </blockquote
</div <br
</div </div </div
</div </div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/...
br <br </blockquote </div
<br </div </div
</div </div </body
</html
--_000_CO2PR0801MB0743270D75299B04BDB7AEFFA6730CO2PR0801MB0743_--
2869
days inactive
2870
days old
8 comments
3 participants
participants (3)
-
Bill Bill -
Maor Lipchuk -
Yaniv Kaul