Re: [ovirt-users] Cannot activate storage domain

2 Mar 2018

      --_000_2CB4E8C8E00E594EA06D4AC427E429920FE5A323fabamailserverf_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi all,

  I managed to get the inactive storage domain to maintenance by stopping a=
ll running VMs that were using it, but I am still not able to activate it.

Trying to activate results in the following events:

For each host:
VDSM <hostname> command GetVGInfoVDS failed: Volume Group does not exist: (=
u'vg_uuid: 813oRe-64r8-mloU-k9G2-LFsS-dXSG-hpN4kf',)

And finally:
VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist=
: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',)

Is there anything I can do to recover this storage domain?

Thank you and all the best,
Simone

Von: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag vo=
n Bruckner, Simone
Gesendet: Donnerstag, 1. M=E4rz 2018 17:57
An: users@ovirt.org
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi,

  we are still struggling getting a storage domain online again. We tried t=
o put the storage domain in maintenance mode, that led to "Failed to update=
 OVF disks 809cc8d7-7687-46cf-a342-3be48674a9b3, OVF data isn't updated on =
those OVF stores".

Trying again with ignoring OVF update failures put the storage domain in "p=
reparing for maintenance". We see the following message on all hosts: "Erro=
r releasing host id 26 for domain b83c159c-4ad6-4613-ba16-bab95ccd10c0 (mon=
itor:578)".

Querying the storage domain using vdsm-client on the SPM resulted in
# vdsm-client StorageDomain getInfo "storagedomainID"=3D"b83c159c-4ad6-4613=
-ba16-bab95ccd10c0"
vdsm-client: Command StorageDomain.getInfo with args {'storagedomainID': 'b=
83c159c-4ad6-4613-ba16-bab95ccd10c0'} failed:
(code=3D358, message=3DStorage domain does not exist: (u'b83c159c-4ad6-4613=
-ba16-bab95ccd10c0',))

Any ideas?

Thank you and all the best,
Simone

Von: users-bounces@ovirt.org<mailto:users-bounces@ovirt.org> [mailto:users-=
bounces@ovirt.org] Im Auftrag von Bruckner, Simone
Gesendet: Mittwoch, 28. Februar 2018 15:52
An: users@ovirt.org<mailto:users@ovirt.org>
Betreff: [ovirt-users] Cannot activate storage domain

Hi all,

  we run a small oVirt installation that we also use for automated testing =
(automatically creating, dropping vms).

We got an inactive FC storage domain that we cannot activate any more. We s=
ee several events at that time starting with:

VM perftest-c17 is down with error. Exit message: Unable to get volume size=
 for domain b83c159c-4ad6-4613-ba16-bab95ccd10c0 volume 686376c1-4be1-44c3-=
89a3-0a8addc8fdf2.

Trying to activate the strorage domain results in the following alert event=
 for each host:

VDSM <hostname> command GetVGInfoVDS failed: Volume Group does not exist: (=
u'vg_uuid: 813oRe-64r8-mloU-k9G2-LFsS-dXSG-hpN4kf',)

And after those messages from all hosts we get:

VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist=
: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',)
Failed to activate Storage Domain VMHOST_LUN_205 (Data Center Production) b=
y <user>
Invalid status on Data Center Production. Setting status to Non Responsive.
Storage Pool Manager runs on Host vmhost003.fabagl.fabasoft.com (Address: v=
mhost003.fabagl.fabasoft.com), Data Center Production.

Checking the hosts with multipath -ll we see the LUN without errors.

We run oVirt 4.2.1 on CentOS 7.4. Hosts are CentOS 7.4 hosts with oVirt ins=
talled using oVirt engine.
Hosts are connected to about 30 FC LUNs (8 TB each) on two all-flash storag=
e arrays.

Thank you,
Simone Bruckner

--_000_2CB4E8C8E00E594EA06D4AC427E429920FE5A323fabamailserverf_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
	{mso-style-name:msonormal;
	mso-margin-top-alt:auto;
	margin-right:0cm;
	mso-margin-bottom-alt:auto;
	margin-left:0cm;
	font-size:12.0pt;
	font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.E-MailFormatvorlage19
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.E-MailFormatvorlage20
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.E-MailFormatvorlage21
	{mso-style-type:personal;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
span.E-MailFormatvorlage22
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"DE" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Hi all,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">  I managed to get the ina=
ctive storage domain to maintenance by stopping all running VMs that were u=
sing it, but I am still not able to activate it.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Trying to activate results in t=
he following events:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">For each host:<o:p></o:p></span=
...
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-size:10.0pt;font-=
family:"Courier New"">VDSM <hostname> command GetVGInfoVDS =
failed: Volume Group does not exist: (u'vg_uuid: 813oRe-64r8-mloU-k9G2-LFsS=
-dXSG-hpN4kf',)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">And finally:<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-size:10.0pt;font-=
family:"Courier New"">VDSM command ActivateStorageDomainVDS faile=
d: Storage domain does not exist: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',=
)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Is there anything I can do to r=
ecover this storage domain?<o:p></o:p></span></p>
<div>
<p class=3D"MsoNormal"><span lang=3D"FR" style=3D"mso-fareast-language:DE-A=
T"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"FR" style=3D"mso-fareast-language:DE-A=
T">Thank you and all the best,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"FR" style=3D"mso-fareast-language:DE-A=
T">Simone</span><span lang=3D"EN-GB" style=3D"font-size:9.0pt;font-family:&=
quot;Arial",sans-serif;mso-fareast-language:DE-AT"><o:p></o:p></span><=
/p>
</div>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<div>
<div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal"><b><span style=3D"mso-fareast-language:DE">Von:</spa=
n></b><span style=3D"mso-fareast-language:DE"> users-bounces@ovirt.org [mai=
lto:users-bounces@ovirt.org]
<b>Im Auftrag von </b>Bruckner, Simone<br>
<b>Gesendet:</b> Donnerstag, 1. M=E4rz 2018 17:57<br>
<b>An:</b> users@ovirt.org<br>
<b>Betreff:</b> Re: [ovirt-users] Cannot activate storage domain<o:p></o:p>=
</span></p>
</div>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Hi,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">  we are still struggling =
getting a storage domain online again. We tried to put the storage domain i=
n maintenance mode, that led to “Failed to update OVF disks 809cc8d7-=
7687-46cf-a342-3be48674a9b3, OVF data isn't updated
 on those OVF stores”.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Trying again with ignoring OVF =
update failures put the storage domain in “preparing for maintenance&=
#8221;. We see the following message on all hosts: “Error releasing h=
ost id 26 for domain b83c159c-4ad6-4613-ba16-bab95ccd10c0
 (monitor:578)”.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Querying the storage domain usi=
ng vdsm-client on the SPM resulted in
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-size:10.0pt;font-=
family:"Courier New""># vdsm-client StorageDomain getInfo "s=
toragedomainID"=3D"b83c159c-4ad6-4613-ba16-bab95ccd10c0"<o:p=
</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-size:10.0pt;font-=
family:"Courier New"">vdsm-client: Command StorageDomain.getInfo =
with args {'storagedomainID': 'b83c159c-4ad6-4613-ba16-bab95ccd10c0'} faile=
d:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB" style=3D"font-size:10.0pt;font-=
family:"Courier New"">(code=3D358, message=3DStorage domain does =
not exist: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',))<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Any ideas?<o:p></o:p></span></p=
...
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<div>
<p class=3D"MsoNormal"><span lang=3D"FR" style=3D"mso-fareast-language:DE-A=
T">Thank you and all the best,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"FR" style=3D"mso-fareast-language:DE-A=
T">Simone</span><span lang=3D"EN-GB" style=3D"font-size:9.0pt;font-family:&=
quot;Arial",sans-serif;mso-fareast-language:DE-AT"><o:p></o:p></span><=
/p>
</div>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<div>
<div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal"><b><span lang=3D"EN-GB" style=3D"mso-fareast-languag=
e:DE">Von:</span></b><span lang=3D"EN-GB" style=3D"mso-fareast-language:DE"=
...
...
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Invalid status on Data Center P=
roduction. Setting status to Non Responsive.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Storage Pool Manager runs on Ho=
st vmhost003.fabagl.fabasoft.com (Address: vmhost003.fabagl.fabasoft.com), =
Data Center Production.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Checking the hosts with multipa=
<a href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a> [<a =
href=3D"mailto:users-bounces@ovirt.org">mailto:users-bounces@ovirt<span lan=
g=3D"DE">.org</span></a></span><span style=3D"mso-fareast-language:DE">]
<b>Im Auftrag von </b>Bruckner, Simone<br>
<b>Gesendet:</b> Mittwoch, 28. Februar 2018 15:52<br>
<b>An:</b> <a href=3D"mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Betreff:</b> [ovirt-users] Cannot activate storage domain<o:p></o:p></sp=
an></p>
</div>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Hi all,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">  we run a small oVirt ins=
tallation that we also use for automated testing (automatically creating, d=
ropping vms).<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">We got an inactive FC storage d=
omain that we cannot activate any more. We see several events at that time =
starting with:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">VM perftest-c17 is down with er=
ror. Exit message: Unable to get volume size for domain b83c159c-4ad6-4613-=
ba16-bab95ccd10c0 volume 686376c1-4be1-44c3-89a3-0a8addc8fdf2.<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Trying to activate the strorage=
 domain results in the following alert event for each host:<o:p></o:p></spa=
n></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">VDSM <hostname> command G=
etVGInfoVDS failed: Volume Group does not exist: (u'vg_uuid: 813oRe-64r8-ml=
oU-k9G2-LFsS-dXSG-hpN4kf',)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">And after those messages from a=
ll hosts we get:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">VDSM command ActivateStorageDom=
ainVDS failed: Storage domain does not exist: (u'b83c159c-4ad6-4613-ba16-ba=
b95ccd10c0',)<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Failed to activate Storage Doma=
in VMHOST_LUN_205 (Data Center Production) by <user><o:p></o:p></span=
th –ll we see the LUN without errors.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">We run oVirt 4.2.1 on CentOS 7.=
4. Hosts are CentOS 7.4 hosts with oVirt installed using oVirt engine.<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB">Hosts are connected to about 30=
 FC LUNs (8 TB each) on two all-flash storage arrays.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"DE-AT" style=3D"mso-fareast-language:D=
E">Thank you,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"DE-AT" style=3D"mso-fareast-language:D=
E">Simone Bruckner</span><span style=3D"font-size:8.0pt;font-family:"A=
rial",sans-serif;mso-fareast-language:DE"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:"Ari=
al",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"DE-AT"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p>
</div>
</body>
</html>

--_000_2CB4E8C8E00E594EA06D4AC427E429920FE5A323fabamailserverf_--