storage domain don't go active anymore after 3.5.3 update

Hello, since the update to 3.5.3, my master data domain fails to activate. I don't know what to do, and it is critical for production. Vms are still up but I can't interacte with them anymore If I shutdown any of them, I won't be able to recover them. My datacenter is a FC one, and all of my hosts have been upgraded to vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed. Thank you for your help. -- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr

------=_Part_23772839_1377633902.1434546623326 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Can you attach engine and vdsm logs ?=20 Regards,=20 __________________________________________________=20 Aharon Canan=20 ----- Original Message -----
From: "Nathana=C3=ABl Blanchet" <blanchet@abes.fr> To: users@ovirt.org Sent: Wednesday, June 17, 2015 4:03:29 PM Subject: [ovirt-users] storage domain don't go active anymore after 3.5.3 update
Hello, since the update to 3.5.3, my master data domain fails to activate= . I don't know what to do, and it is critical for production. Vms are still up but I can't interacte with them anymore If I shutdown any of them, I won't be able to recover them.
My datacenter is a FC one, and all of my hosts have been upgraded to vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.
Thank you for your help.
-- Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
------=_Part_23772839_1377633902.1434546623326 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: comic sans ms,comic sans,sans-serif;= font-size: 10pt; color: #000000"><div>Can you attach engine and vdsm logs = ?<br></div><div><br></div><div><br></div><div><span name=3D"x"></span><div>= <br></div><div><br></div><div>Regards, <br>________________________________= __________________<br><strong><span data-mce-style=3D"color: #3366ff;" styl= e=3D"color: rgb(51, 102, 255);">Aharon Canan</span></strong><br><span style= =3D"color: rgb(51, 102, 255); font-size: small;"></span><br></div><span nam= e=3D"x"></span><br></div><hr id=3D"zwchr"><blockquote style=3D"border-left:= 2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:n= ormal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sa= ns-serif;font-size:12pt;"><b>From: </b>"Nathana=C3=ABl Blanchet" <blanch= et@abes.fr><br><b>To: </b>users@ovirt.org<br><b>Sent: </b>Wednesday, Jun= e 17, 2015 4:03:29 PM<br><b>Subject: </b>[ovirt-users] storage domain don't= go active anymore after 3.5.3 &nb= sp;update<br><div><br></div>Hello, since the update to 3.5.3, my master dat= a domain fails to activate.<br>I don't know what to do, and it is critical = for production.<br>Vms are still up but I can't interacte with them anymore= <br>If I shutdown any of them, I won't be able to recover them.<br><div><br=
</div>My datacenter is a FC one, and all of my hosts have been upgraded to= <br>vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.<b= r><div><br></div>Thank you for your help.<br><div><br></div>-- <br>Nathana= =C3=ABl Blanchet<br><div><br></div>Supervision r=C3=A9seau<br>P=C3=B4le Inf= rastrutures Informatiques<br>227 avenue Professeur-Jean-Louis-Viala<br>3419= 3 MONTPELLIER CEDEX 5 <br>T= =C3=A9l. 33 (0)4 67 54 84 55<br>Fax 33 (0)4 67 54 84 14<br>blanchet@a= bes.fr<br><div><br></div>_______________________________________________<br= Users mailing list<br>Users@ovirt.org<br>http://lists.ovirt.org/mailman/li= stinfo/users<br></blockquote><div><br></div></div></body></html> ------=_Part_23772839_1377633902.1434546623326--

Hello Aharon, I have 3 hosts in that domain : fuji, fuego, siple. thank you for your help Le 17/06/2015 15:10, Aharon Canan a écrit :
Can you attach engine and vdsm logs ?
Regards, __________________________________________________ *Aharon Canan*
------------------------------------------------------------------------
*From: *"Nathanaël Blanchet" <blanchet@abes.fr> *To: *users@ovirt.org *Sent: *Wednesday, June 17, 2015 4:03:29 PM *Subject: *[ovirt-users] storage domain don't go active anymore after 3.5.3 update
Hello, since the update to 3.5.3, my master data domain fails to activate. I don't know what to do, and it is critical for production. Vms are still up but I can't interacte with them anymore If I shutdown any of them, I won't be able to recover them.
My datacenter is a FC one, and all of my hosts have been upgraded to vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.
Thank you for your help.
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr

This is a multi-part message in MIME format. --------------060901050100060706050900 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Just to specify that in addition of this datacenter (cines), I have a=20 second FC data center (abes) in the same engine but which deals only=20 with el6 hosts and upgrade to vdsm was fine for itself. Le 17/06/2015 15:17, Nathana=EBl Blanchet a =E9crit :
Hello Aharon, I have 3 hosts in that domain : fuji, fuego, siple. thank you for your help
Le 17/06/2015 15:10, Aharon Canan a =E9crit :
Can you attach engine and vdsm logs ?
Regards, __________________________________________________ *Aharon Canan*
----------------------------------------------------------------------= --
*From: *"Nathana=EBl Blanchet" <blanchet@abes.fr> *To: *users@ovirt.org *Sent: *Wednesday, June 17, 2015 4:03:29 PM *Subject: *[ovirt-users] storage domain don't go active anymore after 3.5.3 update
Hello, since the update to 3.5.3, my master data domain fails to activate. I don't know what to do, and it is critical for production. Vms are still up but I can't interacte with them anymore If I shutdown any of them, I won't be able to recover them.
My datacenter is a FC one, and all of my hosts have been upgraded = to vdsm 4.16.20, I restarted vdsmd and engine, but nothing has change= d.
Thank you for your help.
--=20 Nathana=EBl Blanchet
Supervision r=E9seau P=F4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=E9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--=20 Nathana=EBl Blanchet
Supervision r=E9seau P=F4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=E9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr =20
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--=20 Nathana=EBl Blanchet Supervision r=E9seau P=F4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=E9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr --------------060901050100060706050900 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dwindows-1252" http-equiv=3D"Content-Type"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> Just to specify that in addition of this datacenter (cines), I have a second FC data center (abes) in the same engine but which deals only with el6 hosts and upgrade to vdsm was fine for itself.<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:17, Nathana=EBl Blanc= het a =E9crit=A0:<br> </div> <blockquote cite=3D"mid:55817375.9070006@abes.fr" type=3D"cite"> <meta content=3D"text/html; charset=3Dwindows-1252" http-equiv=3D"Content-Type"> Hello Aharon, I have 3 hosts in that domain : fuji, fuego, siple.<b= r> thank you for your help<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:10, Aharon Canan a =E9crit=A0:<br> </div> <blockquote cite=3D"mid:1871792607.23772840.1434546623327.JavaMail.zimbra@red= hat.com" type=3D"cite"> <div style=3D"font-family: comic sans ms,comic sans,sans-serif; font-size: 10pt; color: #000000"> <div>Can you attach engine and vdsm logs ?<br> </div> <div><br> </div> <div><br> </div> <div><span name=3D"x"></span> <div><br> </div> <div><br> </div> <div>Regards, <br> __________________________________________________<br> <strong><span data-mce-style=3D"color: #3366ff;" style=3D"color: rgb(51, 102, 255);">Aharon Canan</span>= </strong><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;"=
</span><br> </div> <span name=3D"x"></span><br> </div> <hr id=3D"zwchr"> <blockquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;fo= nt-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-ser= if;font-size:12pt;"><b>From:
</b>"Nathana=EBl Blanchet" <a moz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:blanchet@abes.fr"><blanchet@abes.fr></= a><br> <b>To: </b><a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mailto:users@ovirt.org">users@ovirt.org</a><br> <b>Sent: </b>Wednesday, June 17, 2015 4:03:29 PM<br> <b>Subject: </b>[ovirt-users] storage domain don't go active anymore after 3.5.3=A0=A0=A0=A0=A0=A0=A0=A0update<br> <div><br> </div> Hello, since the update to 3.5.3, my master data domain fails to activate.<br> I don't know what to do, and it is critical for production.<b= r> Vms are still up but I can't interacte with them anymore<br> If I shutdown any of them, I won't be able to recover them.<b= r> <div><br> </div> My datacenter is a FC one, and all of my hosts have been upgraded to <br> vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.<br> <div><br> </div> Thank you for your help.<br> <div><br> </div> -- <br> Nathana=EBl Blanchet<br> <div><br> </div> Supervision r=E9seau<br> P=F4le Infrastrutures Informatiques<br> 227 avenue Professeur-Jean-Louis-Viala<br> 34193 MONTPELLIER CEDEX 5 =A0=A0=A0=A0=A0=A0=A0=A0<br> T=E9l. 33 (0)4 67 54 84 55<br> Fax =A033 (0)4 67 54 84 14<br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated= " href=3D"mailto:blanchet@abes.fr">blanchet@abes.fr</a><br> <div><br> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated= " href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/listinfo/users">http= ://lists.ovirt.org/mailman/listinfo/users</a><br> </blockquote> <div><br> </div> </div> </blockquote> <br> <pre class=3D"moz-signature" cols=3D"72">--=20 Nathana=EBl Blanchet Supervision r=E9seau P=F4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=E9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma= ilto:blanchet@abes.fr">blanchet@abes.fr</a> </pre> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre wrap=3D"">_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Use= rs@ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman= /listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <pre class=3D"moz-signature" cols=3D"72">--=20 Nathana=EBl Blanchet Supervision r=E9seau P=F4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=E9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr">bl= anchet@abes.fr</a> </pre> </body> </html> --------------060901050100060706050900--

------=_Part_23882720_387904612.1434555802939 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi=20 from logs it seems like StorageDomain ID 229c74b0-75b0-48a5-b017-d794343b53= 5b is missing=20 engine -=20 2015-06-17 14:37:49,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Act= ivateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7dc07cff= ] START, ActivateStorageDomainVDSCommand( storagePoolId =3D 1a71a643-8234-4= 137-800e-d7a9957857d3, ignoreFailoverLimit =3D false, storageDomainId =3D 2= 29c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff5345=20 vdsm -=20 Thread-80::ERROR::2015-06-17 15:14:17,907::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 229c74b0-75= b0-48a5-b017-d794343b535b monitoring information=20 Traceback (most recent call last):=20 File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomai= n=20 self.domain.selftest()=20 File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__=20 return getattr(self.getRealDomain(), attrName)=20 File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain=20 return self._cache._realProduce(self._sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce=20 domain =3D self._findDomain(sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain=20 dom =3D findMethod(sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain=20 raise se.StorageDomainDoesNotExist(sdUUID)=20 StorageDomainDoesNotExist: Storage domain does not exist: (u'229c74b0-75b0-= 48a5-b017-d794343b535b',)=20 Can you please check if this VG ('229c74b0-75b0-48a5-b017-d794343b535b') ex= ist in the host and all it's PVs exist as well?=20 if one is missing try to check if the host can see the missing lun.=20 Regards,=20 __________________________________________________=20 Aharon Canan=20 int phone - 8272036=20 ext phone - +97297692036=20 email - acanan@redhat.com=20 ----- Original Message -----
From: "Nathana=C3=ABl Blanchet" <blanchet@abes.fr> To: "Aharon Canan" <acanan@redhat.com> Cc: users@ovirt.org Sent: Wednesday, June 17, 2015 4:33:21 PM Subject: Re: [ovirt-users] storage domain don't go active anymore after 3= .5.3 update
Just to specify that in addition of this datacenter (cines), I have a sec= ond FC data center (abes) in the same engine but which deals only with el6 ho= sts and upgrade to vdsm was fine for itself.
Le 17/06/2015 15:17, Nathana=C3=ABl Blanchet a =C3=A9crit :
Hello Aharon, I have 3 hosts in that domain : fuji, fuego, siple. =20 thank you for your help =20
Le 17/06/2015 15:10, Aharon Canan a =C3=A9crit : =20
Can you attach engine and vdsm logs ? =20 =20
Regards, =20 =20 __________________________________________________ =20 =20 Aharon Canan =20 =20
----- Original Message ----- =20 =20
From: "Nathana=C3=ABl Blanchet" <blanchet@abes.fr> =20 =20 =20 To: users@ovirt.org =20 =20 =20 Sent: Wednesday, June 17, 2015 4:03:29 PM =20 =20 =20 Subject: [ovirt-users] storage domain don't go active anymore after 3.5.3 update =20 =20 =20
Hello, since the update to 3.5.3, my master data domain fails to activate. =20 =20 =20 I don't know what to do, and it is critical for production. =20 =20 =20 Vms are still up but I can't interacte with them anymore =20 =20 =20 If I shutdown any of them, I won't be able to recover them. =20 =20 =20
My datacenter is a FC one, and all of my hosts have been upgraded t= o =20 =20 =20 vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed= . =20 =20 =20
Thank you for your help. =20 =20 =20
-- =20 =20 =20 Nathana=C3=ABl Blanchet =20 =20 =20
Supervision r=C3=A9seau =20 =20 =20 P=C3=B4le Infrastrutures Informatiques =20 =20 =20 227 avenue Professeur-Jean-Louis-Viala =20 =20 =20 34193 MONTPELLIER CEDEX 5 =20 =20 =20 T=C3=A9l. 33 (0)4 67 54 84 55 =20 =20 =20 Fax 33 (0)4 67 54 84 14 =20 =20 =20 blanchet@abes.fr =20 =20 =20
_______________________________________________ =20 =20 =20 Users mailing list =20 =20 =20 Users@ovirt.org =20 =20 =20 http://lists.ovirt.org/mailman/listinfo/users =20 =20 =20
-- =20 Nathana=C3=ABl Blanchet =20
Supervision r=C3=A9seau =20 P=C3=B4le Infrastrutures Informatiques =20 227 avenue Professeur-Jean-Louis-Viala =20 34193 MONTPELLIER CEDEX 5 =20 T=C3=A9l. 33 (0)4 67 54 84 55 =20 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr =20
_______________________________________________ =20 Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =20
-- Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
------=_Part_23882720_387904612.1434555802939 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: comic sans ms,comic sans,sans-serif;= font-size: 10pt; color: #000000"><div>Hi<br></div><div><br></div><div>from= logs it seems like StorageDomain ID 229c74b0-75b0-48a5-b017-d794343b5= 35b is missing<br></div><div><br></div><div>engine - <br></div><div>2015-06= -17 14:37:49,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Acti= vateStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7dc07cff]= START, ActivateStorageDomainVDSCommand( storagePoolId =3D 1a71a643-8234-41= 37-800e-d7a9957857d3, ignoreFailoverLimit =3D false, storageDomainId =3D 22= 9c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff5345<br><br></div><div>vds= m - <br></div><div>Thread-80::ERROR::2015-06-17 15:14:17,907::domainMonitor= ::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting= domain 229c74b0-75b0-48a5-b017-d794343b535b monitoring information<br>Trac= eback (most recent call last):<br> File "/usr/share/vdsm/storage/doma= inMonitor.py", line 232, in _monitorDomain<br> self.domai= n.selftest()<br> File "/usr/share/vdsm/storage/sdc.py", line 49, in _= _getattr__<br> return getattr(self.getRealDomain(), attrN= ame)<br> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDo= main<br> return self._cache._realProduce(self._sdUUID)<br=
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<br= domain =3D self._findDomain(sdUUID)<br> File "/us= r/share/vdsm/storage/sdc.py", line 141, in _findDomain<br>  = ; dom =3D findMethod(sdUUID)<br> File "/usr/share/vdsm/storage/sdc.py= ", line 171, in _findUnfetchedDomain<br> raise se.Storage= DomainDoesNotExist(sdUUID)<br>StorageDomainDoesNotExist: Storage domain doe= s not exist: (u'229c74b0-75b0-48a5-b017-d794343b535b',)</div><div><br></div= <div><span name=3D"x"></span><div><br></div><div>Can you please check if t= his VG ('229c74b0-75b0-48a5-b017-d794343b535b') exist in the host and all i= t's PVs exist as well?<br></div><div>if one is missing try to check if the = host can see the missing lun.<br></div><div><br></div><div><br></div><div>R= egards, <br>__________________________________________________<br><strong><= span data-mce-style=3D"color: #3366ff;" style=3D"color: rgb(51, 102, 255);"= Aharon Canan</span></strong><br><span style=3D"color: rgb(51, 102, 255); f= ont-size: small;">int phone - 8272036</span><br><span style=3D"color: rgb(5= 1, 102, 255); font-size: small;">ext phone - +97297692036</span><br><span s= tyle=3D"color: rgb(51, 102, 255); font-size: small;">email - acanan@redhat.= com</span><br></div><span name=3D"x"></span><br></div><hr id=3D"zwchr"><blo= ckquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left= :5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;f= ont-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Nathan= a=C3=ABl Blanchet" <blanchet@abes.fr><br><b>To: </b>"Aharon Canan" &l= t;acanan@redhat.com><br><b>Cc: </b>users@ovirt.org<br><b>Sent: </b>Wedne= sday, June 17, 2015 4:33:21 PM<br><b>Subject: </b>Re: [ovirt-users] storage= domain don't go active anymore after 3.5.3 update<br><div><br></div> =20 =20 =20 =20 Just to specify that in addition of this datacenter (cines), I have a second FC data center (abes) in the same engine but which deals only with el6 hosts and upgrade to vdsm was fine for itself.<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:17, Nathana=C3=ABl Blan= chet a =C3=A9crit :<br> </div> <blockquote cite=3D"mid:55817375.9070006@abes.fr"> =20 Hello Aharon, I have 3 hosts in that domain : fuji, fuego, siple.<br> thank you for your help<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:10, Aharon Canan a =C3=A9crit :<br> </div> <blockquote cite=3D"mid:1871792607.23772840.1434546623327.JavaMail.zi= mbra@redhat.com"> <div style=3D"font-family: comic sans ms,comic sans,sans-serif; font-size: 10pt; color: #000000"> <div>Can you attach engine and vdsm logs ?<br> </div> <div><br> </div> <div><br> </div> <div><span></span> <div><br> </div> <div><br> </div> <div>Regards, <br> __________________________________________________<br> <strong><span style=3D"color: rgb(51, 102, 255);">Aharon Cana= n</span></strong><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;"><= /span><br> </div> <span></span><br> </div> <hr id=3D"zwchr"> <blockquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font= -style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;f= ont-size:12pt;"><b>From:
</b>"Nathana=C3=ABl Blanchet" <a class=3D"moz-txt-link-rfc2396E= " href=3D"mailto:blanchet@abes.fr" target=3D"_blank"><blanchet@abes.fr&g= t;</a><br> <b>To: </b><a class=3D"moz-txt-link-abbreviated" href=3D"mailto= :users@ovirt.org" target=3D"_blank">users@ovirt.org</a><br> <b>Sent: </b>Wednesday, June 17, 2015 4:03:29 PM<br> <b>Subject: </b>[ovirt-users] storage domain don't go active anymore after 3.5.3 &= nbsp; update<br> <div><br> </div> Hello, since the update to 3.5.3, my master data domain fails to activate.<br> I don't know what to do, and it is critical for production.<br> Vms are still up but I can't interacte with them anymore<br> If I shutdown any of them, I won't be able to recover them.<br> <div><br> </div> My datacenter is a FC one, and all of my hosts have been upgraded to <br> vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.<br> <div><br> </div> Thank you for your help.<br> <div><br> </div> -- <br> Nathana=C3=ABl Blanchet<br> <div><br> </div> Supervision r=C3=A9seau<br> P=C3=B4le Infrastrutures Informatiques<br> 227 avenue Professeur-Jean-Louis-Viala<br> 34193 MONTPELLIER CEDEX 5 &= nbsp; <br> T=C3=A9l. 33 (0)4 67 54 84 55<br> Fax 33 (0)4 67 54 84 14<br> <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@a= bes.fr" target=3D"_blank">blanchet@abes.fr</a><br> <div><br> </div> _______________________________________________<br> Users mailing list<br> <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovir= t.org" target=3D"_blank">Users@ovirt.org</a><br> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.o= rg/mailman/listinfo/users" target=3D"_blank">http://lists.ovirt.org/mailman= /listinfo/users</a><br> </blockquote> <div><br> </div> </div> </blockquote> <br> <pre class=3D"moz-signature">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr" targ= et=3D"_blank">blanchet@abes.fr</a> </pre> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre>_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org" targe= t=3D"_blank">Users@ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users" target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/us= ers</a> </pre> </blockquote> <br> <pre class=3D"moz-signature">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr" targ= et=3D"_blank">blanchet@abes.fr</a> </pre> =20 </blockquote><div><br></div></div></body></html> ------=_Part_23882720_387904612.1434555802939--

Hi
from logs it seems like StorageDomain=20 ID 229c74b0-75b0-48a5-b017-d794343b535b is missing
engine - 2015-06-17 14:37:49,918 INFO=20 [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSComm= and]=20 (org.ovirt.thread.pool-8-thread-45) [7dc07cff] START,=20 ActivateStorageDomainVDSCommand( storagePoolId =3D=20 1a71a643-8234-4137-800e-d7a9957857d3, ignoreFailoverLimit =3D false,=20 storageDomainId =3D 229c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff= 5345
vdsm - Thread-80::ERROR::2015-06-17=20 15:14:17,907::domainMonitor::256::Storage.DomainMonitorThread::(_monito= rDomain)=20 Error while collecting domain 229c74b0-75b0-48a5-b017-d794343b535b=20 monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in=20 _monitorDomain self.domain.selftest() File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDom= ain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist:=20 (u'229c74b0-75b0-48a5-b017-d794343b535b',)
Can you please check if this VG=20 ('229c74b0-75b0-48a5-b017-d794343b535b') exist in the host and all=20 it's PVs exist as well? if one is missing try to check if the host can see the missing lun.
Regards, __________________________________________________ *Aharon Canan* int phone - 8272036 ext phone - +97297692036 email - acanan@redhat.com
-----------------------------------------------------------------------=
This is a multi-part message in MIME format. --------------000604080204050203020709 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Thank you for reply, I found some reason to the issue : I installed a physical host through=20 an automated kickstart process and this host took active lun from ovirt=20 lun. During the install process, it created a LVM centos-home which=20 included this ovirt lun, but it seems that it formatted this LV in XFS.=20 Vms still run without issue an d we can see their LVM id in their own=20 host. But if one stops one vm, they can't boot anymore. More, if I reboot a physical host, this host loses all its LVM but we=20 still can see the lun multipath. And instead of ovirt LVM, I see now the=20 centos-home LVM .... :( So my question is : Does it exist a tool to repair the initial ovirt lvm tags? If I can't do such a thing, my plan is to dd the /dev/mapper/[vmdevid]=20 in the same time the vm is running... I would prefer to do this when the=20 vm is down or when I get a LVM snapshot. Do you think this to be possible= ? Thank you for your help. Le 17/06/2015 17:43, Aharon Canan a =C3=A9crit : -
*From: *"Nathana=C3=ABl Blanchet" <blanchet@abes.fr> *To: *"Aharon Canan" <acanan@redhat.com> *Cc: *users@ovirt.org *Sent: *Wednesday, June 17, 2015 4:33:21 PM *Subject: *Re: [ovirt-users] storage domain don't go active anymore after 3.5.3 update
Just to specify that in addition of this datacenter (cines), I have a second FC data center (abes) in the same engine but which deals only with el6 hosts and upgrade to vdsm was fine for itself.
Le 17/06/2015 15:17, Nathana=C3=ABl Blanchet a =C3=A9crit :
Hello Aharon, I have 3 hosts in that domain : fuji, fuego, sipl=
e.
thank you for your help
Le 17/06/2015 15:10, Aharon Canan a =C3=A9crit :
Can you attach engine and vdsm logs ?
Regards, __________________________________________________ *Aharon Canan*
-----------------------------------------------------------=
-------------
*From: *"Nathana=C3=ABl Blanchet" <blanchet@abes.fr> *To: *users@ovirt.org *Sent: *Wednesday, June 17, 2015 4:03:29 PM *Subject: *[ovirt-users] storage domain don't go active anymore after 3.5.3 update
Hello, since the update to 3.5.3, my master data domain fails to activate. I don't know what to do, and it is critical for production. Vms are still up but I can't interacte with them anymor=
e
If I shutdown any of them, I won't be able to recover them.
My datacenter is a FC one, and all of my hosts have been upgraded to vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.
Thank you for your help.
--=20 Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--=20 Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr =20
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--=20 Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr =20
--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr --------------000604080204050203020709 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html> <head> <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty= pe"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> Thank you for reply,<br> <br> I found some reason to the issue : I installed a=C2=A0 physical host through an automated kickstart process and this host took active lun from ovirt lun. During the install process, it created a LVM centos-home which included this ovirt lun, but it seems that it formatted this LV in XFS. Vms still run without issue an d we can see their LVM id in their own host. But if one stops one vm, they can't boot anymore. <br> More, if I reboot a physical host, this host loses all its LVM but we still can see the lun multipath. And instead of ovirt LVM, I see now the centos-home LVM .... :(<br> <br> So my question is : <br> Does it exist a tool to repair the initial ovirt lvm tags? <br> <br> If I can't do such a thing, my plan is to dd the /dev/mapper/[vmdevid] in the same time the vm is running... I would prefer to do this when the vm is down or when I get a LVM snapshot. Do you think this to be possible?<br> <br> Thank you for your help.<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 17:43, Aharon Canan a =C3=A9crit=C2=A0:<br> </div> <blockquote cite=3D"mid:1899945239.23882721.1434555802940.JavaMail.zimbra@redha= t.com" type=3D"cite"> <div style=3D"font-family: comic sans ms,comic sans,sans-serif; font-size: 10pt; color: #000000"> <div>Hi<br> </div> <div><br> </div> <div>from logs it seems like StorageDomain ID=C2=A0229c74b0-75b0-48a5-b017-d794343b535b is missing<br> </div> <div><br> </div> <div>engine - <br> </div> <div>2015-06-17 14:37:49,918 INFO=C2=A0 [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomai= nVDSCommand] (org.ovirt.thread.pool-8-thread-45) [7dc07cff] START, ActivateStorageDomainVDSCommand( storagePoolId =3D 1a71a643-8234-4137-800e-d7a9957857d3, ignoreFailoverLimit =3D false, storageDomainId =3D 229c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff5345<br> <br> </div> <div>vdsm - <br> </div> <div>Thread-80::ERROR::2015-06-17 15:14:17,907::domainMonitor::256::Storage.DomainMonitorThread::= (_monitorDomain) Error while collecting domain 229c74b0-75b0-48a5-b017-d794343b535b monitoring information<br> Traceback (most recent call last):<br> =C2=A0 File "/usr/share/vdsm/storage/domainMonitor.py", line 23= 2, in _monitorDomain<br> =C2=A0=C2=A0=C2=A0 self.domain.selftest()<br> =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__<br> =C2=A0=C2=A0=C2=A0 return getattr(self.getRealDomain(), attrNam= e)<br> =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain<br> =C2=A0=C2=A0=C2=A0 return self._cache._realProduce(self._sdUUID= )<br> =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce<br> =C2=A0=C2=A0=C2=A0 domain =3D self._findDomain(sdUUID)<br> =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain<br> =C2=A0=C2=A0=C2=A0 dom =3D findMethod(sdUUID)<br> =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain<br> =C2=A0=C2=A0=C2=A0 raise se.StorageDomainDoesNotExist(sdUUID)<b= r> StorageDomainDoesNotExist: Storage domain does not exist: (u'229c74b0-75b0-48a5-b017-d794343b535b',)</div> <div><br> </div> <div><span name=3D"x"></span> <div><br> </div> <div>Can you please check if this VG ('229c74b0-75b0-48a5-b017-d794343b535b') exist in the host and all it's PVs exist as well?<br> </div> <div>if one is missing try to check if the host can see the missing lun.<br> </div> <div><br> </div> <div><br> </div> <div>Regards, <br> __________________________________________________<br> <strong><span data-mce-style=3D"color: #3366ff;" style=3D"col= or: rgb(51, 102, 255);">Aharon Canan</span></strong><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;">i= nt phone - 8272036</span><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;">e= xt phone - +97297692036</span><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;">e= mail - <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:acan= an@redhat.com">acanan@redhat.com</a></span><br> </div> <span name=3D"x"></span><br> </div> <hr id=3D"zwchr"> <blockquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;fo= nt-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-ser= if;font-size:12pt;"><b>From: </b>"Nathana=C3=ABl Blanchet" <a class=3D"moz-txt-link-rfc2396E= " href=3D"mailto:blanchet@abes.fr"><blanchet@abes.fr></a><br> <b>To: </b>"Aharon Canan" <a class=3D"moz-txt-link-rfc2396E" hr= ef=3D"mailto:acanan@redhat.com"><acanan@redhat.com></a><br> <b>Cc: </b><a class=3D"moz-txt-link-abbreviated" href=3D"mailto= :users@ovirt.org">users@ovirt.org</a><br> <b>Sent: </b>Wednesday, June 17, 2015 4:33:21 PM<br> <b>Subject: </b>Re: [ovirt-users] storage domain don't go active anymore after 3.5.3 update<br> <div><br> </div> Just to specify that in addition of this datacenter (cines), I have a second FC data center (abes) in the same engine but which deals only with el6 hosts and upgrade to vdsm was fine for itself.<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:17, Nathana=C3=AB= l Blanchet a =C3=A9crit=C2=A0:<br> </div> <blockquote cite=3D"mid:55817375.9070006@abes.fr"> Hello Aharon= , I have 3 hosts in that domain : fuji, fuego, siple.<br> thank you for your help<br> <br> <div class=3D"moz-cite-prefix">Le 17/06/2015 15:10, Aharon Canan a =C3=A9crit=C2=A0:<br> </div> <blockquote cite=3D"mid:1871792607.23772840.1434546623327.JavaMail.zimb= ra@redhat.com"> <div style=3D"font-family: comic sans ms,comic sans,sans-serif; font-size: 10pt; color: #000000"> <div>Can you attach engine and vdsm logs ?<br> </div> <div><br> </div> <div><br> </div> <div><span></span> <div><br> </div> <div><br> </div> <div>Regards, <br> __________________________________________________<br=
<strong><span style=3D"color: rgb(51, 102, 255);">Aha= ron Canan</span></strong><br> <span style=3D"color: rgb(51, 102, 255); font-size: small;"></span><br> </div> <span></span><br> </div> <hr id=3D"zwchr"> <blockquote style=3D"border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;fo= nt-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-ser= if;font-size:12pt;"><b>From: </b>"Nathana=C3=ABl Blanchet" <a moz-do-not-send=3D"tru= e" class=3D"moz-txt-link-rfc2396E" href=3D"mailto:blanchet@abes.fr" target=3D"_blank">&l= t;blanchet@abes.fr></a><br> <b>To: </b><a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mailto:users@ovirt.org" target=3D"_blank">use= rs@ovirt.org</a><br> <b>Sent: </b>Wednesday, June 17, 2015 4:03:29 PM<br> <b>Subject: </b>[ovirt-users] storage domain don't go active anymore after 3.5.3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0update<br> <div><br> </div> Hello, since the update to 3.5.3, my master data domain fails to activate.<br> I don't know what to do, and it is critical for production.<br> Vms are still up but I can't interacte with them anymore<br> If I shutdown any of them, I won't be able to recover them.<br> <div><br> </div> My datacenter is a FC one, and all of my hosts have been upgraded to <br> vdsm 4.16.20, I restarted vdsmd and engine, but nothing has changed.<br> <div><br> </div> Thank you for your help.<br> <div><br> </div> -- <br> Nathana=C3=ABl Blanchet<br> <div><br> </div> Supervision r=C3=A9seau<br> P=C3=B4le Infrastrutures Informatiques<br> 227 avenue Professeur-Jean-Louis-Viala<br> 34193 MONTPELLIER CEDEX 5 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0<br> T=C3=A9l. 33 (0)4 67 54 84 55<br> Fax =C2=A033 (0)4 67 54 84 14<br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr" target=3D"_blank">bl= anchet@abes.fr</a><br> <div><br> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org" target=3D"_blank">Use= rs@ovirt.org</a><br> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/listinfo/users= " target=3D"_blank">http://lists.ovirt.org/mailman/list= info/users</a><br> </blockquote> <div><br> </div> </div> </blockquote> <br> <pre class=3D"moz-signature">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma= ilto:blanchet@abes.fr" target=3D"_blank">blanchet@abes.fr</a> </pre> <br> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma= ilto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a> <a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http:= //lists.ovirt.org/mailman/listinfo/users" target=3D"_blank">http://lists.= ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <pre class=3D"moz-signature">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma= ilto:blanchet@abes.fr" target=3D"_blank">blanchet@abes.fr</a> </pre> </blockquote> <div><br> </div> </div> </blockquote> <br> <pre class=3D"moz-signature" cols=3D"72">--=20 Nathana=C3=ABl Blanchet Supervision r=C3=A9seau P=C3=B4le Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 =09 T=C3=A9l. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr">bl= anchet@abes.fr</a> </pre> </body> </html> --------------000604080204050203020709--
participants (2)
-
Aharon Canan
-
Nathanaël Blanchet