Re: [ovirt-users] storage domain don't go active anymore after 3.5.3 update

17 Jun 2015

      ...
Hi
from logs it seems like StorageDomain=20
ID 229c74b0-75b0-48a5-b017-d794343b535b is missing
engine -
2015-06-17 14:37:49,918 INFO=20
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSComm=
and]=20
(org.ovirt.thread.pool-8-thread-45) [7dc07cff] START,=20
ActivateStorageDomainVDSCommand( storagePoolId =3D=20
1a71a643-8234-4137-800e-d7a9957857d3, ignoreFailoverLimit =3D false,=20
storageDomainId =3D 229c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff=
5345
vdsm -
Thread-80::ERROR::2015-06-17=20
15:14:17,907::domainMonitor::256::Storage.DomainMonitorThread::(_monito=
rDomain)=20
Error while collecting domain 229c74b0-75b0-48a5-b017-d794343b535b=20
monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in=20
_monitorDomain
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
    domain =3D self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom =3D findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDom=
ain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:=20
(u'229c74b0-75b0-48a5-b017-d794343b535b',)
Can you please check if this VG=20
('229c74b0-75b0-48a5-b017-d794343b535b') exist in the host and all=20
it's PVs exist as well?
if one is missing try to check if the host can see the missing lun.
Regards,
__________________________________________________
*Aharon Canan*
int phone - 8272036
ext phone - +97297692036
email - acanan@redhat.com
-----------------------------------------------------------------------=
This is a multi-part message in MIME format.
--------------000604080204050203020709
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Thank you for reply,

I found some reason to the issue : I installed a  physical host through=20
an automated kickstart process and this host took active lun from ovirt=20
lun. During the install process, it created a LVM centos-home which=20
included this ovirt lun, but it seems that it formatted this LV in XFS.=20
Vms still run without issue an d we can see their LVM id in their own=20
host. But if one stops one vm, they can't boot anymore.
More, if I reboot a physical host, this host loses all its LVM but we=20
still can see the lun multipath. And instead of ovirt LVM, I see now the=20
centos-home LVM .... :(

So my question is :
Does it exist a tool to repair the initial ovirt lvm tags?

If I can't do such a thing, my plan is to dd the /dev/mapper/[vmdevid]=20
in the same time the vm is running... I would prefer to do this when the=20
vm is down or when I get a LVM snapshot. Do you think this to be possible=
?

Thank you for your help.

Le 17/06/2015 17:43, Aharon Canan a =C3=A9crit :
-
...
*From: *"Nathana=C3=ABl Blanchet" <blanchet@abes.fr>
    *To: *"Aharon Canan" <acanan@redhat.com>
    *Cc: *users@ovirt.org
    *Sent: *Wednesday, June 17, 2015 4:33:21 PM
    *Subject: *Re: [ovirt-users] storage domain don't go active
    anymore after 3.5.3 update
Just to specify that in addition of this datacenter (cines), I
    have a second FC data center (abes) in the same engine but which
    deals only with el6 hosts and upgrade to vdsm was fine for itself.
Le 17/06/2015 15:17, Nathana=C3=ABl Blanchet a =C3=A9crit :
Hello Aharon, I have 3 hosts in that domain : fuji, fuego, sipl=
e.
...
thank you for your help
Le 17/06/2015 15:10, Aharon Canan a =C3=A9crit :
Can you attach engine and vdsm logs ?
Regards,
            __________________________________________________
            *Aharon Canan*
-----------------------------------------------------------=
-------------
...
*From: *"Nathana=C3=ABl Blanchet" <blanchet@abes.fr>
                *To: *users@ovirt.org
                *Sent: *Wednesday, June 17, 2015 4:03:29 PM
                *Subject: *[ovirt-users] storage domain don't go
                active anymore after 3.5.3        update
Hello, since the update to 3.5.3, my master data
                domain fails to activate.
                I don't know what to do, and it is critical for
                production.
                Vms are still up but I can't interacte with them anymor=
e
...
If I shutdown any of them, I won't be able to recover
                them.
My datacenter is a FC one, and all of my hosts have
                been upgraded to
                vdsm 4.16.20, I restarted vdsmd and engine, but
                nothing has changed.
Thank you for your help.
--=20
                Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau
                P=C3=B4le Infrastrutures Informatiques
                227 avenue Professeur-Jean-Louis-Viala
                34193 MONTPELLIER CEDEX 5
                T=C3=A9l. 33 (0)4 67 54 84 55
                Fax  33 (0)4 67 54 84 14
                blanchet@abes.fr
_______________________________________________
                Users mailing list
                Users@ovirt.org
                http://lists.ovirt.org/mailman/listinfo/users
--=20
        Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau
        P=C3=B4le Infrastrutures Informatiques
        227 avenue Professeur-Jean-Louis-Viala
        34193 MONTPELLIER CEDEX 5 =09
        T=C3=A9l. 33 (0)4 67 54 84 55
        Fax  33 (0)4 67 54 84 14
        blanchet@abes.fr =20
_______________________________________________
        Users mailing list
        Users@ovirt.org
        http://lists.ovirt.org/mailman/listinfo/users
--=20
    Nathana=C3=ABl Blanchet
Supervision r=C3=A9seau
    P=C3=B4le Infrastrutures Informatiques
    227 avenue Professeur-Jean-Louis-Viala
    34193 MONTPELLIER CEDEX 5 =09
    T=C3=A9l. 33 (0)4 67 54 84 55
    Fax  33 (0)4 67 54 84 14
    blanchet@abes.fr =20
--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
blanchet@abes.fr

--------------000604080204050203020709
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html>
  <head>
    <meta content=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content-Ty=
pe">
  </head>
  <body text=3D"#000000" bgcolor=3D"#FFFFFF">
    Thank you for reply,<br>
    <br>
    I found some reason to the issue : I installed a=C2=A0 physical host
    through an automated kickstart process and this host took active lun
    from ovirt lun. During the install process, it created a LVM
    centos-home which included this ovirt lun, but it seems that it
    formatted this LV in XFS. Vms still run without issue an d we can
    see their LVM id in their own host. But if one stops one vm, they
    can't boot anymore. <br>
    More, if I reboot a physical host, this host loses all its LVM but
    we still can see the lun multipath. And instead of ovirt LVM, I see
    now the centos-home LVM .... :(<br>
    <br>
    So my question is : <br>
    Does it exist a tool to repair the initial ovirt lvm tags? <br>
    <br>
    If I can't do such a thing, my plan is to dd the
    /dev/mapper/[vmdevid] in the same time the vm is running... I would
    prefer to do this when the vm is down or when I get a LVM snapshot.
    Do you think this to be possible?<br>
    <br>
    Thank you for your help.<br>
    <br>
    <div class=3D"moz-cite-prefix">Le 17/06/2015 17:43, Aharon Canan a
      =C3=A9crit=C2=A0:<br>
    </div>
    <blockquote
      cite=3D"mid:1899945239.23882721.1434555802940.JavaMail.zimbra@redha=
t.com"
      type=3D"cite">
      <div style=3D"font-family: comic sans ms,comic sans,sans-serif;
        font-size: 10pt; color: #000000">
        <div>Hi<br>
        </div>
        <div><br>
        </div>
        <div>from logs it seems like StorageDomain
          ID=C2=A0229c74b0-75b0-48a5-b017-d794343b535b is missing<br>
        </div>
        <div><br>
        </div>
        <div>engine - <br>
        </div>
        <div>2015-06-17 14:37:49,918 INFO=C2=A0
          [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomai=
nVDSCommand]
          (org.ovirt.thread.pool-8-thread-45) [7dc07cff] START,
          ActivateStorageDomainVDSCommand( storagePoolId =3D
          1a71a643-8234-4137-800e-d7a9957857d3, ignoreFailoverLimit =3D
          false, storageDomainId =3D
          229c74b0-75b0-48a5-b017-d794343b535b), log id: 72ff5345<br>
          <br>
        </div>
        <div>vdsm - <br>
        </div>
        <div>Thread-80::ERROR::2015-06-17
          15:14:17,907::domainMonitor::256::Storage.DomainMonitorThread::=
(_monitorDomain)
          Error while collecting domain
          229c74b0-75b0-48a5-b017-d794343b535b monitoring information<br>
          Traceback (most recent call last):<br>
          =C2=A0 File "/usr/share/vdsm/storage/domainMonitor.py", line 23=
2,
          in _monitorDomain<br>
          =C2=A0=C2=A0=C2=A0 self.domain.selftest()<br>
          =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 49, in
          __getattr__<br>
          =C2=A0=C2=A0=C2=A0 return getattr(self.getRealDomain(), attrNam=
e)<br>
          =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 52, in
          getRealDomain<br>
          =C2=A0=C2=A0=C2=A0 return self._cache._realProduce(self._sdUUID=
)<br>
          =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 122, in
          _realProduce<br>
          =C2=A0=C2=A0=C2=A0 domain =3D self._findDomain(sdUUID)<br>
          =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 141, in
          _findDomain<br>
          =C2=A0=C2=A0=C2=A0 dom =3D findMethod(sdUUID)<br>
          =C2=A0 File "/usr/share/vdsm/storage/sdc.py", line 171, in
          _findUnfetchedDomain<br>
          =C2=A0=C2=A0=C2=A0 raise se.StorageDomainDoesNotExist(sdUUID)<b=
r>
          StorageDomainDoesNotExist: Storage domain does not exist:
          (u'229c74b0-75b0-48a5-b017-d794343b535b',)</div>
        <div><br>
        </div>
        <div><span name=3D"x"></span>
          <div><br>
          </div>
          <div>Can you please check if this VG
            ('229c74b0-75b0-48a5-b017-d794343b535b') exist in the host
            and all it's PVs exist as well?<br>
          </div>
          <div>if one is missing try to check if the host can see the
            missing lun.<br>
          </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>Regards, <br>
            __________________________________________________<br>
            <strong><span data-mce-style=3D"color: #3366ff;" style=3D"col=
or:
                rgb(51, 102, 255);">Aharon Canan</span></strong><br>
            <span style=3D"color: rgb(51, 102, 255); font-size: small;">i=
nt
              phone - 8272036</span><br>
            <span style=3D"color: rgb(51, 102, 255); font-size: small;">e=
xt
              phone - +97297692036</span><br>
            <span style=3D"color: rgb(51, 102, 255); font-size: small;">e=
mail
              - <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:acan=
an@redhat.com">acanan@redhat.com</a></span><br>
          </div>
          <span name=3D"x"></span><br>
        </div>
        <hr id=3D"zwchr">
        <blockquote style=3D"border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;fo=
nt-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-ser=
if;font-size:12pt;"><b>From:
          </b>"Nathana=C3=ABl Blanchet" <a class=3D"moz-txt-link-rfc2396E=
" href=3D"mailto:blanchet@abes.fr"><blanchet@abes.fr></a><br>
          <b>To: </b>"Aharon Canan" <a class=3D"moz-txt-link-rfc2396E" hr=
ef=3D"mailto:acanan@redhat.com"><acanan@redhat.com></a><br>
          <b>Cc: </b><a class=3D"moz-txt-link-abbreviated" href=3D"mailto=
:users@ovirt.org">users@ovirt.org</a><br>
          <b>Sent: </b>Wednesday, June 17, 2015 4:33:21 PM<br>
          <b>Subject: </b>Re: [ovirt-users] storage domain don't go
          active anymore after 3.5.3 update<br>
          <div><br>
          </div>
          Just to specify that in addition of this datacenter (cines), I
          have a second FC data center (abes) in the same engine but
          which deals only with el6 hosts and upgrade to vdsm was fine
          for itself.<br>
          <br>
          <div class=3D"moz-cite-prefix">Le 17/06/2015 15:17, Nathana=C3=AB=
l
            Blanchet a =C3=A9crit=C2=A0:<br>
          </div>
          <blockquote cite=3D"mid:55817375.9070006@abes.fr"> Hello Aharon=
,
            I have 3 hosts in that domain : fuji, fuego, siple.<br>
            thank you for your help<br>
            <br>
            <div class=3D"moz-cite-prefix">Le 17/06/2015 15:10, Aharon
              Canan a =C3=A9crit=C2=A0:<br>
            </div>
            <blockquote
              cite=3D"mid:1871792607.23772840.1434546623327.JavaMail.zimb=
ra@redhat.com">
              <div style=3D"font-family: comic sans ms,comic
                sans,sans-serif; font-size: 10pt; color: #000000">
                <div>Can you attach engine and vdsm logs ?<br>
                </div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div><span></span>
                  <div><br>
                  </div>
                  <div><br>
                  </div>
                  <div>Regards, <br>
                    __________________________________________________<br=
...
<strong><span style=3D"color: rgb(51, 102, 255);">Aha=
ron
                        Canan</span></strong><br>
                    <span style=3D"color: rgb(51, 102, 255); font-size:
                      small;"></span><br>
                  </div>
                  <span></span><br>
                </div>
                <hr id=3D"zwchr">
                <blockquote style=3D"border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;fo=
nt-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-ser=
if;font-size:12pt;"><b>From:

                  </b>"Nathana=C3=ABl Blanchet" <a moz-do-not-send=3D"tru=
e"
                    class=3D"moz-txt-link-rfc2396E"
                    href=3D"mailto:blanchet@abes.fr" target=3D"_blank">&l=
t;blanchet@abes.fr></a><br>
                  <b>To: </b><a moz-do-not-send=3D"true"
                    class=3D"moz-txt-link-abbreviated"
                    href=3D"mailto:users@ovirt.org" target=3D"_blank">use=
rs@ovirt.org</a><br>
                  <b>Sent: </b>Wednesday, June 17, 2015 4:03:29 PM<br>
                  <b>Subject: </b>[ovirt-users] storage domain don't go
                  active anymore after 3.5.3=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0update<br>
                  <div><br>
                  </div>
                  Hello, since the update to 3.5.3, my master data
                  domain fails to activate.<br>
                  I don't know what to do, and it is critical for
                  production.<br>
                  Vms are still up but I can't interacte with them
                  anymore<br>
                  If I shutdown any of them, I won't be able to recover
                  them.<br>
                  <div><br>
                  </div>
                  My datacenter is a FC one, and all of my hosts have
                  been upgraded to <br>
                  vdsm 4.16.20, I restarted vdsmd and engine, but
                  nothing has changed.<br>
                  <div><br>
                  </div>
                  Thank you for your help.<br>
                  <div><br>
                  </div>
                  -- <br>
                  Nathana=C3=ABl Blanchet<br>
                  <div><br>
                  </div>
                  Supervision r=C3=A9seau<br>
                  P=C3=B4le Infrastrutures Informatiques<br>
                  227 avenue Professeur-Jean-Louis-Viala<br>
                  34193 MONTPELLIER CEDEX 5 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0<br>
                  T=C3=A9l. 33 (0)4 67 54 84 55<br>
                  Fax =C2=A033 (0)4 67 54 84 14<br>
                  <a moz-do-not-send=3D"true"
                    class=3D"moz-txt-link-abbreviated"
                    href=3D"mailto:blanchet@abes.fr" target=3D"_blank">bl=
anchet@abes.fr</a><br>
                  <div><br>
                  </div>
                  _______________________________________________<br>
                  Users mailing list<br>
                  <a moz-do-not-send=3D"true"
                    class=3D"moz-txt-link-abbreviated"
                    href=3D"mailto:Users@ovirt.org" target=3D"_blank">Use=
rs@ovirt.org</a><br>
                  <a moz-do-not-send=3D"true"
                    class=3D"moz-txt-link-freetext"
                    href=3D"http://lists.ovirt.org/mailman/listinfo/users=
"
                    target=3D"_blank">http://lists.ovirt.org/mailman/list=
info/users</a><br>
                </blockquote>
                <div><br>
                </div>
              </div>
            </blockquote>
            <br>
            <pre class=3D"moz-signature">--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma=
ilto:blanchet@abes.fr" target=3D"_blank">blanchet@abes.fr</a> </pre>
            <br>
            <fieldset class=3D"mimeAttachmentHeader"></fieldset>
            <br>
            <pre>_______________________________________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma=
ilto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext" href=3D"http:=
//lists.ovirt.org/mailman/listinfo/users" target=3D"_blank">http://lists.=
ovirt.org/mailman/listinfo/users</a>
</pre>
          </blockquote>
          <br>
          <pre class=3D"moz-signature">--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated" href=3D"ma=
ilto:blanchet@abes.fr" target=3D"_blank">blanchet@abes.fr</a> </pre>
        </blockquote>
        <div><br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class=3D"moz-signature" cols=3D"72">--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr">bl=
anchet@abes.fr</a> </pre>
  </body>
</html>

--------------000604080204050203020709--

Re: [ovirt-users] storage domain don't go active anymore after 3.5.3 update

Nathanaël Blanchet