Error mounting hosted engine Volume (Glusterfs) via VDSM

25 Jun 2016

      This is a multi-part message in MIME format.
--------------490BE4DF07D0809CD094F385
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hello,

I think options for mounting the hosted-engine Volume changed in latest
vdsm to support mounting from backup-volfile-servers.

[root@microcloud28 ~]# rpm -qi vdsm
Name        : vdsm
Version     : 4.17.28
Release     : 1.el7
Architecture: noarch
Install Date: Fri 10 Jun 2016 11:17:37 AM CEST
Group       : Applications/System
Size        : 3828639
License     : GPLv2+
Signature   : RSA/SHA1, Fri 03 Jun 2016 12:53:20 AM CEST, Key ID
7aebbe8261e8806c
Source RPM  : vdsm-4.17.28-1.el7.src.rpm

Now my hosts have problems to mount the Volume. On hosted-engine setup I
configured the (Replica 3) volume to be
"glusterfs.rxmgmt.databay.de:/engine" which ist a Round-Robin DNS to my
gluster hosts and _not_ the DNS-Name of any gluster-brick.

Now VDSM logs:

jsonrpc.Executor/3::DEBUG::2016-06-25
19:40:02,520::fileUtils::143::Storage.fileUtils::(createdir) Cre
ating directory:
/rhev/data-center/mnt/glusterSD/glusterfs.rxmgmt.databay.de:_engine
mode: None
jsonrpc.Executor/3::DEBUG::2016-06-25
19:40:02,520::storageServer::364::Storage.StorageServer.MountCon
nection::(_get_backup_servers_option) Using bricks:
['microcloud21.rxmgmt.databay.de', 'microcloud24.r
xmgmt.databay.de', 'microcloud27.rxmgmt.databay.de']
jsonrpc.Executor/3::WARNING::2016-06-25
19:40:02,520::storageServer::370::Storage.StorageServer.MountC
onnection::(_get_backup_servers_option) gluster server
u'glusterfs.rxmgmt.databay.de' is not in bricks
 ['microcloud21.rxmgmt.databay.de', 'microcloud24.rxmgmt.databay.de',
'microcloud27.rxmgmt.databay.de'
], possibly mounting duplicate servers
jsonrpc.Executor/3::DEBUG::2016-06-25
19:40:02,520::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bi
n/taskset --cpu-list 0-7 /usr/bin/sudo -n /usr/bin/systemd-run --scope
--slice=3Dvdsm-glusterfs /usr/bin
/mount -o
backup-volfile-servers=3Dmicrocloud21.rxmgmt.databay.de:microcloud24.rxmg=
mt.databay.de:microcl
oud27.rxmgmt.databay.de glusterfs.rxmgmt.databay.de:/engine
/rhev/data-center/mnt/glusterSD/glusterfs.
rxmgmt.databay.de:_engine (cwd None)
jsonrpc.Executor/3::ERROR::2016-06-25
19:40:02,540::hsm::2473::Storage.HSM::(connectStorageServer) Cou
ld not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2470, in connectStorageServ=
er
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 237, in connect
    six.reraise(t, v, tb)
  File "/usr/share/vdsm/storage/storageServer.py", line 229, in connect
    self._mount.mount(self.options, self._vfsType, cgroup=3Dself.CGROUP)
  File "/usr/share/vdsm/storage/mount.py", line 225, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 241, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';Running scope as unit run-13461.scope.\nmount.nfs: an
incorrect mount option was specified\n')

So the mount is tried as NFS which hasn't the option "-o
backup-volfile-servers=3D...".

As a consequence the host is disabled in engine. The only way to get it
up is mounting the volume manually to
/rhev/data-center/mnt/glusterSD/glusterfs.
rxmgmt.databay.de:_engine and activate it manually in Management-GUI.

Should and can I change the hosted_storage entry-point globally to i.e.
"microcloud21.rxmgmt.databay.de" or wouldn't it be better globally that
VDSM uses "-t glusterfs" to try to mount the gluster-Volume regardless
which DNS Name for the gluster-service is used ?

Ovirt is:

ovirt-release36.noarch                             1:3.6.6-1

Bye
--=20

*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs@databay.de* <mailto:rs@databay.de>
	  =09
*Databay AG*
Jens-Otto-Krag-Stra=C3=9Fe 11
D-52146 W=C3=BCrselen
*www.databay.de* <http://www.databay.de>

Sitz/Amtsgericht Aachen =E2=80=A2 HRB:8437 =E2=80=A2 USt-IdNr.: DE 210844=
202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Klaus Scholzen (RA)

------------------------------------------------------------------------

--------------490BE4DF07D0809CD094F385
Content-Type: multipart/related;
 boundary="------------9EB1620A3DC18A1D6499F226"

--------------9EB1620A3DC18A1D6499F226
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hello,</p>
    <p>I think options for mounting the hosted-engine Volume changed in
      latest vdsm to support mounting from backup-volfile-servers.</p>
    <p><tt>[root@microcloud28 ~]# rpm -qi vdsm</tt><tt><br>
      </tt><tt>Name        : vdsm</tt><tt><br>
      </tt><tt>Version     : 4.17.28</tt><tt><br>
      </tt><tt>Release     : 1.el7</tt><tt><br>
      </tt><tt>Architecture: noarch</tt><tt><br>
      </tt><tt>Install Date: Fri 10 Jun 2016 11:17:37 AM CEST</tt><tt><br>
      </tt><tt>Group       : Applications/System</tt><tt><br>
      </tt><tt>Size        : 3828639</tt><tt><br>
      </tt><tt>License     : GPLv2+</tt><tt><br>
      </tt><tt>Signature   : RSA/SHA1, Fri 03 Jun 2016 12:53:20 AM CEST,
        Key ID 7aebbe8261e8806c</tt><tt><br>
      </tt><tt>Source RPM  : vdsm-4.17.28-1.el7.src.rpm</tt><tt><br>
      </tt></p>
    <p>Now my hosts have problems to mount the Volume. On hosted-engine
      setup I configured the (Replica 3) volume to be
      "glusterfs.rxmgmt.databay.de:/engine" which ist a Round-Robin DNS
      to my gluster hosts and _not_ the DNS-Name of any gluster-brick.<br>
    </p>
    <p>Now VDSM logs:</p>
    <p><tt>jsonrpc.Executor/3::DEBUG::2016-06-25
        19:40:02,520::fileUtils::143::Storage.fileUtils::(createdir) Cre</tt><tt><br>
      </tt><tt>ating directory:
        /rhev/data-center/mnt/glusterSD/glusterfs.rxmgmt.databay.de:_engine
        mode: None</tt><tt><br>
      </tt><tt>jsonrpc.Executor/3::DEBUG::2016-06-25
        19:40:02,520::storageServer::364::Storage.StorageServer.MountCon</tt><tt><br>
      </tt><tt>nection::(_get_backup_servers_option) Using bricks:
        ['microcloud21.rxmgmt.databay.de', 'microcloud24.r</tt><tt><br>
      </tt><tt>xmgmt.databay.de', 'microcloud27.rxmgmt.databay.de']</tt><tt><br>
      </tt><tt>jsonrpc.Executor/3::WARNING::2016-06-25
        19:40:02,520::storageServer::370::Storage.StorageServer.MountC</tt><tt><br>
      </tt><tt>onnection::(_get_backup_servers_option) gluster server
        u'glusterfs.rxmgmt.databay.de' is not in bricks</tt><tt><br>
      </tt><tt> ['microcloud21.rxmgmt.databay.de',
        'microcloud24.rxmgmt.databay.de',
        'microcloud27.rxmgmt.databay.de'</tt><tt><br>
      </tt><tt>], possibly mounting duplicate servers</tt><tt><br>
      </tt><tt>jsonrpc.Executor/3::DEBUG::2016-06-25
        19:40:02,520::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bi</tt><tt><br>
      </tt><tt>n/taskset --cpu-list 0-7 /usr/bin/sudo -n
        /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin</tt><tt><br>
      </tt><tt>/mount -o
backup-volfile-servers=microcloud21.rxmgmt.databay.de:microcloud24.rxmgmt.databay.de:microcl</tt><tt><br>
      </tt><tt>oud27.rxmgmt.databay.de
        glusterfs.rxmgmt.databay.de:/engine
        /rhev/data-center/mnt/glusterSD/glusterfs.</tt><tt><br>
      </tt><tt>rxmgmt.databay.de:_engine (cwd None)</tt><tt><br>
      </tt><tt>jsonrpc.Executor/3::ERROR::2016-06-25
        19:40:02,540::hsm::2473::Storage.HSM::(connectStorageServer) Cou</tt><tt><br>
      </tt><tt>ld not connect to storageServer</tt><tt><br>
      </tt><tt>Traceback (most recent call last):</tt><tt><br>
      </tt><tt>  File "/usr/share/vdsm/storage/hsm.py", line 2470, in
        connectStorageServer</tt><tt><br>
      </tt><tt>    conObj.connect()</tt><tt><br>
      </tt><tt>  File "/usr/share/vdsm/storage/storageServer.py", line
        237, in connect</tt><tt><br>
      </tt><tt>    six.reraise(t, v, tb)</tt><tt><br>
      </tt><tt>  File "/usr/share/vdsm/storage/storageServer.py", line
        229, in connect</tt><tt><br>
      </tt><tt>    self._mount.mount(self.options, self._vfsType,
        cgroup=self.CGROUP)</tt><tt><br>
      </tt><tt>  File "/usr/share/vdsm/storage/mount.py", line 225, in
        mount</tt><tt><br>
      </tt><tt>    return self._runcmd(cmd, timeout)</tt><tt><br>
      </tt><tt>  File "/usr/share/vdsm/storage/mount.py", line 241, in
        _runcmd</tt><tt><br>
      </tt><tt>    raise MountError(rc, ";".join((out, err)))</tt><tt><br>
      </tt><tt>MountError: (32, ';Running scope as unit
        run-13461.scope.\nmount.nfs: an incorrect mount option was
        specified\n')</tt><tt><br>
      </tt></p>
    <p>So the mount is tried as NFS which hasn't the option "-o
      backup-volfile-servers=...".</p>
    <p>As a consequence the host is disabled in engine. The only way to
      get it up is mounting the volume manually to
      /rhev/data-center/mnt/glusterSD/glusterfs.<br>
      rxmgmt.databay.de:_engine and activate it manually in
      Management-GUI.<br>
    </p>
    <p>Should and can I change the hosted_storage entry-point globally
      to i.e. "microcloud21.rxmgmt.databay.de" or wouldn't it be better
      globally that VDSM uses "-t glusterfs" to try to mount the
      gluster-Volume regardless which DNS Name for the gluster-service
      is used ?</p>
    <p>Ovirt is:</p>
    <p><tt>ovirt-release36.noarch                             1:3.6.6-1</tt><br>
    </p>
    Bye<br>
    <div class="moz-signature">-- <br>
      <p>
      </p>
      <table border="0" cellpadding="0" cellspacing="0">
        <tbody>
          <tr>
            <td colspan="3"><img
                src="cid:part1.2FE99626.D833EF80@databay.de" height="30"
                border="0" width="151"></td>
          </tr>
          <tr>
            <td valign="top"> <font face="Verdana, Arial, sans-serif"
                size="-1"><br>
                <b>Ralf Schenk</b><br>
                fon +49 (0) 24 05 / 40 83 70<br>
                fax +49 (0) 24 05 / 40 83 759<br>
                mail <a href="mailto:rs@databay.de"><font
                    color="#FF0000"><b>rs@databay.de</b></font></a><br>
              </font> </td>
            <td width="30"> </td>
            <td valign="top"> <font face="Verdana, Arial, sans-serif"
                size="-1"><br>
                <b>Databay AG</b><br>
                Jens-Otto-Krag-Straße 11<br>
                D-52146 Würselen<br>
                <a href="http://www.databay.de"><font color="#FF0000"><b>www.databay.de</b></font></a>
              </font> </td>
          </tr>
          <tr>
            <td colspan="3" valign="top"> <font face="Verdana, Arial,
                sans-serif" size="1"><br>
                Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE
                210844202<br>
                Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch
                Yavari, Dipl.-Kfm. Philipp Hermanns<br>
                Aufsichtsratsvorsitzender: Klaus Scholzen (RA) </font>
            </td>
          </tr>
        </tbody>
      </table>
      <hr color="#000000" noshade="noshade" size="1" width="100%">
    </div>
  </body>
</html>

--------------9EB1620A3DC18A1D6499F226
Content-Type: image/gif;
 name="logo_databay_email.gif"
Content-Transfer-Encoding: base64
Content-ID: <part1.2FE99626.D833EF80@databay.de>
Content-Disposition: inline;
 filename="logo_databay_email.gif"

R0lGODlhlwAeAMQAAObm5v9QVf/R0oKBgfDw8NfX105MTLi3t/r6+sfHx/+rrf98gC0sLP8L
EhIQEKalpf/g4ZmYmHd2dmppaf8uNP/y8v8cIv+Ym//AwkE/P46NjRwbG11cXP8ABwUDA///
/yH5BAAAAAAALAAAAACXAB4AAAX/4CeOYnUJZKqubOu+cCzPNA0tVnfVfO//wGAKk+t0Ap+K
QMFUYCDCqHRKJVUWDaPRUsFktZ1G4AKtms9o1gKsFVS+7I5ll67bpd647hPQawNld4KDMQJF
bA07F35aFBiEkJEpfXEBjx8KjI0Vkp2DEIdaCySgFBShbEgrCQOtrq+uEQcALQewrQUjEbe8
rgkkD7y5KhMZB3drqSoVFQhdlHGXKQYe1dbX2BvHKwzY1RMiAN7j1xEjBeTmKeIeD3cYCxRf
FigvChRxFJwkBBvk5A7cpZhAjgGCDwn+kfslgto4CSoSehh2BwEEBQvowDAUR0EKdArHZTg4
4oDCXBFC/3qj9SEluZEpHnjYQFIGgpo1KgSasYjNKBImrzF4NaFbNgIjCGRQeIyVKwneOLzS
cLCAg38OWI4Y4GECgQcSOEwYcADnh6/FNjAwoGFYAQ0atI4AAFeEFwsLFLiJUQEfGH0kNGAD
x8+oNQdIRQg+7NCaOhIgD8sVgYADNsPVGI5YWjRqzQTdHDDIYHRDLokaUhCglkFEJi0NKJhl
0RP2TsvXUg88KiLBVWsZrF6DmMKlNYMqglqTik1guN8OBgAgkGCpB+L9ugK4iSCBvwEfECw1
kILrBpa1jVCQIQBRvbP+rlEcQVAoSevWyv6uhpwE12uEkQAAZucpVw1xIsjkgf8B863mQVYt
eQATCZYJZJ5WBfij2wfpHcEeHGG8Z+BMszVWDXkfKLhceJhBSAJ+1ThH32AfRFZNayNAtUFi
wFSTSwEHJIYAAQU84IADwyjIEALU9MchG+vFgIF7W2GDI2T7HfjBgNcgKQKMHmwjgnCSpeCb
ULRkdxhF1CDY40RjgmUAA/v1J5FAKW2gGSZscBFDMraNgJs1AYpAAGYP5jJoNQ4Y4Gh8jpFg
HH9mgbmWo1l6oA4C3Ygp6UwEIFBfNRtkMIBlKMLnAXgAXLWhXXH85EIFqMhGGZgDEKArABGA
ed0HI4bk5qgnprCYSt88B6dqS0FEEAMPJDCdCJYViur/B1BlwGMJqDTwnhqxJgUpo0ceOQ4D
0yEakpMm/jqCRMgWm2I1j824Y6vLvuuPjHnqOJkIgP6xzwp5sCFNsCFp88Gxh11lrjfDcNrc
CEx64/CD3iAHlQcMUEQXvcA+qBkBB4Q2X1CusjBlJdKMYAKI6g28MbKN5hJsBAXknHOwutn4
oFYqkpqAzjnPbE0u1PxmwAQGXLWBbvhuIIEGEnRjlAHO4SvhbCNAkwoGzEBwgV9U0lfu2WiX
OkDEGaCdKgl0nk2YkWdPOCDabvaGdkAftL1LlgwCM+7Tq11V71IO7LkM2XE0YAHMYMhqqK6U
V165CpaHukLmiXFO8XSVzzakX+UH6TrmAajPNxfqByTQec41AeBPvSwIALkmAnuiexCsca3C
BajgfsROuxcPA8kHQJX4DAIwjnsAvhsvfXHWKEwDAljg7sj03L9wwAQTxOWD2AE0YP75eCkw
cPfs+xACADs=
--------------9EB1620A3DC18A1D6499F226--

--------------490BE4DF07D0809CD094F385--

Ralf Schenk

Ralf Schenk

tags

participants (1)