Re: [ovirt-users] [Gluster-users] 4.0 - 2nd node fails on deploy

Wednesday, 5 October 2016

------4VSAMIL16A4UO07LK9ROSU1PLADG1A
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: quoted-printable

"no route to host" is a network problem. Looks like quorum loss is approp=
riate.=20

On October 5, 2016 12:31:18 PM GMT+02:00, Sahina Bose <sabose(a)redhat.com&gt;=
 wrote:
...
On Wed, Oct 5, 2016 at 1:56 PM, Jason Jeffrey <jason(a)sudo.co.uk&gt;
wrote:

> HI,
>
>
>
> Logs attached
>

Have you probed 2 interfaces for same host, that is - dcasrv02 and
dcastor02? Does "gluster peer status" understand both names as for same
host?

From glusterd logs and the mount logs - the connection between the
peers is
lost, and quorum is lost, which is reaffirming what Simone said
earlier.
Logs seem to indicate network issues - check the direct link setup. See
below

From mount logs:
[2016-10-04 17:26:15.718300] E [socket.c:2292:socket_connect_finish]
0-engine-client-2: connection to 10.100.103.3:24007 failed (No route to
host)
[2016-10-04 17:26:15.718345] W [MSGID: 108001]
[afr-common.c:4379:afr_notify] 0-engine-replicate-0: Client-quorum is
not
met
[2016-10-04 17:26:16.428290] E [socket.c:2292:socket_connect_finish]
0-engine-client-1: connection to 10.100.101.2:24007 failed (No route to
host)
[2016-10-04 17:26:16.428336] E [MSGID: 108006]
[afr-common.c:4321:afr_notify] 0-engine-replicate-0: All subvolumes are
down. Going offline until atleast one of them comes back up

And in glusterd logs:
[2016-10-04 17:24:39.522402] E [socket.c:2292:socket_connect_finish]
0-management: connection to 10.100.50.82:24007 failed (No route to
host)
[2016-10-04 17:24:39.522578] I [MSGID: 106004]
[glusterd-handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer
<dcasrv02> (<1e788fc9-dfe9-4753-92c7-76a95c8d0891>), in state <Peer in
Cluster>, has disconnected from glusterd.
[2016-10-04 17:24:39.523272] C [MSGID: 106002]
[glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action]
0-management: Server quorum lost for volume engine. Stopping local
bricks.
[2016-10-04 17:24:39.523314] I [MSGID: 106132]
[glusterd-utils.c:1560:glusterd_service_stop] 0-management: brick
already
stopped
[2016-10-04 17:24:39.526188] E [socket.c:2292:socket_connect_finish]
0-management: connection to 10.100.103.3:24007 failed (No route to
host)
[2016-10-04 17:24:39.526219] I [MSGID: 106004]
[glusterd-handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer
<dcastor03> (<9a9c037e-96cd-4f73-9800-a1df5cdd2818>), in state <Peer in
Cluster>, has disconnected from glusterd.

> Thanks
>
>
>
> *From:* Sahina Bose [mailto:sabose@redhat.com]
> *Sent:* 05 October 2016 08:11
> *To:* Jason Jeffrey <jason(a)sudo.co.uk>; gluster-users(a)gluster.org;
> Ravishankar Narayanankutty <ravishankar(a)redhat.com&gt;
> *Cc:* Simone Tiraboschi <stirabos(a)redhat.com>; users
<users(a)ovirt.org&gt;
>
> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
>
>
>
> [Adding gluster-users ML]
>
> The brick logs are filled with errors :
> [2016-10-05 19:30:28.659061] E [MSGID: 113077]
[posix-handle.c:309:posix_handle_pump]
> 0-engine-posix: malformed internal link /var/run/vdsm/storage/
>
0a021563-91b5-4f49-9c6b-fff45e85a025/d84f0551-0f2b-457c-808c-6369c6708d4= 3/
...
> 1b5a5e34-818c-4914-8192-2f05733b5583 for /xpool/engine/brick/.
> glusterfs/b9/8e/b98ed8d2-3bf9-4b11-92fd-ca5324e131a8
> [2016-10-05 19:30:28.659069] E [MSGID: 113091]
[posix.c:180:posix_lookup]
> 0-engine-posix: Failed to create inode handle for path
> <gfid:b98ed8d2-3bf9-4b11-92fd-ca5324e131a8>
> The message "E [MSGID: 113018] [posix.c:198:posix_lookup]
0-engine-posix:
> lstat on null failed" repeated 3 times between [2016-10-05
19:30:28.656529]
> and [2016-10-05 19:30:28.659076]
> [2016-10-05 19:30:28.659087] W [MSGID: 115005]
> [server-resolve.c:126:resolve_gfid_cbk] 0-engine-server:
> b98ed8d2-3bf9-4b11-92fd-ca5324e131a8: failed to resolve (Success)
>
> - Ravi, the above are from the data brick of the arbiter volume. Can
you
> take a look?
>
>
>
> Jason,
>
> Could you also provide the mount logs from the first host
> (/var/log/glusterfs/rhev-data-center-mnt-glusterSD*engine.log) and
> glusterd log (/var/log/glusterfs/etc-glusterfs-glusterd.vol.log)
around
> the same time frame.
>
>
>
>
>
> On Wed, Oct 5, 2016 at 3:28 AM, Jason Jeffrey <jason(a)sudo.co.uk&gt;
wrote:
>
> Hi,
>
>
>
> Servers are powered  off  when I=E2=80=99m not looking at the problem.
>
>
>
> There may have been instances where all three were not powered on,
during
> the same period.
>
>
>
> Glusterhd log attached, the xpool-engine-brick log is over 1 GB in
size,
> I=E2=80=99ve taken a sample of the last  couple days, looks to be high= ly
...
repative.
>
>
>
> Cheers
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
> *From:* Simone Tiraboschi [mailto:stirabos@redhat.com]
> *Sent:* 04 October 2016 16:50
>
>
> *To:* Jason Jeffrey <jason(a)sudo.co.uk&gt;
> *Cc:* users <users(a)ovirt.org&gt;
> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
>
>
>
>
>
>
>
> On Tue, Oct 4, 2016 at 5:22 PM, Jason Jeffrey <jason(a)sudo.co.uk&gt;
wrote:
>
> Hi,
>
>
>
> DCASTORXX is a hosts entry for dedicated  direct 10GB links (each
private
> /28) between the x3 servers  i.e 1=3D> 2&3, 2=3D> 1&3, etc) planned to=
 be
...
used
> solely for storage.
>
>
>
> I,e
>
>
>
> 10.100.50.81    dcasrv01
>
> 10.100.101.1    dcastor01
>
> 10.100.50.82    dcasrv02
>
> 10.100.101.2    dcastor02
>
> 10.100.50.83    dcasrv03
>
> 10.100.103.3    dcastor03
>
>
>
> These were setup with the gluster commands
>
>
>
> =C2=B7         gluster volume create iso replica 3 arbiter 1
> dcastor01:/xpool/iso/brick   dcastor02:/xpool/iso/brick
> dcastor03:/xpool/iso/brick
>
> =C2=B7         gluster volume create export replica 3 arbiter 1
> dcastor02:/xpool/export/brick  dcastor03:/xpool/export/brick
> dcastor01:/xpool/export/brick
>
> =C2=B7         gluster volume create engine replica 3 arbiter 1
> dcastor01:/xpool/engine/brick dcastor02:/xpool/engine/brick
> dcastor03:/xpool/engine/brick
>
> =C2=B7         gluster volume create data replica 3 arbiter 1
> dcastor01:/xpool/data/brick  dcastor03:/xpool/data/brick
> dcastor02:/xpool/data/bricky
>
>
>
>
>
> So yes, DCASRV01 is the server (pri) and have local bricks access
through
> DCASTOR01 interface
>
>
>
> Is the issue here not the incorrect soft link ?
>
>
>
> No, this should be fine.
>
>
>
> The issue is that periodically your gluster volume losses its server
> quorum and become unavailable.
>
> It happened more than once from your logs.
>
>
>
> Can you please attach also gluster logs for that volume?
>
>
>
>
>
> lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 hosted-engine.metadata ->
>
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a= -
...
> 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93
>
> [root@dcasrv01 /]# ls -al
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-
> 76a4876ecaaf/
>
> ls: cannot access
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/:
> No such file or directory
>
> But the data does exist
>
> [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al
>
> drwxr-xr-x. 2 vdsm kvm    4096 Oct  3 17:17 .
>
> drwxr-xr-x. 6 vdsm kvm    4096 Oct  3 17:17 ..
>
> -rw-rw----. 2 vdsm kvm 1028096 Oct  3 20:48 cee9440c-4eb8-453b-bc04-
> c47e6f9cbc93
>
> -rw-rw----. 2 vdsm kvm 1048576 Oct  3 17:17 cee9440c-4eb8-453b-bc04-
> c47e6f9cbc93.lease
>
> -rw-r--r--. 2 vdsm kvm     283 Oct  3 17:17
cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta
>
>
>
>
> Thanks
>
>
>
> Jason
>
>
>
>
>
>
>
> *From:* Simone Tiraboschi [mailto:stirabos@redhat.com]
> *Sent:* 04 October 2016 14:40
>
>
> *To:* Jason Jeffrey <jason(a)sudo.co.uk&gt;
> *Cc:* users <users(a)ovirt.org&gt;
> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
>
>
>
>
>
>
>
> On Tue, Oct 4, 2016 at 10:51 AM, Simone Tiraboschi
<stirabos(a)redhat.com&gt;
> wrote:
>
>
>
>
>
> On Mon, Oct 3, 2016 at 11:56 PM, Jason Jeffrey <jason(a)sudo.co.uk&gt;
wrote:
>
> Hi,
>
>
>
> Another problem has appeared, after rebooting the primary the VM will
not
> start.
>
>
>
> Appears the symlink is broken between gluster mount ref and vdsm
>
>
>
> The first host was correctly deployed but it seas that you are facing
some
> issue connecting the storage.
>
> Can you please attach vdsm logs and /var/log/messages from the first
host?
>
>
>
> Thanks Jason,
>
> I suspect that your issue is related to this:
>
> Oct  4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]:
[2016-10-04
> 17:24:39.522620] C [MSGID: 106002] [glusterd-server-quorum.c:351:
> glusterd_do_volume_quorum_action] 0-management: Server quorum lost
for
> volume data. Stopping local bricks.
>
> Oct  4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]:
[2016-10-04
> 17:24:39.523272] C [MSGID: 106002] [glusterd-server-quorum.c:351:
> glusterd_do_volume_quorum_action] 0-management: Server quorum lost
for
> volume engine. Stopping local bricks.
>
>
>
> and for some time your gluster volume has been working.
>
>
>
> But then:
>
> Oct  4 19:02:09 dcasrv01 systemd: Started /usr/bin/mount -t glusterfs
-o
> backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine
> /rhev/data-center/mnt/glusterSD/dcastor01:engine.
>
> Oct  4 19:02:09 dcasrv01 systemd: Starting /usr/bin/mount -t
glusterfs -o
> backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine
> /rhev/data-center/mnt/glusterSD/dcastor01:engine.
>
> Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-
> packages/yajsonrpc/stomp.py:352: DeprecationWarning:
Dispatcher.pending
> is deprecated. Use Dispatcher.socket.pending instead.
>
> Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D
getattr(dispatcher,
> 'pending', lambda: 0)
>
> Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-
> packages/yajsonrpc/stomp.py:352: DeprecationWarning:
Dispatcher.pending
> is deprecated. Use Dispatcher.socket.pending instead.
>
> Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D
getattr(dispatcher,
> 'pending', lambda: 0)
>
> Oct  4 19:02:11 dcasrv01 journal: vdsm vds.dispatcher ERROR SSL error
> during reading data: unexpected eof
>
> Oct  4 19:02:11 dcasrv01 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to
> storage server failed' - trying to restart agent
>
> Oct  4 19:02:11 dcasrv01 ovirt-ha-agent:
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error:
> 'Connection to storage server failed' - trying to restart agent
>
> Oct  4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]:
[2016-10-04
> 18:02:12.384611] C [MSGID: 106003] [glusterd-server-quorum.c:346:
> glusterd_do_volume_quorum_action] 0-management: Server quorum
regained
> for volume data. Starting local bricks.
>
> Oct  4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]:
[2016-10-04
> 18:02:12.388981] C [MSGID: 106003] [glusterd-server-quorum.c:346:
> glusterd_do_volume_quorum_action] 0-management: Server quorum
regained
> for volume engine. Starting local bricks.
>
>
>
> And at that point VDSM started complaining that the
hosted-engine-storage
> domain doesn't exist anymore:
>
> Oct  4 19:02:30 dcasrv01 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.lib.image.Image ERROR Error fetching volumes
list:
> Storage domain does not exist:
(u'bbb70623-194a-46d2-a164-76a4876ecaaf',)
>
> Oct  4 19:02:30 dcasrv01 ovirt-ha-agent:
ERROR:ovirt_hosted_engine_ha.lib.image.Image:Error
> fetching volumes list: Storage domain does not exist:
> (u'bbb70623-194a-46d2-a164-76a4876ecaaf',)
>
>
>
> I see from the logs that the ovirt-ha-agent is trying to mount the
> hosted-engine storage domain as:
>
> /usr/bin/mount -t glusterfs -o
backup-volfile-servers=3Ddcastor02:dcastor03
> dcastor01:engine /rhev/data-center/mnt/glusterSD/dcastor01:engine.
>
>
>
> Pointing to dcastor01, dcastor02 and dcastor03 while your server is
> dcasrv01.
>
> But at the same time it seams that also dcasrv01 has local bricks for
the
> same engine volume.
>
>
>
> So, is dcasrv01 just an alias fro dcastor01? if not you probably have
some
> issue with the configuration of your gluster volume.
>
>
>
>
>
>
>
> From broker.log
>
>
>
> Thread-169::ERROR::2016-10-04 22:44:16,189::storage_broker::138::
> ovirt_hosted_engine_ha.broker.storage_broker.
> StorageBroker::(get_raw_stats_for_service_type) Failed to read
metadata
> from /rhev/data-center/mnt/glusterSD/dcastor01:engine/
> bbb70623-194a-46d2-a164-76a4876ecaaf/ha_agent/hosted-engine.metadata
>
>
>
> [root@dcasrv01 ovirt-hosted-engine-ha]# ls -al /rhev/data-center/mnt/
>
glusterSD/dcastor01\:engine/bbb70623-194a-46d2-a164-76a4876ecaaf/ha_agen= t/
...
>
> total 9
>
> drwxrwx---. 2 vdsm kvm 4096 Oct  3 17:27 .
>
> drwxr-xr-x. 5 vdsm kvm 4096 Oct  3 17:17 ..
>
> lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 hosted-engine.lockspace ->
>
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/23d81b73-bcb7= -
...
> 4742-abde-128522f43d78/11d6a3e1-1817-429d-b2e0-9051a3cf41a4
>
> lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 hosted-engine.metadata ->
>
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a= -
...
> 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93
>
>
>
> [root@dcasrv01 /]# ls -al
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-
> 76a4876ecaaf/
>
> ls: cannot access
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/:
> No such file or directory
>
>
>
> Though file appears to be there
>
>
>
> Gluster is setup as xpool/engine
>
>
>
> [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# pwd
>
> /xpool/engine/brick/bbb70623-194a-46d2-a164-76a4876ecaaf/
> images/fd44dbf9-473a-496a-9996-c8abe3278390
>
> [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al
>
> total 2060
>
> drwxr-xr-x. 2 vdsm kvm    4096 Oct  3 17:17 .
>
> drwxr-xr-x. 6 vdsm kvm    4096 Oct  3 17:17 ..
>
> -rw-rw----. 2 vdsm kvm 1028096 Oct  3 20:48 cee9440c-4eb8-453b-bc04-
> c47e6f9cbc93
>
> -rw-rw----. 2 vdsm kvm 1048576 Oct  3 17:17 cee9440c-4eb8-453b-bc04-
> c47e6f9cbc93.lease
>
> -rw-r--r--. 2 vdsm kvm     283 Oct  3 17:17
cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta
>
>
>
>
>
>
> [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume
info
>
>
>
> Volume Name: data
>
> Type: Replicate
>
> Volume ID: 54fbcafc-fed9-4bce-92ec-fa36cdcacbd4
>
> Status: Started
>
> Number of Bricks: 1 x (2 + 1) =3D 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: dcastor01:/xpool/data/brick
>
> Brick2: dcastor03:/xpool/data/brick
>
> Brick3: dcastor02:/xpool/data/bricky (arbiter)
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> performance.quick-read: off
>
> performance.read-ahead: off
>
> performance.io-cache: off
>
> performance.stat-prefetch: off
>
> cluster.eager-lock: enable
>
> network.remote-dio: enable
>
> cluster.quorum-type: auto
>
> cluster.server-quorum-type: server
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
>
>
> Volume Name: engine
>
> Type: Replicate
>
> Volume ID: dd4c692d-03aa-4fc6-9011-a8dad48dad96
>
> Status: Started
>
> Number of Bricks: 1 x (2 + 1) =3D 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: dcastor01:/xpool/engine/brick
>
> Brick2: dcastor02:/xpool/engine/brick
>
> Brick3: dcastor03:/xpool/engine/brick (arbiter)
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> performance.quick-read: off
>
> performance.read-ahead: off
>
> performance.io-cache: off
>
> performance.stat-prefetch: off
>
> cluster.eager-lock: enable
>
> network.remote-dio: enable
>
> cluster.quorum-type: auto
>
> cluster.server-quorum-type: server
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
>
>
> Volume Name: export
>
> Type: Replicate
>
> Volume ID: 23f14730-d264-4cc2-af60-196b943ecaf3
>
> Status: Started
>
> Number of Bricks: 1 x (2 + 1) =3D 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: dcastor02:/xpool/export/brick
>
> Brick2: dcastor03:/xpool/export/brick
>
> Brick3: dcastor01:/xpool/export/brick (arbiter)
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
>
>
> Volume Name: iso
>
> Type: Replicate
>
> Volume ID: b2d3d7e2-9919-400b-8368-a0443d48e82a
>
> Status: Started
>
> Number of Bricks: 1 x (2 + 1) =3D 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: dcastor01:/xpool/iso/brick
>
> Brick2: dcastor02:/xpool/iso/brick
>
> Brick3: dcastor03:/xpool/iso/brick (arbiter)
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
>
>
>
>
> [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume
> status
>
> Status of volume: data
>
> Gluster process                             TCP Port  RDMA Port=20
Online
> Pid
>
> ------------------------------------------------------------
> ------------------
>
> Brick dcastor01:/xpool/data/brick           49153     0          Y
> 3076
>
> Brick dcastor03:/xpool/data/brick           49153     0          Y
> 3019
>
> Brick dcastor02:/xpool/data/bricky          49153     0          Y
> 3857
>
> NFS Server on localhost                     2049      0          Y
>     3097
>
> Self-heal Daemon on localhost               N/A       N/A        Y
> 3088
>
> NFS Server on dcastor03                     2049      0          Y
> 3039
>
> Self-heal Daemon on dcastor03               N/A       N/A        Y
> 3114
>
> NFS Server on dcasrv02                      2049      0          Y
> 3871
>
> Self-heal Daemon on dcasrv02                N/A       N/A        Y
> 3864
>
>
>
> Task Status of Volume data
>
> ------------------------------------------------------------
> ------------------
>
> There are no active volume tasks
>
>
>
> Status of volume: engine
>
> Gluster process                             TCP Port  RDMA Port=20
Online
> Pid
>
> ------------------------------------------------------------
> ------------------
>
> Brick dcastor01:/xpool/engine/brick         49152     0          Y
> 3131
>
> Brick dcastor02:/xpool/engine/brick         49152     0          Y
> 3852
>
> Brick dcastor03:/xpool/engine/brick         49152     0          Y
> 2992
>
> NFS Server on localhost                     2049      0          Y
> 3097
>
> Self-heal Daemon on localhost               N/A       N/A        Y
> 3088
>
> NFS Server on dcastor03                     2049      0          Y
> 3039
>
> Self-heal Daemon on dcastor03               N/A       N/A        Y
> 3114
>
> NFS Server on dcasrv02                      2049      0          Y
> 3871
>
> Self-heal Daemon on dcasrv02                N/A       N/A        Y
> 3864
>
>
>
> Task Status of Volume engine
>
> ------------------------------------------------------------
> ------------------
>
> There are no active volume tasks
>
>
>
> Status of volume: export
>
> Gluster process                             TCP Port  RDMA Port=20
Online
> Pid
>
> ------------------------------------------------------------
> ------------------
>
> Brick dcastor02:/xpool/export/brick         49155     0          Y
> 3872
>
> Brick dcastor03:/xpool/export/brick         49155     0          Y
> 3147
>
> Brick dcastor01:/xpool/export/brick         49155     0          Y
> 3150
>
> NFS Server on localhost                     2049      0          Y
> 3097
>
> Self-heal Daemon on localhost               N/A       N/A        Y
> 3088
>
> NFS Server on dcastor03                     2049      0          Y
> 3039
>
> Self-heal Daemon on dcastor03               N/A       N/A        Y
> 3114
>
> NFS Server on dcasrv02                      2049      0          Y
> 3871
>
> Self-heal Daemon on dcasrv02                N/A       N/A        Y
> 3864
>
>
>
> Task Status of Volume export
>
> ------------------------------------------------------------
> ------------------
>
> There are no active volume tasks
>
>
>
> Status of volume: iso
>
> Gluster process                             TCP Port  RDMA Port=20
Online
> Pid
>
> ------------------------------------------------------------
> ------------------
>
> Brick dcastor01:/xpool/iso/brick            49154     0          Y
> 3152
>
> Brick dcastor02:/xpool/iso/brick            49154     0          Y
> 3881
>
> Brick dcastor03:/xpool/iso/brick            49154     0          Y
> 3146
>
> NFS Server on localhost                     2049      0          Y
> 3097
>
> Self-heal Daemon on localhost               N/A       N/A        Y
> 3088
>
> NFS Server on dcastor03                     2049      0          Y
> 3039
>
> Self-heal Daemon on dcastor03               N/A       N/A        Y
> 3114
>
> NFS Server on dcasrv02                      2049      0          Y
> 3871
>
> Self-heal Daemon on dcasrv02                N/A       N/A        Y
> 3864
>
>
>
> Task Status of Volume iso
>
> ------------------------------------------------------------
> ------------------
>
> There are no active volume tasks
>
>
>
>
> Thanks
>
>
>
> Jason
>
>
>
>
>
>
>
> *From:* users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] *On
> Behalf Of *Jason Jeffrey
> *Sent:* 03 October 2016 18:40
> *To:* users(a)ovirt.org
>
>
> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
>
>
>
> Hi,
>
>
>
> Setup log attached for primary
>
>
>
> Regards
>
>
>
> Jason
>
>
>
> *From:* Simone Tiraboschi [mailto:stirabos@redhat.com
> <stirabos(a)redhat.com>]
> *Sent:* 03 October 2016 09:27
> *To:* Jason Jeffrey <jason(a)sudo.co.uk&gt;
> *Cc:* users <users(a)ovirt.org&gt;
> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
>
>
>
>
>
>
>
> On Mon, Oct 3, 2016 at 12:45 AM, Jason Jeffrey <jason(a)sudo.co.uk&gt;
wrote:
>
> Hi,
>
>
>
> I am trying to build a x3 HC cluster, with a self hosted engine using
> gluster.
>
>
>
> I have successful built the 1st node,  however when I attempt to run
> hosted-engine =E2=80=93deploy on node 2, I get the following error
>
>
>
> [WARNING] A configuration file must be supplied to deploy Hosted
Engine on
> an additional host.
>
> [ ERROR ] 'version' is not stored in the HE configuration image
>
> [ ERROR ] Unable to get the answer file from the shared storage
>
> [ ERROR ] Failed to execute stage 'Environment customization': Unable
to
> get the answer file from the shared storage
>
> [ INFO  ] Stage: Clean up
>
> [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-
> setup/answers/answers-20161002232505.conf'
>
> [ INFO  ] Stage: Pre-termination
>
> [ INFO  ] Stage: Termination
>
> [ ERROR ] Hosted Engine deployment failed
>
>
>
> Looking at the failure in the log file..
>
>
>
> Can you please attach hosted-engine-setup logs from the first host?
>
>
>
>
>
> 2016-10-02 23:25:05 WARNING
otopi.plugins.gr_he_common.core.remote_answerfile
> remote_answerfile._customization:151 A configuration
>
> file must be supplied to deploy Hosted Engine on an additional host.
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> remote_answerfile._fetch_answer_file:61 _fetch_answer_f
>
> ile
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> remote_answerfile._fetch_answer_file:69 fetching from:
>
>
/rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b= -
...
> fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/7
>
> 8cb2527-a2e2-489a-9fad-465a72221b37
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> heconflib._dd_pipe_tar:69 executing: 'sudo -u vdsm dd i
>
> f=3D/rhev/data-center/mnt/glusterSD/dcastor02:engine/
> 0a021563-91b5-4f49-9c6b-fff45e85a025/images/f055216c-
> 02f9-4cd1-a22c-d6b56a0a8e9b
>
> /78cb2527-a2e2-489a-9fad-465a72221b37 bs=3D4k'
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> heconflib._dd_pipe_tar:70 executing: 'tar -tvf -'
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> heconflib._dd_pipe_tar:88 stdout:
>
> 2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_common.core.remote_answerfile
> heconflib._dd_pipe_tar:89 stderr:
>
> 2016-10-02 23:25:05 ERROR
otopi.plugins.gr_he_common.core.remote_answerfile
> heconflib.validateConfImage:111 'version' is not stored
>
> in the HE configuration image
>
> 2016-10-02 23:25:05 ERROR
otopi.plugins.gr_he_common.core.remote_answerfile
> remote_answerfile._fetch_answer_file:73 Unable to get t
>
> he answer file from the shared storage
>
>
>
> Looking at the detected gluster path - /rhev/data-center/mnt/
> glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-
> fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/
>
>
>
> [root@dcasrv02 ~]# ls -al /rhev/data-center/mnt/
> glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-
> fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/
>
> total 1049609
>
> drwxr-xr-x. 2 vdsm kvm       4096 Oct  2 04:46 .
>
> drwxr-xr-x. 6 vdsm kvm       4096 Oct  2 04:46 ..
>
> -rw-rw----. 1 vdsm kvm 1073741824 Oct  2 04:46
78cb2527-a2e2-489a-9fad-
> 465a72221b37
>
> -rw-rw----. 1 vdsm kvm    1048576 Oct  2 04:46
78cb2527-a2e2-489a-9fad-
> 465a72221b37.lease
>
> -rw-r--r--. 1 vdsm kvm        294 Oct  2 04:46
78cb2527-a2e2-489a-9fad-465a72221b37.meta
>
>
>
>
> 78cb2527-a2e2-489a-9fad-465a72221b37 is  a 1 GB file, is this the
engine
> VM ?
>
>
>
> Copying the answers file form primary
(/etc/ovirt-hosted-engine/answers.conf
> ) to  node 2 and rerunning produces the same error : (
>
> (hosted-engine --deploy  --config-append=3D/root/answers.conf )
>
>
>
> Also tried on node 3, same issues
>
>
>
> Happy to provide logs and other debugs
>
>
>
> Thanks
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>

------------------------------------------------------------------------

_______________________________________________
Gluster-users mailing list
Gluster-users(a)gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users 
--=20
Sent from my Android device with K-9 Mail. Please excuse my brevity.
------4VSAMIL16A4UO07LK9ROSU1PLADG1A
Content-Type: text/html;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head></head><body>&quot;no route to host&quot; is a
network proble=
m. Looks like quorum loss is appropriate. <br><br><div
class=3D"gmail_quo=
te">On October 5, 2016 12:31:18 PM GMT+02:00, Sahina Bose &lt;sabose@redh=
at.com&gt; wrote:<blockquote class=3D"gmail_quote" style=3D"margin:
0pt 0=
pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1e=
x;">
<div dir=3D"ltr"><br /><div
class=3D"gmail_extra"><br /><div class=3D"gma=
il_quote">On Wed, Oct 5, 2016 at 1:56 PM, Jason Jeffrey <span
dir=3D"ltr"=
...
&lt;<a target=3D"_blank"
href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.=
uk</a>&gt;</span> wrote:<br /><blockquote
style=3D"margin:0px 0px 0px 0.8=
ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmai=
l_quote"><div lang=3D"EN-GB"><div><p
class=3D"MsoNormal"><span style=3D"f=
ont-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,=
125)">HI,<u></u><u></u></span></p><p
class=3D"MsoNormal"><span style=3D"f=
ont-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,=
125)"><u></u>=C2=A0<u></u></span></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">Logs
attached</span></p></div></div></blockquote><div><br
/>Have =
you probed 2 interfaces for same host, that is - dcasrv02 and=20
dcastor02? Does &quot;gluster peer status&quot; understand both names as =
for same=20
host?<br /><br /></div><div>From glusterd logs and the mount logs
- the c=
onnection between the peers is lost, and quorum is lost, which is reaffir=
ming what Simone said earlier.<br /></div><div>Logs seem to indicate
netw=
ork issues - check the direct link setup. See below<br
/></div><div></div=
...
<div><br /></div><div>From mount logs:<br
/>[2016-10-04 17:26:15.718300]=  E [socket.c:2292:socket_connect_finish]
0-engine-client-2: connection to=
 <a href=3D"http://10.100.103.3:24007">10.100.103.3:24007</a> failed
(No =
route to host)<br />[2016-10-04 17:26:15.718345] W [MSGID: 108001] [afr-c=
ommon.c:4379:afr_notify] 0-engine-replicate-0: Client-quorum is not met<b=
r />[2016-10-04 17:26:16.428290] E [socket.c:2292:socket_connect_finish] =
0-engine-client-1: connection to <a
href=3D"http://10.100.101.2:24007">10=
.100.101.2:24007</a> failed (No route to host)<br />[2016-10-04 17:26:16.=
428336] E [MSGID: 108006] [afr-common.c:4321:afr_notify] 0-engine-replica=
te-0: All subvolumes are down. Going offline until
atleast one of them comes back up<br /><br /></div><div>And in
glusterd l=
ogs:</div>[2016-10-04 17:24:39.522402] E [socket.c:2292:socket_connect_fi=
nish] 0-management: connection to <a
href=3D"http://10.100.50.82:24007">1=
0.100.50.82:24007</a> failed (No route to host)<br />[2016-10-04 17:24:39=
.522578] I [MSGID: 106004] [glusterd-handler.c:5201:__glusterd_peer_rpc_n=
otify] 0-management: Peer &lt;dcasrv02&gt; (&lt;1e788fc9-dfe9-4753-92c7-7=
6a95c8d0891&gt;), in state &lt;Peer in Cluster&gt;, has disconnected from=
 glusterd.<br />[2016-10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-=
server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management: Serve=
r quorum lost for volume engine. Stopping local bricks.<br />[2016-10-04 =
17:24:39.523314] I [MSGID: 106132] [glusterd-utils.c:1560:glusterd_servic=
e_stop] 0-management: brick already stopped<br />[2016-10-04 17:24:39.526=
188] E [socket.c:2292:socket_connect_finish] 0-management: connection to =
<a
href=3D"http://10.100.103.3:24007">10.100.103.3:24007</a> failed (No
rout=
e to host)<br />[2016-10-04 17:24:39.526219] I [MSGID: 106004] [glusterd-=
handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer &lt;dcastor=
03&gt; (&lt;9a9c037e-96cd-4f73-9800-a1df5cdd2818&gt;), in state &lt;Peer
=
in Cluster&gt;, has disconnected from glusterd.<br /><br /><br
/><blockqu=
ote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex" class=3D"gmail_quote"><div
lang=3D"EN-GB"><div><p c=
lass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibr=
i&quot;,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p><p
clas=
s=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&q=
uot;,sans-serif;color:rgb(31,73,125)"><br
/></span></p></div></div></bloc=
kquote><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div
lang=3D"E=
N-GB"><div><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)"><u></u></span></p><p
class=3D"MsoNormal"><span style=3D"f=
ont-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,=
125)">Thanks <u></u><u></u></span></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)"><u></u>=C2=A0<u></u></span></p><p
class=3D"MsoNormal"><b><span st=
yle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif"
lang=3D=
"EN-US">From:</span></b><span
style=3D"font-size:11pt;font-family:&quot;c=
alibri&quot;,sans-serif" lang=3D"EN-US"> Sahina Bose [mailto:<a
target=3D=
"_blank"
href=3D"mailto:sabose@redhat.com">sabose@redhat.com</a>] <br
/><=
b>Sent:</b> 05 October 2016 08:11<br /><b>To:</b> Jason Jeffrey
&lt;<a ta=
rget=3D"_blank"
href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&gt;=
; <a target=3D"_blank"
href=3D"mailto:gluster-users@gluster.org">gluster-=
users(a)gluster.org&lt;/a&gt;; Ravishankar Narayanankutty &lt;<a
target=3D"_blank=
"
href=3D"mailto:ravishankar@redhat.com">ravishankar@redhat.com</a>&gt;<br
=
/><b>Cc:</b> Simone Tiraboschi &lt;<a target=3D"_blank"
href=3D"mailto:st=
irabos@redhat.com&quot;&gt;stirabos(a)redhat.com&lt;/a&gt;&amp;gt;; users &lt;<a
target=3D"_b=
lank"
href=3D"mailto:users@ovirt.org">users@ovirt.org</a>&gt;</span></p><=
div><div class=3D"gmail-h5"><br /><b>Subject:</b> Re:
[ovirt-users] 4.0 -=
 2nd node fails on
deploy<u></u><u></u></div></div><p></p><div><div
class=
=3D"gmail-h5"><p
class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><div><div><d=
iv><p class=3D"MsoNormal">[Adding gluster-users
ML]<u></u><u></u></p></di=
v><p style=3D"margin-bottom:12pt" class=3D"MsoNormal">The
brick logs are =
filled with errors :<br />[2016-10-05 19:30:28.659061] E [MSGID: 113077] =
[posix-handle.c:309:posix_<wbr />handle_pump] 0-engine-posix: malformed i=
nternal link /var/run/vdsm/storage/<wbr />0a021563-91b5-4f49-9c6b-<wbr />=
fff45e85a025/d84f0551-0f2b-<wbr />457c-808c-6369c6708d43/<wbr />1b5a5e34-=
818c-4914-8192-<wbr />2f05733b5583 for /xpool/engine/brick/.<wbr
/>glusterfs/b9/8e/b98ed8d2-3bf9-<wbr />4b11-92fd-ca5324e131a8 <br />[2016=
-10-05 19:30:28.659069] E [MSGID: 113091] [posix.c:180:posix_lookup] 0-en=
gine-posix: Failed to create inode handle for path &lt;gfid:b98ed8d2-3bf9=
-4b11-92fd-<wbr />ca5324e131a8&gt; <br />The message &quot;E [MSGID:
1130=
18] [posix.c:198:posix_lookup] 0-engine-posix: lstat on null failed&quot;=
 repeated 3 times between [2016-10-05 19:30:28.656529] and [2016-10-05 19=
:30:28.659076] <br />[2016-10-05 19:30:28.659087] W [MSGID: 115005] [serv=
er-resolve.c:126:resolve_<wbr />gfid_cbk] 0-engine-server: b98ed8d2-3bf9-=
4b11-92fd-<wbr />ca5324e131a8: failed to resolve (Success)
<u></u><u></u>=
</p></div><p class=3D"MsoNormal">- Ravi, the above are from
the data bric=
k of the arbiter volume. Can you take a
look?<u></u><u></u></p><div><p cl=
ass=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div><div><p
class=3D"MsoNorma=
l">Jason,<u></u><u></u></p></div><div><div><div><p
style=3D"margin-bottom=
:12pt" class=3D"MsoNormal">Could you also provide the mount
logs from the first host (/var/log/glusterfs/rhev-data-<wbr />center-mnt-=
glusterSD*engine.<wbr />log) and glusterd log (/var/log/glusterfs/etc-<wb=
r />glusterfs-glusterd.vol.log) around the same time
frame.<u></u><u></u>=
</p></div><div><p
class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div=
...
</div></div><div><p
class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><div><p
= class=3D"MsoNormal">On Wed, Oct 5, 2016 at 3:28 AM, Jason Jeffrey
&lt;<a =
target=3D"_blank"
href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&g=
t; wrote:<u></u><u></u></p><blockquote
style=3D"border-width:medium mediu=
m medium 1pt;border-style:none none none solid;border-color:-moz-use-text=
-color -moz-use-text-color -moz-use-text-color rgb(204,204,204);padding:0=
cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"><div><div><p
class=3D"=
MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,=
sans-serif;color:rgb(31,73,125)">Hi,</span><u></u><u></u></p><p
class=3D"=
MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;col=
or:rgb(31,73,125)">Servers are powered =C2=A0off =C2=A0when I=E2=80=99m n=
ot looking at the
problem.</span><u></u><u></u></p><p
class=3D"MsoNormal"=
...
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif=
;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-se=
rif;color:rgb(31,73,125)">There may have been instances where all three w=
ere not powered on, during the same
period.</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">Glusterhd log attached, the xp=
ool-engine-brick log is
over 1 GB in size, I=E2=80=99ve taken a sample of the last =C2=A0couple d=
ays, looks to be highly
repative.</span><u></u><u></u></p><p
class=3D"Mso=
Normal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,san=
s-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"=
MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,=
sans-serif;color:rgb(31,73,125)">Cheers</span><u></u><u></u></p><p
class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=
=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&qu=
ot;,sans-serif;color:rgb(31,73,125)">Jason</span><u></u><u></u></p><p
cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
=
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>=
<p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>=
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;ca=
libri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u><=
/p><p class=3D"MsoNormal"><b><span
style=3D"font-size:11pt;font-family:&q=
uot;calibri&quot;,sans-serif"
lang=3D"EN-US">From:</span></b><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif"
lang=3D"EN-US=
"> Simone Tiraboschi [mailto:<a target=3D"_blank"
href=3D"mailto:stirabos=
@redhat.com&quot;&gt;stirabos(a)redhat.com&lt;/a>] <br /><b>Sent:</b>
04 October 2016 =
16:50</span><u></u><u></u></p><div><div><p
class=3D"MsoNormal"><br /><b>T=
o:</b> Jason Jeffrey &lt;<a target=3D"_blank"
href=3D"mailto:jason@sudo.c=
o.uk&quot;&gt;jason(a)sudo.co.uk&lt;/a&gt;&amp;gt;&lt;br /><b>Cc:</b> users
&lt;<a target=3D"_bl=
ank"
href=3D"mailto:users@ovirt.org">users@ovirt.org</a>&gt;<br
/><b>Subj=
ect:</b> Re: [ovirt-users] 4.0 - 2nd node fails on
deploy<u></u><u></u></p></div></div><div><div><p
class=3D"MsoNormal">=C2=A0=
<u></u><u></u></p><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><di=
v><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p
class=3D"MsoNorm=
al">On Tue, Oct 4, 2016 at 5:22 PM, Jason Jeffrey &lt;<a
target=3D"_blank=
" href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&gt;
wrote:<u></u>=
<u></u></p><blockquote style=3D"border-width:medium medium
medium 1pt;bor=
der-style:none none none solid;border-color:-moz-use-text-color -moz-use-=
text-color -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;m=
argin:5pt 0cm 5pt 4.8pt"><div><div><p
class=3D"MsoNormal"><span style=3D"=
font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73=
,125)">Hi,</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"=
font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73=
,125)">=C2=A0</span><u></u><u></u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">DCASTORXX is a hosts entry for dedicated =C2=A0direct 10G=
B links (each private /28) between the x3 servers=C2=A0 i.e 1=3D&gt; 2&am=
p;3, 2=3D&gt; 1&amp;3, etc) planned to be used solely for storage.</span>=
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;fo=
nt-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</sp=
an><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt=
;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">I,e
</s=
pan><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11p=
t;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0=
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">10.=
100.50.81=C2=A0=C2=A0=C2=A0
dcasrv01</span><u></u><u></u></p><p
class=3D"=
MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">10.100.101.1=C2=A0=C2=A0=C2=A0
dcastor01</span><u></u><u>=
</u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:=
&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">10.100.50.82=C2=A0=C2=
=A0=C2=A0 dcasrv02</span><u></u><u></u></p><p
class=3D"MsoNormal"><span s=
tyle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:r=
gb(31,73,125)">10.100.101.2=C2=A0=C2=A0=C2=A0
dcastor02</span><u></u><u><=
/u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&=
quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">10.100.50.83=C2=A0=C2=
=A0=C2=A0 dcasrv03</span><u></u><u></u></p><p
class=3D"MsoNormal"><span s=
tyle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:r=
gb(31,73,125)">10.100.103.3=C2=A0=C2=A0=C2=A0 dcastor03=C2=A0
</span><u><=
/u><u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-f=
amily:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><=
u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">These were setup with the gluster
commands</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></=
u><u></u></p><p><span
style=3D"font-size:11pt;font-family:symbol;color:rg=
b(31,73,125)">=C2=B7</span><span
style=3D"font-size:7pt;color:rgb(31,73,1=
25)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span
style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">gluster volume create iso replica 3 arbiter 1=C2=A0 dcastor01:/xp=
ool/iso/brick=C2=A0=C2=A0 dcastor02:/xpool/iso/brick=C2=A0=C2=A0 dcastor0=
3:/xpool/iso/brick</span><u></u><u></u></p><p><span
style=3D"font-size:11=
pt;font-family:symbol;color:rgb(31,73,125)">=C2=B7</span><span
style=3D"f=
ont-size:7pt;color:rgb(31,73,125)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 </span><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">gluster volume create export repli=
ca 3 arbiter
1=C2=A0 dcastor02:/xpool/export/brick=C2=A0 dcastor03:/xpool/export/brick=
=C2=A0 dcastor01:/xpool/export/brick=C2=A0
</span><u></u><u></u></p><p><s=
pan style=3D"font-size:11pt;font-family:symbol;color:rgb(31,73,125)">=C2=B7=
</span><span
style=3D"font-size:7pt;color:rgb(31,73,125)">=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span style=3D"font-size:11pt;fo=
nt-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">gluster
vo=
lume create engine replica 3 arbiter 1 dcastor01:/xpool/engine/brick dcas=
tor02:/xpool/engine/brick
dcastor03:/xpool/engine/brick</span><u></u><u><=
/u></p><p><span
style=3D"font-size:11pt;font-family:symbol;color:rgb(31,7=
3,125)">=C2=B7</span><span
style=3D"font-size:7pt;color:rgb(31,73,125)">=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span style=3D"font-=
size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)=
">gluster volume create data replica 3 arbiter 1=C2=A0 dcastor01:/xpool/d=
ata/brick=C2=A0 dcastor03:/xpool/data/brick=C2=A0 dcastor02:/xpool/data/b=
ricky</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;col=
or:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal">=
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;=
color:rgb(31,73,125)">So yes, DCASRV01 is the server (pri) and have local=
 bricks access through DCASTOR01 interface
</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">Is the issue here not the inco=
rrect soft link
?</span><u></u><u></u></p></div></div></blockquote><div><=
p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoN=
ormal">No, this should be
fine.<u></u><u></u></p></div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNor=
mal">The issue is that periodically your gluster volume losses its server=
 quorum and become
unavailable.<u></u><u></u></p></div><div><p
class=3D"M=
soNormal">It happened more than once from your
logs.<u></u><u></u></p></d=
iv><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
clas=
s=3D"MsoNormal">Can you please attach also gluster logs for that
volume?<=
u></u><u></u></p></div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></=
p></div><blockquote style=3D"border-width:medium medium medium
1pt;border=
-style:none none none solid;border-color:-moz-use-text-color -moz-use-tex=
t-color -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;marg=
in:5pt 0cm 5pt 4.8pt"><div><div><p
class=3D"MsoNormal"><span style=3D"fon=
t-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,12=
5)">=C2=A0</span><u></u><u></u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">lrwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:27 hosted=
-engine.metadata -&gt; /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a1=
64-<wbr />76a4876ecaaf/fd44dbf9-473a-<wbr />496a-9996-c8abe3278390/<wbr /=
...
cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93=C2=A0=C2=A0=C2=A0
</span><u>= </u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;font-=
family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">[root@dcasrv0=
1 /]# ls -al /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a164-<wbr />=
76a4876ecaaf/</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">ls: cannot access /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2=
-a164-<wbr />76a4876ecaaf/: No such file or directory=C2=A0=C2=A0 </span>=
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;fo=
nt-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">But the
da=
ta does exist
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">[ro=
ot@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# ls -al</span><u=
...
</u><u></u></p><p><span
style=3D"font-size:11pt;font-family:&quot;calibr=
i&quot;,sans-serif;color:rgb(31,73,125)">drwxr-xr-x. 2 vdsm
kvm=C2=A0=C2=A0=
=C2=A0 4096 Oct=C2=A0 3 17:17
.</span><u></u><u></u></p><p><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 3 17:17 .=
.</span><u></u><u></u></p><p><span
style=3D"font-size:11pt;font-family:&q=
uot;calibri&quot;,sans-serif;color:rgb(31,73,125)">-rw-rw----. 2 vdsm kvm=
 1028096 Oct=C2=A0 3 20:48 cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93</s=
pan><u></u><u></u></p><p><span
style=3D"font-size:11pt;font-family:&quot;=
calibri&quot;,sans-serif;color:rgb(31,73,125)">-rw-rw----. 2 vdsm kvm 104=
8576 Oct=C2=A0 3 17:17 cee9440c-4eb8-453b-bc04-<wbr
/>c47e6f9cbc93.lease</span><u></u><u></u></p><p><span
style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">-rw=
-r--r--. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0 283 Oct=C2=A0 3 17:17 cee9440=
c-4eb8-453b-bc04-<wbr />c47e6f9cbc93.meta=C2=A0=C2=A0
</span><u></u><u></=
u></p><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><p
class=3D"MsoNorma=
l">Thanks <u></u><u></u></p><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></=
p><p class=3D"MsoNormal">Jason
<u></u><u></u></p><p
class=3D"MsoNormal"><=
span style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;c=
olor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-seri=
f;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNor=
mal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-s=
erif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"Mso=
Normal"><b><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,=
sans-serif" lang=3D"EN-US">From:</span></b><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif"
lang=3D=
"EN-US"> Simone Tiraboschi [mailto:<a target=3D"_blank"
href=3D"mailto:st=
irabos@redhat.com&quot;&gt;stirabos(a)redhat.com&lt;/a>] <br
/><b>Sent:</b> 04 October=
 2016
14:40</span><u></u><u></u></p><div><div><p
class=3D"MsoNormal"><br =
/><b>To:</b> Jason Jeffrey &lt;<a target=3D"_blank"
href=3D"mailto:jason@=
sudo.co.uk&quot;&gt;jason(a)sudo.co.uk&lt;/a&gt;&amp;gt;&lt;br /><b>Cc:</b>
users &lt;<a target=3D=
"_blank"
href=3D"mailto:users@ovirt.org">users@ovirt.org</a>&gt;<br
/><b>=
Subject:</b> Re: [ovirt-users] 4.0 - 2nd node fails on
deploy<u></u><u></=
u></p></div></div><div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></=
p><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p
class=3D"Ms=
oNormal">=C2=A0<u></u><u></u></p><div><p
class=3D"MsoNormal">On Tue, Oct =
4, 2016 at 10:51 AM, Simone Tiraboschi &lt;<a target=3D"_blank"
href=3D"m=
ailto:stirabos@redhat.com">stirabos@redhat.com</a>&gt;
wrote:<u></u><u></=
u></p><blockquote style=3D"border-width:medium medium medium
1pt;border-s=
tyle:none none
none solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-=
text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.=
8pt"><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p
class=3D=
"MsoNormal">=C2=A0<u></u><u></u></p><div><p
class=3D"MsoNormal">On Mon, O=
ct 3, 2016 at 11:56 PM, Jason Jeffrey &lt;<a target=3D"_blank"
href=3D"ma=
ilto:jason@sudo.co.uk">jason@sudo.co.uk</a>&gt;
wrote:<u></u><u></u></p><=
blockquote style=3D"border-width:medium medium medium 1pt;border-style:no=
ne none none solid;border-color:-moz-use-text-color -moz-use-text-color -=
moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0c=
m 5pt 4.8pt"><div><div><p class=3D"MsoNormal"><span
style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Hi,</=
span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0=
</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Another problem has appeared, after rebooting the primary=
 the VM will not start.</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"=
...
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif=
;color:rgb(31,73,125)">Appears the symlink is broken between gluster moun=
t ref and
vdsm</span><u></u><u></u></p></div></div></blockquote><div><p
c=
lass=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNorm=
al">The first host was correctly deployed but it seas that you are facing=
 some issue connecting the
storage.<u></u><u></u></p></div><div><p
class=3D=
"MsoNormal">Can you please attach vdsm logs and /var/log/messages from th=
e first
host?<u></u><u></u></p></div></div></div></div></blockquote><div>=
<p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Thanks
Jason,<u></u><u></u></p></div><div><p
class=3D=
"MsoNormal">I suspect that your issue is related to
this:<u></u><u></u></=
p></div><div><div><p class=3D"MsoNormal">Oct =C2=A04
18:24:39 dcasrv01 et=
c-glusterfs-glusterd.vol[<wbr />2252]: [2016-10-04 17:24:39.522620] C [MS=
GID: 106002] [glusterd-server-quorum.c:351:<wbr />glusterd_do_volume_quor=
um_<wbr />action] 0-management: Server quorum lost for volume data. Stopp=
ing local
bricks.<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Oct =
=C2=A04 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[<wbr />2252]: [2016-=
10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-server-quorum.c:351:<w=
br />glusterd_do_volume_quorum_<wbr />action] 0-management: Server quorum=
 lost for volume engine. Stopping local
bricks.<u></u><u></u></p></div></=
div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
cla=
ss=3D"MsoNormal">and for some time your gluster volume has been
working.<=
u></u><u></u></p></div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNor=
mal">But
then:<u></u><u></u></p></div><div><div><p
class=3D"MsoNormal">Oc=
t =C2=A04 19:02:09 dcasrv01 systemd: Started /usr/bin/mount -t glusterfs =
-o backup-volfile-servers=3D<wbr />dcastor02:dcastor03 dcastor01:engine /=
rhev/data-center/mnt/<wbr
/>glusterSD/dcastor01:engine.<u></u><u></u></p>=
</div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:09
dcasrv01 systemd: =
Starting /usr/bin/mount -t glusterfs -o backup-volfile-servers=3D<wbr />d=
castor02:dcastor03 dcastor01:engine /rhev/data-center/mnt/<wbr />glusterS=
D/dcastor01:engine.<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Oc=
t =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-<wbr =
/>packages/yajsonrpc/stomp.py:<wbr />352: DeprecationWarning: Dispatcher.=
pending is deprecated. Use Dispatcher.socket.pending
instead.<u></u><u></=
u></p></div><div><p class=3D"MsoNormal">Oct =C2=A04
19:02:11 dcasrv01 ovi=
rt-ha-agent: pending =3D getattr(dispatcher, &#39;pending&#39;, lambda:
0)<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Oct =C2=A04 19:02:1=
1 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-<wbr />packages/yajson=
rpc/stomp.py:<wbr />352: DeprecationWarning: Dispatcher.pending is deprec=
ated. Use Dispatcher.socket.pending
instead.<u></u><u></u></p></div><div>=
<p class=3D"MsoNormal">Oct =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent:
pend=
ing =3D getattr(dispatcher, &#39;pending&#39;, lambda:
0)<u></u><u></u></=
p></div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:11
dcasrv01 journal=
: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof=
<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Oct =C2=A04 19:02:11 =
dcasrv01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.<wbr />agen=
t.Agent ERROR Error: &#39;Connection to storage server failed&#39; - tryi=
ng to restart
agent<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Oc=
t =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.=
<wbr />agent.agent.Agent:Error: &#39;Connection to storage server failed&=
#39; -
trying to restart
agent<u></u><u></u></p></div><div><p
class=3D"MsoNormal=
">Oct =C2=A04 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[<wbr />2252]: =
[2016-10-04 18:02:12.384611] C [MSGID: 106003] [glusterd-server-quorum.c:=
346:<wbr />glusterd_do_volume_quorum_<wbr />action] 0-management: Server =
quorum regained for volume data. Starting local
bricks.<u></u><u></u></p>=
</div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:12
dcasrv01 etc-glust=
erfs-glusterd.vol[<wbr />2252]: [2016-10-04 18:02:12.388981] C [MSGID: 10=
6003] [glusterd-server-quorum.c:346:<wbr />glusterd_do_volume_quorum_<wbr=
 />action] 0-management: Server quorum regained for volume engine. Starti=
ng local
bricks.<u></u><u></u></p></div></div><div><p
class=3D"MsoNormal"=
...
=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNormal">And at that po= int VDSM started complaining that
the hosted-engine-storage domain doesn&=
#39;t exist
anymore:<u></u><u></u></p></div><div><div><p
class=3D"MsoNorm=
al">Oct =C2=A04 19:02:30 dcasrv01 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.lib.<wbr />image.Image ERROR Error fetching volume=
s list: Storage domain does not exist: (u&#39;bbb70623-194a-46d2-a164-<wb=
r
/>76a4876ecaaf&#39;,)<u></u><u></u></p></div><div><p
class=3D"MsoNormal=
">Oct =C2=A04 19:02:30 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine=
_ha.<wbr />lib.image.Image:Error fetching volumes list: Storage domain do=
es not exist: (u&#39;bbb70623-194a-46d2-a164-<wbr
/>76a4876ecaaf&#39;,)<u=
...
</u><u></u></p></div></div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u><=
/u></p></div><div><p class=3D"MsoNormal">I see
from the logs that the ovi=
rt-ha-agent is trying to mount the hosted-engine storage domain as:<u></u=
...
<u></u></p></div><div><p
class=3D"MsoNormal">/usr/bin/mount -t glusterfs=  -o
backup-volfile-servers=3D<wbr />dcastor02:dcastor03 dcastor01:engine =
/rhev/data-center/mnt/<wbr
/>glusterSD/dcastor01:engine.<u></u><u></u></p=
...
</div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
= class=3D"MsoNormal">Pointing to dcastor01, dcastor02 and dcastor03
while
your server is
dcasrv01.<u></u><u></u></p></div><div><p
class=3D"MsoNorma=
l">But at the same time it seams that also dcasrv01 has local bricks for =
the same engine
volume.<u></u><u></u></p></div><div><p
class=3D"MsoNormal=
">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNormal">So, is dcasrv=
01 just an alias fro dcastor01? if not you probably have some issue with =
the configuration of your gluster
volume.<u></u><u></u></p></div><div><p =
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNor=
mal">=C2=A0<u></u><u></u></p></div><blockquote
style=3D"border-width:medi=
um medium medium 1pt;border-style:none none none solid;border-color:-moz-=
use-text-color -moz-use-text-color -moz-use-text-color rgb(204,204,204);p=
adding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt
4.8pt"><div><div><div><div><div=
...
<blockquote style=3D"border-width:medium medium medium
1pt;border-style:= none none none solid;border-color:-moz-use-text-color
-moz-use-text-color=
 -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt
0cm 5pt 4.8pt"><div><div><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
...
From
broker.log</span><u></u><u></u></p><p
class=3D"MsoNormal"><span styl=
e=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(=
31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span s=
tyle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:r=
gb(31,73,125)">Thread-169::ERROR::2016-10-04 22:44:16,189::storage_broker=
::<wbr />138::<a target=3D"_blank"
href=3D"http://ovirt_hosted_engine_ha.=
br">ovirt_hosted_engine_ha.br</a><wbr />oker.storage_broker.<wbr
/>Storag=
eBroker::(get_raw_stats_<wbr />for_service_type) Failed to read metadata =
from /rhev/data-center/mnt/<wbr />glusterSD/dcastor01:engine/<wbr />bbb70=
623-194a-46d2-a164-<wbr />76a4876ecaaf/ha_agent/hosted-<wbr
/>engine.metadata</span><u></u><u></u></p><p
class=3D"MsoNormal"><span st=
yle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rg=
b(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span=
 style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color=
:rgb(31,73,125)">[root@dcasrv01 ovirt-hosted-engine-ha]# ls -al /rhev/dat=
a-center/mnt/<wbr />glusterSD/dcastor01\:engine/<wbr />bbb70623-194a-46d2=
-a164-<wbr
/>76a4876ecaaf/ha_agent/</span><u></u><u></u></p><p
class=3D"M=
soNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,s=
ans-serif;color:rgb(31,73,125)">total
9</span><u></u><u></u></p><p class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">drwxrwx---. 2 vdsm kvm 4096 Oct=C2=A0 3=
 17:27 .</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"fo=
nt-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,1=
25)">drwxr-xr-x. 5 vdsm kvm 4096 Oct=C2=A0 3 17:17
..</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-siz=
e:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">l=
rwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:27 hosted-engine.lockspace=
 -&gt; /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a164-<wbr
/>76a487=
6ecaaf/23d81b73-bcb7-<wbr />4742-abde-128522f43d78/<wbr />11d6a3e1-1817-4=
29d-b2e0-<wbr
/>9051a3cf41a4</span><u></u><u></u></p><p
class=3D"MsoNorma=
l"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-ser=
if;color:rgb(31,73,125)">lrwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:=
27 hosted-engine.metadata -&gt; /var/run/vdsm/storage/<wbr />bbb70623-194=
a-46d2-a164-<wbr />76a4876ecaaf/fd44dbf9-473a-<wbr />496a-9996-c8abe32783=
90/<wbr />cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93=C2=A0=C2=A0=C2=A0 <=
/span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:1=
1pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0=
</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">[root@dcasrv01 /]# ls -al /var/run/vdsm/storage/<wbr />bb=
b70623-194a-46d2-a164-<wbr
/>76a4876ecaaf/</span><u></u><u></u></p><p cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">ls: cannot access /var/run/vdsm/st=
orage/<wbr />bbb70623-194a-46d2-a164-<wbr />76a4876ecaaf/: No such file o=
r directory=C2=A0=C2=A0 </span><u></u><u></u></p><p
class=3D"MsoNormal"><=
span style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;c=
olor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-seri=
f;color:rgb(31,73,125)">Though file appears to be there
</span><u></u><u>=
</u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:=
&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u>=
<u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Gluster is setup as xpool/engine
</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;c=
alibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u>=
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quo=
t;calibri&quot;,sans-serif;color:rgb(31,73,125)">[root@dcasrv01 fd44dbf9-=
473a-496a-9996-<wbr />c8abe3278390]#
pwd</span><u></u><u></u></p><p class=
=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&qu=
ot;,sans-serif;color:rgb(31,73,125)">/xpool/engine/brick/bbb70623-<wbr />=
194a-46d2-a164-76a4876ecaaf/<wbr />images/fd44dbf9-473a-496a-<wbr />9996-=
c8abe3278390</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# ls -=
al</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">total
2060</span><u></u><u></u></p><p
class=3D"MsoNormal"=
...
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif=
;color:rgb(31,73,125)">drwxr-xr-x. 2 vdsm kvm=C2=A0=C2=A0=C2=A0 4096
Oct=C2=
=A0 3 17:17 .</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 3 17:17 .=
.</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size=
:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">-r=
w-rw----. 2 vdsm kvm 1028096 Oct=C2=A0 3 20:48 cee9440c-4eb8-453b-bc04-<w=
br />c47e6f9cbc93</span><u></u><u></u></p><p
class=3D"MsoNormal"><span st=
yle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rg=
b(31,73,125)">-rw-rw----. 2 vdsm kvm 1048576 Oct=C2=A0 3 17:17 cee9440c-4=
eb8-453b-bc04-<wbr
/>c47e6f9cbc93.lease</span><u></u><u></u></p><p
class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">-rw-r--r--. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0 283 Oct=C2=
=A0 3 17:17 cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93.meta=C2=A0=C2=A0 =
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font=
-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125=
)">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# gluster v=
olume info</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"=
font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73=
,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">Volume Name:
data</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-s=
ize:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)"=
...
Type:
Replicate</span><u></u><u></u></p><p
class=3D"MsoNormal"><span sty=
le=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb=
(31,73,125)">Volume ID: 54fbcafc-fed9-4bce-92ec-<wbr
/>fa36cdcacbd4</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Status:
S=
tarted</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font=
-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125=
)">Number of Bricks: 1 x (2 + 1) =3D
3</span><u></u><u></u></p><p class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">Transport-type:
tcp</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Bricks:</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">Brick1:
dcastor01:/xpool/data/brick</span><u></u><u><=
/u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&=
quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick2: dcastor03:/xp=
ool/data/brick</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">Brick3: dcastor02:/xpool/data/bricky
(arbiter)</span><u></u><u=
...
</u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family=
:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Options
Reconfigure=
d:</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-siz=
e:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">p=
erformance.readdir-ahead:
on</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">performance.quick-read: off</sp=
an><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt=
;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">perform=
ance.read-ahead: off</span><u></u><u></u></p><p
class=3D"MsoNormal"><span=
 style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color=
:rgb(31,73,125)">performance.io-cache:
off</span><u></u><u></u></p><p cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">performance.stat-prefetch: off</sp=
an><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt=
;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">cluster=
.eager-lock: enable</span><u></u><u></u></p><p
class=3D"MsoNormal"><span =
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">network.remote-dio:
enable</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">cluster.quorum-type: auto</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">cluster.s=
erver-quorum-type:
server</span><u></u><u></u></p><p
class=3D"MsoNormal">=
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;=
color:rgb(31,73,125)">storage.owner-uid:
36</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">storage.owner-gid:
36</span><u></=
u><u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-fa=
mily:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u=
...
</u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;font=
-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Volume
Name:=
 engine</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Type:
Replicate</span><u></u><u></u></p><p
class=3D"MsoNo=
rmal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-=
serif;color:rgb(31,73,125)">Volume ID: dd4c692d-03aa-4fc6-9011-<wbr />a8d=
ad48dad96</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"f=
ont-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,=
125)">Status:
Started</span><u></u><u></u></p><p
class=3D"MsoNormal"><spa=
n style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;colo=
r:rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D
3</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&qu=
ot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Transport-type:
tcp</sp=
an><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt=
;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Bricks:=
</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Brick1:
dcastor01:/xpool/engine/brick</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&qu=
ot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick2:
dcastor02:/xpoo=
l/engine/brick</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">Brick3: dcastor03:/xpool/engine/brick
(arbiter)</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Options
Reconfigur=
ed:</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
performance.readdir-ahead:
on</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-se=
rif;color:rgb(31,73,125)">performance.quick-read:
off</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
performance.read-ahead:
off</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-seri=
f;color:rgb(31,73,125)">performance.io-cache:
off</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;c=
alibri&quot;,sans-serif;color:rgb(31,73,125)">performance.stat-prefetch:
=
off</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
cluster.eager-lock:
enable</span><u></u><u></u></p><p
class=3D"MsoNormal"=
...
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif=
;color:rgb(31,73,125)">network.remote-dio:
enable</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;c=
alibri&quot;,sans-serif;color:rgb(31,73,125)">cluster.quorum-type:
auto</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-s=
ize:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)"=
...
cluster.server-quorum-type:
server</span><u></u><u></u></p><p class=3D"M=
soNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,s=
ans-serif;color:rgb(31,73,125)">storage.owner-uid:
36</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&qu=
ot;calibri&quot;,sans-serif;color:rgb(31,73,125)">storage.owner-gid:
36</=
span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0=
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Vol=
ume Name: export</span><u></u><u></u></p><p
class=3D"MsoNormal"><span sty=
le=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb=
(31,73,125)">Type:
Replicate</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">Volume ID: 23f14730-d264-4cc2-a=
f60-<wbr
/>196b943ecaf3</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">Status:
Started</span><u></u><u></u></p><p
class=3D"M=
soNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,s=
ans-serif;color:rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D 3</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Transport=
-type: tcp</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"=
font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73=
,125)">Bricks:</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">Brick1:
dcastor02:/xpool/export/brick</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-se=
rif;color:rgb(31,73,125)">Brick2:
dcastor03:/xpool/export/brick</span><u>=
</u><u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-=
family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick3:
dcast=
or01:/xpool/export/brick
(arbiter)</span><u></u><u></u></p><p
class=3D"Ms=
oNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sa=
ns-serif;color:rgb(31,73,125)">Options
Reconfigured:</span><u></u><u></u>=
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quo=
t;calibri&quot;,sans-serif;color:rgb(31,73,125)">performance.readdir-ahea=
d: on</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-=
size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)=
">storage.owner-uid:
36</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">storage.owner-gid:
36</span><u></u><u></u></p><p class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=
=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&qu=
ot;,sans-serif;color:rgb(31,73,125)">Volume Name:
iso</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&qu=
ot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Type:
Replicate</span><=
u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;fon=
t-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Volume ID:
=
b2d3d7e2-9919-400b-8368-<wbr
/>a0443d48e82a</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">Status:
Started</span><u></u><u><=
/u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D
3</span><u></u><u></u><=
/p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot=
;calibri&quot;,sans-serif;color:rgb(31,73,125)">Transport-type: tcp</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Bricks:</=
span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick=
1:
dcastor01:/xpool/iso/brick</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-se=
rif;color:rgb(31,73,125)">Brick2:
dcastor02:/xpool/iso/brick</span><u></u=
...
<u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;font-fam=
ily:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick3:
dcastor0=
3:/xpool/iso/brick
(arbiter)</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">Options
Reconfigured:</span><u>=
</u><u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-=
family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">performance.r=
eaddir-ahead: on</span><u></u><u></u></p><p
class=3D"MsoNormal"><span sty=
le=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb=
(31,73,125)">storage.owner-uid:
36</span><u></u><u></u></p><p class=3D"Ms=
oNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sa=
ns-serif;color:rgb(31,73,125)">storage.owner-gid: 36=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<=
wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;col=
or:rgb(31,73,125)">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe327=
8390]# gluster volume
status</span><u></u><u></u></p><p
class=3D"MsoNorma=
l"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-ser=
if;color:rgb(31,73,125)">Status of volume:
data</span><u></u><u></u></p><=
p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cal=
ibri&quot;,sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0 Pid</span><u></u=
...
<u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;font-fam=
ily:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">----------------=
--------------<wbr />------------------------------<wbr />---------------=
---</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">Brick dcastor01:/xpool/data/bri=
ck=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 491=
53=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3076</span><u></u><u></u>=
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quo=
t;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick dcastor03:/xpool/d=
ata/brick=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 49153=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3019</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick
dcastor02:/x=
pool/data/bricky=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 49153=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3857</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">NFS Server on
loca=
lhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=
=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A03097</span><u></u><u></u></p><p
class=3D"MsoNorma=
l"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-ser=
if;color:rgb(31,73,125)">Self-heal Daemon on localhost=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3088</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 3039</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">Self-heal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3114</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 3871</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3864</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;c=
alibri&quot;,sans-serif;color:rgb(31,73,125)">Task Status of Volume
data<=
/span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:1=
1pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">----=
--------------------------<wbr />------------------------------<wbr />---=
---------------</span><u></u><u></u></p><p
class=3D"MsoNormal"><span styl=
e=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(=
31,73,125)">There are no active volume
tasks</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-=
size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)=
">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"fo=
nt-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,1=
25)">Status of volume:
engine</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-se=
rif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TCP P=
ort=C2=A0 RDMA Port=C2=A0 Online=C2=A0
Pid</span><u></u><u></u></p><p cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">------------------------------<wbr=
 />------------------------------<wbr
/>------------------</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick
dcastor01:/x=
pool/engine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
49152=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3131</span><u></u><u><=
/u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&=
quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick dcastor02:/xpoo=
l/engine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 491=
52=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3852</span><u></u><u></u>=
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quo=
t;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick dcastor03:/xpool/e=
ngine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49152=C2=
=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
2992</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">NFS Server on localhost=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 3097</span><u></u><u></u></p><p
class=3D"MsoNormal"><s=
pan style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;co=
lor:rgb(31,73,125)">Self-heal Daemon on
localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3088=
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">NFS=
 Server on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3039</span><u></u><u></u>=
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quo=
t;calibri&quot;,sans-serif;color:rgb(31,73,125)">Self-heal Daemon on dcas=
tor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3114</s=
pan><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11p=
t;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">NFS
Se=
rver on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 204=
9=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3871</span><u></u><u><=
/u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3864</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">Task Status of Volume engine</=
span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">-----=
-------------------------<wbr />------------------------------<wbr />----=
--------------</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">There are no active volume
tasks</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;col=
or:rgb(31,73,125)">Status of volume:
export</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri=
&quot;,sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0
Pid</span><u></u><u=
...
</u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family=
:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">-------------------=
-----------<wbr />------------------------------<wbr />------------------=
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Bri=
ck dcastor02:/xpool/export/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3872</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-s=
ize:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)"=
...
Brick dcastor03:/xpool/export/brick=C2=A0<wbr
/>=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0
0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3147=
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Bri=
ck dcastor01:/xpool/export/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3150</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">NFS
Serve=
r on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3097</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;cali=
bri&quot;,sans-serif;color:rgb(31,73,125)">Self-heal Daemon on localhost=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0
Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3088</span><u></u><u></u></p><p cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 3039</span><u></u><u></u></p><p
class=3D"MsoNormal"><span=
 style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color=
:rgb(31,73,125)">Self-heal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
3114</span><u></u><u></u></p><p class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 3871</span><u></u><u></u></p><p
class=3D"MsoNormal"><span=
 style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color=
:rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0
N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 3864</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span st=
yle=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rg=
b(31,73,125)">Task Status of Volume
export</span><u></u><u></u></p><p cla=
ss=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&=
quot;,sans-serif;color:rgb(31,73,125)">------------------------------<wbr=
 />------------------------------<wbr
/>------------------</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-famil=
y:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">There are no
activ=
e volume tasks</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">=C2=A0</span><u></u><u></u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">Status of volume:
iso</span><u></u><u></u></p><p class=3D=
"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;=
,sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0
Pid</span><u></u><u></u><=
/p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot=
;calibri&quot;,sans-serif;color:rgb(31,73,125)">-------------------------=
-----<wbr />------------------------------<wbr />------------------</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Brick
dca=
stor01:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3=
152</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
Brick
dcastor02:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 3881</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-=
size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)=
">Brick dcastor03:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 3146</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">NFS Server on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3097</span=
...
<u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11pt;f=
ont-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Self-heal=
 Daemon on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/=
A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 3088</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 =C2=A02049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3039</s=
pan><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11p=
t;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Self-h=
eal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 3114</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3871</s=
pan><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11p=
t;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Self-h=
eal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 3864</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>=
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;ca=
libri&quot;,sans-serif;color:rgb(31,73,125)">Task Status of Volume iso</s=
pan><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:11p=
t;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">------=
------------------------<wbr />------------------------------<wbr />-----=
-------------</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">There are no active volume
tasks</span><u></u><u></u></p><p class=
=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&qu=
ot;,sans-serif;color:rgb(31,73,125)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr=
 />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-size:=
11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">Tha=
nks</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font=
-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125=
)">Jason</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"fo=
nt-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,1=
25)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,7=
3,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(3=
1,73,125)">=C2=A0</span><u></u><u></u></p><div><div
style=3D"border-width=
:1pt medium medium;border-style:solid none none;border-color:rgb(225,225,=
225)
-moz-use-text-color -moz-use-text-color;padding:3pt 0cm 0cm"><p
class=3D"=
MsoNormal"><b><span
style=3D"font-size:11pt;font-family:&quot;calibri&quo=
t;,sans-serif" lang=3D"EN-US">From:</span></b><span
style=3D"font-size:11=
pt;font-family:&quot;calibri&quot;,sans-serif" lang=3D"EN-US">
<a target=3D=
"_blank"
href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org<=
/a> [mailto:<a target=3D"_blank"
href=3D"mailto:users-bounces@ovirt.org">=
users-bounces(a)ovirt.&lt;wbr />org</a>] <b>On Behalf Of </b>Jason
Jeffrey<br =
/><b>Sent:</b> 03 October 2016 18:40<br /><b>To:</b>
<a target=3D"_blank"=

href=3D"mailto:users@ovirt.org">users@ovirt.org</a></span><u></u><u></u>=
</p><div><div><p class=3D"MsoNormal"><br
/><b>Subject:</b> Re: [ovirt-use=
rs] 4.0 - 2nd node fails on
deploy<u></u><u></u></p></div></div></div></d=
iv><div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><p
class=3D"M=
soNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,s=
ans-serif;color:rgb(31,73,125)">Hi,</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calib=
ri&quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>=
<p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;ca=
libri&quot;,sans-serif;color:rgb(31,73,125)">Setup log attached for prima=
ry</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-siz=
e:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"font-si=
ze:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,125)">=
Regards</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"fon=
t-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73,12=
5)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"=
font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:rgb(31,73=
,125)">Jason </span><u></u><u></u...
</p><p class=3D"MsoNormal"><span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif;color:=
rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><b>=
<span
style=3D"font-size:11pt;font-family:&quot;calibri&quot;,sans-serif"=
 lang=3D"EN-US">From:</span></b><span
style=3D"font-size:11pt;font-family=
:&quot;calibri&quot;,sans-serif" lang=3D"EN-US"> Simone
Tiraboschi [<a ta=
rget=3D"_blank"
href=3D"mailto:stirabos@redhat.com">mailto:stirabos@redha=
t.com</a>] <br /><b>Sent:</b> 03 October 2016 09:27<br
/><b>To:</b> Jason=
 Jeffrey &lt;<a target=3D"_blank"
href=3D"mailto:jason@sudo.co.uk">jason@=
sudo.co.uk</a>&gt;<br /><b>Cc:</b> users &lt;<a
target=3D"_blank" href=3D=
"mailto:users@ovirt.org">users@ovirt.org</a>&gt;<br
/><b>Subject:</b> Re:=
 [ovirt-users] 4.0 - 2nd node fails on
deploy</span><u></u><u></u></p><p =
class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p
class=3D"MsoNormal">=C2=
=A0<u></u><u></u></p><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p>=
<div><p class=3D"MsoNormal">On Mon, Oct 3, 2016 at 12:45 AM, Jason
Jeffre=
y &lt;<a
target=3D"_blank"
href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&g=
t; wrote:<u></u><u></u></p><blockquote
style=3D"border-width:medium mediu=
m medium 1pt;border-style:none none none solid;border-color:-moz-use-text=
-color -moz-use-text-color -moz-use-text-color rgb(204,204,204);padding:0=
cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><p
class=3D"MsoNormal"=
...
<span
lang=3D"EN-US">Hi,</span><u></u><u></u></p><p
class=3D"MsoNormal">= <span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span lang=3D"EN-US">I am trying to build a x3 HC cluster, with
a self =
hosted engine using
gluster.</span><u></u><u></u></p><p
class=3D"MsoNorma=
l"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNor=
mal"><span lang=3D"EN-US">I have successful built the
1<sup>st</sup> node=
,=C2=A0 however when I attempt to run hosted-engine =E2=80=93deploy on no=
de 2, I get the following
error</span><u></u><u></u></p><p
class=3D"MsoNo=
rmal"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"Mso=
Normal"><span
lang=3D"EN-US">[WARNING] A configuration file must be supplied to deploy =
Hosted Engine on an additional
host.</span><u></u><u></u></p><p class=3D"=
MsoNormal"><span lang=3D"EN-US">[ ERROR ]
&#39;version&#39; is not stored=
 in the HE configuration
image</span><u></u><u></u></p><p
class=3D"MsoNor=
mal"><span lang=3D"EN-US">[ ERROR ] Unable to get the answer file
from th=
e shared storage</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lan=
g=3D"EN-US">[ ERROR ] Failed to execute stage &#39;Environment
customizat=
ion&#39;: Unable to get the answer file from the shared
storage</span><u>=
</u><u></u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">[ INFO=C2=A0 ]=
 Stage: Clean up</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lan=
g=3D"EN-US">[ INFO=C2=A0 ] Generating answer file
&#39;/var/lib/ovirt-hos=
ted-engine-<wbr />setup/answers/answers-<wbr
/>20161002232505.conf&#39;</=
span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">[ INFO=
=C2=A0 ] Stage:
Pre-termination</span><u></u><u></u></p><p
class=3D"MsoNo=
rmal"><span
lang=3D"EN-US">[ INFO=C2=A0 ] Stage:
Termination</span><u></u><u></u></p>=
<p class=3D"MsoNormal"><span lang=3D"EN-US">[ ERROR ]
Hosted Engine deplo=
yment failed=C2=A0=C2=A0=C2=A0
</span><u></u><u></u></p><p class=3D"MsoNo=
rmal"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"Mso=
Normal"><span lang=3D"EN-US">Looking at the failure in the log
file..</sp=
an><u></u><u></u></p></div></div></blockquote><div><p
class=3D"MsoNormal"=
...
=C2=A0<u></u><u></u></p></div><div><p
class=3D"MsoNormal">Can you please=  attach hosted-engine-setup logs
from the first host?<u></u><u></u></p></=
div><div><p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><blockquote=
 style=3D"border-width:medium medium medium 1pt;border-style:none none no=
ne solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-te=
xt-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8p=
t"><div><div><p class=3D"MsoNormal"><span
lang=3D"EN-US">=C2=A0</span><u>=
</u><u></u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">2016-10-02 23:=
25:05 WARNING
otopi.plugins.gr_he_common.<wbr />core.remote_answerfile remote_answerfil=
e._<wbr />customization:151 A
configuration</span><u></u><u></u></p><p cl=
ass=3D"MsoNormal"><span lang=3D"EN-US">file must be supplied
to deploy Ho=
sted Engine on an additional
host.</span><u></u><u></u></p><p class=3D"Ms=
oNormal"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_=
he_common.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />a=
nswer_file:61
_fetch_answer_f</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span
lang=3D"EN-US">ile</span><u></u><u></u></p><p
class=3D"MsoNorma=
l"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_he_com=
mon.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />answer_=
file:69 fetching from:</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an lang=3D"EN-US">/rhev/data-center/mnt/<wbr
/>glusterSD/dcastor02:engine=
/<wbr />0a021563-91b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr =
/>02f9-4cd1-a22c-d6b56a0a8e9b/7</span><u></u><u></u></p><p
class=3D"MsoNormal"><span
lang=3D"EN-US">8cb2527-a2e2-489a-9fad-<wbr />46=
5a72221b37</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"E=
N-US">2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.<wbr />core.re=
mote_answerfile heconflib._dd_pipe_tar:69 executing: &#39;sudo -u vdsm dd=
 i</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">f=3D=
/rhev/data-center/mnt/<wbr />glusterSD/dcastor02:engine/<wbr />0a021563-9=
1b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr />02f9-4cd1-a22c-d=
6b56a0a8e9b</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"=
EN-US">/78cb2527-a2e2-489a-9fad-<wbr />465a72221b37
bs=3D4k&#39;</span><u=
...
</u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">2016-10-02 23=
:25:05 DEBUG otopi.plugins.gr_he_common.<wbr />core.remote_answerfile hec=
onflib._dd_pipe_tar:70 executing: &#39;tar -tvf
-&#39;</span><u></u><u></=
u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">2016-10-02 23:25:05 DEB=
UG otopi.plugins.gr_he_common.<wbr />core.remote_answerfile
heconflib._dd_pipe_tar:88
stdout:</span><u></u><u></u></p><p
class=3D"Mso=
Normal"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG
otopi.plugins.gr_h=
e_common.<wbr />core.remote_answerfile heconflib._dd_pipe_tar:89 stderr:<=
/span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">2016-=
10-02 23:25:05 ERROR otopi.plugins.gr_he_common.<wbr />core.remote_answer=
file heconflib.validateConfImage:<wbr />111 &#39;version&#39; is not stor=
ed</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">in=
 the HE configuration
image</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span lang=3D"EN-US">2016-10-02 23:25:05 ERROR
otopi.plugins.gr_he_comm=
on.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />answer_f=
ile:73 Unable to get t</span><u></u><u></u></p><p
class=3D"MsoNormal"><sp=
an lang=3D"EN-US">he answer file from the shared
storage</span><u></u><u>=
</u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">=C2=A0</span><u></u><=
u></u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">Looking at the dete=
cted
gluster path - /rhev/data-center/mnt/<wbr />glusterSD/dcastor02:engine/<w=
br />0a021563-91b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr />0=
2f9-4cd1-a22c-d6b56a0a8e9b/</span><u></u><u></u></p><p
class=3D"MsoNormal=
"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNorm=
al"><span lang=3D"EN-US">[root@dcasrv02 ~]# ls -al
/rhev/data-center/mnt/=
<wbr />glusterSD/dcastor02:engine/<wbr />0a021563-91b5-4f49-9c6b-<wbr
/>f=
ff45e85a025/images/f055216c-<wbr
/>02f9-4cd1-a22c-d6b56a0a8e9b/</span><u>=
</u><u></u></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">total 1049609<=
/span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">drwxr=
-xr-x. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 2 04=
:46 .</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US"=
...
drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4096
Oct=C2=A0=  2 04:46
..</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"=
EN-US">-rw-rw----. 1 vdsm kvm 1073741824 Oct=C2=A0 2 04:46 78cb2527-a2e2-=
489a-9fad-<wbr
/>465a72221b37</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">-rw-rw----. 1 vdsm
kvm=C2=A0=C2=A0=
=C2=A0 1048576 Oct=C2=A0 2 04:46 78cb2527-a2e2-489a-9fad-<wbr />465a72221=
b37.lease</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN=
-US">-rw-r--r--. 1 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 294=
 Oct=C2=A0 2 04:46 78cb2527-a2e2-489a-9fad-<wbr />465a72221b37.meta=C2=A0=
 </span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"EN-US">7=
8cb2527-a2e2-489a-9fad-<wbr />465a72221b37 is=C2=A0 a 1 GB file, is this =
the engine VM ?</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=
=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span la=
ng=3D"EN-US">Copying the answers file form primary (/etc/ovirt-hosted-eng=
ine/<wbr />answers.conf ) to=C2=A0 node 2 and rerunning produces the same=
 error : (</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D"E=
N-US">(hosted-engine --deploy=C2=A0 --config-append=3D/root/answers.<wbr =
/>conf )</span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><spa=
n lang=3D"EN-US">Also tried on node 3, same issues
</span><u></u><u></u><=
/p><p class=3D"MsoNormal"><span
lang=3D"EN-US">=C2=A0</span><u></u><u></u=
...
</p><p class=3D"MsoNormal"><span
lang=3D"EN-US">Happy to provide logs an= d other
debugs</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D=
"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span lang=3D=
"EN-US">Thanks
</span><u></u><u></u></p><p
class=3D"MsoNormal"><span styl=
e=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></=
p><p class=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)" lang=3D"E=
N-US">Jason </span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US=
">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"co=
lor:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
cl=
ass=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></=
u></p><p class=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)" lang=3D=
"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)" lang=3D"EN=
-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US=
">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"co=
lor:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
cl=
ass=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"color:r=
gb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D=
"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0<=
/span><u></u><u></u...
</p><p
class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></=
u></p><p class=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)" lang=3D=
"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=
=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p=
...
<p class=3D"MsoNormal"><span
style=3D"color:rgb(136,136,136)" lang=3D"EN=
-US">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D=
"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p=
 class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US=
">=C2=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"co=
lor:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
cl=
ass=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=
=A0</span><u></u><u></u></p><p
class=3D"MsoNormal"><span style=3D"color:r=
gb(136,136,136)"
lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p
class=3D=
"MsoNormal"><span style=3D"color:rgb(136,136,136)"
lang=3D"EN-US">=C2=A0<=
/span><u></u><u></u></p></div></div><p
style=3D"margin-bottom:12pt"
class=3D"MsoNormal"><br />______________________________<wbr
/>__________=
_______<br />Users mailing list<br /><a target=3D"_blank"
href=3D"mailto:=
Users@ovirt.org&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br /><a
target=3D"_blank" href=3D"ht=
tp://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.o... =
/>mailman/listinfo/users</a><u></u><u></u></p></blockquote></div><p
class=
=3D"MsoNormal">=C2=A0<u></u><u></u></p></div></div></div></div></div></di=
v><p style=3D"margin-bottom:12pt" class=3D"MsoNormal"><br
/>_____________=
_________________<wbr />_________________<br />Users mailing list<br
/><a=
 target=3D"_blank"
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br=
 /><a target=3D"_blank"
href=3D"http://lists.ovirt.org/mailman/listinfo/u=
sers">http://lists.ovirt.org/<wbr
/>mailman/listinfo/users</a><u></u><u><=
/u></p></blockquote></div></div></div><p
class=3D"MsoNormal">=C2=A0<u></u=
...
<u></u></p></div></div></blockquote></div><p
class=3D"MsoNormal">=C2=A0<=
u></u><u></u></p></div></div></div></div></div></div></blockquote></div><=
p
class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div></div></div></div></div=
...
</div><p style=3D"margin-bottom:12pt"
class=3D"MsoNormal"><br />________= ______________________<wbr
/>_________________<br />Users mailing list<br=
 /><a target=3D"_blank"
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</=
a><br /><a target=3D"_blank"
href=3D"http://lists.ovirt.org/mailman/listi=
nfo/users">http://lists.ovirt.org/<wbr
/>mailman/listinfo/users</a><u></u=
...
<u></u></p></blockquote></div><p
class=3D"MsoNormal"><u></u>=C2=A0<u></u=
</p></div></div></div></div></div></blockquote></div><br
/></div></div> <p style=3D"margin-top: 2.5em; margin-bottom:
1em; border-bottom: 1px sol=
id #000"></p><pre class=3D"k9mail"><hr /><br
/>Gluster-users mailing list=
<br /&gt;Gluster-users(a)gluster.org&lt;br /><a
href=3D"http://www.gluster.org/ma=
ilman/listinfo/gluster-users">http://www.gluster.org/mailman/list...
ster-users</a></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</bod=
y></html>
------4VSAMIL16A4UO07LK9ROSU1PLADG1A--

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] [Gluster-users] 4.0 - 2nd node fails on deploy