On Wed, Oct 5, 2016 at 1:56 PM, Jason Jeffrey <jason@sudo.co.uk> wrote:
HI,
Logs attached
Have you probed 2 interfaces for same host, that is - dcasrv02 and dcastor02? Does "gluster peer status" understand both names as for same host?
From glusterd logs and the mount logs - the connection between the peers is lost, and quorum is lost, which is reaffirming what Simone said earlier. Logs seem to indicate network issues - check the direct link setup. See below
From mount logs: [2016-10-04 17:26:15.718300] E [socket.c:2292:socket_connect_finish] 0-engine-client-2: connection to 10.100.103.3:24007 failed (No route to host) [2016-10-04 17:26:15.718345] W [MSGID: 108001] [afr-common.c:4379:afr_notify] 0-engine-replicate-0: Client-quorum is not met [2016-10-04 17:26:16.428290] E [socket.c:2292:socket_connect_finish] 0-engine-client-1: connection to 10.100.101.2:24007 failed (No route to host) [2016-10-04 17:26:16.428336] E [MSGID: 108006] [afr-common.c:4321:afr_notify] 0-engine-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up
And in glusterd logs: [2016-10-04 17:24:39.522402] E [socket.c:2292:socket_connect_finish] 0-management: connection to 10.100.50.82:24007 failed (No route to host) [2016-10-04 17:24:39.522578] I [MSGID: 106004] [glusterd-handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer <dcasrv02> (<1e788fc9-dfe9-4753-92c7-76a95c8d0891>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume engine. Stopping local bricks. [2016-10-04 17:24:39.523314] I [MSGID: 106132] [glusterd-utils.c:1560:glusterd_service_stop] 0-management: brick already stopped [2016-10-04 17:24:39.526188] E [socket.c:2292:socket_connect_finish] 0-management: connection to 10.100.103.3:24007 failed (No route to host) [2016-10-04 17:24:39.526219] I [MSGID: 106004] [glusterd-handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer <dcastor03> (<9a9c037e-96cd-4f73-9800-a1df5cdd2818>), in state <Peer in Cluster>, has disconnected from glusterd.
Thanks
*From:* Sahina Bose [mailto:sabose@redhat.com] *Sent:* 05 October 2016 08:11 *To:* Jason Jeffrey <jason@sudo.co.uk>; gluster-users@gluster.org; Ravishankar Narayanankutty <ravishankar@redhat.com> *Cc:* Simone Tiraboschi <stirabos@redhat.com>; users <users@ovirt.org>
*Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
[Adding gluster-users ML]
The brick logs are filled with errors : [2016-10-05 19:30:28.659061] E [MSGID: 113077] [posix-handle.c:309:posix_handle_pump] 0-engine-posix: malformed internal link /var/run/vdsm/storage/
1b5a5e34-818c-4914-8192-2f05733b5583 for /xpool/engine/brick/. glusterfs/b9/8e/b98ed8d2-3bf9-4b11-92fd-ca5324e131a8 [2016-10-05 19:30:28.659069] E [MSGID: 113091] [posix.c:180:posix_lookup] 0-engine-posix: Failed to create inode handle for path <gfid:b98ed8d2-3bf9-4b11-92fd-ca5324e131a8> The message "E [MSGID: 113018] [posix.c:198:posix_lookup] 0-engine-posix: lstat on null failed" repeated 3 times between [2016-10-05 19:30:28.656529] and [2016-10-05 19:30:28.659076] [2016-10-05 19:30:28.659087] W [MSGID: 115005] [server-resolve.c:126:resolve_gfid_cbk] 0-engine-server: b98ed8d2-3bf9-4b11-92fd-ca5324e131a8: failed to resolve (Success)
- Ravi, the above are from the data brick of the arbiter volume. Can you take a look?
Jason,
Could you also provide the mount logs from the first host (/var/log/glusterfs/rhev-data-center-mnt-glusterSD*engine.log) and glusterd log (/var/log/glusterfs/etc-glusterfs-glusterd.vol.log) around the same time frame.
On Wed, Oct 5, 2016 at 3:28 AM, Jason Jeffrey <jason@sudo.co.uk> wrote:
Hi,
Servers are powered off when I=E2=80=99m not looking at the problem.
There may have been instances where all three were not powered on, during the same period.
Glusterhd log attached, the xpool-engine-brick log is over 1 GB in size, I=E2=80=99ve taken a sample of the last couple days, looks to be high= ly repative.
Cheers
Jason
*From:* Simone Tiraboschi [mailto:stirabos@redhat.com] *Sent:* 04 October 2016 16:50
*To:* Jason Jeffrey <jason@sudo.co.uk> *Cc:* users <users@ovirt.org> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
On Tue, Oct 4, 2016 at 5:22 PM, Jason Jeffrey <jason@sudo.co.uk> wrote:
Hi,
DCASTORXX is a hosts entry for dedicated direct 10GB links (each
/28) between the x3 servers i.e 1=3D> 2&3, 2=3D> 1&3, etc) planned to= be used solely for storage.
I,e
10.100.50.81 dcasrv01
10.100.101.1 dcastor01
10.100.50.82 dcasrv02
10.100.101.2 dcastor02
10.100.50.83 dcasrv03
10.100.103.3 dcastor03
These were setup with the gluster commands
=C2=B7 gluster volume create iso replica 3 arbiter 1 dcastor01:/xpool/iso/brick dcastor02:/xpool/iso/brick dcastor03:/xpool/iso/brick
=C2=B7 gluster volume create export replica 3 arbiter 1 dcastor02:/xpool/export/brick dcastor03:/xpool/export/brick dcastor01:/xpool/export/brick
=C2=B7 gluster volume create engine replica 3 arbiter 1 dcastor01:/xpool/engine/brick dcastor02:/xpool/engine/brick dcastor03:/xpool/engine/brick
=C2=B7 gluster volume create data replica 3 arbiter 1 dcastor01:/xpool/data/brick dcastor03:/xpool/data/brick dcastor02:/xpool/data/bricky
So yes, DCASRV01 is the server (pri) and have local bricks access
0a021563-91b5-4f49-9c6b-fff45e85a025/d84f0551-0f2b-457c-808c-6369c6708d4= 3/ private through
DCASTOR01 interface
Is the issue here not the incorrect soft link ?
No, this should be fine.
The issue is that periodically your gluster volume losses its server quorum and become unavailable.
It happened more than once from your logs.
Can you please attach also gluster logs for that volume?
lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.metadata ->
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a=
496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93
[root@dcasrv01 /]# ls -al /var/run/vdsm/storage/bbb70623-194a-46d2-a164- 76a4876ecaaf/
ls: cannot access /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such file or directory
But the data does exist
[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al
drwxr-xr-x. 2 vdsm kvm 4096 Oct 3 17:17 .
drwxr-xr-x. 6 vdsm kvm 4096 Oct 3 17:17 ..
-rw-rw----. 2 vdsm kvm 1028096 Oct 3 20:48 cee9440c-4eb8-453b-bc04- c47e6f9cbc93
-rw-rw----. 2 vdsm kvm 1048576 Oct 3 17:17 cee9440c-4eb8-453b-bc04- c47e6f9cbc93.lease
-rw-r--r--. 2 vdsm kvm 283 Oct 3 17:17 cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta
Thanks
Jason
*From:* Simone Tiraboschi [mailto:stirabos@redhat.com] *Sent:* 04 October 2016 14:40
*To:* Jason Jeffrey <jason@sudo.co.uk> *Cc:* users <users@ovirt.org> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
On Tue, Oct 4, 2016 at 10:51 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 3, 2016 at 11:56 PM, Jason Jeffrey <jason@sudo.co.uk> wrote:
Hi,
Another problem has appeared, after rebooting the primary the VM will not start.
Appears the symlink is broken between gluster mount ref and vdsm
The first host was correctly deployed but it seas that you are facing some issue connecting the storage.
Can you please attach vdsm logs and /var/log/messages from the first host?
Thanks Jason,
I suspect that your issue is related to this:
Oct 4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 17:24:39.522620] C [MSGID: 106002] [glusterd-server-quorum.c:351: glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume data. Stopping local bricks.
Oct 4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-server-quorum.c:351: glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume engine. Stopping local bricks.
and for some time your gluster volume has been working.
But then:
Oct 4 19:02:09 dcasrv01 systemd: Started /usr/bin/mount -t glusterfs -o backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine /rhev/data-center/mnt/glusterSD/dcastor01:engine.
Oct 4 19:02:09 dcasrv01 systemd: Starting /usr/bin/mount -t glusterfs -o backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine /rhev/data-center/mnt/glusterSD/dcastor01:engine.
Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site- packages/yajsonrpc/stomp.py:352: DeprecationWarning: Dispatcher.pending is deprecated. Use Dispatcher.socket.pending instead.
Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D getattr(dispatcher, 'pending', lambda: 0)
Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site- packages/yajsonrpc/stomp.py:352: DeprecationWarning: Dispatcher.pending is deprecated. Use Dispatcher.socket.pending instead.
Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D getattr(dispatcher, 'pending', lambda: 0)
Oct 4 19:02:11 dcasrv01 journal: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
Oct 4 19:02:11 dcasrv01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to storage server failed' - trying to restart agent
Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to storage server failed' - trying to restart agent
Oct 4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 18:02:12.384611] C [MSGID: 106003] [glusterd-server-quorum.c:346: glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume data. Starting local bricks.
Oct 4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 18:02:12.388981] C [MSGID: 106003] [glusterd-server-quorum.c:346: glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume engine. Starting local bricks.
And at that point VDSM started complaining that the hosted-engine-storage domain doesn't exist anymore:
Oct 4 19:02:30 dcasrv01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.lib.image.Image ERROR Error fetching volumes list: Storage domain does not exist: (u'bbb70623-194a-46d2-a164-76a4876ecaaf',)
Oct 4 19:02:30 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.lib.image.Image:Error fetching volumes list: Storage domain does not exist: (u'bbb70623-194a-46d2-a164-76a4876ecaaf',)
I see from the logs that the ovirt-ha-agent is trying to mount the hosted-engine storage domain as:
/usr/bin/mount -t glusterfs -o backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine /rhev/data-center/mnt/glusterSD/dcastor01:engine.
Pointing to dcastor01, dcastor02 and dcastor03 while your server is dcasrv01.
But at the same time it seams that also dcasrv01 has local bricks for the same engine volume.
So, is dcasrv01 just an alias fro dcastor01? if not you probably have some issue with the configuration of your gluster volume.
From broker.log
Thread-169::ERROR::2016-10-04 22:44:16,189::storage_broker::138:: ovirt_hosted_engine_ha.broker.storage_broker. StorageBroker::(get_raw_stats_for_service_type) Failed to read metadata from /rhev/data-center/mnt/glusterSD/dcastor01:engine/ bbb70623-194a-46d2-a164-76a4876ecaaf/ha_agent/hosted-engine.metadata
[root@dcasrv01 ovirt-hosted-engine-ha]# ls -al /rhev/data-center/mnt/
glusterSD/dcastor01\:engine/bbb70623-194a-46d2-a164-76a4876ecaaf/ha_agen= t/
total 9
drwxrwx---. 2 vdsm kvm 4096 Oct 3 17:27 .
drwxr-xr-x. 5 vdsm kvm 4096 Oct 3 17:17 ..
lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.lockspace ->
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/23d81b73-bcb7=
4742-abde-128522f43d78/11d6a3e1-1817-429d-b2e0-9051a3cf41a4
lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.metadata ->
/var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a=
496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93
[root@dcasrv01 /]# ls -al /var/run/vdsm/storage/bbb70623-194a-46d2-a164- 76a4876ecaaf/
ls: cannot access /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such file or directory
Though file appears to be there
Gluster is setup as xpool/engine
[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# pwd
/xpool/engine/brick/bbb70623-194a-46d2-a164-76a4876ecaaf/ images/fd44dbf9-473a-496a-9996-c8abe3278390
[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al
total 2060
drwxr-xr-x. 2 vdsm kvm 4096 Oct 3 17:17 .
drwxr-xr-x. 6 vdsm kvm 4096 Oct 3 17:17 ..
-rw-rw----. 2 vdsm kvm 1028096 Oct 3 20:48 cee9440c-4eb8-453b-bc04- c47e6f9cbc93
-rw-rw----. 2 vdsm kvm 1048576 Oct 3 17:17 cee9440c-4eb8-453b-bc04- c47e6f9cbc93.lease
-rw-r--r--. 2 vdsm kvm 283 Oct 3 17:17 cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta
[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume info
Volume Name: data
Type: Replicate
Volume ID: 54fbcafc-fed9-4bce-92ec-fa36cdcacbd4
Status: Started
Number of Bricks: 1 x (2 + 1) =3D 3
Transport-type: tcp
Bricks:
Brick1: dcastor01:/xpool/data/brick
Brick2: dcastor03:/xpool/data/brick
Brick3: dcastor02:/xpool/data/bricky (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
Volume Name: engine
Type: Replicate
Volume ID: dd4c692d-03aa-4fc6-9011-a8dad48dad96
Status: Started
Number of Bricks: 1 x (2 + 1) =3D 3
Transport-type: tcp
Bricks:
Brick1: dcastor01:/xpool/engine/brick
Brick2: dcastor02:/xpool/engine/brick
Brick3: dcastor03:/xpool/engine/brick (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
Volume Name: export
Type: Replicate
Volume ID: 23f14730-d264-4cc2-af60-196b943ecaf3
Status: Started
Number of Bricks: 1 x (2 + 1) =3D 3
Transport-type: tcp
Bricks:
Brick1: dcastor02:/xpool/export/brick
Brick2: dcastor03:/xpool/export/brick
Brick3: dcastor01:/xpool/export/brick (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
storage.owner-uid: 36
storage.owner-gid: 36
Volume Name: iso
Type: Replicate
Volume ID: b2d3d7e2-9919-400b-8368-a0443d48e82a
Status: Started
Number of Bricks: 1 x (2 + 1) =3D 3
Transport-type: tcp
Bricks:
Brick1: dcastor01:/xpool/iso/brick
Brick2: dcastor02:/xpool/iso/brick
Brick3: dcastor03:/xpool/iso/brick (arbiter)
Options Reconfigured:
performance.readdir-ahead: on
storage.owner-uid: 36
storage.owner-gid: 36
[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port=20 Online Pid
------------------------------------------------------------ ------------------
Brick dcastor01:/xpool/data/brick 49153 0 Y 3076
Brick dcastor03:/xpool/data/brick 49153 0 Y 3019
Brick dcastor02:/xpool/data/bricky 49153 0 Y 3857
NFS Server on localhost 2049 0 Y 3097
Self-heal Daemon on localhost N/A N/A Y 3088
NFS Server on dcastor03 2049 0 Y 3039
Self-heal Daemon on dcastor03 N/A N/A Y 3114
NFS Server on dcasrv02 2049 0 Y 3871
Self-heal Daemon on dcasrv02 N/A N/A Y 3864
Task Status of Volume data
------------------------------------------------------------ ------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port=20 Online Pid
------------------------------------------------------------ ------------------
Brick dcastor01:/xpool/engine/brick 49152 0 Y 3131
Brick dcastor02:/xpool/engine/brick 49152 0 Y 3852
Brick dcastor03:/xpool/engine/brick 49152 0 Y 2992
NFS Server on localhost 2049 0 Y 3097
Self-heal Daemon on localhost N/A N/A Y 3088
NFS Server on dcastor03 2049 0 Y 3039
Self-heal Daemon on dcastor03 N/A N/A Y 3114
NFS Server on dcasrv02 2049 0 Y 3871
Self-heal Daemon on dcasrv02 N/A N/A Y 3864
Task Status of Volume engine
------------------------------------------------------------ ------------------
There are no active volume tasks
Status of volume: export
Gluster process TCP Port RDMA Port=20 Online Pid
------------------------------------------------------------ ------------------
Brick dcastor02:/xpool/export/brick 49155 0 Y 3872
Brick dcastor03:/xpool/export/brick 49155 0 Y 3147
Brick dcastor01:/xpool/export/brick 49155 0 Y 3150
NFS Server on localhost 2049 0 Y 3097
Self-heal Daemon on localhost N/A N/A Y 3088
NFS Server on dcastor03 2049 0 Y 3039
Self-heal Daemon on dcastor03 N/A N/A Y 3114
NFS Server on dcasrv02 2049 0 Y 3871
Self-heal Daemon on dcasrv02 N/A N/A Y 3864
Task Status of Volume export
------------------------------------------------------------ ------------------
There are no active volume tasks
Status of volume: iso
Gluster process TCP Port RDMA Port=20 Online Pid
------------------------------------------------------------ ------------------
Brick dcastor01:/xpool/iso/brick 49154 0 Y 3152
Brick dcastor02:/xpool/iso/brick 49154 0 Y 3881
Brick dcastor03:/xpool/iso/brick 49154 0 Y 3146
NFS Server on localhost 2049 0 Y 3097
Self-heal Daemon on localhost N/A N/A Y 3088
NFS Server on dcastor03 2049 0 Y 3039
Self-heal Daemon on dcastor03 N/A N/A Y 3114
NFS Server on dcasrv02 2049 0 Y 3871
Self-heal Daemon on dcasrv02 N/A N/A Y 3864
Task Status of Volume iso
------------------------------------------------------------ ------------------
There are no active volume tasks
Thanks
Jason
*From:* users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] *On Behalf Of *Jason Jeffrey *Sent:* 03 October 2016 18:40 *To:* users@ovirt.org
*Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
Hi,
Setup log attached for primary
Regards
Jason
*From:* Simone Tiraboschi [mailto:stirabos@redhat.com <stirabos@redhat.com>] *Sent:* 03 October 2016 09:27 *To:* Jason Jeffrey <jason@sudo.co.uk> *Cc:* users <users@ovirt.org> *Subject:* Re: [ovirt-users] 4.0 - 2nd node fails on deploy
On Mon, Oct 3, 2016 at 12:45 AM, Jason Jeffrey <jason@sudo.co.uk> wrote:
Hi,
I am trying to build a x3 HC cluster, with a self hosted engine using gluster.
I have successful built the 1st node, however when I attempt to run hosted-engine =E2=80=93deploy on node 2, I get the following error
[WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host.
[ ERROR ] 'version' is not stored in the HE configuration image
[ ERROR ] Unable to get the answer file from the shared storage
[ ERROR ] Failed to execute stage 'Environment customization': Unable to get the answer file from the shared storage
[ INFO ] Stage: Clean up
[ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine- setup/answers/answers-20161002232505.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
Looking at the failure in the log file..
Can you please attach hosted-engine-setup logs from the first host?
2016-10-02 23:25:05 WARNING otopi.plugins.gr_he_common.core.remote_answerfile remote_answerfile._customization:151 A configuration
file must be supplied to deploy Hosted Engine on an additional host.
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile remote_answerfile._fetch_answer_file:61 _fetch_answer_f
ile
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile remote_answerfile._fetch_answer_file:69 fetching from:
/rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b=
------4VSAMIL16A4UO07LK9ROSU1PLADG1A Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable "no route to host" is a network problem. Looks like quorum loss is approp= riate.=20 On October 5, 2016 12:31:18 PM GMT+02:00, Sahina Bose <sabose@redhat.com>= wrote: - - - -
fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/7
8cb2527-a2e2-489a-9fad-465a72221b37
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile heconflib._dd_pipe_tar:69 executing: 'sudo -u vdsm dd i
f=3D/rhev/data-center/mnt/glusterSD/dcastor02:engine/ 0a021563-91b5-4f49-9c6b-fff45e85a025/images/f055216c- 02f9-4cd1-a22c-d6b56a0a8e9b
/78cb2527-a2e2-489a-9fad-465a72221b37 bs=3D4k'
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile heconflib._dd_pipe_tar:70 executing: 'tar -tvf -'
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile heconflib._dd_pipe_tar:88 stdout:
2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.core.remote_answerfile heconflib._dd_pipe_tar:89 stderr:
2016-10-02 23:25:05 ERROR otopi.plugins.gr_he_common.core.remote_answerfile heconflib.validateConfImage:111 'version' is not stored
in the HE configuration image
2016-10-02 23:25:05 ERROR otopi.plugins.gr_he_common.core.remote_answerfile remote_answerfile._fetch_answer_file:73 Unable to get t
he answer file from the shared storage
Looking at the detected gluster path - /rhev/data-center/mnt/ glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b- fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/
[root@dcasrv02 ~]# ls -al /rhev/data-center/mnt/ glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b- fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/
total 1049609
drwxr-xr-x. 2 vdsm kvm 4096 Oct 2 04:46 .
drwxr-xr-x. 6 vdsm kvm 4096 Oct 2 04:46 ..
-rw-rw----. 1 vdsm kvm 1073741824 Oct 2 04:46 78cb2527-a2e2-489a-9fad- 465a72221b37
-rw-rw----. 1 vdsm kvm 1048576 Oct 2 04:46 78cb2527-a2e2-489a-9fad- 465a72221b37.lease
-rw-r--r--. 1 vdsm kvm 294 Oct 2 04:46 78cb2527-a2e2-489a-9fad-465a72221b37.meta
78cb2527-a2e2-489a-9fad-465a72221b37 is a 1 GB file, is this the engine VM ?
Copying the answers file form primary (/etc/ovirt-hosted-engine/answers.conf ) to node 2 and rerunning produces the same error : (
(hosted-engine --deploy --config-append=3D/root/answers.conf )
Also tried on node 3, same issues
Happy to provide logs and other debugs
Thanks
Jason
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
------------------------------------------------------------------------
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
<<a target=3D"_blank" href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.= uk</a>></span> wrote:<br /><blockquote style=3D"margin:0px 0px 0px 0.8= ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class=3D"gmai= l_quote"><div lang=3D"EN-GB"><div><p class=3D"MsoNormal"><span style=3D"f= ont-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,= 125)">HI,<u></u><u></u></span></p><p class=3D"MsoNormal"><span style=3D"f= ont-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,= 125)"><u></u>=C2=A0<u></u></span></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">Logs attached</span></p></div></div></blockquote><div><br />Have = you probed 2 interfaces for same host, that is - dcasrv02 and=20 dcastor02? Does "gluster peer status" understand both names as = for same=20 host?<br /><br /></div><div>From glusterd logs and the mount logs - the c= onnection between the peers is lost, and quorum is lost, which is reaffir= ming what Simone said earlier.<br /></div><div>Logs seem to indicate netw= ork issues - check the direct link setup. See below<br /></div><div></div= <div><br /></div><div>From mount logs:<br />[2016-10-04 17:26:15.718300]= E [socket.c:2292:socket_connect_finish] 0-engine-client-2: connection to= <a href=3D"http://10.100.103.3:24007">10.100.103.3:24007</a> failed (No = route to host)<br />[2016-10-04 17:26:15.718345] W [MSGID: 108001] [afr-c= ommon.c:4379:afr_notify] 0-engine-replicate-0: Client-quorum is not met<b= r />[2016-10-04 17:26:16.428290] E [socket.c:2292:socket_connect_finish] = 0-engine-client-1: connection to <a href=3D"http://10.100.101.2:24007">10= .100.101.2:24007</a> failed (No route to host)<br />[2016-10-04 17:26:16.= 428336] E [MSGID: 108006] [afr-common.c:4321:afr_notify] 0-engine-replica= te-0: All subvolumes are down. Going offline until atleast one of them comes back up<br /><br /></div><div>And in glusterd l= ogs:</div>[2016-10-04 17:24:39.522402] E [socket.c:2292:socket_connect_fi= nish] 0-management: connection to <a href=3D"http://10.100.50.82:24007">1= 0.100.50.82:24007</a> failed (No route to host)<br />[2016-10-04 17:24:39= .522578] I [MSGID: 106004] [glusterd-handler.c:5201:__glusterd_peer_rpc_n= otify] 0-management: Peer <dcasrv02> (<1e788fc9-dfe9-4753-92c7-7= 6a95c8d0891>), in state <Peer in Cluster>, has disconnected from= glusterd.<br />[2016-10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-= server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management: Serve= r quorum lost for volume engine. Stopping local bricks.<br />[2016-10-04 = 17:24:39.523314] I [MSGID: 106132] [glusterd-utils.c:1560:glusterd_servic= e_stop] 0-management: brick already stopped<br />[2016-10-04 17:24:39.526= 188] E [socket.c:2292:socket_connect_finish] 0-management: connection to = <a
</div></div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><div><p = class=3D"MsoNormal">On Wed, Oct 5, 2016 at 3:28 AM, Jason Jeffrey <<a = target=3D"_blank" href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&g= t; wrote:<u></u><u></u></p><blockquote style=3D"border-width:medium mediu= m medium 1pt;border-style:none none none solid;border-color:-moz-use-text= -color -moz-use-text-color -moz-use-text-color rgb(204,204,204);padding:0= cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm"><div><div><p class=3D"= MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",= sans-serif;color:rgb(31,73,125)">Hi,</span><u></u><u></u></p><p class=3D"= MsoNormal"><span
<span style=3D"font-size:11pt;font-family:"calibri",sans-serif= ;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span style=3D"font-size:11pt;font-family:"calibri",sans-se= rif;color:rgb(31,73,125)">There may have been instances where all three w= ere not powered on, during the same period.</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">Glusterhd log attached, the xp= ool-engine-brick log is over 1 GB in size, I=E2=80=99ve taken a sample of the last =C2=A0couple d= ays, looks to be highly repative.</span><u></u><u></u></p><p class=3D"Mso= Normal"><span style=3D"font-size:11pt;font-family:"calibri",san= s-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"= MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",= sans-serif;color:rgb(31,73,125)">Cheers</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class= =3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&qu= ot;,sans-serif;color:rgb(31,73,125)">Jason</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p = class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>= <p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>= <p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"ca=
cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93=C2=A0=C2=A0=C2=A0 </span><u>= </u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-= family:"calibri",sans-serif;color:rgb(31,73,125)">[root@dcasrv0= 1 /]# ls -al /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a164-<wbr />= 76a4876ecaaf/</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">ls: cannot access /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2= -a164-<wbr />76a4876ecaaf/: No such file or directory=C2=A0=C2=A0 </span>= <u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;fo= nt-family:"calibri",sans-serif;color:rgb(31,73,125)">But the da= ta does exist </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">[ro= ot@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# ls -al</span><u= </u><u></u></p><p><span style=3D"font-size:11pt;font-family:"calibr= i",sans-serif;color:rgb(31,73,125)">drwxr-xr-x. 2 vdsm kvm=C2=A0=C2=A0= =C2=A0 4096 Oct=C2=A0 3 17:17 .</span><u></u><u></u></p><p><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 3 17:17 .= .</span><u></u><u></u></p><p><span style=3D"font-size:11pt;font-family:&q= uot;calibri",sans-serif;color:rgb(31,73,125)">-rw-rw----. 2 vdsm kvm= 1028096 Oct=C2=A0 3 20:48 cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93</s=
<span style=3D"font-size:11pt;font-family:"calibri",sans-serif= ;color:rgb(31,73,125)">Appears the symlink is broken between gluster moun= t ref and vdsm</span><u></u><u></u></p></div></div></blockquote><div><p c= lass=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNorm= al">The first host was correctly deployed but it seas that you are facing= some issue connecting the storage.<u></u><u></u></p></div><div><p class=3D= "MsoNormal">Can you please attach vdsm logs and /var/log/messages from th= e first host?<u></u><u></u></p></div></div></div></div></blockquote><div>= <p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNormal">Thanks Jason,<u></u><u></u></p></div><div><p class=3D= "MsoNormal">I suspect that your issue is related to this:<u></u><u></u></=
=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNormal">And at that po= int VDSM started complaining that the hosted-engine-storage domain doesn&= #39;t exist anymore:<u></u><u></u></p></div><div><div><p class=3D"MsoNorm= al">Oct =C2=A04 19:02:30 dcasrv01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.lib.<wbr />image.Image ERROR Error fetching volume= s list: Storage domain does not exist: (u'bbb70623-194a-46d2-a164-<wb= r />76a4876ecaaf',)<u></u><u></u></p></div><div><p class=3D"MsoNormal= ">Oct =C2=A04 19:02:30 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine= _ha.<wbr />lib.image.Image:Error fetching volumes list: Storage domain do= es not exist: (u'bbb70623-194a-46d2-a164-<wbr />76a4876ecaaf',)<u= </u><u></u></p></div></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u><= /u></p></div><div><p class=3D"MsoNormal">I see from the logs that the ovi= rt-ha-agent is trying to mount the hosted-engine storage domain as:<u></u= <u></u></p></div><div><p class=3D"MsoNormal">/usr/bin/mount -t glusterfs= -o backup-volfile-servers=3D<wbr />dcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/<wbr />glusterSD/dcastor01:engine.<u></u><u></u></p= </div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p = class=3D"MsoNormal">Pointing to dcastor01, dcastor02 and dcastor03 while your server is dcasrv01.<u></u><u></u></p></div><div><p class=3D"MsoNorma= l">But at the same time it seams that also dcasrv01 has local bricks for =
<blockquote style=3D"border-width:medium medium medium 1pt;border-style:= none none none solid;border-color:-moz-use-text-color -moz-use-text-color= -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= From broker.log</span><u></u><u></u></p><p class=3D"MsoNormal"><span styl= e=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(= 31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span s= tyle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:r= gb(31,73,125)">Thread-169::ERROR::2016-10-04 22:44:16,189::storage_broker= ::<wbr />138::<a target=3D"_blank" href=3D"http://ovirt_hosted_engine_ha.= br">ovirt_hosted_engine_ha.br</a><wbr />oker.storage_broker.<wbr />Storag= eBroker::(get_raw_stats_<wbr />for_service_type) Failed to read metadata = from /rhev/data-center/mnt/<wbr />glusterSD/dcastor01:engine/<wbr />bbb70= 623-194a-46d2-a164-<wbr />76a4876ecaaf/ha_agent/hosted-<wbr />engine.metadata</span><u></u><u></u></p><p class=3D"MsoNormal"><span st= yle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rg= b(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span=
<p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"c= alibri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u>= </p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&quo= t;calibri",sans-serif;color:rgb(31,73,125)">[root@dcasrv01 fd44dbf9-= 473a-496a-9996-<wbr />c8abe3278390]# pwd</span><u></u><u></u></p><p class= =3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&qu= ot;,sans-serif;color:rgb(31,73,125)">/xpool/engine/brick/bbb70623-<wbr />= 194a-46d2-a164-76a4876ecaaf/<wbr />images/fd44dbf9-473a-496a-<wbr />9996-= c8abe3278390</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# ls -= al</span><u></u><u></u></p><p class=3D"MsoNormal"><span
<span style=3D"font-size:11pt;font-family:"calibri",sans-serif= ;color:rgb(31,73,125)">drwxr-xr-x. 2 vdsm kvm=C2=A0=C2=A0=C2=A0 4096 Oct=C2= =A0 3 17:17 .</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 3 17:17 .= .</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size= :11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">-r= w-rw----. 2 vdsm kvm 1028096 Oct=C2=A0 3 20:48 cee9440c-4eb8-453b-bc04-<w= br />c47e6f9cbc93</span><u></u><u></u></p><p class=3D"MsoNormal"><span st= yle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rg= b(31,73,125)">-rw-rw----. 2 vdsm kvm 1048576 Oct=C2=A0 3 17:17 cee9440c-4= eb8-453b-bc04-<wbr />c47e6f9cbc93.lease</span><u></u><u></u></p><p class=3D= "MsoNormal"><span
Type: Replicate</span><u></u><u></u></p><p class=3D"MsoNormal"><span sty= le=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb= (31,73,125)">Volume ID: 54fbcafc-fed9-4bce-92ec-<wbr />fa36cdcacbd4</span= <u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">Status: S= tarted</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font= -size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125= )">Number of Bricks: 1 x (2 + 1) =3D 3</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">Transport-type: tcp</span><u></u><u></u= </p><p class=3D"MsoNormal"><span
</u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family= :"calibri",sans-serif;color:rgb(31,73,125)">Options Reconfigure= d:</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-siz= e:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">p= erformance.readdir-ahead: on</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">performance.quick-read: off</sp= an><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt= ;font-family:"calibri",sans-serif;color:rgb(31,73,125)">perform= ance.read-ahead: off</span><u></u><u></u></p><p class=3D"MsoNormal"><span=
--=20 Sent from my Android device with K-9 Mail. Please excuse my brevity. ------4VSAMIL16A4UO07LK9ROSU1PLADG1A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><head></head><body>"no route to host" is a network proble= m. Looks like quorum loss is appropriate. <br><br><div class=3D"gmail_quo= te">On October 5, 2016 12:31:18 PM GMT+02:00, Sahina Bose <sabose@redh= at.com> wrote:<blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0= pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1e= x;"> <div dir=3D"ltr"><br /><div class=3D"gmail_extra"><br /><div class=3D"gma= il_quote">On Wed, Oct 5, 2016 at 1:56 PM, Jason Jeffrey <span dir=3D"ltr"= href=3D"http://10.100.103.3:24007">10.100.103.3:24007</a> failed (No rout= e to host)<br />[2016-10-04 17:24:39.526219] I [MSGID: 106004] [glusterd-= handler.c:5201:__glusterd_peer_rpc_notify] 0-management: Peer <dcastor= 03> (<9a9c037e-96cd-4f73-9800-a1df5cdd2818>), in state <Peer = in Cluster>, has disconnected from glusterd.<br /><br /><br /><blockqu= ote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex" class=3D"gmail_quote"><div lang=3D"EN-GB"><div><p c= lass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibr= i",sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p><p clas= s=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&q= uot;,sans-serif;color:rgb(31,73,125)"><br /></span></p></div></div></bloc= kquote><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli= d rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div lang=3D"E= N-GB"><div><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)"><u></u></span></p><p class=3D"MsoNormal"><span style=3D"f= ont-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,= 125)">Thanks <u></u><u></u></span></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)"><u></u>=C2=A0<u></u></span></p><p class=3D"MsoNormal"><b><span st= yle=3D"font-size:11pt;font-family:"calibri",sans-serif" lang=3D= "EN-US">From:</span></b><span style=3D"font-size:11pt;font-family:"c= alibri",sans-serif" lang=3D"EN-US"> Sahina Bose [mailto:<a target=3D= "_blank" href=3D"mailto:sabose@redhat.com">sabose@redhat.com</a>] <br /><= b>Sent:</b> 05 October 2016 08:11<br /><b>To:</b> Jason Jeffrey <<a ta= rget=3D"_blank" href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>>= ; <a target=3D"_blank" href=3D"mailto:gluster-users@gluster.org">gluster-= users@gluster.org</a>; Ravishankar Narayanankutty <<a target=3D"_blank= " href=3D"mailto:ravishankar@redhat.com">ravishankar@redhat.com</a>><br = /><b>Cc:</b> Simone Tiraboschi <<a target=3D"_blank" href=3D"mailto:st= irabos@redhat.com">stirabos@redhat.com</a>>; users <<a target=3D"_b= lank" href=3D"mailto:users@ovirt.org">users@ovirt.org</a>></span></p><= div><div class=3D"gmail-h5"><br /><b>Subject:</b> Re: [ovirt-users] 4.0 -= 2nd node fails on deploy<u></u><u></u></div></div><p></p><div><div class= =3D"gmail-h5"><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p><div><div><d= iv><p class=3D"MsoNormal">[Adding gluster-users ML]<u></u><u></u></p></di= v><p style=3D"margin-bottom:12pt" class=3D"MsoNormal">The brick logs are = filled with errors :<br />[2016-10-05 19:30:28.659061] E [MSGID: 113077] = [posix-handle.c:309:posix_<wbr />handle_pump] 0-engine-posix: malformed i= nternal link /var/run/vdsm/storage/<wbr />0a021563-91b5-4f49-9c6b-<wbr />= fff45e85a025/d84f0551-0f2b-<wbr />457c-808c-6369c6708d43/<wbr />1b5a5e34-= 818c-4914-8192-<wbr />2f05733b5583 for /xpool/engine/brick/.<wbr />glusterfs/b9/8e/b98ed8d2-3bf9-<wbr />4b11-92fd-ca5324e131a8 <br />[2016= -10-05 19:30:28.659069] E [MSGID: 113091] [posix.c:180:posix_lookup] 0-en= gine-posix: Failed to create inode handle for path <gfid:b98ed8d2-3bf9= -4b11-92fd-<wbr />ca5324e131a8> <br />The message "E [MSGID: 1130= 18] [posix.c:198:posix_lookup] 0-engine-posix: lstat on null failed"= repeated 3 times between [2016-10-05 19:30:28.656529] and [2016-10-05 19= :30:28.659076] <br />[2016-10-05 19:30:28.659087] W [MSGID: 115005] [serv= er-resolve.c:126:resolve_<wbr />gfid_cbk] 0-engine-server: b98ed8d2-3bf9-= 4b11-92fd-<wbr />ca5324e131a8: failed to resolve (Success) <u></u><u></u>= </p></div><p class=3D"MsoNormal">- Ravi, the above are from the data bric= k of the arbiter volume. Can you take a look?<u></u><u></u></p><div><p cl= ass=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div><div><p class=3D"MsoNorma= l">Jason,<u></u><u></u></p></div><div><div><div><p style=3D"margin-bottom= :12pt" class=3D"MsoNormal">Could you also provide the mount logs from the first host (/var/log/glusterfs/rhev-data-<wbr />center-mnt-= glusterSD*engine.<wbr />log) and glusterd log (/var/log/glusterfs/etc-<wb= r />glusterfs-glusterd.vol.log) around the same time frame.<u></u><u></u>= </p></div><div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p></div></div= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an style=3D"font-size:11pt;font-family:"calibri",sans-serif;col= or:rgb(31,73,125)">Servers are powered =C2=A0off =C2=A0when I=E2=80=99m n= ot looking at the problem.</span><u></u><u></u></p><p class=3D"MsoNormal"= libri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u><= /p><p class=3D"MsoNormal"><b><span style=3D"font-size:11pt;font-family:&q= uot;calibri",sans-serif" lang=3D"EN-US">From:</span></b><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif" lang=3D"EN-US= "> Simone Tiraboschi [mailto:<a target=3D"_blank" href=3D"mailto:stirabos= @redhat.com">stirabos@redhat.com</a>] <br /><b>Sent:</b> 04 October 2016 = 16:50</span><u></u><u></u></p><div><div><p class=3D"MsoNormal"><br /><b>T= o:</b> Jason Jeffrey <<a target=3D"_blank" href=3D"mailto:jason@sudo.c= o.uk">jason@sudo.co.uk</a>><br /><b>Cc:</b> users <<a target=3D"_bl= ank" href=3D"mailto:users@ovirt.org">users@ovirt.org</a>><br /><b>Subj= ect:</b> Re: [ovirt-users] 4.0 - 2nd node fails on deploy<u></u><u></u></p></div></div><div><div><p class=3D"MsoNormal">=C2=A0= <u></u><u></u></p><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><di= v><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p class=3D"MsoNorm= al">On Tue, Oct 4, 2016 at 5:22 PM, Jason Jeffrey <<a target=3D"_blank= " href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>> wrote:<u></u>= <u></u></p><blockquote style=3D"border-width:medium medium medium 1pt;bor= der-style:none none none solid;border-color:-moz-use-text-color -moz-use-= text-color -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;m= argin:5pt 0cm 5pt 4.8pt"><div><div><p class=3D"MsoNormal"><span style=3D"= font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73= ,125)">Hi,</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"= font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73= ,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">DCASTORXX is a hosts entry for dedicated =C2=A0direct 10G= B links (each private /28) between the x3 servers=C2=A0 i.e 1=3D> 2&am= p;3, 2=3D> 1&3, etc) planned to be used solely for storage.</span>= <u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;fo= nt-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0</sp= an><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt= ;font-family:"calibri",sans-serif;color:rgb(31,73,125)">I,e </s= pan><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11p= t;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">10.= 100.50.81=C2=A0=C2=A0=C2=A0 dcasrv01</span><u></u><u></u></p><p class=3D"= MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">10.100.101.1=C2=A0=C2=A0=C2=A0 dcastor01</span><u></u><u>= </u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:= "calibri",sans-serif;color:rgb(31,73,125)">10.100.50.82=C2=A0=C2= =A0=C2=A0 dcasrv02</span><u></u><u></u></p><p class=3D"MsoNormal"><span s= tyle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:r= gb(31,73,125)">10.100.101.2=C2=A0=C2=A0=C2=A0 dcastor02</span><u></u><u><= /u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&= quot;calibri",sans-serif;color:rgb(31,73,125)">10.100.50.83=C2=A0=C2= =A0=C2=A0 dcasrv03</span><u></u><u></u></p><p class=3D"MsoNormal"><span s= tyle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:r= gb(31,73,125)">10.100.103.3=C2=A0=C2=A0=C2=A0 dcastor03=C2=A0 </span><u><= /u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-f= amily:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><= u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">These were setup with the gluster commands</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></= u><u></u></p><p><span style=3D"font-size:11pt;font-family:symbol;color:rg= b(31,73,125)">=C2=B7</span><span style=3D"font-size:7pt;color:rgb(31,73,1= 25)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">gluster volume create iso replica 3 arbiter 1=C2=A0 dcastor01:/xp= ool/iso/brick=C2=A0=C2=A0 dcastor02:/xpool/iso/brick=C2=A0=C2=A0 dcastor0= 3:/xpool/iso/brick</span><u></u><u></u></p><p><span style=3D"font-size:11= pt;font-family:symbol;color:rgb(31,73,125)">=C2=B7</span><span style=3D"f= ont-size:7pt;color:rgb(31,73,125)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 </span><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">gluster volume create export repli= ca 3 arbiter 1=C2=A0 dcastor02:/xpool/export/brick=C2=A0 dcastor03:/xpool/export/brick= =C2=A0 dcastor01:/xpool/export/brick=C2=A0 </span><u></u><u></u></p><p><s= pan style=3D"font-size:11pt;font-family:symbol;color:rgb(31,73,125)">=C2=B7= </span><span style=3D"font-size:7pt;color:rgb(31,73,125)">=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span style=3D"font-size:11pt;fo= nt-family:"calibri",sans-serif;color:rgb(31,73,125)">gluster vo= lume create engine replica 3 arbiter 1 dcastor01:/xpool/engine/brick dcas= tor02:/xpool/engine/brick dcastor03:/xpool/engine/brick</span><u></u><u><= /u></p><p><span style=3D"font-size:11pt;font-family:symbol;color:rgb(31,7= 3,125)">=C2=B7</span><span style=3D"font-size:7pt;color:rgb(31,73,125)">=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><span style=3D"font-= size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)= ">gluster volume create data replica 3 arbiter 1=C2=A0 dcastor01:/xpool/d= ata/brick=C2=A0 dcastor03:/xpool/data/brick=C2=A0 dcastor02:/xpool/data/b= ricky</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an style=3D"font-size:11pt;font-family:"calibri",sans-serif;col= or:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal">= <span style=3D"font-size:11pt;font-family:"calibri",sans-serif;= color:rgb(31,73,125)">So yes, DCASRV01 is the server (pri) and have local= bricks access through DCASTOR01 interface </span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">Is the issue here not the inco= rrect soft link ?</span><u></u><u></u></p></div></div></blockquote><div><= p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoN= ormal">No, this should be fine.<u></u><u></u></p></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNor= mal">The issue is that periodically your gluster volume losses its server= quorum and become unavailable.<u></u><u></u></p></div><div><p class=3D"M= soNormal">It happened more than once from your logs.<u></u><u></u></p></d= iv><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p clas= s=3D"MsoNormal">Can you please attach also gluster logs for that volume?<= u></u><u></u></p></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></= p></div><blockquote style=3D"border-width:medium medium medium 1pt;border= -style:none none none solid;border-color:-moz-use-text-color -moz-use-tex= t-color -moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;marg= in:5pt 0cm 5pt 4.8pt"><div><div><p class=3D"MsoNormal"><span style=3D"fon= t-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,12= 5)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">lrwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:27 hosted= -engine.metadata -> /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a1= 64-<wbr />76a4876ecaaf/fd44dbf9-473a-<wbr />496a-9996-c8abe3278390/<wbr /= pan><u></u><u></u></p><p><span style=3D"font-size:11pt;font-family:"= calibri",sans-serif;color:rgb(31,73,125)">-rw-rw----. 2 vdsm kvm 104= 8576 Oct=C2=A0 3 17:17 cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93.lease</span><u></u><u></u></p><p><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">-rw= -r--r--. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0 283 Oct=C2=A0 3 17:17 cee9440= c-4eb8-453b-bc04-<wbr />c47e6f9cbc93.meta=C2=A0=C2=A0 </span><u></u><u></= u></p><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><p class=3D"MsoNorma= l">Thanks <u></u><u></u></p><p class=3D"MsoNormal">=C2=A0<u></u><u></u></= p><p class=3D"MsoNormal">Jason <u></u><u></u></p><p class=3D"MsoNormal"><= span style=3D"font-size:11pt;font-family:"calibri",sans-serif;c= olor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal= "><span style=3D"font-size:11pt;font-family:"calibri",sans-seri= f;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNor= mal"><span style=3D"font-size:11pt;font-family:"calibri",sans-s= erif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"Mso= Normal"><b><span style=3D"font-size:11pt;font-family:"calibri",= sans-serif" lang=3D"EN-US">From:</span></b><span style=3D"font-size:11pt;font-family:"calibri",sans-serif" lang=3D= "EN-US"> Simone Tiraboschi [mailto:<a target=3D"_blank" href=3D"mailto:st= irabos@redhat.com">stirabos@redhat.com</a>] <br /><b>Sent:</b> 04 October= 2016 14:40</span><u></u><u></u></p><div><div><p class=3D"MsoNormal"><br = /><b>To:</b> Jason Jeffrey <<a target=3D"_blank" href=3D"mailto:jason@= sudo.co.uk">jason@sudo.co.uk</a>><br /><b>Cc:</b> users <<a target=3D= "_blank" href=3D"mailto:users@ovirt.org">users@ovirt.org</a>><br /><b>= Subject:</b> Re: [ovirt-users] 4.0 - 2nd node fails on deploy<u></u><u></= u></p></div></div><div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></= p><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p class=3D"Ms= oNormal">=C2=A0<u></u><u></u></p><div><p class=3D"MsoNormal">On Tue, Oct = 4, 2016 at 10:51 AM, Simone Tiraboschi <<a target=3D"_blank" href=3D"m= ailto:stirabos@redhat.com">stirabos@redhat.com</a>> wrote:<u></u><u></= u></p><blockquote style=3D"border-width:medium medium medium 1pt;border-s= tyle:none none none solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-= text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.= 8pt"><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p class=3D= "MsoNormal">=C2=A0<u></u><u></u></p><div><p class=3D"MsoNormal">On Mon, O= ct 3, 2016 at 11:56 PM, Jason Jeffrey <<a target=3D"_blank" href=3D"ma= ilto:jason@sudo.co.uk">jason@sudo.co.uk</a>> wrote:<u></u><u></u></p><= blockquote style=3D"border-width:medium medium medium 1pt;border-style:no= ne none none solid;border-color:-moz-use-text-color -moz-use-text-color -= moz-use-text-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0c= m 5pt 4.8pt"><div><div><p class=3D"MsoNormal"><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Hi,</= span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Another problem has appeared, after rebooting the primary= the VM will not start.</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"= p></div><div><div><p class=3D"MsoNormal">Oct =C2=A04 18:24:39 dcasrv01 et= c-glusterfs-glusterd.vol[<wbr />2252]: [2016-10-04 17:24:39.522620] C [MS= GID: 106002] [glusterd-server-quorum.c:351:<wbr />glusterd_do_volume_quor= um_<wbr />action] 0-management: Server quorum lost for volume data. Stopp= ing local bricks.<u></u><u></u></p></div><div><p class=3D"MsoNormal">Oct = =C2=A04 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[<wbr />2252]: [2016-= 10-04 17:24:39.523272] C [MSGID: 106002] [glusterd-server-quorum.c:351:<w= br />glusterd_do_volume_quorum_<wbr />action] 0-management: Server quorum= lost for volume engine. Stopping local bricks.<u></u><u></u></p></div></= div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p cla= ss=3D"MsoNormal">and for some time your gluster volume has been working.<= u></u><u></u></p></div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNor= mal">But then:<u></u><u></u></p></div><div><div><p class=3D"MsoNormal">Oc= t =C2=A04 19:02:09 dcasrv01 systemd: Started /usr/bin/mount -t glusterfs = -o backup-volfile-servers=3D<wbr />dcastor02:dcastor03 dcastor01:engine /= rhev/data-center/mnt/<wbr />glusterSD/dcastor01:engine.<u></u><u></u></p>= </div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:09 dcasrv01 systemd: = Starting /usr/bin/mount -t glusterfs -o backup-volfile-servers=3D<wbr />d= castor02:dcastor03 dcastor01:engine /rhev/data-center/mnt/<wbr />glusterS= D/dcastor01:engine.<u></u><u></u></p></div><div><p class=3D"MsoNormal">Oc= t =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-<wbr = />packages/yajsonrpc/stomp.py:<wbr />352: DeprecationWarning: Dispatcher.= pending is deprecated. Use Dispatcher.socket.pending instead.<u></u><u></= u></p></div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:11 dcasrv01 ovi= rt-ha-agent: pending =3D getattr(dispatcher, 'pending', lambda: 0)<u></u><u></u></p></div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:1= 1 dcasrv01 ovirt-ha-agent: /usr/lib/python2.7/site-<wbr />packages/yajson= rpc/stomp.py:<wbr />352: DeprecationWarning: Dispatcher.pending is deprec= ated. Use Dispatcher.socket.pending instead.<u></u><u></u></p></div><div>= <p class=3D"MsoNormal">Oct =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent: pend= ing =3D getattr(dispatcher, 'pending', lambda: 0)<u></u><u></u></= p></div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:11 dcasrv01 journal= : vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof= <u></u><u></u></p></div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:11 = dcasrv01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.<wbr />agen= t.Agent ERROR Error: 'Connection to storage server failed' - tryi= ng to restart agent<u></u><u></u></p></div><div><p class=3D"MsoNormal">Oc= t =C2=A04 19:02:11 dcasrv01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.= <wbr />agent.agent.Agent:Error: 'Connection to storage server failed&= #39; - trying to restart agent<u></u><u></u></p></div><div><p class=3D"MsoNormal= ">Oct =C2=A04 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[<wbr />2252]: = [2016-10-04 18:02:12.384611] C [MSGID: 106003] [glusterd-server-quorum.c:= 346:<wbr />glusterd_do_volume_quorum_<wbr />action] 0-management: Server = quorum regained for volume data. Starting local bricks.<u></u><u></u></p>= </div><div><p class=3D"MsoNormal">Oct =C2=A04 19:02:12 dcasrv01 etc-glust= erfs-glusterd.vol[<wbr />2252]: [2016-10-04 18:02:12.388981] C [MSGID: 10= 6003] [glusterd-server-quorum.c:346:<wbr />glusterd_do_volume_quorum_<wbr= />action] 0-management: Server quorum regained for volume engine. Starti= ng local bricks.<u></u><u></u></p></div></div><div><p class=3D"MsoNormal"= the same engine volume.<u></u><u></u></p></div><div><p class=3D"MsoNormal= ">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNormal">So, is dcasrv= 01 just an alias fro dcastor01? if not you probably have some issue with = the configuration of your gluster volume.<u></u><u></u></p></div><div><p = class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNor= mal">=C2=A0<u></u><u></u></p></div><blockquote style=3D"border-width:medi= um medium medium 1pt;border-style:none none none solid;border-color:-moz-= use-text-color -moz-use-text-color -moz-use-text-color rgb(204,204,204);p= adding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><div><div><div= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color= :rgb(31,73,125)">[root@dcasrv01 ovirt-hosted-engine-ha]# ls -al /rhev/dat= a-center/mnt/<wbr />glusterSD/dcastor01\:engine/<wbr />bbb70623-194a-46d2= -a164-<wbr />76a4876ecaaf/ha_agent/</span><u></u><u></u></p><p class=3D"M= soNormal"><span style=3D"font-size:11pt;font-family:"calibri",s= ans-serif;color:rgb(31,73,125)">total 9</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">drwxrwx---. 2 vdsm kvm 4096 Oct=C2=A0 3= 17:27 .</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"fo= nt-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,1= 25)">drwxr-xr-x. 5 vdsm kvm 4096 Oct=C2=A0 3 17:17 ..</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-siz= e:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">l= rwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:27 hosted-engine.lockspace= -> /var/run/vdsm/storage/<wbr />bbb70623-194a-46d2-a164-<wbr />76a487= 6ecaaf/23d81b73-bcb7-<wbr />4742-abde-128522f43d78/<wbr />11d6a3e1-1817-4= 29d-b2e0-<wbr />9051a3cf41a4</span><u></u><u></u></p><p class=3D"MsoNorma= l"><span style=3D"font-size:11pt;font-family:"calibri",sans-ser= if;color:rgb(31,73,125)">lrwxrwxrwx. 1 vdsm kvm=C2=A0 132 Oct=C2=A0 3 17:= 27 hosted-engine.metadata -> /var/run/vdsm/storage/<wbr />bbb70623-194= a-46d2-a164-<wbr />76a4876ecaaf/fd44dbf9-473a-<wbr />496a-9996-c8abe32783= 90/<wbr />cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93=C2=A0=C2=A0=C2=A0 <= /span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:1= 1pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">[root@dcasrv01 /]# ls -al /var/run/vdsm/storage/<wbr />bb= b70623-194a-46d2-a164-<wbr />76a4876ecaaf/</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">ls: cannot access /var/run/vdsm/st= orage/<wbr />bbb70623-194a-46d2-a164-<wbr />76a4876ecaaf/: No such file o= r directory=C2=A0=C2=A0 </span><u></u><u></u></p><p class=3D"MsoNormal"><= span style=3D"font-size:11pt;font-family:"calibri",sans-serif;c= olor:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal= "><span style=3D"font-size:11pt;font-family:"calibri",sans-seri= f;color:rgb(31,73,125)">Though file appears to be there </span><u></u><u>= </u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:= "calibri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u>= <u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Gluster is setup as xpool/engine </span><u></u><u></u></p= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">total 2060</span><u></u><u></u></p><p class=3D"MsoNormal"= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">-rw-r--r--. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0 283 Oct=C2= =A0 3 17:17 cee9440c-4eb8-453b-bc04-<wbr />c47e6f9cbc93.meta=C2=A0=C2=A0 = </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= =C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font= -size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125= )">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe3278390]# gluster v= olume info</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"= font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73= ,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">Volume Name: data</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-s= ize:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)"= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Bricks:</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">Brick1: dcastor01:/xpool/data/brick</span><u></u><u><= /u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&= quot;calibri",sans-serif;color:rgb(31,73,125)">Brick2: dcastor03:/xp= ool/data/brick</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">Brick3: dcastor02:/xpool/data/bricky (arbiter)</span><u></u><u= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color= :rgb(31,73,125)">performance.io-cache: off</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">performance.stat-prefetch: off</sp= an><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt= ;font-family:"calibri",sans-serif;color:rgb(31,73,125)">cluster= .eager-lock: enable</span><u></u><u></u></p><p class=3D"MsoNormal"><span = style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">network.remote-dio: enable</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">cluster.quorum-type: auto</span=
<u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">cluster.s= erver-quorum-type: server</span><u></u><u></u></p><p class=3D"MsoNormal">= <span style=3D"font-size:11pt;font-family:"calibri",sans-serif;=
color:rgb(31,73,125)">storage.owner-uid: 36</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">storage.owner-gid: 36</span><u></= u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-fa= mily:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u= ></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font= -family:"calibri",sans-serif;color:rgb(31,73,125)">Volume Name:= engine</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Type: Replicate</span><u></u><u></u></p><p class=3D"MsoNo= rmal"><span style=3D"font-size:11pt;font-family:"calibri",sans-= serif;color:rgb(31,73,125)">Volume ID: dd4c692d-03aa-4fc6-9011-<wbr />a8d= ad48dad96</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"f= ont-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,= 125)">Status: Started</span><u></u><u></u></p><p class=3D"MsoNormal"><spa= n style=3D"font-size:11pt;font-family:"calibri",sans-serif;colo= r:rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D 3</span><u></u><u></u= ></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&qu= ot;calibri",sans-serif;color:rgb(31,73,125)">Transport-type: tcp</sp= an><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt= ;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Bricks:= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Brick1: dcastor01:/xpool/engine/brick</span><u></u><u></u= ></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&qu= ot;calibri",sans-serif;color:rgb(31,73,125)">Brick2: dcastor02:/xpoo= l/engine/brick</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">Brick3: dcastor03:/xpool/engine/brick (arbiter)</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">Options Reconfigur= ed:</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= performance.readdir-ahead: on</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span style=3D"font-size:11pt;font-family:"calibri",sans-se= rif;color:rgb(31,73,125)">performance.quick-read: off</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= performance.read-ahead: off</span><u></u><u></u></p><p class=3D"MsoNormal= "><span style=3D"font-size:11pt;font-family:"calibri",sans-seri= f;color:rgb(31,73,125)">performance.io-cache: off</span><u></u><u></u></p= ><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"c= alibri",sans-serif;color:rgb(31,73,125)">performance.stat-prefetch: = off</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= cluster.eager-lock: enable</span><u></u><u></u></p><p class=3D"MsoNormal"= ><span style=3D"font-size:11pt;font-family:"calibri",sans-serif= ;color:rgb(31,73,125)">network.remote-dio: enable</span><u></u><u></u></p= ><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"c= alibri",sans-serif;color:rgb(31,73,125)">cluster.quorum-type: auto</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-s= ize:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)"= >cluster.server-quorum-type: server</span><u></u><u></u></p><p class=3D"M= soNormal"><span style=3D"font-size:11pt;font-family:"calibri",s= ans-serif;color:rgb(31,73,125)">storage.owner-uid: 36</span><u></u><u></u= ></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&qu= ot;calibri",sans-serif;color:rgb(31,73,125)">storage.owner-gid: 36</= span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2=A0= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Vol= ume Name: export</span><u></u><u></u></p><p class=3D"MsoNormal"><span sty= le=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb= (31,73,125)">Type: Replicate</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">Volume ID: 23f14730-d264-4cc2-a= f60-<wbr />196b943ecaf3</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">Status: Started</span><u></u><u></u></p><p class=3D"M= soNormal"><span style=3D"font-size:11pt;font-family:"calibri",s= ans-serif;color:rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D 3</span= ><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">Transport= -type: tcp</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"= font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73= ,125)">Bricks:</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">Brick1: dcastor02:/xpool/export/brick</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span style=3D"font-size:11pt;font-family:"calibri",sans-se= rif;color:rgb(31,73,125)">Brick2: dcastor03:/xpool/export/brick</span><u>= </u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-= family:"calibri",sans-serif;color:rgb(31,73,125)">Brick3: dcast= or01:/xpool/export/brick (arbiter)</span><u></u><u></u></p><p class=3D"Ms= oNormal"><span style=3D"font-size:11pt;font-family:"calibri",sa= ns-serif;color:rgb(31,73,125)">Options Reconfigured:</span><u></u><u></u>= </p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&quo= t;calibri",sans-serif;color:rgb(31,73,125)">performance.readdir-ahea= d: on</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-= size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)= ">storage.owner-uid: 36</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">storage.owner-gid: 36</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class= =3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&qu= ot;,sans-serif;color:rgb(31,73,125)">Volume Name: iso</span><u></u><u></u= ></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&qu= ot;calibri",sans-serif;color:rgb(31,73,125)">Type: Replicate</span><= u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;fon= t-family:"calibri",sans-serif;color:rgb(31,73,125)">Volume ID: = b2d3d7e2-9919-400b-8368-<wbr />a0443d48e82a</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">Status: Started</span><u></u><u><= /u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Number of Bricks: 1 x (2 + 1) =3D 3</span><u></u><u></u><= /p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"= ;calibri",sans-serif;color:rgb(31,73,125)">Transport-type: tcp</span= ><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">Bricks:</= span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Brick= 1: dcastor01:/xpool/iso/brick</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span style=3D"font-size:11pt;font-family:"calibri",sans-se= rif;color:rgb(31,73,125)">Brick2: dcastor02:/xpool/iso/brick</span><u></u= ><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-fam= ily:"calibri",sans-serif;color:rgb(31,73,125)">Brick3: dcastor0= 3:/xpool/iso/brick (arbiter)</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">Options Reconfigured:</span><u>= </u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-= family:"calibri",sans-serif;color:rgb(31,73,125)">performance.r= eaddir-ahead: on</span><u></u><u></u></p><p class=3D"MsoNormal"><span sty= le=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb= (31,73,125)">storage.owner-uid: 36</span><u></u><u></u></p><p class=3D"Ms= oNormal"><span style=3D"font-size:11pt;font-family:"calibri",sa= ns-serif;color:rgb(31,73,125)">storage.owner-gid: 36=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<= wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an style=3D"font-size:11pt;font-family:"calibri",sans-serif;col= or:rgb(31,73,125)">[root@dcasrv01 fd44dbf9-473a-496a-9996-<wbr />c8abe327= 8390]# gluster volume status</span><u></u><u></u></p><p class=3D"MsoNorma= l"><span style=3D"font-size:11pt;font-family:"calibri",sans-ser= if;color:rgb(31,73,125)">Status of volume: data</span><u></u><u></u></p><= p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cal= ibri",sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0 Pid</span><u></u= ><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-fam= ily:"calibri",sans-serif;color:rgb(31,73,125)">----------------= --------------<wbr />------------------------------<wbr />---------------= ---</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">Brick dcastor01:/xpool/data/bri= ck=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 491= 53=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3076</span><u></u><u></u>= </p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&quo= t;calibri",sans-serif;color:rgb(31,73,125)">Brick dcastor03:/xpool/d= ata/brick=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 49153=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3019</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">Brick dcastor02:/x= pool/data/bricky=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 49153=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3857</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">NFS Server on loca= lhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A03097</span><u></u><u></u></p><p class=3D"MsoNorma= l"><span style=3D"font-size:11pt;font-family:"calibri",sans-ser= if;color:rgb(31,73,125)">Self-heal Daemon on localhost=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3088</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 3039</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">Self-heal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3114</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 3871</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3864</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p= ><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"c= alibri",sans-serif;color:rgb(31,73,125)">Task Status of Volume data<= /span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:1= 1pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">----= --------------------------<wbr />------------------------------<wbr />---= ---------------</span><u></u><u></u></p><p class=3D"MsoNormal"><span styl= e=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(= 31,73,125)">There are no active volume tasks</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-= size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)= ">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"fo= nt-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,1= 25)">Status of volume: engine</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span style=3D"font-size:11pt;font-family:"calibri",sans-se= rif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TCP P= ort=C2=A0 RDMA Port=C2=A0 Online=C2=A0 Pid</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">------------------------------<wbr= />------------------------------<wbr />------------------</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">Brick dcastor01:/x= pool/engine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49152=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3131</span><u></u><u><= /u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&= quot;calibri",sans-serif;color:rgb(31,73,125)">Brick dcastor02:/xpoo= l/engine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 491= 52=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3852</span><u></u><u></u>= </p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&quo= t;calibri",sans-serif;color:rgb(31,73,125)">Brick dcastor03:/xpool/e= ngine/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49152=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2992</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">NFS Server on localhost=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 3097</span><u></u><u></u></p><p class=3D"MsoNormal"><s= pan style=3D"font-size:11pt;font-family:"calibri",sans-serif;co= lor:rgb(31,73,125)">Self-heal Daemon on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3088= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">NFS= Server on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3039</span><u></u><u></u>= </p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:&quo= t;calibri",sans-serif;color:rgb(31,73,125)">Self-heal Daemon on dcas= tor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3114</s= pan><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11p= t;font-family:"calibri",sans-serif;color:rgb(31,73,125)">NFS Se= rver on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 204= 9=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3871</span><u></u><u><= /u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3864</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">Task Status of Volume engine</= span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">-----= -------------------------<wbr />------------------------------<wbr />----= --------------</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">There are no active volume tasks</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an style=3D"font-size:11pt;font-family:"calibri",sans-serif;col= or:rgb(31,73,125)">Status of volume: export</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri= ",sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0 Pid</span><u></u><u= ></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family= :"calibri",sans-serif;color:rgb(31,73,125)">-------------------= -----------<wbr />------------------------------<wbr />------------------= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Bri= ck dcastor02:/xpool/export/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3872</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-s= ize:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)"= >Brick dcastor03:/xpool/export/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3147= </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Bri= ck dcastor01:/xpool/export/brick=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 49155=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3150</span= ><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">NFS Serve= r on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3097</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"cali= bri",sans-serif;color:rgb(31,73,125)">Self-heal Daemon on localhost=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3088</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 3039</span><u></u><u></u></p><p class=3D"MsoNormal"><span= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color= :rgb(31,73,125)">Self-heal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3114</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 3871</span><u></u><u></u></p><p class=3D"MsoNormal"><span= style=3D"font-size:11pt;font-family:"calibri",sans-serif;color= :rgb(31,73,125)">Self-heal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 3864</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span st= yle=3D"font-size:11pt;font-family:"calibri",sans-serif;color:rg= b(31,73,125)">Task Status of Volume export</span><u></u><u></u></p><p cla= ss=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&= quot;,sans-serif;color:rgb(31,73,125)">------------------------------<wbr= />------------------------------<wbr />------------------</span><u></u><= u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-famil= y:"calibri",sans-serif;color:rgb(31,73,125)">There are no activ= e volume tasks</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">Status of volume: iso</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri"= ,sans-serif;color:rgb(31,73,125)">Gluster process=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 TCP Port=C2=A0 RDMA Port=C2=A0 Online=C2=A0 Pid</span><u></u><u></u><= /p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"= ;calibri",sans-serif;color:rgb(31,73,125)">-------------------------= -----<wbr />------------------------------<wbr />------------------</span= ><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">Brick dca= stor01:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3= 152</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= Brick dcastor02:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 3881</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-= size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)= ">Brick dcastor03:/xpool/iso/brick=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49154=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 3146</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">NFS Server on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3097</span= ><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;f= ont-family:"calibri",sans-serif;color:rgb(31,73,125)">Self-heal= Daemon on localhost=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/= A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 3088</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">NFS Server on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 =C2=A02049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3039</s= pan><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11p= t;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Self-h= eal Daemon on dcastor03=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 3114</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">NFS Server on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 2049=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3871</s= pan><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11p= t;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Self-h= eal Daemon on dcasrv02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 N/A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Y=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 3864</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>= <p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"ca= libri",sans-serif;color:rgb(31,73,125)">Task Status of Volume iso</s= pan><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11p= t;font-family:"calibri",sans-serif;color:rgb(31,73,125)">------= ------------------------<wbr />------------------------------<wbr />-----= -------------</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">There are no active volume tasks</span><u></u><u></u></p><p class= =3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri&qu= ot;,sans-serif;color:rgb(31,73,125)">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr= />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<wbr />=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:= 11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">Tha= nks</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= =C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font= -size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125= )">Jason</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"fo= nt-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,1= 25)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,7= 3,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"font-size:11pt;font-family:"calibri",sans-serif;color:rgb(3= 1,73,125)">=C2=A0</span><u></u><u></u></p><div><div style=3D"border-width= :1pt medium medium;border-style:solid none none;border-color:rgb(225,225,= 225) -moz-use-text-color -moz-use-text-color;padding:3pt 0cm 0cm"><p class=3D"= MsoNormal"><b><span style=3D"font-size:11pt;font-family:"calibri&quo= t;,sans-serif" lang=3D"EN-US">From:</span></b><span style=3D"font-size:11= pt;font-family:"calibri",sans-serif" lang=3D"EN-US"> <a target=3D= "_blank" href=3D"mailto:users-bounces@ovirt.org">users-bounces@ovirt.org<= /a> [mailto:<a target=3D"_blank" href=3D"mailto:users-bounces@ovirt.org">= users-bounces@ovirt.<wbr />org</a>] <b>On Behalf Of </b>Jason Jeffrey<br = /><b>Sent:</b> 03 October 2016 18:40<br /><b>To:</b> <a target=3D"_blank"= href=3D"mailto:users@ovirt.org">users@ovirt.org</a></span><u></u><u></u>= </p><div><div><p class=3D"MsoNormal"><br /><b>Subject:</b> Re: [ovirt-use= rs] 4.0 - 2nd node fails on deploy<u></u><u></u></p></div></div></div></d= iv><div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p><p class=3D"M= soNormal"><span style=3D"font-size:11pt;font-family:"calibri",s= ans-serif;color:rgb(31,73,125)">Hi,</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calib= ri",sans-serif;color:rgb(31,73,125)">=C2=A0</span><u></u><u></u></p>= <p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"ca= libri",sans-serif;color:rgb(31,73,125)">Setup log attached for prima= ry</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-siz= e:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-si= ze:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,125)">= Regards</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"fon= t-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73,12= 5)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"= font-size:11pt;font-family:"calibri",sans-serif;color:rgb(31,73= ,125)">Jason </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"font-size:11pt;font-family:"calibri",sans-serif;color:= rgb(31,73,125)">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><b>= <span style=3D"font-size:11pt;font-family:"calibri",sans-serif"= lang=3D"EN-US">From:</span></b><span style=3D"font-size:11pt;font-family= :"calibri",sans-serif" lang=3D"EN-US"> Simone Tiraboschi [<a ta= rget=3D"_blank" href=3D"mailto:stirabos@redhat.com">mailto:stirabos@redha= t.com</a>] <br /><b>Sent:</b> 03 October 2016 09:27<br /><b>To:</b> Jason= Jeffrey <<a target=3D"_blank" href=3D"mailto:jason@sudo.co.uk">jason@= sudo.co.uk</a>><br /><b>Cc:</b> users <<a target=3D"_blank" href=3D= "mailto:users@ovirt.org">users@ovirt.org</a>><br /><b>Subject:</b> Re:= [ovirt-users] 4.0 - 2nd node fails on deploy</span><u></u><u></u></p><p = class=3D"MsoNormal">=C2=A0<u></u><u></u></p><div><p class=3D"MsoNormal">=C2= =A0<u></u><u></u></p><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p>= <div><p class=3D"MsoNormal">On Mon, Oct 3, 2016 at 12:45 AM, Jason Jeffre= y <<a target=3D"_blank" href=3D"mailto:jason@sudo.co.uk">jason@sudo.co.uk</a>&g= t; wrote:<u></u><u></u></p><blockquote style=3D"border-width:medium mediu= m medium 1pt;border-style:none none none solid;border-color:-moz-use-text= -color -moz-use-text-color -moz-use-text-color rgb(204,204,204);padding:0= cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt"><div><div><p class=3D"MsoNormal"= ><span lang=3D"EN-US">Hi,</span><u></u><u></u></p><p class=3D"MsoNormal">= <span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal= "><span lang=3D"EN-US">I am trying to build a x3 HC cluster, with a self = hosted engine using gluster.</span><u></u><u></u></p><p class=3D"MsoNorma= l"><span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNor= mal"><span lang=3D"EN-US">I have successful built the 1<sup>st</sup> node= ,=C2=A0 however when I attempt to run hosted-engine =E2=80=93deploy on no= de 2, I get the following error</span><u></u><u></u></p><p class=3D"MsoNo= rmal"><span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"Mso= Normal"><span lang=3D"EN-US">[WARNING] A configuration file must be supplied to deploy = Hosted Engine on an additional host.</span><u></u><u></u></p><p class=3D"= MsoNormal"><span lang=3D"EN-US">[ ERROR ] 'version' is not stored= in the HE configuration image</span><u></u><u></u></p><p class=3D"MsoNor= mal"><span lang=3D"EN-US">[ ERROR ] Unable to get the answer file from th= e shared storage</span><u></u><u></u></p><p class=3D"MsoNormal"><span lan= g=3D"EN-US">[ ERROR ] Failed to execute stage 'Environment customizat= ion': Unable to get the answer file from the shared storage</span><u>= </u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">[ INFO=C2=A0 ]= Stage: Clean up</span><u></u><u></u></p><p class=3D"MsoNormal"><span lan= g=3D"EN-US">[ INFO=C2=A0 ] Generating answer file '/var/lib/ovirt-hos= ted-engine-<wbr />setup/answers/answers-<wbr />20161002232505.conf'</= span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">[ INFO= =C2=A0 ] Stage: Pre-termination</span><u></u><u></u></p><p class=3D"MsoNo= rmal"><span lang=3D"EN-US">[ INFO=C2=A0 ] Stage: Termination</span><u></u><u></u></p>= <p class=3D"MsoNormal"><span lang=3D"EN-US">[ ERROR ] Hosted Engine deplo= yment failed=C2=A0=C2=A0=C2=A0 </span><u></u><u></u></p><p class=3D"MsoNo= rmal"><span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"Mso= Normal"><span lang=3D"EN-US">Looking at the failure in the log file..</sp= an><u></u><u></u></p></div></div></blockquote><div><p class=3D"MsoNormal"= >=C2=A0<u></u><u></u></p></div><div><p class=3D"MsoNormal">Can you please= attach hosted-engine-setup logs from the first host?<u></u><u></u></p></= div><div><p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div><blockquote= style=3D"border-width:medium medium medium 1pt;border-style:none none no= ne solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-te= xt-color rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8p= t"><div><div><p class=3D"MsoNormal"><span lang=3D"EN-US">=C2=A0</span><u>= </u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">2016-10-02 23:= 25:05 WARNING otopi.plugins.gr_he_common.<wbr />core.remote_answerfile remote_answerfil= e._<wbr />customization:151 A configuration</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span lang=3D"EN-US">file must be supplied to deploy Ho= sted Engine on an additional host.</span><u></u><u></u></p><p class=3D"Ms= oNormal"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG otopi.plugins.gr_= he_common.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />a= nswer_file:61 _fetch_answer_f</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span lang=3D"EN-US">ile</span><u></u><u></u></p><p class=3D"MsoNorma= l"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_com= mon.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />answer_= file:69 fetching from:</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an lang=3D"EN-US">/rhev/data-center/mnt/<wbr />glusterSD/dcastor02:engine= /<wbr />0a021563-91b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr = />02f9-4cd1-a22c-d6b56a0a8e9b/7</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">8cb2527-a2e2-489a-9fad-<wbr />46= 5a72221b37</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"E= N-US">2016-10-02 23:25:05 DEBUG otopi.plugins.gr_he_common.<wbr />core.re= mote_answerfile heconflib._dd_pipe_tar:69 executing: 'sudo -u vdsm dd= i</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">f=3D= /rhev/data-center/mnt/<wbr />glusterSD/dcastor02:engine/<wbr />0a021563-9= 1b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr />02f9-4cd1-a22c-d= 6b56a0a8e9b</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"= EN-US">/78cb2527-a2e2-489a-9fad-<wbr />465a72221b37 bs=3D4k'</span><u= ></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">2016-10-02 23= :25:05 DEBUG otopi.plugins.gr_he_common.<wbr />core.remote_answerfile hec= onflib._dd_pipe_tar:70 executing: 'tar -tvf -'</span><u></u><u></= u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">2016-10-02 23:25:05 DEB= UG otopi.plugins.gr_he_common.<wbr />core.remote_answerfile heconflib._dd_pipe_tar:88 stdout:</span><u></u><u></u></p><p class=3D"Mso= Normal"><span lang=3D"EN-US">2016-10-02 23:25:05 DEBUG otopi.plugins.gr_h= e_common.<wbr />core.remote_answerfile heconflib._dd_pipe_tar:89 stderr:<= /span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">2016-= 10-02 23:25:05 ERROR otopi.plugins.gr_he_common.<wbr />core.remote_answer= file heconflib.validateConfImage:<wbr />111 'version' is not stor= ed</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">in= the HE configuration image</span><u></u><u></u></p><p class=3D"MsoNormal= "><span lang=3D"EN-US">2016-10-02 23:25:05 ERROR otopi.plugins.gr_he_comm= on.<wbr />core.remote_answerfile remote_answerfile._fetch_<wbr />answer_f= ile:73 Unable to get t</span><u></u><u></u></p><p class=3D"MsoNormal"><sp= an lang=3D"EN-US">he answer file from the shared storage</span><u></u><u>= </u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">=C2=A0</span><u></u><= u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">Looking at the dete= cted gluster path - /rhev/data-center/mnt/<wbr />glusterSD/dcastor02:engine/<w= br />0a021563-91b5-4f49-9c6b-<wbr />fff45e85a025/images/f055216c-<wbr />0= 2f9-4cd1-a22c-d6b56a0a8e9b/</span><u></u><u></u></p><p class=3D"MsoNormal= "><span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNorm= al"><span lang=3D"EN-US">[root@dcasrv02 ~]# ls -al /rhev/data-center/mnt/= <wbr />glusterSD/dcastor02:engine/<wbr />0a021563-91b5-4f49-9c6b-<wbr />f= ff45e85a025/images/f055216c-<wbr />02f9-4cd1-a22c-d6b56a0a8e9b/</span><u>= </u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">total 1049609<= /span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">drwxr= -xr-x. 2 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0 2 04= :46 .</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US"= >drwxr-xr-x. 6 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4096 Oct=C2=A0= 2 04:46 ..</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"= EN-US">-rw-rw----. 1 vdsm kvm 1073741824 Oct=C2=A0 2 04:46 78cb2527-a2e2-= 489a-9fad-<wbr />465a72221b37</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">-rw-rw----. 1 vdsm kvm=C2=A0=C2=A0= =C2=A0 1048576 Oct=C2=A0 2 04:46 78cb2527-a2e2-489a-9fad-<wbr />465a72221= b37.lease</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN= -US">-rw-r--r--. 1 vdsm kvm=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 294= Oct=C2=A0 2 04:46 78cb2527-a2e2-489a-9fad-<wbr />465a72221b37.meta=C2=A0= </span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">7= 8cb2527-a2e2-489a-9fad-<wbr />465a72221b37 is=C2=A0 a 1 GB file, is this = the engine VM ?</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang= =3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span la= ng=3D"EN-US">Copying the answers file form primary (/etc/ovirt-hosted-eng= ine/<wbr />answers.conf ) to=C2=A0 node 2 and rerunning produces the same= error : (</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"E= N-US">(hosted-engine --deploy=C2=A0 --config-append=3D/root/answers.<wbr = />conf )</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><spa= n lang=3D"EN-US">Also tried on node 3, same issues </span><u></u><u></u><= /p><p class=3D"MsoNormal"><span lang=3D"EN-US">=C2=A0</span><u></u><u></u= ></p><p class=3D"MsoNormal"><span lang=3D"EN-US">Happy to provide logs an= d other debugs</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D= "EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span lang=3D= "EN-US">Thanks </span><u></u><u></u></p><p class=3D"MsoNormal"><span styl= e=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></= p><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"E= N-US">Jason </span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US= ">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"co= lor:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></= u></p><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D= "EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p= ><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN= -US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US= ">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"co= lor:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"color:r= gb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0<= /span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></= u></p><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D= "EN-US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style= =3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p= ><p class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN= -US">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D= "color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p= class=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US= ">=C2=A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"co= lor:rgb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p cl= ass=3D"MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2= =A0</span><u></u><u></u></p><p class=3D"MsoNormal"><span style=3D"color:r= gb(136,136,136)" lang=3D"EN-US">=C2=A0</span><u></u><u></u></p><p class=3D= "MsoNormal"><span style=3D"color:rgb(136,136,136)" lang=3D"EN-US">=C2=A0<= /span><u></u><u></u></p></div></div><p style=3D"margin-bottom:12pt" class=3D"MsoNormal"><br />______________________________<wbr />__________= _______<br />Users mailing list<br /><a target=3D"_blank" href=3D"mailto:= Users@ovirt.org">Users@ovirt.org</a><br /><a target=3D"_blank" href=3D"ht= tp://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/<wbr = />mailman/listinfo/users</a><u></u><u></u></p></blockquote></div><p class= =3D"MsoNormal">=C2=A0<u></u><u></u></p></div></div></div></div></div></di= v><p style=3D"margin-bottom:12pt" class=3D"MsoNormal"><br />_____________= _________________<wbr />_________________<br />Users mailing list<br /><a= target=3D"_blank" href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br= /><a target=3D"_blank" href=3D"http://lists.ovirt.org/mailman/listinfo/u= sers">http://lists.ovirt.org/<wbr />mailman/listinfo/users</a><u></u><u><= /u></p></blockquote></div></div></div><p class=3D"MsoNormal">=C2=A0<u></u= ><u></u></p></div></div></blockquote></div><p class=3D"MsoNormal">=C2=A0<= u></u><u></u></p></div></div></div></div></div></div></blockquote></div><= p class=3D"MsoNormal">=C2=A0<u></u><u></u></p></div></div></div></div></div= ></div><p style=3D"margin-bottom:12pt" class=3D"MsoNormal"><br />________= ______________________<wbr />_________________<br />Users mailing list<br= /><a target=3D"_blank" href=3D"mailto:Users@ovirt.org">Users@ovirt.org</= a><br /><a target=3D"_blank" href=3D"http://lists.ovirt.org/mailman/listi= nfo/users">http://lists.ovirt.org/<wbr />mailman/listinfo/users</a><u></u= ><u></u></p></blockquote></div><p class=3D"MsoNormal"><u></u>=C2=A0<u></u= ></p></div></div></div></div></div></blockquote></div><br /></div></div> <p style=3D"margin-top: 2.5em; margin-bottom: 1em; border-bottom: 1px sol= id #000"></p><pre class=3D"k9mail"><hr /><br />Gluster-users mailing list= <br />Gluster-users@gluster.org<br /><a href=3D"http://www.gluster.org/ma= ilman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/glu= ster-users</a></pre></blockquote></div><br> -- <br> Sent from my Android device with K-9 Mail. Please excuse my brevity.</bod= y></html> ------4VSAMIL16A4UO07LK9ROSU1PLADG1A--