Re: [ovirt-users] 4.0 - 2nd node fails on deploy

This is a multipart message in MIME format. ------=_NextPart_000_1D1B_01D21E5B.76B9C8F0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi, =20 DCASTORXX is a hosts entry for dedicated direct 10GB links (each = private /28) between the x3 servers i.e 1=3D> 2&3, 2=3D> 1&3, etc) = planned to be used solely for storage. =20 I,e=20 =20 10.100.50.81 dcasrv01 10.100.101.1 dcastor01 10.100.50.82 dcasrv02 10.100.101.2 dcastor02 10.100.50.83 dcasrv03 10.100.103.3 dcastor03 =20 =20 These were setup with the gluster commands =20 * gluster volume create iso replica 3 arbiter 1 = dcastor01:/xpool/iso/brick dcastor02:/xpool/iso/brick = dcastor03:/xpool/iso/brick * gluster volume create export replica 3 arbiter 1 = dcastor02:/xpool/export/brick dcastor03:/xpool/export/brick = dcastor01:/xpool/export/brick =20 * gluster volume create engine replica 3 arbiter 1 = dcastor01:/xpool/engine/brick dcastor02:/xpool/engine/brick = dcastor03:/xpool/engine/brick * gluster volume create data replica 3 arbiter 1 = dcastor01:/xpool/data/brick dcastor03:/xpool/data/brick = dcastor02:/xpool/data/bricky =20 =20 So yes, DCASRV01 is the server (pri) and have local bricks access = through DCASTOR01 interface=20 =20 Is the issue here not the incorrect soft link ? =20 lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.metadata -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a-= 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93 =20 [root@dcasrv01 /]# ls -al = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/ ls: cannot access = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such = file or directory =20 But the data does exist=20 [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al drwxr-xr-x. 2 vdsm kvm 4096 Oct 3 17:17 . drwxr-xr-x. 6 vdsm kvm 4096 Oct 3 17:17 .. -rw-rw----. 2 vdsm kvm 1028096 Oct 3 20:48 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93 -rw-rw----. 2 vdsm kvm 1048576 Oct 3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.lease -rw-r--r--. 2 vdsm kvm 283 Oct 3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta =20 =20 Thanks=20 =20 Jason=20 =20 =20 =20 From: Simone Tiraboschi [mailto:stirabos@redhat.com]=20 Sent: 04 October 2016 14:40 To: Jason Jeffrey <jason@sudo.co.uk> Cc: users <users@ovirt.org> Subject: Re: [ovirt-users] 4.0 - 2nd node fails on deploy =20 =20 =20 On Tue, Oct 4, 2016 at 10:51 AM, Simone Tiraboschi <stirabos@redhat.com = <mailto:stirabos@redhat.com> > wrote: =20 =20 On Mon, Oct 3, 2016 at 11:56 PM, Jason Jeffrey <jason@sudo.co.uk = <mailto:jason@sudo.co.uk> > wrote: Hi, =20 Another problem has appeared, after rebooting the primary the VM will = not start. =20 Appears the symlink is broken between gluster mount ref and vdsm =20 The first host was correctly deployed but it seas that you are facing = some issue connecting the storage. Can you please attach vdsm logs and /var/log/messages from the first = host? =20 Thanks Jason, I suspect that your issue is related to this: Oct 4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 17:24:39.522620] C [MSGID: 106002] = [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] = 0-management: Server quorum lost for volume data. Stopping local bricks. Oct 4 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 17:24:39.523272] C [MSGID: 106002] = [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] = 0-management: Server quorum lost for volume engine. Stopping local = bricks. =20 and for some time your gluster volume has been working. =20 But then: Oct 4 19:02:09 dcasrv01 systemd: Started /usr/bin/mount -t glusterfs -o = backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine. Oct 4 19:02:09 dcasrv01 systemd: Starting /usr/bin/mount -t glusterfs = -o backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine. Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: = /usr/lib/python2.7/site-packages/yajsonrpc/stomp.py:352: = DeprecationWarning: Dispatcher.pending is deprecated. Use = Dispatcher.socket.pending instead. Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D getattr(dispatcher, = 'pending', lambda: 0) Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: = /usr/lib/python2.7/site-packages/yajsonrpc/stomp.py:352: = DeprecationWarning: Dispatcher.pending is deprecated. Use = Dispatcher.socket.pending instead. Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: pending =3D getattr(dispatcher, = 'pending', lambda: 0) Oct 4 19:02:11 dcasrv01 journal: vdsm vds.dispatcher ERROR SSL error = during reading data: unexpected eof Oct 4 19:02:11 dcasrv01 journal: ovirt-ha-agent = ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to = storage server failed' - trying to restart agent Oct 4 19:02:11 dcasrv01 ovirt-ha-agent: = ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to = storage server failed' - trying to restart agent Oct 4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 18:02:12.384611] C [MSGID: 106003] = [glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action] = 0-management: Server quorum regained for volume data. Starting local = bricks. Oct 4 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 18:02:12.388981] C [MSGID: 106003] = [glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action] = 0-management: Server quorum regained for volume engine. Starting local = bricks. =20 And at that point VDSM started complaining that the = hosted-engine-storage domain doesn't exist anymore: Oct 4 19:02:30 dcasrv01 journal: ovirt-ha-agent = ovirt_hosted_engine_ha.lib.image.Image ERROR Error fetching volumes = list: Storage domain does not exist: = (u'bbb70623-194a-46d2-a164-76a4876ecaaf',) Oct 4 19:02:30 dcasrv01 ovirt-ha-agent: = ERROR:ovirt_hosted_engine_ha.lib.image.Image:Error fetching volumes = list: Storage domain does not exist: = (u'bbb70623-194a-46d2-a164-76a4876ecaaf',) =20 I see from the logs that the ovirt-ha-agent is trying to mount the = hosted-engine storage domain as: /usr/bin/mount -t glusterfs -o = backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine. =20 Pointing to dcastor01, dcastor02 and dcastor03 while your server is = dcasrv01. But at the same time it seams that also dcasrv01 has local bricks for = the same engine volume. =20 So, is dcasrv01 just an alias fro dcastor01? if not you probably have = some issue with the configuration of your gluster volume. =20 =20 =20

From broker.log

lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 = hosted-engine.metadata -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a-= 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93  &n= bsp; </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 /]# ls -al = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/</span><o:p></= o:p></p><p class=3DMsoNormal =

ls: cannot access = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such = file or directory   <o:p></o:p></span></p><p class=3DMsoNormal =

But the data does exist <o:p></o:p></span></p><p class=3DMsoNormal =

[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls = -al</span><o:p></o:p></p><p class=3DMsoNormalCxSpMiddle =

drwxr-xr-x. 2 vdsm kvm    4096 Oct  3 17:17 = .</span><o:p></o:p></p><p class=3DMsoNormalCxSpMiddle =

drwxr-xr-x. 6 vdsm kvm    4096 Oct  3 17:17 = ..</span><o:p></o:p></p><p class=3DMsoNormalCxSpMiddle =

-rw-rw----. 2 vdsm kvm 1028096 Oct  3 20:48 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93</span><o:p></o:p></p><p = class=3DMsoNormalCxSpMiddle =

-rw-rw----. 2 vdsm kvm 1048576 Oct  3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.lease</span><o:p></o:p></p><p = class=3DMsoNormalCxSpMiddle =

-rw-r--r--. 2 vdsm kvm     283 Oct  3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta   = </span><o:p></o:p></p><p class=3DMsoNormal =

Hi,</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Another problem has appeared, after rebooting the primary the VM will = not start.</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Appears the symlink is broken between gluster mount ref and = vdsm</span><o:p></o:p></p></div></div></blockquote><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>The first host was correctly deployed but it seas that = you are facing some issue connecting the = storage.<o:p></o:p></p></div><div><p class=3DMsoNormal>Can you please = attach vdsm logs and /var/log/messages from the first = host?<o:p></o:p></p></div></div></div></div></blockquote><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>Thanks Jason,<o:p></o:p></p></div><div><p = class=3DMsoNormal>I suspect that your issue is related to =

<p class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>I see from the logs that the ovirt-ha-agent is trying = to mount the hosted-engine storage domain = as:<o:p></o:p></p></div><div><p class=3DMsoNormal>/usr/bin/mount -t = glusterfs -o backup-volfile-servers=3Ddcastor02:dcastor03 = dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine.<o:p></o:p></p></div><di= v><p class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>Pointing to dcastor01, dcastor02 and dcastor03 while = your server is dcasrv01.<o:p></o:p></p></div><div><p = class=3DMsoNormal>But at the same time it seams that also dcasrv01 has = local bricks for the same engine volume.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>So, is dcasrv01 just an alias fro dcastor01? if not = you probably have some issue with the configuration of your gluster = volume.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal> <o:p></o:p></p></div><blockquote =

 </span><o:p></o:p></p><p class=3DMsoNormal =

From broker.log</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Thread-169::ERROR::2016-10-04 22:44:16,189::storage_broker::138::<a =

 </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 ovirt-hosted-engine-ha]# ls -al = /rhev/data-center/mnt/glusterSD/dcastor01\:engine/bbb70623-194a-46d2-a164= -76a4876ecaaf/ha_agent/</span><o:p></o:p></p><p class=3DMsoNormal =

total 9</span><o:p></o:p></p><p class=3DMsoNormal =

drwxrwx---. 2 vdsm kvm 4096 Oct  3 17:27 .</span><o:p></o:p></p><p = class=3DMsoNormal =

drwxr-xr-x. 5 vdsm kvm 4096 Oct  3 17:17 = ..</span><o:p></o:p></p><p class=3DMsoNormal =

lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 = hosted-engine.lockspace -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/23d81b73-bcb7-= 4742-abde-128522f43d78/11d6a3e1-1817-429d-b2e0-9051a3cf41a4</span><o:p></= o:p></p><p class=3DMsoNormal =

lrwxrwxrwx. 1 vdsm kvm  132 Oct  3 17:27 = hosted-engine.metadata -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a-= 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93  &n= bsp; </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 /]# ls -al = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/</span><o:p></= o:p></p><p class=3DMsoNormal =

ls: cannot access = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such = file or directory   </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Though file appears to be there </span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Gluster is setup as xpool/engine </span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# =

/xpool/engine/brick/bbb70623-194a-46d2-a164-76a4876ecaaf/images/fd44dbf9= -473a-496a-9996-c8abe3278390</span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls = -al</span><o:p></o:p></p><p class=3DMsoNormal =

total 2060</span><o:p></o:p></p><p class=3DMsoNormal =

drwxr-xr-x. 2 vdsm kvm    4096 Oct  3 17:17 = .</span><o:p></o:p></p><p class=3DMsoNormal =

drwxr-xr-x. 6 vdsm kvm    4096 Oct  3 17:17 = ..</span><o:p></o:p></p><p class=3DMsoNormal =

-rw-rw----. 2 vdsm kvm 1028096 Oct  3 20:48 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93</span><o:p></o:p></p><p = class=3DMsoNormal =

-rw-rw----. 2 vdsm kvm 1048576 Oct  3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.lease</span><o:p></o:p></p><p = class=3DMsoNormal =

-rw-r--r--. 2 vdsm kvm     283 Oct  3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta   = </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume = info</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Volume Name: data</span><o:p></o:p></p><p class=3DMsoNormal =

Type: Replicate</span><o:p></o:p></p><p class=3DMsoNormal =

Volume ID: 54fbcafc-fed9-4bce-92ec-fa36cdcacbd4</span><o:p></o:p></p><p = class=3DMsoNormal =

Status: Started</span><o:p></o:p></p><p class=3DMsoNormal =

Number of Bricks: 1 x (2 + 1) =3D 3</span><o:p></o:p></p><p = class=3DMsoNormal =

Transport-type: tcp</span><o:p></o:p></p><p class=3DMsoNormal =

Bricks:</span><o:p></o:p></p><p class=3DMsoNormal =

Brick1: dcastor01:/xpool/data/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick2: dcastor03:/xpool/data/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick3: dcastor02:/xpool/data/bricky (arbiter)</span><o:p></o:p></p><p = class=3DMsoNormal =

Options Reconfigured:</span><o:p></o:p></p><p class=3DMsoNormal =

performance.readdir-ahead: on</span><o:p></o:p></p><p class=3DMsoNormal =

performance.quick-read: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.read-ahead: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.io-cache: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.stat-prefetch: off</span><o:p></o:p></p><p = class=3DMsoNormal =

cluster.eager-lock: enable</span><o:p></o:p></p><p class=3DMsoNormal =

network.remote-dio: enable</span><o:p></o:p></p><p class=3DMsoNormal =

cluster.quorum-type: auto</span><o:p></o:p></p><p class=3DMsoNormal =

cluster.server-quorum-type: server</span><o:p></o:p></p><p = class=3DMsoNormal =

storage.owner-uid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-gid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Volume Name: engine</span><o:p></o:p></p><p class=3DMsoNormal =

Type: Replicate</span><o:p></o:p></p><p class=3DMsoNormal =

Volume ID: dd4c692d-03aa-4fc6-9011-a8dad48dad96</span><o:p></o:p></p><p = class=3DMsoNormal =

Status: Started</span><o:p></o:p></p><p class=3DMsoNormal =

Number of Bricks: 1 x (2 + 1) =3D 3</span><o:p></o:p></p><p = class=3DMsoNormal =

Transport-type: tcp</span><o:p></o:p></p><p class=3DMsoNormal =

Bricks:</span><o:p></o:p></p><p class=3DMsoNormal =

Brick1: dcastor01:/xpool/engine/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick2: dcastor02:/xpool/engine/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick3: dcastor03:/xpool/engine/brick (arbiter)</span><o:p></o:p></p><p = class=3DMsoNormal =

Options Reconfigured:</span><o:p></o:p></p><p class=3DMsoNormal =

performance.readdir-ahead: on</span><o:p></o:p></p><p class=3DMsoNormal =

performance.quick-read: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.read-ahead: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.io-cache: off</span><o:p></o:p></p><p class=3DMsoNormal =

performance.stat-prefetch: off</span><o:p></o:p></p><p = class=3DMsoNormal =

cluster.eager-lock: enable</span><o:p></o:p></p><p class=3DMsoNormal =

network.remote-dio: enable</span><o:p></o:p></p><p class=3DMsoNormal =

cluster.quorum-type: auto</span><o:p></o:p></p><p class=3DMsoNormal =

cluster.server-quorum-type: server</span><o:p></o:p></p><p = class=3DMsoNormal =

storage.owner-uid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-gid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Volume Name: export</span><o:p></o:p></p><p class=3DMsoNormal =

Type: Replicate</span><o:p></o:p></p><p class=3DMsoNormal =

Volume ID: 23f14730-d264-4cc2-af60-196b943ecaf3</span><o:p></o:p></p><p = class=3DMsoNormal =

Status: Started</span><o:p></o:p></p><p class=3DMsoNormal =

Number of Bricks: 1 x (2 + 1) =3D 3</span><o:p></o:p></p><p = class=3DMsoNormal =

Transport-type: tcp</span><o:p></o:p></p><p class=3DMsoNormal =

Bricks:</span><o:p></o:p></p><p class=3DMsoNormal =

Brick1: dcastor02:/xpool/export/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick2: dcastor03:/xpool/export/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick3: dcastor01:/xpool/export/brick (arbiter)</span><o:p></o:p></p><p = class=3DMsoNormal =

Options Reconfigured:</span><o:p></o:p></p><p class=3DMsoNormal =

performance.readdir-ahead: on</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-uid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-gid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Volume Name: iso</span><o:p></o:p></p><p class=3DMsoNormal =

Type: Replicate</span><o:p></o:p></p><p class=3DMsoNormal =

Volume ID: b2d3d7e2-9919-400b-8368-a0443d48e82a</span><o:p></o:p></p><p = class=3DMsoNormal =

Status: Started</span><o:p></o:p></p><p class=3DMsoNormal =

Number of Bricks: 1 x (2 + 1) =3D 3</span><o:p></o:p></p><p = class=3DMsoNormal =

Transport-type: tcp</span><o:p></o:p></p><p class=3DMsoNormal =

Bricks:</span><o:p></o:p></p><p class=3DMsoNormal =

Brick1: dcastor01:/xpool/iso/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick2: dcastor02:/xpool/iso/brick</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick3: dcastor03:/xpool/iso/brick (arbiter)</span><o:p></o:p></p><p = class=3DMsoNormal =

Options Reconfigured:</span><o:p></o:p></p><p class=3DMsoNormal =

performance.readdir-ahead: on</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-uid: 36</span><o:p></o:p></p><p class=3DMsoNormal =

storage.owner-gid: = 36            = ;            =            = </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

[root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume = status</span><o:p></o:p></p><p class=3DMsoNormal =

Status of volume: data</span><o:p></o:p></p><p class=3DMsoNormal =

Gluster =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

Brick = dcastor01:/xpool/data/brick       &nbs=

Brick = dcastor03:/xpool/data/brick       &nbs=

Brick = dcastor02:/xpool/data/bricky       &nb= sp;  49153     = 0          = Y       3857</span><o:p></o:p></p><p = class=3DMsoNormal =

NFS Server on = localhost          &nbs=

Self-heal Daemon on = localhost          &nbs=

NFS Server on = dcastor03          &nbs=

Self-heal Daemon on = dcastor03          &nbs=

NFS Server on = dcasrv02           = ;           = 2049      = 0          = Y       3871</span><o:p></o:p></p><p = class=3DMsoNormal =

Self-heal Daemon on = dcasrv02           = ;     N/A       = N/A        = Y       3864</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Task Status of Volume data</span><o:p></o:p></p><p class=3DMsoNormal =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

There are no active volume tasks</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Status of volume: engine</span><o:p></o:p></p><p class=3DMsoNormal =

Gluster =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

Brick = dcastor01:/xpool/engine/brick       &n= bsp; 49152     = 0          = Y       3131</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor02:/xpool/engine/brick       &n= bsp; 49152     = 0          = Y       3852</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor03:/xpool/engine/brick       &n= bsp; 49152     = 0          = Y       2992</span><o:p></o:p></p><p = class=3DMsoNormal =

NFS Server on = localhost          &nbs=

Self-heal Daemon on = localhost          &nbs=

NFS Server on = dcastor03          &nbs=

Self-heal Daemon on = dcastor03          &nbs=

NFS Server on = dcasrv02           = ;           = 2049      = 0          = Y       3871</span><o:p></o:p></p><p = class=3DMsoNormal =

Self-heal Daemon on = dcasrv02           = ;     N/A       = N/A        = Y       3864</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Task Status of Volume engine</span><o:p></o:p></p><p class=3DMsoNormal =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

There are no active volume tasks</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Status of volume: export</span><o:p></o:p></p><p class=3DMsoNormal =

Gluster =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

Brick = dcastor02:/xpool/export/brick       &n= bsp; 49155     = 0          = Y       3872</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor03:/xpool/export/brick       &n= bsp; 49155     = 0          = Y       3147</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor01:/xpool/export/brick       &n= bsp; 49155     = 0          = Y       3150</span><o:p></o:p></p><p = class=3DMsoNormal =

NFS Server on = localhost          &nbs=

Self-heal Daemon on = localhost          &nbs=

NFS Server on = dcastor03          &nbs=

Self-heal Daemon on = dcastor03          &nbs=

NFS Server on = dcasrv02           = ;           = 2049      = 0          = Y       3871</span><o:p></o:p></p><p = class=3DMsoNormal =

Self-heal Daemon on = dcasrv02           = ;     N/A       = N/A        = Y       3864</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Task Status of Volume export</span><o:p></o:p></p><p class=3DMsoNormal =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

There are no active volume tasks</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Status of volume: iso</span><o:p></o:p></p><p class=3DMsoNormal =

Gluster =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

Brick = dcastor01:/xpool/iso/brick        = ;    49154     = 0          = Y       3152</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor02:/xpool/iso/brick        = ;    49154     = 0          = Y       3881</span><o:p></o:p></p><p = class=3DMsoNormal =

Brick = dcastor03:/xpool/iso/brick        = ;    49154     = 0          = Y       3146</span><o:p></o:p></p><p = class=3DMsoNormal =

NFS Server on = localhost          &nbs=

Self-heal Daemon on = localhost          &nbs=

NFS Server on = dcastor03          &nbs=

Self-heal Daemon on = dcastor03          &nbs=

NFS Server on = dcasrv02           = ;           = 2049      = 0          = Y       3871</span><o:p></o:p></p><p = class=3DMsoNormal =

Self-heal Daemon on = dcasrv02           = ;     N/A       = N/A        = Y       3864</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Task Status of Volume iso</span><o:p></o:p></p><p class=3DMsoNormal =

------------------------------------------------------------------------= ------</span><o:p></o:p></p><p class=3DMsoNormal =

There are no active volume tasks</span><o:p></o:p></p><p = class=3DMsoNormal =

            =             &= nbsp;           &n= bsp;           &nb= sp;           &nbs=

Thanks</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Jason</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><div><div =

Hi,</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Setup log attached for primary</span><o:p></o:p></p><p = class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Regards</span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

Jason </span><o:p></o:p></p><p class=3DMsoNormal =

 </span><o:p></o:p></p><p class=3DMsoNormal =

=20 Thread-169::ERROR::2016-10-04 = 22:44:16,189::storage_broker::138::ovirt_hosted_engine_ha.br = <http://ovirt_hosted_engine_ha.br> = oker.storage_broker.StorageBroker::(get_raw_stats_for_service_type) = Failed to read metadata from = /rhev/data-center/mnt/glusterSD/dcastor01:engine/bbb70623-194a-46d2-a164-= 76a4876ecaaf/ha_agent/hosted-engine.metadata =20 [root@dcasrv01 ovirt-hosted-engine-ha]# ls -al = /rhev/data-center/mnt/glusterSD/dcastor01\:engine/bbb70623-194a-46d2-a164= -76a4876ecaaf/ha_agent/ total 9 drwxrwx---. 2 vdsm kvm 4096 Oct 3 17:27 . drwxr-xr-x. 5 vdsm kvm 4096 Oct 3 17:17 .. lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.lockspace -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/23d81b73-bcb7-= 4742-abde-128522f43d78/11d6a3e1-1817-429d-b2e0-9051a3cf41a4 lrwxrwxrwx. 1 vdsm kvm 132 Oct 3 17:27 hosted-engine.metadata -> = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/fd44dbf9-473a-= 496a-9996-c8abe3278390/cee9440c-4eb8-453b-bc04-c47e6f9cbc93 =20 =20 [root@dcasrv01 /]# ls -al = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/ ls: cannot access = /var/run/vdsm/storage/bbb70623-194a-46d2-a164-76a4876ecaaf/: No such = file or directory =20 =20 Though file appears to be there=20 =20 Gluster is setup as xpool/engine=20 =20 [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# pwd /xpool/engine/brick/bbb70623-194a-46d2-a164-76a4876ecaaf/images/fd44dbf9-= 473a-496a-9996-c8abe3278390 [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# ls -al total 2060 drwxr-xr-x. 2 vdsm kvm 4096 Oct 3 17:17 . drwxr-xr-x. 6 vdsm kvm 4096 Oct 3 17:17 .. -rw-rw----. 2 vdsm kvm 1028096 Oct 3 20:48 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93 -rw-rw----. 2 vdsm kvm 1048576 Oct 3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.lease -rw-r--r--. 2 vdsm kvm 283 Oct 3 17:17 = cee9440c-4eb8-453b-bc04-c47e6f9cbc93.meta =20 =20 =20 [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume = info =20 Volume Name: data Type: Replicate Volume ID: 54fbcafc-fed9-4bce-92ec-fa36cdcacbd4 Status: Started Number of Bricks: 1 x (2 + 1) =3D 3 Transport-type: tcp Bricks: Brick1: dcastor01:/xpool/data/brick Brick2: dcastor03:/xpool/data/brick Brick3: dcastor02:/xpool/data/bricky (arbiter) Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 =20 Volume Name: engine Type: Replicate Volume ID: dd4c692d-03aa-4fc6-9011-a8dad48dad96 Status: Started Number of Bricks: 1 x (2 + 1) =3D 3 Transport-type: tcp Bricks: Brick1: dcastor01:/xpool/engine/brick Brick2: dcastor02:/xpool/engine/brick Brick3: dcastor03:/xpool/engine/brick (arbiter) Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 =20 Volume Name: export Type: Replicate Volume ID: 23f14730-d264-4cc2-af60-196b943ecaf3 Status: Started Number of Bricks: 1 x (2 + 1) =3D 3 Transport-type: tcp Bricks: Brick1: dcastor02:/xpool/export/brick Brick2: dcastor03:/xpool/export/brick Brick3: dcastor01:/xpool/export/brick (arbiter) Options Reconfigured: performance.readdir-ahead: on storage.owner-uid: 36 storage.owner-gid: 36 =20 Volume Name: iso Type: Replicate Volume ID: b2d3d7e2-9919-400b-8368-a0443d48e82a Status: Started Number of Bricks: 1 x (2 + 1) =3D 3 Transport-type: tcp Bricks: Brick1: dcastor01:/xpool/iso/brick Brick2: dcastor02:/xpool/iso/brick Brick3: dcastor03:/xpool/iso/brick (arbiter) Options Reconfigured: performance.readdir-ahead: on storage.owner-uid: 36 storage.owner-gid: 36 =20 =20 =20 [root@dcasrv01 fd44dbf9-473a-496a-9996-c8abe3278390]# gluster volume = status Status of volume: data Gluster process TCP Port RDMA Port Online = Pid -------------------------------------------------------------------------= ----- Brick dcastor01:/xpool/data/brick 49153 0 Y = 3076 Brick dcastor03:/xpool/data/brick 49153 0 Y = 3019 Brick dcastor02:/xpool/data/bricky 49153 0 Y = 3857 NFS Server on localhost 2049 0 Y = 3097 Self-heal Daemon on localhost N/A N/A Y = 3088 NFS Server on dcastor03 2049 0 Y = 3039 Self-heal Daemon on dcastor03 N/A N/A Y = 3114 NFS Server on dcasrv02 2049 0 Y = 3871 Self-heal Daemon on dcasrv02 N/A N/A Y = 3864 =20 Task Status of Volume data -------------------------------------------------------------------------= ----- There are no active volume tasks =20 Status of volume: engine Gluster process TCP Port RDMA Port Online = Pid -------------------------------------------------------------------------= ----- Brick dcastor01:/xpool/engine/brick 49152 0 Y = 3131 Brick dcastor02:/xpool/engine/brick 49152 0 Y = 3852 Brick dcastor03:/xpool/engine/brick 49152 0 Y = 2992 NFS Server on localhost 2049 0 Y = 3097 Self-heal Daemon on localhost N/A N/A Y = 3088 NFS Server on dcastor03 2049 0 Y = 3039 Self-heal Daemon on dcastor03 N/A N/A Y = 3114 NFS Server on dcasrv02 2049 0 Y = 3871 Self-heal Daemon on dcasrv02 N/A N/A Y = 3864 =20 Task Status of Volume engine -------------------------------------------------------------------------= ----- There are no active volume tasks =20 Status of volume: export Gluster process TCP Port RDMA Port Online = Pid -------------------------------------------------------------------------= ----- Brick dcastor02:/xpool/export/brick 49155 0 Y = 3872 Brick dcastor03:/xpool/export/brick 49155 0 Y = 3147 Brick dcastor01:/xpool/export/brick 49155 0 Y = 3150 NFS Server on localhost 2049 0 Y = 3097 Self-heal Daemon on localhost N/A N/A Y = 3088 NFS Server on dcastor03 2049 0 Y = 3039 Self-heal Daemon on dcastor03 N/A N/A Y = 3114 NFS Server on dcasrv02 2049 0 Y = 3871 Self-heal Daemon on dcasrv02 N/A N/A Y = 3864 =20 Task Status of Volume export -------------------------------------------------------------------------= ----- There are no active volume tasks =20 Status of volume: iso Gluster process TCP Port RDMA Port Online = Pid -------------------------------------------------------------------------= ----- Brick dcastor01:/xpool/iso/brick 49154 0 Y = 3152 Brick dcastor02:/xpool/iso/brick 49154 0 Y = 3881 Brick dcastor03:/xpool/iso/brick 49154 0 Y = 3146 NFS Server on localhost 2049 0 Y = 3097 Self-heal Daemon on localhost N/A N/A Y = 3088 NFS Server on dcastor03 2049 0 Y = 3039 Self-heal Daemon on dcastor03 N/A N/A Y = 3114 NFS Server on dcasrv02 2049 0 Y = 3871 Self-heal Daemon on dcasrv02 N/A N/A Y = 3864 =20 Task Status of Volume iso -------------------------------------------------------------------------= ----- There are no active volume tasks = =20 Thanks =20 Jason =20 =20 =20 From: users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> = [mailto:users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> ] On = Behalf Of Jason Jeffrey Sent: 03 October 2016 18:40 To: users@ovirt.org <mailto:users@ovirt.org>=20 Subject: Re: [ovirt-users] 4.0 - 2nd node fails on deploy =20 Hi, =20 Setup log attached for primary =20 Regards =20 Jason=20 =20 From: Simone Tiraboschi [mailto:stirabos@redhat.com]=20 Sent: 03 October 2016 09:27 To: Jason Jeffrey <jason@sudo.co.uk <mailto:jason@sudo.co.uk> > Cc: users <users@ovirt.org <mailto:users@ovirt.org> > Subject: Re: [ovirt-users] 4.0 - 2nd node fails on deploy =20 =20 =20 On Mon, Oct 3, 2016 at 12:45 AM, Jason Jeffrey <jason@sudo.co.uk = <mailto:jason@sudo.co.uk> > wrote: Hi, =20 I am trying to build a x3 HC cluster, with a self hosted engine using = gluster. =20 I have successful built the 1st node, however when I attempt to run = hosted-engine =E2=80=93deploy on node 2, I get the following error =20 [WARNING] A configuration file must be supplied to deploy Hosted Engine = on an additional host. [ ERROR ] 'version' is not stored in the HE configuration image [ ERROR ] Unable to get the answer file from the shared storage [ ERROR ] Failed to execute stage 'Environment customization': Unable to = get the answer file from the shared storage [ INFO ] Stage: Clean up [ INFO ] Generating answer file = '/var/lib/ovirt-hosted-engine-setup/answers/answers-20161002232505.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed =20 =20 Looking at the failure in the log file.. =20 Can you please attach hosted-engine-setup logs from the first host? =20 =20 2016-10-02 23:25:05 WARNING = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._customization:151 A configuration file must be supplied to deploy Hosted Engine on an additional host. 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:61 _fetch_answer_f ile 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:69 fetching from: /rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-= fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/7 8cb2527-a2e2-489a-9fad-465a72221b37 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:69 executing: 'sudo -u vdsm dd i f=3D/rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9= c6b-fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b /78cb2527-a2e2-489a-9fad-465a72221b37 bs=3D4k' 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:70 executing: 'tar -tvf -' 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:88 stdout: 2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:89 stderr: 2016-10-02 23:25:05 ERROR = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib.validateConfImage:111 'version' is not stored in the HE configuration image 2016-10-02 23:25:05 ERROR = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:73 Unable to get t he answer file from the shared storage =20 Looking at the detected gluster path - = /rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-= fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/ =20 [root@dcasrv02 ~]# ls -al = /rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-= fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/ total 1049609 drwxr-xr-x. 2 vdsm kvm 4096 Oct 2 04:46 . drwxr-xr-x. 6 vdsm kvm 4096 Oct 2 04:46 .. -rw-rw----. 1 vdsm kvm 1073741824 Oct 2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37 -rw-rw----. 1 vdsm kvm 1048576 Oct 2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37.lease -rw-r--r--. 1 vdsm kvm 294 Oct 2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37.meta =20 =20 78cb2527-a2e2-489a-9fad-465a72221b37 is a 1 GB file, is this the engine = VM ? =20 Copying the answers file form primary = (/etc/ovirt-hosted-engine/answers.conf ) to node 2 and rerunning = produces the same error : ( (hosted-engine --deploy --config-append=3D/root/answers.conf ) =20 Also tried on node 3, same issues=20 =20 Happy to provide logs and other debugs =20 Thanks=20 =20 Jason=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org>=20 http://lists.ovirt.org/mailman/listinfo/users =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org>=20 http://lists.ovirt.org/mailman/listinfo/users =20 =20 ------=_NextPart_000_1D1B_01D21E5B.76B9C8F0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta = name=3DGenerator content=3D"Microsoft Word 15 (filtered = medium)"><style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman",serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p {mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0cm; mso-margin-bottom-alt:auto; margin-left:0cm; font-size:12.0pt; font-family:"Times New Roman",serif;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0cm; margin-right:0cm; margin-bottom:0cm; margin-left:36.0pt; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman",serif;} span.gmail- {mso-style-name:gmail-;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:952053295; mso-list-type:hybrid; mso-list-template-ids:-675874350 134807553 134807555 134807557 = 134807553 134807555 134807557 134807553 134807555 134807557;} @list l0:level1 {mso-level-number-format:bullet; mso-level-text:=EF=82=B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Symbol;} @list l0:level2 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level3 {mso-level-number-format:bullet; mso-level-text:=EF=82=A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} @list l0:level4 {mso-level-number-format:bullet; mso-level-text:=EF=82=B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Symbol;} @list l0:level5 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level6 {mso-level-number-format:bullet; mso-level-text:=EF=82=A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} @list l0:level7 {mso-level-number-format:bullet; mso-level-text:=EF=82=B7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Symbol;} @list l0:level8 {mso-level-number-format:bullet; mso-level-text:o; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:"Courier New";} @list l0:level9 {mso-level-number-format:bullet; mso-level-text:=EF=82=A7; mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-18.0pt; font-family:Wingdings;} ol {margin-bottom:0cm;} ul {margin-bottom:0cm;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-GB link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>Hi,<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>DCASTORXX is a hosts entry for dedicated = =C2=A0direct 10GB links (each private /28) between the x3 servers=C2=A0 = i.e 1=3D> 2&3, 2=3D> 1&3, etc) planned to be used solely = for storage.<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>I,e <o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.50.81=C2=A0=C2=A0=C2=A0 = dcasrv01<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.101.1=C2=A0=C2=A0=C2=A0 = dcastor01<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.50.82=C2=A0=C2=A0=C2=A0 = dcasrv02<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.101.2=C2=A0=C2=A0=C2=A0 = dcastor02<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.50.83=C2=A0=C2=A0=C2=A0 = dcasrv03<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>10.100.103.3=C2=A0=C2=A0=C2=A0 = dcastor03=C2=A0 <o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>These were setup with the gluster = commands<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoListParagraph style=3D'text-indent:-18.0pt;mso-list:l0 level1 = lfo1'><![if !supportLists]><span = style=3D'font-size:11.0pt;font-family:Symbol;color:#1F497D;mso-fareast-la= nguage:EN-US'><span style=3D'mso-list:Ignore'>=C2=B7<span = style=3D'font:7.0pt "Times New = Roman"'>         = </span></span></span><![endif]><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>gluster volume create iso replica 3 arbiter = 1=C2=A0 dcastor01:/xpool/iso/brick=C2=A0=C2=A0 = dcastor02:/xpool/iso/brick=C2=A0=C2=A0 = dcastor03:/xpool/iso/brick<o:p></o:p></span></p><p = class=3DMsoListParagraph style=3D'text-indent:-18.0pt;mso-list:l0 level1 = lfo1'><![if !supportLists]><span = style=3D'font-size:11.0pt;font-family:Symbol;color:#1F497D;mso-fareast-la= nguage:EN-US'><span style=3D'mso-list:Ignore'>=C2=B7<span = style=3D'font:7.0pt "Times New = Roman"'>         = </span></span></span><![endif]><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>gluster volume create export replica 3 = arbiter 1=C2=A0 dcastor02:/xpool/export/brick=C2=A0 = dcastor03:/xpool/export/brick=C2=A0 dcastor01:/xpool/export/brick=C2=A0 = <o:p></o:p></span></p><p class=3DMsoListParagraph = style=3D'text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if = !supportLists]><span = style=3D'font-size:11.0pt;font-family:Symbol;color:#1F497D;mso-fareast-la= nguage:EN-US'><span style=3D'mso-list:Ignore'>=C2=B7<span = style=3D'font:7.0pt "Times New = Roman"'>         = </span></span></span><![endif]><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>gluster volume create engine replica 3 = arbiter 1 dcastor01:/xpool/engine/brick dcastor02:/xpool/engine/brick = dcastor03:/xpool/engine/brick<o:p></o:p></span></p><p = class=3DMsoListParagraph style=3D'text-indent:-18.0pt;mso-list:l0 level1 = lfo1'><![if !supportLists]><span = style=3D'font-size:11.0pt;font-family:Symbol;color:#1F497D;mso-fareast-la= nguage:EN-US'><span style=3D'mso-list:Ignore'>=C2=B7<span = style=3D'font:7.0pt "Times New = Roman"'>         = </span></span></span><![endif]><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>gluster volume create data replica 3 arbiter = 1=C2=A0 dcastor01:/xpool/data/brick=C2=A0 = dcastor03:/xpool/data/brick=C2=A0 = dcastor02:/xpool/data/bricky<o:p></o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>So yes, DCASRV01 is the server (pri) and = have local bricks access through DCASTOR01 interface = <o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>Is the issue here not the incorrect soft = link ?<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-add-space= :auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-add-space= :auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-add-space= :auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-add-space= :auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-add-space= :auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> <= /o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks = <o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> <= /o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Jason = <o:p></o:p></p><p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p> </o:p></span></p><p = class=3DMsoNormal><b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span><= /b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> Simone = Tiraboschi [mailto:stirabos@redhat.com] <br><b>Sent:</b> 04 October 2016 = 14:40<br><b>To:</b> Jason Jeffrey <jason@sudo.co.uk><br><b>Cc:</b> = users <users@ovirt.org><br><b>Subject:</b> Re: [ovirt-users] 4.0 - = 2nd node fails on deploy<o:p></o:p></span></p><p = class=3DMsoNormal><o:p> </o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal>On Tue, = Oct 4, 2016 at 10:51 AM, Simone Tiraboschi <<a = href=3D"mailto:stirabos@redhat.com" = target=3D"_blank">stirabos@redhat.com</a>> = wrote:<o:p></o:p></p><blockquote style=3D'border:none;border-left:solid = #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p = class=3DMsoNormal><o:p> </o:p></p><div><p class=3DMsoNormal><span = class=3Dgmail->On Mon, Oct 3, 2016 at 11:56 PM, Jason Jeffrey <<a = href=3D"mailto:jason@sudo.co.uk" = target=3D"_blank">jason@sudo.co.uk</a>> = wrote:</span><o:p></o:p></p><blockquote = style=3D'border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= this:<o:p></o:p></p></div><div><div><p class=3DMsoNormal>Oct  4 = 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 17:24:39.522620] C [MSGID: 106002] = [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] = 0-management: Server quorum lost for volume data. Stopping local = bricks.<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 = 18:24:39 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 17:24:39.523272] C [MSGID: 106002] = [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] = 0-management: Server quorum lost for volume engine. Stopping local = bricks.<o:p></o:p></p></div></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>and for some time your gluster volume has been = working.<o:p></o:p></p></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>But then:<o:p></o:p></p></div><div><div><p = class=3DMsoNormal>Oct  4 19:02:09 dcasrv01 systemd: Started = /usr/bin/mount -t glusterfs -o = backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine.<o:p></o:p></p></div><di= v><p class=3DMsoNormal>Oct  4 19:02:09 dcasrv01 systemd: Starting = /usr/bin/mount -t glusterfs -o = backup-volfile-servers=3Ddcastor02:dcastor03 dcastor01:engine = /rhev/data-center/mnt/glusterSD/dcastor01:engine.<o:p></o:p></p></div><di= v><p class=3DMsoNormal>Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: = /usr/lib/python2.7/site-packages/yajsonrpc/stomp.py:352: = DeprecationWarning: Dispatcher.pending is deprecated. Use = Dispatcher.socket.pending instead.<o:p></o:p></p></div><div><p = class=3DMsoNormal>Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: pending = =3D getattr(dispatcher, 'pending', lambda: = 0)<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 19:02:11 = dcasrv01 ovirt-ha-agent: = /usr/lib/python2.7/site-packages/yajsonrpc/stomp.py:352: = DeprecationWarning: Dispatcher.pending is deprecated. Use = Dispatcher.socket.pending instead.<o:p></o:p></p></div><div><p = class=3DMsoNormal>Oct  4 19:02:11 dcasrv01 ovirt-ha-agent: pending = =3D getattr(dispatcher, 'pending', lambda: = 0)<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 19:02:11 = dcasrv01 journal: vdsm vds.dispatcher ERROR SSL error during reading = data: unexpected eof<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct =  4 19:02:11 dcasrv01 journal: ovirt-ha-agent = ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to = storage server failed' - trying to restart = agent<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 19:02:11 = dcasrv01 ovirt-ha-agent: = ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to = storage server failed' - trying to restart = agent<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 19:02:12 = dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 18:02:12.384611] = C [MSGID: 106003] = [glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action] = 0-management: Server quorum regained for volume data. Starting local = bricks.<o:p></o:p></p></div><div><p class=3DMsoNormal>Oct  4 = 19:02:12 dcasrv01 etc-glusterfs-glusterd.vol[2252]: [2016-10-04 = 18:02:12.388981] C [MSGID: 106003] = [glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action] = 0-management: Server quorum regained for volume engine. Starting local = bricks.<o:p></o:p></p></div></div><div><p = class=3DMsoNormal><o:p> </o:p></p></div><div><p = class=3DMsoNormal>And at that point VDSM started complaining that the = hosted-engine-storage domain doesn't exist = anymore:<o:p></o:p></p></div><div><div><p class=3DMsoNormal>Oct  4 = 19:02:30 dcasrv01 journal: ovirt-ha-agent = ovirt_hosted_engine_ha.lib.image.Image ERROR Error fetching volumes = list: Storage domain does not exist: = (u'bbb70623-194a-46d2-a164-76a4876ecaaf',)<o:p></o:p></p></div><div><p = class=3DMsoNormal>Oct  4 19:02:30 dcasrv01 ovirt-ha-agent: = ERROR:ovirt_hosted_engine_ha.lib.image.Image:Error fetching volumes = list: Storage domain does not exist: = (u'bbb70623-194a-46d2-a164-76a4876ecaaf',)<o:p></o:p></p></div></div><div= style=3D'border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><div><div><div><block= quote style=3D'border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm = 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= href=3D"http://ovirt_hosted_engine_ha.br">ovirt_hosted_engine_ha.br</a>ok= er.storage_broker.StorageBroker::(get_raw_stats_for_service_type) Failed = to read metadata from = /rhev/data-center/mnt/glusterSD/dcastor01:engine/bbb70623-194a-46d2-a164-= 76a4876ecaaf/ha_agent/hosted-engine.metadata</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= pwd</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= process           =             &= nbsp;     TCP Port  RDMA Port  = Online  Pid</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;   49153     = 0          = Y       3076</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;   49153     = 0          = Y       3019</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          Y   =     3097</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3088</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3039</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3114</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= process           =             &= nbsp;     TCP Port  RDMA Port  = Online  Pid</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3097</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3088</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3039</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3114</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= process           =             &= nbsp;     TCP Port  RDMA Port  = Online  Pid</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3097</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3088</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3039</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3114</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= process           =             &= nbsp;     TCP Port  RDMA Port  = Online  Pid</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;          = 2049      = 0          = Y       3097</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3088</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;         =  2049      = 0          = Y       3039</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;    N/A       = N/A        = Y       3114</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= p;            = ;         = </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm = 0cm 0cm'><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span><= /b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> <a = href=3D"mailto:users-bounces@ovirt.org" = target=3D"_blank">users-bounces@ovirt.org</a> [mailto:<a = href=3D"mailto:users-bounces@ovirt.org" = target=3D"_blank">users-bounces@ovirt.org</a>] <b>On Behalf Of </b>Jason = Jeffrey<br><b>Sent:</b> 03 October 2016 18:40<br><b>To:</b> <a = href=3D"mailto:users@ovirt.org" = target=3D"_blank">users@ovirt.org</a></span><o:p></o:p></p><div><div><p = class=3DMsoNormal><br><b>Subject:</b> Re: [ovirt-users] 4.0 - 2nd node = fails on deploy<o:p></o:p></p></div></div></div></div><div><div><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'= style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span><= /b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> Simone = Tiraboschi [<a href=3D"mailto:stirabos@redhat.com" = target=3D"_blank">mailto:stirabos@redhat.com</a>] <br><b>Sent:</b> 03 = October 2016 09:27<br><b>To:</b> Jason Jeffrey <<a = href=3D"mailto:jason@sudo.co.uk" = target=3D"_blank">jason@sudo.co.uk</a>><br><b>Cc:</b> users <<a = href=3D"mailto:users@ovirt.org" = target=3D"_blank">users@ovirt.org</a>><br><b>Subject:</b> Re: = [ovirt-users] 4.0 - 2nd node fails on deploy</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>On Mon, Oct = 3, 2016 at 12:45 AM, Jason Jeffrey <<a = href=3D"mailto:jason@sudo.co.uk" = target=3D"_blank">jason@sudo.co.uk</a>> = wrote:<o:p></o:p></p><blockquote style=3D'border:none;border-left:solid = #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5= .0pt'><div><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Hi,</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>I am trying to build a x3 HC cluster, with a self hosted = engine using gluster.</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>I have successful built the 1^st node,  = however when I attempt to run hosted-engine =E2=80=93deploy on node 2, I = get the following error</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[WARNING] A configuration file must be supplied to deploy = Hosted Engine on an additional host.</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ ERROR ] 'version' is not stored in the HE configuration = image</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ ERROR ] Unable to get the answer file from the shared = storage</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ ERROR ] Failed to execute stage 'Environment = customization': Unable to get the answer file from the shared = storage</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ INFO  ] Stage: Clean up</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ INFO  ] Generating answer file = '/var/lib/ovirt-hosted-engine-setup/answers/answers-20161002232505.conf'<= /span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ INFO  ] Stage: = Pre-termination</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ INFO  ] Stage: Termination</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[ ERROR ] Hosted Engine deployment failed    = </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Looking at the failure in the log = file..</span><o:p></o:p></p></div></div></blockquote><div><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p></div><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Can you = please attach hosted-engine-setup logs from the first = host?<o:p></o:p></p></div><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p></div><blockquote style=3D'border:none;border-left:solid = #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5= .0pt'><div><div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 WARNING = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._customization:151 A = configuration</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>file must be supplied to deploy Hosted Engine on an = additional host.</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:61 = _fetch_answer_f</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>ile</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:69 fetching = from:</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>/rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91= b5-4f49-9c6b-fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/7</= span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>8cb2527-a2e2-489a-9fad-465a72221b37</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:69 executing: 'sudo -u vdsm dd = i</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>f=3D/rhev/data-center/mnt/glusterSD/dcastor02:engine/0a02156= 3-91b5-4f49-9c6b-fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b= </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>/78cb2527-a2e2-489a-9fad-465a72221b37 = bs=3D4k'</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:70 executing: 'tar -tvf = -'</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:88 stdout:</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 DEBUG = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib._dd_pipe_tar:89 stderr:</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 ERROR = otopi.plugins.gr_he_common.core.remote_answerfile = heconflib.validateConfImage:111 'version' is not = stored</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>in the HE configuration image</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>2016-10-02 23:25:05 ERROR = otopi.plugins.gr_he_common.core.remote_answerfile = remote_answerfile._fetch_answer_file:73 Unable to get = t</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>he answer file from the shared = storage</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Looking at the detected gluster path - = /rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-= fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/</span><o:p></o:= p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>[root@dcasrv02 ~]# ls -al = /rhev/data-center/mnt/glusterSD/dcastor02:engine/0a021563-91b5-4f49-9c6b-= fff45e85a025/images/f055216c-02f9-4cd1-a22c-d6b56a0a8e9b/</span><o:p></o:= p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>total 1049609</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>drwxr-xr-x. 2 vdsm kvm       = 4096 Oct  2 04:46 .</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>drwxr-xr-x. 6 vdsm kvm       = 4096 Oct  2 04:46 ..</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>-rw-rw----. 1 vdsm kvm 1073741824 Oct  2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>-rw-rw----. 1 vdsm kvm    1048576 Oct  = 2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37.lease</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>-rw-r--r--. 1 vdsm = kvm        294 Oct  2 04:46 = 78cb2527-a2e2-489a-9fad-465a72221b37.meta  </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>78cb2527-a2e2-489a-9fad-465a72221b37 is  a 1 GB file, = is this the engine VM ?</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Copying the answers file form primary = (/etc/ovirt-hosted-engine/answers.conf ) to  node 2 and rerunning = produces the same error : (</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>(hosted-engine --deploy  = --config-append=3D/root/answers.conf )</span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Also tried on node 3, same issues </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Happy to provide logs and other = debugs</span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US> </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US>Thanks </span><o:p></o:p></p><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'>Jason </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US style=3D'color:#888888'> </span><o:p></o:p></p><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = lang=3DEN-US = style=3D'color:#888888'> </span><o:p></o:p></p></div></div><p = class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;margin-bottom:12.0pt'><br>______________= _________________________________<br>Users mailing list<br><a = href=3D"mailto:Users@ovirt.org" = target=3D"_blank">Users@ovirt.org</a><br><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/users</a><o:p><= /o:p></p></blockquote></div><p class=3DMsoNormal = style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p><= /o:p></p></div></div></div></div></div></div><p class=3DMsoNormal = style=3D'margin-bottom:12.0pt'><br>______________________________________= _________<br>Users mailing list<br><a href=3D"mailto:Users@ovirt.org" = target=3D"_blank">Users@ovirt.org</a><br><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = target=3D"_blank">http://lists.ovirt.org/mailman/listinfo/users</a><o:p><= /o:p></p></blockquote></div></div></div><p = class=3DMsoNormal><o:p> </o:p></p></div></div></blockquote></div><p = class=3DMsoNormal><o:p> </o:p></p></div></div></div></body></html> ------=_NextPart_000_1D1B_01D21E5B.76B9C8F0--