4:22 a.m.
Hi,
i have a Six node Stretched HCI cluster with 2 x 3 =6 distributed Replicated bricks, want
to achieve Active - Active Disaster Recovery , if my three hosts in DC power Down or
Failed my DR Site Three node should take care of running virtual machines.
During Split-brain scenario Ovirt-Hosted-Engine VM goes in PAUSE mode and not restarting
on DR hosts and waited for long time, due to this my all production VMs are not restarting
on DR Site.
please guide to achieve Active-Active Disaster Recovery with 6 node stretched cluster. i
referred document hosted official site but did not get success, it would be great help of
anybody can extend help or guide me for the same.
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/...
My Setup insight.
root@phnode06 /]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.1.1:/gluster_bricks/data/data 58744 0 Y 1746524
Brick 172.16.1.2:/gluster_bricks/data/data 58732 0 Y 2933385
Brick 172.16.1.3:/gluster_bricks/data/data 60442 0 Y 6158
Brick phnode04..local:/gluster_bricks
/data/data1 51661 0 Y 5905
Brick phnode05..local:/gluster_bricks
/data/data1 52177 0 Y 6178
Brick phnode06..local:/gluster_bricks
/data/data1 59910 0 Y 6208
Self-heal Daemon on localhost N/A N/A Y 5878
Self-heal Daemon on phnode03..local N/A N/A Y 6133
Self-heal Daemon on gluster01..local N/A N/A Y 6072
Self-heal Daemon on phnode05..local N/A N/A Y 5751
Self-heal Daemon on gluster02..local N/A N/A Y 5746
Self-heal Daemon on phnode04..local N/A N/A Y 4734
Task Status of Volume data
------------------------------------------------------------------------------
Task : Rebalance
ID : 1c30fbfa-2707-457e-b844-ccaf2e07aa7f
Status : completed
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.1.1:/gluster_bricks/engine/eng
ine 53723 0 Y 1746535
Brick 172.16.1.2:/gluster_bricks/engine/eng
ine 49937 0 Y 2933395
Brick 172.16.1.3:/gluster_bricks/engine/eng
ine 53370 0 Y 6171
Brick phnode04..local:/gluster_bricks
/engine/engine1 52024 0 Y 5916
Brick phnode05..local:/gluster_bricks
/engine/engine1 51785 0 Y 6189
Brick phnode06..local:/gluster_bricks
/engine/engine1 55443 0 Y 6221
Self-heal Daemon on localhost N/A N/A Y 5878
Self-heal Daemon on gluster02..local N/A N/A Y 5746
Self-heal Daemon on gluster01..local N/A N/A Y 6072
Self-heal Daemon on phnode05..local N/A N/A Y 5751
Self-heal Daemon on phnode03..local N/A N/A Y 6133
Self-heal Daemon on phnode04..local N/A N/A Y 4734
Task Status of Volume engine
------------------------------------------------------------------------------
Task : Rebalance
ID : 21304de4-b100-4860-b408-5e58103080a2
Status : completed
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 172.16.1.1:/gluster_bricks/vmstore/vm
store 50244 0 Y 1746546
Brick 172.16.1.2:/gluster_bricks/vmstore/vm
store 51607 0 Y 2933405
Brick 172.16.1.3:/gluster_bricks/vmstore/vm
store 49835 0 Y 6182
Brick phnode04..local:/gluster_bricks
/vmstore/vmstore1 54098 0 Y 5927
Brick phnode05..local:/gluster_bricks
/vmstore/vmstore1 56565 0 Y 6200
Brick phnode06..local:/gluster_bricks
/vmstore/vmstore1 58653 0 Y 1853456
Self-heal Daemon on localhost N/A N/A Y 5878
Self-heal Daemon on gluster02..local N/A N/A Y 5746
Self-heal Daemon on gluster01..local N/A N/A Y 6072
Self-heal Daemon on phnode05..local N/A N/A Y 5751
Self-heal Daemon on phnode03..local N/A N/A Y 6133
Self-heal Daemon on phnode04..local N/A N/A Y 4734
Task Status of Volume vmstore
------------------------------------------------------------------------------
Task : Rebalance
ID : 7d1849f8-433a-43b1-93bc-d674a7b0aef3
Status : completed
[root@phnode06 /]# hosted-engine --vm-status
--== Host phnode01..local (id: 1) status ==--
Host ID : 1
Host timestamp : 9113939
Score : 3400
Engine status : {"vm": "up", "health":
"good", "detail": "Up"}
Hostname : phnode01..local
Local maintenance : False
stopped : False
crc32 : 00979124
conf_on_shared_storage : True
local_conf_timestamp : 9113939
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=9113939 (Sat Aug 19 14:39:41 2023)
host-id=1
score=3400
vm_conf_refresh_time=9113939 (Sat Aug 19 14:39:41 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineUp
stopped=False
--== Host phnode03..local (id: 2) status ==--
Host ID : 2
Host timestamp : 405629
Score : 3400
Engine status : {"vm": "down",
"health": "bad", "detail": "unknown",
"reason": "vm not running on this host"}
Hostname : phnode03..local
Local maintenance : False
stopped : False
crc32 : 0b1c7489
conf_on_shared_storage : True
local_conf_timestamp : 405630
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=405629 (Sat Aug 19 14:39:43 2023)
host-id=2
score=3400
vm_conf_refresh_time=405630 (Sat Aug 19 14:39:43 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host phnode02..local (id: 3) status ==--
Host ID : 3
Host timestamp : 9069311
Score : 3400
Engine status : {"vm": "down",
"health": "bad", "detail": "unknown",
"reason": "vm not running on this host"}
Hostname : phnode02..local
Local maintenance : False
stopped : False
crc32 : b84baa31
conf_on_shared_storage : True
local_conf_timestamp : 9069311
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=9069311 (Sat Aug 19 14:39:41 2023)
host-id=3
score=3400
vm_conf_refresh_time=9069311 (Sat Aug 19 14:39:41 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host phnode04..local (id: 4) status ==--
Host ID : 4
Host timestamp : 1055893
Score : 3400
Engine status : {"vm": "down",
"health": "bad", "detail": "unknown",
"reason": "vm not running on this host"}
Hostname : phnode04..local
Local maintenance : False
stopped : False
crc32 : 58a9b84c
conf_on_shared_storage : True
local_conf_timestamp : 1055893
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1055893 (Sat Aug 19 14:39:38 2023)
host-id=4
score=3400
vm_conf_refresh_time=1055893 (Sat Aug 19 14:39:38 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host phnode05..local (id: 5) status ==--
Host ID : 5
Host timestamp : 404187
Score : 3400
Engine status : {"vm": "down",
"health": "bad", "detail": "unknown",
"reason": "vm not running on this host"}
Hostname : phnode05..local
Local maintenance : False
stopped : False
crc32 : 48cf9186
conf_on_shared_storage : True
local_conf_timestamp : 404187
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=404187 (Sat Aug 19 14:39:40 2023)
host-id=5
score=3400
vm_conf_refresh_time=404187 (Sat Aug 19 14:39:40 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host phnode06..local (id: 6) status ==--
Host ID : 6
Host timestamp : 1055902
Score : 3400
Engine status : {"vm": "down",
"health": "bad", "detail": "unknown",
"reason": "vm not running on this host"}
Hostname : phnode06..local
Local maintenance : False
stopped : False
crc32 : 913ff6c2
conf_on_shared_storage : True
local_conf_timestamp : 1055902
Status up-to-date : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1055902 (Sat Aug 19 14:39:48 2023)
host-id=6
score=3400
vm_conf_refresh_time=1055902 (Sat Aug 19 14:39:48 2023)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
[root@phnode06 /]#
8:53 a.m.
New subject: Active Active Stretched Node cluster not working as expected.
On Sat, Aug 19, 2023 at 7:03 PM <gaurang.patel(a)allotgroup.com> wrote:
Hi,
i have a Six node Stretched HCI cluster with 2 x 3 =6 distributed
Replicated bricks, want to achieve Active - Active Disaster Recovery , if
my three hosts in DC power Down or Failed my DR Site Three node should take
care of running virtual machines.
During Split-brain scenario Ovirt-Hosted-Engine VM goes in PAUSE mode and
not restarting on DR hosts and waited for long time, due to this my all
production VMs are not restarting on DR Site.
Hi,
even if the guide mentions GlusterFS as supported storage domain for
Active-Active DR scenario and Self Hosted Engine configuration as a
supported one for Active-Active DR scenario, and it doesn't explicitly
mention that HCI configurations are not supported, I have always
interpreted that the storage replication mentioned inside the docs refers
to "external" replication, provided by the storage array hw architecture,
not by RHV HCI sw solution itself. In that sense your architecture should
not be a supported one for Active-Active DR scenario.
Some wordings inside the docs:
"
...
This configuration requires replicated storage that is writeable on both
sites...
...
The sites require synchronously replicated storage that is writeable on
both sites with shared layer 2 (L2) network connectivity...
...
pre-requisites
Real-time storage replication service to duplicate the storage.
"
Anyway you can open a bugzilla request to ask for confirmation.
HIH,
Gianluca
1:52 a.m.
New subject: Active Active Stretched Node cluster not working as expected.
Hi Gianluca,
Thank you for reply,
I found VMSTORE volume mount options across all six nodes but found only 3 primary nodes
in “Hosted_engine” volume, files are attached.
Volume is “Distributed Replicated” across all 6 nodes, 2 x 3 = 6.
I have edited file “/etc/ovirt-hosted-engine/hosted-engine.conf” to change Mount options
for the Hosted Engine Volume, but did not get success, will check after putting cluster in
maintenance mode and try.
Will follow
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/3NCQ5F7QDNSB...
Suggest any other solution if any.
Thanks in advance.
Gaurang Patel
From: Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
Sent: 20 August 2023 19:24
To: Gaurang Patel <gaurang.patel(a)allotgroup.com>
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Active Active Stretched Node cluster not working as expected.
CAUTION: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe.
On Sat, Aug 19, 2023 at 7:03 PM
<gaurang.patel@allotgroup.com<mailto:gaurang.patel@allotgroup.com>> wrote:
Hi,
i have a Six node Stretched HCI cluster with 2 x 3 =6 distributed Replicated bricks, want
to achieve Active - Active Disaster Recovery , if my three hosts in DC power Down or
Failed my DR Site Three node should take care of running virtual machines.
During Split-brain scenario Ovirt-Hosted-Engine VM goes in PAUSE mode and not restarting
on DR hosts and waited for long time, due to this my all production VMs are not restarting
on DR Site.
Hi,
even if the guide mentions GlusterFS as supported storage domain for Active-Active DR
scenario and Self Hosted Engine configuration as a supported one for Active-Active DR
scenario, and it doesn't explicitly mention that HCI configurations are not supported,
I have always interpreted that the storage replication mentioned inside the docs refers to
"external" replication, provided by the storage array hw architecture, not by
RHV HCI sw solution itself. In that sense your architecture should not be a supported one
for Active-Active DR scenario.
Some wordings inside the docs:
"
...
This configuration requires replicated storage that is writeable on both sites...
...
The sites require synchronously replicated storage that is writeable on both sites with
shared layer 2 (L2) network connectivity...
...
pre-requisites
Real-time storage replication service to duplicate the storage.
"
Anyway you can open a bugzilla request to ask for confirmation.
HIH,
Gianluca
DISCLAIMER : The content of this email is confidential and intended for the recipient
specified in message only. It is strictly forbidden to share any part of this message with
any third party, without a written consent of the sender. If you received this message by
mistake, please reply to this message and follow with its deletion, so that we can ensure
such a mistake does not occur in the future.
487
days inactive
490
days old
2 comments
3 participants
participants (3)
-
Gaurang Patel -
gaurang.patel＠allotgroup.com
-
Gianluca Cecchi