Hi,

I have installed two HCI platforms.
Then I set up the disaster-recovery configuration

On source platform I did :
# gluster volume set all cluster.enable-shared-storage enable
# gluster system:: execute gsec_create
# gluster volume geo-replication source gnodesen2-01.example.com::dest create push-pem
# gluster volume geo-replication source gnodesen2-01.example.com::dest config use_meta_volume true

On target platform I did :
# gluster volume set dest features.shard enable

I then realized these two tests :
1 - Sync data using the schedule_georep.py

   a - Use the : "python3 /usr/share/glusterfs/scripts/schedule_georep.py source gnodesen2-01.example.com dest " command to sync data between the volumes
   b - After the sync is complete, execute "gluster volume dest set features.read-only off"
   c - Execute import (not create) dest volume on the target platform manually through the Admin-UI,  Result : The domain could not be added and I get this error on vdsm log :
"
2020-08-24 10:18:05,644+0100 ERROR (jsonrpc/5) [storage.HSM] Unexpected error (hsm:2843)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 456, in getVersion
    version = self.getMetaParam(DMDK_VERSION)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 452, in getMetaParam
    return self._metadata[key]
  File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line 114, in __getitem__
    return dec(self._dict[key])
  File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line 225, in __getitem__
    return self._metadata[key]
KeyError: 'VERSION'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829, in getStorageDomainsList
    dom = sdCache.produce(sdUUID=sdUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in produce
    domain.getRealDomain()
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in _findDomain
   return findMethod(sdUUID)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/glusterSD.py", line 62, in findDomain
    return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
  File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 385, in __init__
    manifest = self.manifestClass(domainPath)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 162, in __init__
    sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 352, in __init__
    version = self.getVersion()
  File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 458, in getVersion
    raise se.MetaDataKeyNotFoundError("key={}".format(DMDK_VERSION))
vdsm.storage.exception.MetaDataKeyNotFoundError: Meta Data key not found error: ('key=VERSION',)

2020-08-24 10:18:05,644+0100 INFO  (jsonrpc/5) [vdsm.api] FINISH getStorageDomainsList return={'domlist': []} from=::ffff:10.80.101.27,48282, flow_id=5209d576-ee09-4de3-bc46-0c186c6ad52a, task_id=bc93aaae-5449-4d0d-9884-f47ea542a963 (api:54)
2020-08-24 10:18:05,645+0100 INFO  (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStorageDomains succeeded in 3.01 seconds (__init__:312)
2020-08-24 10:18:05,754+0100 INFO  (jsonrpc/4) [vdsm.api] START disconnectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=10.90.101.23:10.90.101.25', 'iqn': '', 'connection': '10.90.101.21:/dest', 'ipv6_enabled': 'false', 'id': 'cc24df41-19b0-41ed-ae85-e091e40cc612', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:10.80.101.27,48282, flow_id=5fcaf7d7-d3b6-4430-b5f9-28e6bd95656a, task_id=73654a43-c86b-4250-95d7-3e3399e5bc72 (api:48)
2020-08-24 10:18:05,754+0100 INFO  (jsonrpc/4) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/10.90.101.21:_dest (mount:215)
"

Note : I had to modify the line 105 from /usr/share/glusterfs/scripts/schedule_georep.py to be able to use it with python3 :
-         ey = "_".join([func.func_name] + list(args))
+         key = "_".join([func.__name__] + list(args))


2 - Sync data using the adminUI :
   a - Create a schedule on the adminUI and wait until the sync is complete.
   b - Switch off read-only on target volume : gluster volume set dest features.read-only off (on the target volume)
   c - Execute ansible failover scripts : ansible-playbook dr-rhv-failover.yml --tags="fail_over" command (on the ansible machine)

Result :
020-08-24 15:51:58,874+0100 INFO (jsonrpc/0) [IOProcessClient] (3a5d378b-8043-4072-9c58-e838343d6028) Starting client (init:308)
2020-08-24 15:51:58,886+0100 INFO (ioprocess/140212) [IOProcess] (3a5d378b-8043-4072-9c58-e838343d6028) Starting ioprocess (init:434)
2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0)** [vdsm.api] FINISH createStorageDomain error=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) from=::ffff:10.80.101.27,45118, flow_id=765c5009, task_id=ed3a34f9-5e81-40f4-a592-78e585c1f937 (api:52)
2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run
return fn(args, *kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
ret = func(args, *kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 95, in create
version)
**File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 60, in _preCreateValidation
raise se.StorageDomainNotEmpty(typeSpecificArg)
vdsm.storage.exception.StorageDomainNotEmpty: Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',)
2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') aborting: Task is aborted: "value=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) abortedcode=361" (task:1190)

2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) (dispatcher:83)
2020-08-24 15:51:58,889+0100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 361) in 0.20 seconds (init:312)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '****', 'vfs_type': 'glusterfs', 'port': '', 'iqn': '', 'connection': 'gnodesen2-01.example.com:/dest', 'ipv6_enabled': 'false', 'id': 'd104177f-f391-45bc-a8a7-905da90d4e7d', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:10.80.101.27,45118, flow_id=f8c31837-fd32-48f2-bed9-6138ea4c6bb0, task_id=badf6391-06cd-4e11-ade5-95dd10b86af5 (api:48)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/gnodesen2-01.example.com:_dest (mount:215)


Any ideas?

Regards.