Hi, I have installed two HCI platforms. Then I set up the disaster-recovery configuration On source platform I did : # gluster volume set all cluster.enable-shared-storage enable # gluster system:: execute gsec_create # gluster volume geo-replication *source* gnodesen2-01.example.com::dest create push-pem # gluster volume geo-replication *source* gnodesen2-01.example.com::dest config use_meta_volume true On target platform I did : # gluster volume set *dest* features.shard enable I then realized these two tests : *1 - Sync data using the schedule_georep.py* a - Use the : "python3 /usr/share/glusterfs/scripts/schedule_georep.py source gnodesen2-01.example.com dest " command to sync data between the volumes b - After the sync is complete, execute "gluster volume dest set features.read-only off" c - Execute *import* (not create) *dest* volume on the target platform manually through the Admin-UI, Result : The domain could not be added and I get this error on vdsm log : " 2020-08-24 10:18:05,644+0100 ERROR (jsonrpc/5) [storage.HSM] Unexpected error (hsm:2843) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 456, in getVersion version = self.getMetaParam(DMDK_VERSION) File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 452, in getMetaParam return self._metadata[key] File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line 114, in __getitem__ return dec(self._dict[key]) File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line 225, in __getitem__ return self._metadata[key] KeyError: 'VERSION' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829, in getStorageDomainsList dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/glusterSD.py", line 62, in findDomain return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 385, in __init__ manifest = self.manifestClass(domainPath) File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 162, in __init__ sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata) File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 352, in __init__ version = self.getVersion() * File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 458, in getVersion raise se.MetaDataKeyNotFoundError("key={}".format(DMDK_VERSION))vdsm.storage.exception.MetaDataKeyNotFoundError: Meta Data key not found error: ('key=VERSION',)* 2020-08-24 10:18:05,644+0100 INFO (jsonrpc/5) [vdsm.api] FINISH getStorageDomainsList return={'domlist': []} from=::ffff:10.80.101.27,48282, flow_id=5209d576-ee09-4de3-bc46-0c186c6ad52a, task_id=bc93aaae-5449-4d0d-9884-f47ea542a963 (api:54) 2020-08-24 10:18:05,645+0100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStorageDomains succeeded in 3.01 seconds (__init__:312) 2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [vdsm.api] START disconnectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=10.90.101.23:10.90.101.25', 'iqn': '', 'connection': '10.90.101.21:/dest', 'ipv6_enabled': 'false', 'id': 'cc24df41-19b0-41ed-ae85-e091e40cc612', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:10.80.101.27,48282, flow_id=5fcaf7d7-d3b6-4430-b5f9-28e6bd95656a, task_id=73654a43-c86b-4250-95d7-3e3399e5bc72 (api:48) 2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/10.90.101.21:_dest (mount:215) " *Note :* I had to modify the line 105 from /usr/share/glusterfs/scripts/schedule_georep.py to be able to use it with python3 : - ey = "_".join([func.func_name] + list(args)) + key = "_".join([func.__name__] + list(args)) *2 - Sync data using the adminUI :* a - Create a schedule on the adminUI and wait until the sync is complete. b - Switch off read-only on target volume : gluster volume set dest features.read-only off (on the target volume) c - Execute ansible failover scripts : ansible-playbook dr-rhv-failover.yml --tags="fail_over" command (on the ansible machine) Result : 020-08-24 15:51:58,874+0100 INFO (jsonrpc/0) [IOProcessClient] (3a5d378b-8043-4072-9c58-e838343d6028) Starting client (init:308) 2020-08-24 15:51:58,886+0100 INFO (ioprocess/140212) [IOProcess] (3a5d378b-8043-4072-9c58-e838343d6028) Starting ioprocess (init:434) 2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0)** [vdsm.api] FINISH createStorageDomain error=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) from=::ffff:10.80.101.27,45118, flow_id=765c5009, task_id=ed3a34f9-5e81-40f4-a592-78e585c1f937 (api:52) 2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') Unexpected error (task:880) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(args, *kargs) File "<decorator-gen-121>", line 2, in createStorageDomain File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method ret = func(args, *kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in createStorageDomain max_hosts=max_hosts) File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 95, in create version) **File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 60, in _preCreateValidation *raise se.StorageDomainNotEmpty(typeSpecificArg)vdsm.storage.exception.StorageDomainNotEmpty: Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',)2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') aborting: Task is aborted: "value=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) abortedcode=361" (task:1190)* 2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH createStorageDomain error=Storage domain is not empty - requires cleaning: ('gnodesen2-01.example.com:/dest',) (dispatcher:83) 2020-08-24 15:51:58,889+0100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC* call StorageDomain.create failed *(error 361) in 0.20 seconds (init:312) 2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=7, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '****', 'vfs_type': 'glusterfs', 'port': '', 'iqn': '', 'connection': 'gnodesen2-01.example.com:/dest', 'ipv6_enabled': 'false', 'id': 'd104177f-f391-45bc-a8a7-905da90d4e7d', 'user': '', 'tpgt': '1'}], options=None) from=::ffff:10.80.101.27,45118, flow_id=f8c31837-fd32-48f2-bed9-6138ea4c6bb0, task_id=badf6391-06cd-4e11-ade5-95dd10b86af5 (api:48) 2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [storage.Mount] unmounting /rhev/data-center/mnt/glusterSD/gnodesen2-01.example.com:_dest (mount:215) Any ideas? Regards.