7:42 p.m.
Hi,
I have installed two HCI platforms.
Then I set up the disaster-recovery configuration
On source platform I did :
# gluster volume set all cluster.enable-shared-storage enable
# gluster system:: execute gsec_create
# gluster volume geo-replication *source* gnodesen2-01.example.com::dest
create push-pem
# gluster volume geo-replication *source* gnodesen2-01.example.com::dest
config use_meta_volume true
On target platform I did :
# gluster volume set *dest* features.shard enable
I then realized these two tests :
*1 - Sync data using the schedule_georep.py*
a - Use the : "python3 /usr/share/glusterfs/scripts/schedule_georep.py
source gnodesen2-01.example.com dest " command to sync data between the
volumes
b - After the sync is complete, execute "gluster volume dest set
features.read-only off"
c - Execute *import* (not create) *dest* volume on the target platform
manually through the Admin-UI, Result : The domain could not be added and
I get this error on vdsm log :
"
2020-08-24 10:18:05,644+0100 ERROR (jsonrpc/5) [storage.HSM] Unexpected
error (hsm:2843)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 456, in
getVersion
version = self.getMetaParam(DMDK_VERSION)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 452, in
getMetaParam
return self._metadata[key]
File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line
114, in __getitem__
return dec(self._dict[key])
File "/usr/lib/python3.6/site-packages/vdsm/storage/persistent.py", line
225, in __getitem__
return self._metadata[key]
KeyError: 'VERSION'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829,
in getStorageDomainsList
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in
produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/glusterSD.py", line
62, in findDomain
return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 385,
in __init__
manifest = self.manifestClass(domainPath)
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 162,
in __init__
sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 352, in
__init__
version = self.getVersion()
* File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 458, in
getVersion raise
se.MetaDataKeyNotFoundError("key={}".format(DMDK_VERSION))vdsm.storage.exception.MetaDataKeyNotFoundError:
Meta Data key not found error: ('key=VERSION',)*
2020-08-24 10:18:05,644+0100 INFO (jsonrpc/5) [vdsm.api] FINISH
getStorageDomainsList return={'domlist': []}
from=::ffff:10.80.101.27,48282,
flow_id=5209d576-ee09-4de3-bc46-0c186c6ad52a,
task_id=bc93aaae-5449-4d0d-9884-f47ea542a963 (api:54)
2020-08-24 10:18:05,645+0100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC
call Host.getStorageDomains succeeded in 3.01 seconds (__init__:312)
2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [vdsm.api] START
disconnectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password':
'********', 'vfs_type': 'glusterfs', 'port': '',
'mnt_options':
'backup-volfile-servers=10.90.101.23:10.90.101.25', 'iqn': '',
'connection': '10.90.101.21:/dest', 'ipv6_enabled':
'false', 'id':
'cc24df41-19b0-41ed-ae85-e091e40cc612', 'user': '',
'tpgt': '1'}],
options=None) from=::ffff:10.80.101.27,48282,
flow_id=5fcaf7d7-d3b6-4430-b5f9-28e6bd95656a,
task_id=73654a43-c86b-4250-95d7-3e3399e5bc72 (api:48)
2020-08-24 10:18:05,754+0100 INFO (jsonrpc/4) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/10.90.101.21:_dest (mount:215)
"
*Note :* I had to modify the line 105 from
/usr/share/glusterfs/scripts/schedule_georep.py to be able to use it with
python3 :
- ey = "_".join([func.func_name] + list(args))
+ key = "_".join([func.__name__] + list(args))
*2 - Sync data using the adminUI :*
a - Create a schedule on the adminUI and wait until the sync is complete.
b - Switch off read-only on target volume : gluster volume set dest
features.read-only off (on the target volume)
c - Execute ansible failover scripts : ansible-playbook
dr-rhv-failover.yml --tags="fail_over" command (on the ansible machine)
Result :
020-08-24 15:51:58,874+0100 INFO (jsonrpc/0) [IOProcessClient]
(3a5d378b-8043-4072-9c58-e838343d6028) Starting client (init:308)
2020-08-24 15:51:58,886+0100 INFO (ioprocess/140212) [IOProcess]
(3a5d378b-8043-4072-9c58-e838343d6028) Starting ioprocess (init:434)
2020-08-24 15:51:58,888+0100 INFO (jsonrpc/0)** [vdsm.api] FINISH
createStorageDomain error=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) from=::ffff:10.80.101.27,45118,
flow_id=765c5009, task_id=ed3a34f9-5e81-40f4-a592-78e585c1f937 (api:52)
2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') Unexpected error (task:880)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in
_run
return fn(args, *kargs)
File "<decorator-gen-121>", line 2, in createStorageDomain
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(args, *kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2644, in
createStorageDomain
max_hosts=max_hosts)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 95, in
create
version)
**File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 60,
in _preCreateValidation
*raise
se.StorageDomainNotEmpty(typeSpecificArg)vdsm.storage.exception.StorageDomainNotEmpty:
Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',)2020-08-24 15:51:58,888+0100 INFO
(jsonrpc/0) [storage.TaskManager.Task]
(Task='ed3a34f9-5e81-40f4-a592-78e585c1f937') aborting: Task is aborted:
"value=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) abortedcode=361" (task:1190)*
2020-08-24 15:51:58,888+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
createStorageDomain error=Storage domain is not empty - requires cleaning:
('gnodesen2-01.example.com:/dest',) (dispatcher:83)
2020-08-24 15:51:58,889+0100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC*
call StorageDomain.create failed *(error 361) in 0.20 seconds (init:312)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [vdsm.api] START
disconnectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password':
'****', 'vfs_type': 'glusterfs', 'port': '',
'iqn': '', 'connection':
'gnodesen2-01.example.com:/dest', 'ipv6_enabled': 'false',
'id':
'd104177f-f391-45bc-a8a7-905da90d4e7d', 'user': '',
'tpgt': '1'}],
options=None) from=::ffff:10.80.101.27,45118,
flow_id=f8c31837-fd32-48f2-bed9-6138ea4c6bb0,
task_id=badf6391-06cd-4e11-ade5-95dd10b86af5 (api:48)
2020-08-24 15:51:58,934+0100 INFO (jsonrpc/2) [storage.Mount] unmounting
/rhev/data-center/mnt/glusterSD/gnodesen2-01.example.com:_dest (mount:215)
Any ideas?
Regards.