9:04 a.m.
Hi all,
I had a catastrophic failure of a host with but the disk with the domain
on it is fine.
I've put the disk in a new machine, mounted it on /backups and I can see
this structure ...
[root@max backups]# ls -laR *
2f2da2d7-596f-4a53-8a1b-a301f84b3b74:
total 20
drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 .
drwxr-xr-x. 4 vdsm kvm 4096 Jun 10 11:53 ..
drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 dom_md
drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 images
drwxr-xr-x. 4 vdsm kvm 4096 Sep 15 2020 master
2f2da2d7-596f-4a53-8a1b-a301f84b3b74/dom_md:
total 32780
drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 .
drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 ..
-rwxr-xr-x. 1 vdsm kvm 0 Sep 15 2020 ids
-rwxr-xr-x. 1 vdsm kvm 16777216 Sep 15 2020 inbox
-rwxr-xr-x. 1 vdsm kvm 0 Sep 15 2020 leases
-rwxr-xr-x. 1 vdsm kvm 486 Sep 15 2020 metadata
-rwxr-xr-x. 1 vdsm kvm 16777216 Sep 15 2020 outbox
2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images:
total 92
drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 .
drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 ..
drwxr-xr-x. 2 vdsm kvm 4096 Feb 3 2021
1eb2b682-ef45-409d-9341-abcf61418619
drwxr-xr-x. 2 vdsm kvm 4096 Aug 11 2022
283838fd-43b5-42b9-a332-a7b980613188
drwxr-xr-x. 2 vdsm kvm 4096 Oct 4 2020
36237183-54a2-4232-b0b3-6d1a9457c23a
drwxr-xr-x. 2 vdsm kvm 4096 Jan 5 2022
366e7b29-2123-45c0-9190-dafead431651
drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29
3ea98df5-3ee8-4d21-8607-94284e7fff37
drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29
44c3611b-1205-48ca-9a84-007eaae03cbb
drwxr-xr-x. 2 vdsm kvm 4096 May 18 23:49
674b364c-b74a-4fff-a38b-ca1089103241
drwxr-xr-x. 2 vdsm kvm 4096 Oct 15 2021
6949a9e0-4cc0-4eee-be99-e88795cd6447
drwxr-xr-x. 2 vdsm kvm 4096 Jan 5 2022
8b618bad-6dfb-48de-90ad-f325bf50cd45
drwxr-xr-x. 2 vdsm kvm 4096 Feb 6 2021
91f1f606-0f26-4ef0-a2fe-cccde6c78e89
drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35
94f4bf41-2b2f-42bc-ae21-0e80205ba19a
drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35
a18f0c48-b3ac-4b89-9e52-fddaddbc8f4a
drwxr-xr-x. 2 vdsm kvm 4096 May 26 15:51
ac4a428d-961e-46ae-a279-f282fd9ecf94
drwxr-xr-x. 2 vdsm kvm 4096 May 26 15:51
ba1a624c-6995-4109-b770-7bbd1efc340e
drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020
dec14b5b-69fe-4c51-abb4-d8296c547a0b
drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020
e7a17f3b-7567-4c90-b58e-18f34d4686c0
drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29
ea786bee-b98e-44c0-a44a-a63af63b8c51
drwxr-xr-x. 2 vdsm kvm 4096 Apr 21 2021
ec37f395-3b4b-41df-8f5b-8a22b7e1273b
drwxr-xr-x. 2 vdsm kvm 4096 Feb 25 2022
f3e641c7-a2cc-42af-bcd1-b5ecdf437e03
drwxr-xr-x. 2 vdsm kvm 4096 May 18 23:55
f4c221a6-6ce4-44cc-8d32-d673e40aa915
drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35
f539c098-3f16-410f-938f-3de6ef37b017
2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images/1eb2b682-ef45-409d-9341-abcf61418619:
total 104857624
drwxr-xr-x. 2 vdsm kvm 4096 Feb 3 2021 .
drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 ..
-rwxr-xr-x. 1 vdsm kvm 107374182400 Feb 3 2021
e29b005f-d272-49f8-a162-41f908ca67a4
-rwxr-xr-x. 1 vdsm kvm 300 Feb 3 2021
e29b005f-d272-49f8-a162-41f908ca67a4.meta
2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images/283838fd-43b5-42b9-a332-a7b980613188:
total 23223760
drwxr-xr-x. 2 vdsm kvm 4096 Aug 11 2022 .
drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 ..
-rwxr-xr-x. 1 vdsm kvm 23781113856 Aug 11 2022
3273221e-1649-4e4a-816c-2ffec18e51c7
-rwxr-xr-x. 1 vdsm kvm 251 Aug 11 2022
3273221e-1649-4e4a-816c-2ffec18e51c7.meta
.
. etc
So... is there a way to get this back ?
I tried "Import Domain" both with the disk manually mounted and
unmounted, both failed and I get errors in the vdsm.log like this...
2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0)
[storage.StorageServer.MountConnection] Creating directory
'/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74'
(storageServer:167)
2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0) [storage.fileUtils]
Creating directory:
/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74
mode: None (fileUtils:201)
2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0) [storage.Mount] mounting
/backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74 at
/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74
(mount:207)
2023-06-15 15:57:55,176+1000 ERROR (jsonrpc/0) [storage.HSM] Could not
connect to storageServer (hsm:2374)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line
2371, in connectStorageServer
conObj.connect()
File
"/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
180, in connect
six.reraise(t, v, tb)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
171, in connect
self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line
210, in mount
cgroup=cgroup)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py",
line 56, in __call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py",
line 54, in <lambda>
**kwargs)
File "<string>", line 2, in mount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t',
'posix',
'/backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74',
'/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74']
failed with rc=32 out=b'' err=b"mount:
/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74:
unknown filesystem type 'posix'.\n"
2023-06-15 15:57:55,176+1000 INFO (jsonrpc/0)
[storage.StorageDomainCache] Invalidating storage domain cache (sdc:74)
2023-06-15 15:57:55,176+1000 INFO (jsonrpc/0) [vdsm.api] FINISH
connectStorageServer return={'statuslist': [{'id':
'00000000-0000-0000-0000-000000000000', 'status': 477}]}
from=::ffff:172.20.1.160,47702,
flow_id=6693c76d-b62b-4e62-9b70-c87f2a199705,
task_id=682abcff-3040-46d7-aaa1-f2445f0a6698 (api:54)
2023-06-15 15:57:55,283+1000 INFO (jsonrpc/2) [vdsm.api] START
disconnectStorageServer(domType=6,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password':
'********', 'vfs_type': 'posix', 'port': '',
'iqn': '', 'connection':
'/backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74', 'ipv6_enabled':
'false', 'id': '00000000-0000-0000-0000-000000000000',
'user': '',
'tpgt': '1'}]) from=::ffff:172.20.1.160,47702,
flow_id=0d3d4a3e-3523-4436-93b3-e22bff47f082,
task_id=11870123-365d-4b40-b481-be5d44098fc2 (api:48)
2023-06-15 15:57:55,284+1000 INFO (jsonrpc/2) [storage.Mount]
unmounting
/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74
(mount:215)
2023-06-15 15:57:55,290+1000 ERROR (jsonrpc/2) [storage.HSM] Could not
disconnect from storageServer (hsm:2480)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line
2476, in disconnectStorageServer
conObj.disconnect()
File
"/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
202, in disconnect
self._mount.umount(True, True)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line
217, in umount
umount(self.fs_file, force=force, lazy=lazy, freeloop=freeloop)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py",
line 56, in __call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py",
line 54, in <lambda>
**kwargs)
File "<string>", line 2, in umount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/umount', '-f',
'-l',
'/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74']
failed with rc=32 out=b'' err=b'umount:
/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74: no
mount point specified.\n'
2023-06-15 15:57:55,290+1000 INFO (jsonrpc/2)
[storage.StorageDomainCache] Refreshing storage domain cache
(resize=False) (sdc:80)
2023-06-15 15:57:55,290+1000 INFO (jsonrpc/2) [storage.ISCSI] Scanning
iSCSI devices (iscsi:442)
2023-06-15 15:57:55,328+1000 INFO (jsonrpc/2) [storage.ISCSI] Scanning
iSCSI devices: 0.04 seconds (utils:390)
2023-06-15 15:57:55,328+1000 INFO (jsonrpc/2) [storage.HBA] Scanning FC
devices (hba:60)
2023-06-15 15:57:55,416+1000 INFO (jsonrpc/2) [storage.HBA] Scanning FC
devices: 0.09 seconds (utils:390)
2023-06-15 15:57:55,416+1000 INFO (jsonrpc/2) [storage.Multipath]
Waiting until multipathd is ready (multipath:112)
2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [storage.Multipath]
Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:139)
2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2)
[storage.StorageDomainCache] Refreshing storage domain cache: 2.17
seconds (utils:390)
2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [vdsm.api] FINISH
disconnectStorageServer return={'statuslist': [{'id':
'00000000-0000-0000-0000-000000000000', 'status': 477}]}
from=::ffff:172.20.1.160,47702,
flow_id=0d3d4a3e-3523-4436-93b3-e22bff47f082,
task_id=11870123-365d-4b40-b481-be5d44098fc2 (api:54)
2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer]
RPC call StoragePool.disconnectStorageServer took more than 1.00 seconds
to succeed: 2.18 (__init__:316)
any help appreciated
thanks
Tim