Trying to recover local domain from failed host
Hi all, I had a catastrophic failure of a host with but the disk with the domain on it is fine. I've put the disk in a new machine, mounted it on /backups and I can see this structure ... [root@max backups]# ls -laR * 2f2da2d7-596f-4a53-8a1b-a301f84b3b74: total 20 drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 . drwxr-xr-x. 4 vdsm kvm 4096 Jun 10 11:53 .. drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 dom_md drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 images drwxr-xr-x. 4 vdsm kvm 4096 Sep 15 2020 master 2f2da2d7-596f-4a53-8a1b-a301f84b3b74/dom_md: total 32780 drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 . drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 .. -rwxr-xr-x. 1 vdsm kvm 0 Sep 15 2020 ids -rwxr-xr-x. 1 vdsm kvm 16777216 Sep 15 2020 inbox -rwxr-xr-x. 1 vdsm kvm 0 Sep 15 2020 leases -rwxr-xr-x. 1 vdsm kvm 486 Sep 15 2020 metadata -rwxr-xr-x. 1 vdsm kvm 16777216 Sep 15 2020 outbox 2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images: total 92 drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 . drwxr-xr-x. 5 vdsm kvm 4096 Sep 15 2020 .. drwxr-xr-x. 2 vdsm kvm 4096 Feb 3 2021 1eb2b682-ef45-409d-9341-abcf61418619 drwxr-xr-x. 2 vdsm kvm 4096 Aug 11 2022 283838fd-43b5-42b9-a332-a7b980613188 drwxr-xr-x. 2 vdsm kvm 4096 Oct 4 2020 36237183-54a2-4232-b0b3-6d1a9457c23a drwxr-xr-x. 2 vdsm kvm 4096 Jan 5 2022 366e7b29-2123-45c0-9190-dafead431651 drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29 3ea98df5-3ee8-4d21-8607-94284e7fff37 drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29 44c3611b-1205-48ca-9a84-007eaae03cbb drwxr-xr-x. 2 vdsm kvm 4096 May 18 23:49 674b364c-b74a-4fff-a38b-ca1089103241 drwxr-xr-x. 2 vdsm kvm 4096 Oct 15 2021 6949a9e0-4cc0-4eee-be99-e88795cd6447 drwxr-xr-x. 2 vdsm kvm 4096 Jan 5 2022 8b618bad-6dfb-48de-90ad-f325bf50cd45 drwxr-xr-x. 2 vdsm kvm 4096 Feb 6 2021 91f1f606-0f26-4ef0-a2fe-cccde6c78e89 drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35 94f4bf41-2b2f-42bc-ae21-0e80205ba19a drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35 a18f0c48-b3ac-4b89-9e52-fddaddbc8f4a drwxr-xr-x. 2 vdsm kvm 4096 May 26 15:51 ac4a428d-961e-46ae-a279-f282fd9ecf94 drwxr-xr-x. 2 vdsm kvm 4096 May 26 15:51 ba1a624c-6995-4109-b770-7bbd1efc340e drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 dec14b5b-69fe-4c51-abb4-d8296c547a0b drwxr-xr-x. 2 vdsm kvm 4096 Sep 15 2020 e7a17f3b-7567-4c90-b58e-18f34d4686c0 drwxr-xr-x. 2 vdsm kvm 4096 May 20 09:29 ea786bee-b98e-44c0-a44a-a63af63b8c51 drwxr-xr-x. 2 vdsm kvm 4096 Apr 21 2021 ec37f395-3b4b-41df-8f5b-8a22b7e1273b drwxr-xr-x. 2 vdsm kvm 4096 Feb 25 2022 f3e641c7-a2cc-42af-bcd1-b5ecdf437e03 drwxr-xr-x. 2 vdsm kvm 4096 May 18 23:55 f4c221a6-6ce4-44cc-8d32-d673e40aa915 drwxr-xr-x. 2 vdsm kvm 4096 May 20 11:35 f539c098-3f16-410f-938f-3de6ef37b017 2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images/1eb2b682-ef45-409d-9341-abcf61418619: total 104857624 drwxr-xr-x. 2 vdsm kvm 4096 Feb 3 2021 . drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 .. -rwxr-xr-x. 1 vdsm kvm 107374182400 Feb 3 2021 e29b005f-d272-49f8-a162-41f908ca67a4 -rwxr-xr-x. 1 vdsm kvm 300 Feb 3 2021 e29b005f-d272-49f8-a162-41f908ca67a4.meta 2f2da2d7-596f-4a53-8a1b-a301f84b3b74/images/283838fd-43b5-42b9-a332-a7b980613188: total 23223760 drwxr-xr-x. 2 vdsm kvm 4096 Aug 11 2022 . drwxr-xr-x. 23 vdsm kvm 4096 May 20 11:35 .. -rwxr-xr-x. 1 vdsm kvm 23781113856 Aug 11 2022 3273221e-1649-4e4a-816c-2ffec18e51c7 -rwxr-xr-x. 1 vdsm kvm 251 Aug 11 2022 3273221e-1649-4e4a-816c-2ffec18e51c7.meta . . etc So... is there a way to get this back ? I tried "Import Domain" both with the disk manually mounted and unmounted, both failed and I get errors in the vdsm.log like this... 2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0) [storage.StorageServer.MountConnection] Creating directory '/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74' (storageServer:167) 2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74 mode: None (fileUtils:201) 2023-06-15 15:57:55,166+1000 INFO (jsonrpc/0) [storage.Mount] mounting /backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74 at /rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74 (mount:207) 2023-06-15 15:57:55,176+1000 ERROR (jsonrpc/0) [storage.HSM] Could not connect to storageServer (hsm:2374) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2371, in connectStorageServer conObj.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 180, in connect six.reraise(t, v, tb) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 171, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in mount cgroup=cgroup) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File "<string>", line 2, in mount File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'posix', '/backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74', '/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74'] failed with rc=32 out=b'' err=b"mount: /rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74: unknown filesystem type 'posix'.\n" 2023-06-15 15:57:55,176+1000 INFO (jsonrpc/0) [storage.StorageDomainCache] Invalidating storage domain cache (sdc:74) 2023-06-15 15:57:55,176+1000 INFO (jsonrpc/0) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '00000000-0000-0000-0000-000000000000', 'status': 477}]} from=::ffff:172.20.1.160,47702, flow_id=6693c76d-b62b-4e62-9b70-c87f2a199705, task_id=682abcff-3040-46d7-aaa1-f2445f0a6698 (api:54) 2023-06-15 15:57:55,283+1000 INFO (jsonrpc/2) [vdsm.api] START disconnectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'password': '********', 'vfs_type': 'posix', 'port': '', 'iqn': '', 'connection': '/backups/2f2da2d7-596f-4a53-8a1b-a301f84b3b74', 'ipv6_enabled': 'false', 'id': '00000000-0000-0000-0000-000000000000', 'user': '', 'tpgt': '1'}]) from=::ffff:172.20.1.160,47702, flow_id=0d3d4a3e-3523-4436-93b3-e22bff47f082, task_id=11870123-365d-4b40-b481-be5d44098fc2 (api:48) 2023-06-15 15:57:55,284+1000 INFO (jsonrpc/2) [storage.Mount] unmounting /rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74 (mount:215) 2023-06-15 15:57:55,290+1000 ERROR (jsonrpc/2) [storage.HSM] Could not disconnect from storageServer (hsm:2480) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2476, in disconnectStorageServer conObj.disconnect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 202, in disconnect self._mount.umount(True, True) File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 217, in umount umount(self.fs_file, force=force, lazy=lazy, freeloop=freeloop) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File "<string>", line 2, in umount File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.mount.MountError: Command ['/usr/bin/umount', '-f', '-l', '/rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74'] failed with rc=32 out=b'' err=b'umount: /rhev/data-center/mnt/_backups_2f2da2d7-596f-4a53-8a1b-a301f84b3b74: no mount point specified.\n' 2023-06-15 15:57:55,290+1000 INFO (jsonrpc/2) [storage.StorageDomainCache] Refreshing storage domain cache (resize=False) (sdc:80) 2023-06-15 15:57:55,290+1000 INFO (jsonrpc/2) [storage.ISCSI] Scanning iSCSI devices (iscsi:442) 2023-06-15 15:57:55,328+1000 INFO (jsonrpc/2) [storage.ISCSI] Scanning iSCSI devices: 0.04 seconds (utils:390) 2023-06-15 15:57:55,328+1000 INFO (jsonrpc/2) [storage.HBA] Scanning FC devices (hba:60) 2023-06-15 15:57:55,416+1000 INFO (jsonrpc/2) [storage.HBA] Scanning FC devices: 0.09 seconds (utils:390) 2023-06-15 15:57:55,416+1000 INFO (jsonrpc/2) [storage.Multipath] Waiting until multipathd is ready (multipath:112) 2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [storage.Multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:139) 2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [storage.StorageDomainCache] Refreshing storage domain cache: 2.17 seconds (utils:390) 2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [vdsm.api] FINISH disconnectStorageServer return={'statuslist': [{'id': '00000000-0000-0000-0000-000000000000', 'status': 477}]} from=::ffff:172.20.1.160,47702, flow_id=0d3d4a3e-3523-4436-93b3-e22bff47f082, task_id=11870123-365d-4b40-b481-be5d44098fc2 (api:54) 2023-06-15 15:57:57,458+1000 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StoragePool.disconnectStorageServer took more than 1.00 seconds to succeed: 2.18 (__init__:316) any help appreciated thanks Tim
participants (1)
-
Tim Tuck