I can also see... kvmr03:~# lvs | grep 927f423a-6689-4ddb-8fda-b3375c3bbca3 /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/36001405063455cf7cd74c20bc06e9304. Metadata location on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/36001405063455cf7cd74c20bc06e9304 Failed to scan VG from /dev/mapper/36001405063455cf7cd74c20bc06e9304 Seems to me like metadata from that VG has been corrupted. Is there a way to recover? El 2021-09-16 11:19, nicolas(a)devels.es escribió: > The most relevant log snippet I have found is the following. I assume > it cannot scan the Storage Domain, but I'm unsure why, as the storage > domain backend is up and running. > > 021-09-16 11:16:58,884+0100 WARN (monitor/219fa16) [storage.LVM] > Command ['/usr/sbin/lvm', 'vgs', '--config', 'devices { > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 > filter=["a|^/dev/mapper/36001405063455cf7cd74c20bc06e9304$|^/dev/mapper/360014056481868b09dd4d05bee5b4185$|^/dev/mapper/360014057d9d4bc57df046888b8d8b6eb$|^/dev/mapper/360014057e612d2079b649d5b539e5f6a$|^/dev/mapper/360014059b49883b502a4fa9b81add3e4$|^/dev/mapper/36001405acece27e83b547e3a873b19e2$|^/dev/mapper/36001405dc03f6be1b8c42219e8912fbd$|^/dev/mapper/36001405f3ab584afde347d3a8855baf0$|^/dev/mapper/3600c0ff00052a0fe013ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe033ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe1b40c65f01000000$|^/dev/mapper/3600c0ff00052a0fe2294c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2394c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2494c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2594c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2694c75f01000000$|^/dev/mapper/3600c0ff00052a0fee293c75f01000000$|^/dev/mapper/3600c0ff00052a0fee493c75f01000000$|^/dev/mapper/3600c0ff00064835b628d306101000000$|^/dev/mapper/3600c0ff00064835b628d306103000000$|^/dev/mapper/3600c0ff000648 > 35b628d306105000000$|^/dev/mapper/3600c0ff00064835b638d306101000000$|^/dev/mapper/3600c0ff00064835b638d306103000000$|^/dev/mapper/3600c0ff00064835b638d306105000000$|^/dev/mapper/3600c0ff00064835b638d306107000000$|^/dev/mapper/3600c0ff00064835b638d306109000000$|^/dev/mapper/3600c0ff00064835b638d30610b000000$|^/dev/mapper/3600c0ff00064835cb98f306101000000$|^/dev/mapper/3600c0ff00064835cb98f306103000000$|^/dev/mapper/3600c0ff00064835cb98f306105000000$|^/dev/mapper/3600c0ff00064835cb98f306107000000$|^/dev/mapper/3600c0ff00064835cb98f306109000000$|^/dev/mapper/3600c0ff00064835cba8f306101000000$|^/dev/mapper/3600c0ff00064835cba8f306103000000$|^/dev/mapper/3600c0ff00064835cba8f306105000000$|^/dev/mapper/3600c0ff00064835cba8f306107000000$|^/dev/mapper/3634b35410019574796dcb0e300000007$|^/dev/mapper/3634b35410019574796dcdffc00000008$|^/dev/mapper/3634b354100195747999c2dc500000003$|^/dev/mapper/3634b354100195747999c3c4a00000004$|^/dev/mapper/3634b3541001957479c2b9c6400000001$|^/dev/mapper/3634 > b3541001957479c2baba500000002$|", "r|.*|"] } global { locking_type=4 > prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { > retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', > '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', > 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', > '--select', 'vg_name = 219fa16f-13c9-44e4-a07d-a40c0a7fe206'] > succeeded with warnings: [' > /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at > offset 2198927383040', " Couldn't read volume group metadata from > /dev/mapper/36001405063455cf7cd74c20bc06e9304.", ' Metadata location > on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 has > invalid summary for VG.', ' Failed to read metadata summary from > /dev/mapper/36001405063455cf7cd74c20bc06e9304', ' Failed to scan VG > from /dev/mapper/36001405063455cf7cd74c20bc06e9304'] (lvm:462) > 2021-09-16 11:16:58,909+0100 ERROR (monitor/219fa16) [storage.Monitor] > Setting up monitor for 219fa16f-13c9-44e4-a07d-a40c0a7fe206 failed > (monitor:330) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 327, in _setupLoop > self._setupMonitor() > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 349, in _setupMonitor > self._produceDomain() > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 159, in > wrapper > value = meth(self, *a, **kw) > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 367, in _produceDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 110, in produce > domain.getRealDomain() > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 134, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 151, in _findDomain > return findMethod(sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 176, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) > StorageDomainDoesNotExist: Storage domain does not exist: > (u'219fa16f-13c9-44e4-a07d-a40c0a7fe206',) > > > El 2021-09-16 08:28, Vojtech Juranek escribió: >> On Wednesday, 15 September 2021 14:52:27 CEST nicolas(a)devels.es wrote: >>> Hi, >>> >>> We're running oVirt 4.3.8 and we recently had a oVirt crash after >>> moving >>> too much disks between storage domains. >>> >>> Concretely, one of the Storage Domains reports status "Unknown", >>> "Total/Free/Guaranteed free spaces" are "[N/A]". >>> >>> After trying to activate it in the Domain Center we see messages like >>> these from all of the hosts: >>> >>> VDSM hostX command GetVGInfoVDS failed: Volume Group does not >>> exist: >>> (u'vg_uuid: Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp',) >>> >>> I tried putting the Storage Domain in maintenance and it fails with >>> messages like: >>> >>> Storage Domain iaasb13 (Data Center KVMRojo) was deactivated by >>> system because it's not visible by any of the hosts. >>> Failed to update OVF disks 8661acd1-d1c4-44a0-a4d4-ddee834844e9, >>> OVF >>> data isn't updated on those OVF stores (Data Center KVMRojo, Storage >>> Domain iaasb13). >>> Failed to update VMs/Templates OVF data for Storage Domain >>> iaasb13 >>> in Data Center KVMRojo. >>> >>> I'm sure the storage domain backend is up and running, and the LUN >>> being >>> exported. >>> >>> Any hints how can I debug this problem and restore the Storage >>> Domain? >> >> I'd suggest to ssh to any of the host from given data center and >> investigate >> manually, if the device is visible to the host (e.g. using lsblk) and >> eventually check /var/log/messages to determine where the problem >> could be. >> >> >>> Thanks. >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ List >>> Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UNXKR7HRCRDTT >>> WLEYO6FFM4WOLD6YATW/ >> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y6BBLM6M5Z5... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5ZXFLCBM3UJ... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NCZEXH7NMTN...

Hi Roman, Unfortunately, step 1 returns nothing: kvmr03:~# vgcfgrestore --list Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp No archives found in /etc/lvm/archive I tried several hosts and noone has a copy. Any other way to get a backup of the VG? El 2021-09-16 13:42, Roman Bednar escribió: > Hi Nicolas, > > You can try to recover VG metadata from a backup or archive which lvm > automatically creates by default. > > 1) To list all available backups for given VG: > > #vgcfgrestore --list Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp > > Select the latest one which sounds right, something with a description > along the lines of "Created *before* lvremove". > You might want to select something older than the latest as lvm does a > backup also *after* running some command. > > 2) Find UUID of your broken PV (filter might not be needed, depends on > your local conf): > > #pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' > /dev/mapper/36001405063455cf7cd74c20bc06e9304 > > 3) Create a new PV on a different partition or disk (/dev/sdX) using > the UUID found in previous step and restorefile option: > > #pvcreate --uuid <ID_OF_BROKEN_PV> --restorefile <PATH_TO_BACKUP_FILE> > <EMPTY_DISK> > > 4) Try to display the VG: > > # vgdisplay Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp > > -Roman > > On Thu, Sep 16, 2021 at 1:47 PM <nicolas(a)devels.es&gt; wrote: > >> I can also see... >> >> kvmr03:~# lvs | grep 927f423a-6689-4ddb-8fda-b3375c3bbca3 >> /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at >> offset 2198927383040 >> Couldn't read volume group metadata from >> /dev/mapper/36001405063455cf7cd74c20bc06e9304. >> Metadata location on >> /dev/mapper/36001405063455cf7cd74c20bc06e9304 at >> 2198927383040 has invalid summary for VG. >> Failed to read metadata summary from >> /dev/mapper/36001405063455cf7cd74c20bc06e9304 >> Failed to scan VG from >> /dev/mapper/36001405063455cf7cd74c20bc06e9304 >> >> Seems to me like metadata from that VG has been corrupted. Is there >> a >> way to recover? >> >> El 2021-09-16 11:19, nicolas(a)devels.es escribió: >>> The most relevant log snippet I have found is the following. I >> assume >>> it cannot scan the Storage Domain, but I'm unsure why, as the >> storage >>> domain backend is up and running. >>> >>> 021-09-16 11:16:58,884+0100 WARN (monitor/219fa16) [storage.LVM] >>> Command ['/usr/sbin/lvm', 'vgs', '--config', 'devices { >>> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 >>> write_cache_state=0 disable_after_error_count=3 >>> >> > filter=["a|^/dev/mapper/36001405063455cf7cd74c20bc06e9304$|^/dev/mapper/360014056481868b09dd4d05bee5b4185$|^/dev/mapper/360014057d9d4bc57df046888b8d8b6eb$|^/dev/mapper/360014057e612d2079b649d5b539e5f6a$|^/dev/mapper/360014059b49883b502a4fa9b81add3e4$|^/dev/mapper/36001405acece27e83b547e3a873b19e2$|^/dev/mapper/36001405dc03f6be1b8c42219e8912fbd$|^/dev/mapper/36001405f3ab584afde347d3a8855baf0$|^/dev/mapper/3600c0ff00052a0fe013ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe033ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe1b40c65f01000000$|^/dev/mapper/3600c0ff00052a0fe2294c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2394c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2494c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2594c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2694c75f01000000$|^/dev/mapper/3600c0ff00052a0fee293c75f01000000$|^/dev/mapper/3600c0ff00052a0fee493c75f01000000$|^/dev/mapper/3600c0ff00064835b628d306101000000$|^/dev/mapper/3600c0ff00064835b628d306103000000$|^/dev/mapper/3600c0ff000648 >>> >> > 35b628d306105000000$|^/dev/mapper/3600c0ff00064835b638d306101000000$|^/dev/mapper/3600c0ff00064835b638d306103000000$|^/dev/mapper/3600c0ff00064835b638d306105000000$|^/dev/mapper/3600c0ff00064835b638d306107000000$|^/dev/mapper/3600c0ff00064835b638d306109000000$|^/dev/mapper/3600c0ff00064835b638d30610b000000$|^/dev/mapper/3600c0ff00064835cb98f306101000000$|^/dev/mapper/3600c0ff00064835cb98f306103000000$|^/dev/mapper/3600c0ff00064835cb98f306105000000$|^/dev/mapper/3600c0ff00064835cb98f306107000000$|^/dev/mapper/3600c0ff00064835cb98f306109000000$|^/dev/mapper/3600c0ff00064835cba8f306101000000$|^/dev/mapper/3600c0ff00064835cba8f306103000000$|^/dev/mapper/3600c0ff00064835cba8f306105000000$|^/dev/mapper/3600c0ff00064835cba8f306107000000$|^/dev/mapper/3634b35410019574796dcb0e300000007$|^/dev/mapper/3634b35410019574796dcdffc00000008$|^/dev/mapper/3634b354100195747999c2dc500000003$|^/dev/mapper/3634b354100195747999c3c4a00000004$|^/dev/mapper/3634b3541001957479c2b9c6400000001$|^/dev/mapper/3634 >>> b3541001957479c2baba500000002$|", "r|.*|"] } global { >> locking_type=4 >>> prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup >> { >>> retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', >>> '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', >>> >> > 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', >>> '--select', 'vg_name = 219fa16f-13c9-44e4-a07d-a40c0a7fe206'] >>> succeeded with warnings: [' >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at >>> offset 2198927383040', " Couldn't read volume group metadata from >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304.", ' Metadata >> location >>> on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 >> has >>> invalid summary for VG.', ' Failed to read metadata summary from >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304', ' Failed to scan >> VG >>> from /dev/mapper/36001405063455cf7cd74c20bc06e9304'] (lvm:462) >>> 2021-09-16 11:16:58,909+0100 ERROR (monitor/219fa16) >> [storage.Monitor] >>> Setting up monitor for 219fa16f-13c9-44e4-a07d-a40c0a7fe206 failed >>> (monitor:330) >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>> line 327, in _setupLoop >>> self._setupMonitor() >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>> line 349, in _setupMonitor >>> self._produceDomain() >>> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 159, >> in >>> wrapper >>> value = meth(self, *a, **kw) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>> line 367, in _produceDomain >>> self.domain = sdCache.produce(self.sdUUID) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >> line >>> 110, in produce >>> domain.getRealDomain() >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >> line >>> 51, in getRealDomain >>> return self._cache._realProduce(self._sdUUID) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >> line >>> 134, in _realProduce >>> domain = self._findDomain(sdUUID) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >> line >>> 151, in _findDomain >>> return findMethod(sdUUID) >>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >> line >>> 176, in _findUnfetchedDomain >>> raise se.StorageDomainDoesNotExist(sdUUID) >>> StorageDomainDoesNotExist: Storage domain does not exist: >>> (u'219fa16f-13c9-44e4-a07d-a40c0a7fe206',) >>> >>> >>> El 2021-09-16 08:28, Vojtech Juranek escribió: >>>> On Wednesday, 15 September 2021 14:52:27 CEST nicolas(a)devels.es >> wrote: >>>>> Hi, >>>>> >>>>> We're running oVirt 4.3.8 and we recently had a oVirt crash >> after >>>>> moving >>>>> too much disks between storage domains. >>>>> >>>>> Concretely, one of the Storage Domains reports status "Unknown", >>>>> "Total/Free/Guaranteed free spaces" are "[N/A]". >>>>> >>>>> After trying to activate it in the Domain Center we see messages >> like >>>>> these from all of the hosts: >>>>> >>>>> VDSM hostX command GetVGInfoVDS failed: Volume Group does >> not >>>>> exist: >>>>> (u'vg_uuid: Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp',) >>>>> >>>>> I tried putting the Storage Domain in maintenance and it fails >> with >>>>> messages like: >>>>> >>>>> Storage Domain iaasb13 (Data Center KVMRojo) was >> deactivated by >>>>> system because it's not visible by any of the hosts. >>>>> Failed to update OVF disks >> 8661acd1-d1c4-44a0-a4d4-ddee834844e9, >>>>> OVF >>>>> data isn't updated on those OVF stores (Data Center KVMRojo, >> Storage >>>>> Domain iaasb13). >>>>> Failed to update VMs/Templates OVF data for Storage Domain >>>>> iaasb13 >>>>> in Data Center KVMRojo. >>>>> >>>>> I'm sure the storage domain backend is up and running, and the >> LUN >>>>> being >>>>> exported. >>>>> >>>>> Any hints how can I debug this problem and restore the Storage >>>>> Domain? >>>> >>>> I'd suggest to ssh to any of the host from given data center and >>>> investigate >>>> manually, if the device is visible to the host (e.g. using lsblk) >> and >>>> eventually check /var/log/messages to determine where the problem >> >>>> could be. >>>> >>>> >>>>> Thanks. >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ List >> >>>>> Archives: >>>>> >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UNXKR7HRCRDTT >>>>> WLEYO6FFM4WOLD6YATW/ >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y6BBLM6M5Z5... >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5ZXFLCBM3UJ... >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NCZEXH7NMTN...

Hi Nicolas, You can try to recover VG metadata from a backup or archive which lvm automatically creates by default. 1) To list all available backups for given VG: #vgcfgrestore --list Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp Select the latest one which sounds right, something with a description along the lines of "Created *before* lvremove". You might want to select something older than the latest as lvm does a backup also *after* running some command.

2) Find UUID of your broken PV (filter might not be needed, depends on your local conf): #pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' /dev/mapper/36001405063455cf7cd74c20bc06e9304

3) Create a new PV on a different partition or disk (/dev/sdX) using the UUID found in previous step and restorefile option: #pvcreate --uuid <ID_OF_BROKEN_PV> --restorefile <PATH_TO_BACKUP_FILE> <EMPTY_DISK>

4) Try to display the VG: # vgdisplay Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp -Roman On Thu, Sep 16, 2021 at 1:47 PM <nicolas(a)devels.es&gt; wrote: > I can also see... > > kvmr03:~# lvs | grep 927f423a-6689-4ddb-8fda-b3375c3bbca3 > /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at > offset 2198927383040 > Couldn't read volume group metadata from > /dev/mapper/36001405063455cf7cd74c20bc06e9304. > Metadata location on > /dev/mapper/36001405063455cf7cd74c20bc06e9304 at > 2198927383040 has invalid summary for VG. > Failed to read metadata summary from > /dev/mapper/36001405063455cf7cd74c20bc06e9304 > Failed to scan VG from > /dev/mapper/36001405063455cf7cd74c20bc06e9304 > > Seems to me like metadata from that VG has been corrupted. Is there > a > way to recover? > > El 2021-09-16 11:19, nicolas(a)devels.es escribió: >> The most relevant log snippet I have found is the following. I > assume >> it cannot scan the Storage Domain, but I'm unsure why, as the > storage >> domain backend is up and running. >> >> 021-09-16 11:16:58,884+0100 WARN (monitor/219fa16) [storage.LVM] >> Command ['/usr/sbin/lvm', 'vgs', '--config', 'devices { >> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 >> write_cache_state=0 disable_after_error_count=3 >> >

>> >

So, I've made several attempts to restore the metadata. In my last e-mail I said in step 2 that the PV ID is: 36001405063455cf7cd74c20bc06e9304, which is incorrect. I'm trying to find out the PV UUID running "pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' /dev/mapper/36001405063455cf7cd74c20bc06e9304". However, it shows no PV UUID. All I get from the command output is: # pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' /dev/mapper/36001405063455cf7cd74c20bc06e9304 /dev/mapper/360014057b367e3a53b44ab392ae0f25f: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/360014057b367e3a53b44ab392ae0f25f. Metadata location on /dev/mapper/360014057b367e3a53b44ab392ae0f25f at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Failed to scan VG from /dev/mapper/360014057b367e3a53b44ab392ae0f25f /dev/mapper/360014057b367e3a53b44ab392ae0f25f: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/360014057b367e3a53b44ab392ae0f25f. Metadata location on /dev/mapper/360014057b367e3a53b44ab392ae0f25f at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Failed to scan VG from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Failed to find device "/dev/mapper/36001405063455cf7cd74c20bc06e9304". I tried running a bare "vgcfgrestore 219fa16f-13c9-44e4-a07d-a40c0a7fe206" command, which returned: # vgcfgrestore 219fa16f-13c9-44e4-a07d-a40c0a7fe206 /dev/mapper/360014057b367e3a53b44ab392ae0f25f: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/360014057b367e3a53b44ab392ae0f25f. Metadata location on /dev/mapper/360014057b367e3a53b44ab392ae0f25f at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Failed to scan VG from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Couldn't find device with uuid Q3xkre-25cg-L3Do-aeMD-iLem-wOHh-fb8fzb. Cannot restore Volume Group 219fa16f-13c9-44e4-a07d-a40c0a7fe206 with 1 PVs marked as missing. Restore failed. Seems that the PV is missing, however, I assume the PV UUID (from output above) is Q3xkre-25cg-L3Do-aeMD-iLem-wOHh-fb8fzb. So I tried running: # pvcreate --uuid Q3xkre-25cg-L3Do-aeMD-iLem-wOHh-fb8fzb --restore /etc/lvm/archive/219fa16f-13c9-44e4-a07d-a40c0a7fe206_00200-1084769199.vg /dev/sdb1 Couldn't find device with uuid Q3xkre-25cg-L3Do-aeMD-iLem-wOHh-fb8fzb. /dev/mapper/360014057b367e3a53b44ab392ae0f25f: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/360014057b367e3a53b44ab392ae0f25f. Metadata location on /dev/mapper/360014057b367e3a53b44ab392ae0f25f at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Failed to scan VG from /dev/mapper/360014057b367e3a53b44ab392ae0f25f Device /dev/sdb1 excluded by a filter. Either the PV UUID is not the one I specified, or the system can't find it (or both). El 2021-09-20 09:21, nicolas(a)devels.es escribió: > Hi Roman and Nir, > > El 2021-09-16 13:42, Roman Bednar escribió: >> Hi Nicolas, >> >> You can try to recover VG metadata from a backup or archive which lvm >> automatically creates by default. >> >> 1) To list all available backups for given VG: >> >> #vgcfgrestore --list Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp >> >> Select the latest one which sounds right, something with a description >> along the lines of "Created *before* lvremove". >> You might want to select something older than the latest as lvm does a >> backup also *after* running some command. >> > > You were right. There actually *are* LV backups, I was specifying an > incorrect ID. > > So the correct command would return: > > # vgcfgrestore --list 219fa16f-13c9-44e4-a07d-a40c0a7fe206 > [...] > > File: /etc/lvm/archive/ 219fa16f-13c9-44e4-a07d-a40c0a7fe206_00202-1152107223.vg > VG name: 219fa16f-13c9-44e4-a07d-a40c0a7fe206 > Description: Created *before* executing 'vgs --noheading --nosuffix > --units b -o +vg_uuid,vg_extent_size' > Backup Time: Sat Sep 11 03:41:25 2021 > [...] > > That one seems ok. > >> 2) Find UUID of your broken PV (filter might not be needed, depends on >> your local conf): >> >> #pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' >> /dev/mapper/36001405063455cf7cd74c20bc06e9304 >> > > As I understand it, the PV won't be listed in the 'pvs' command, this > is just a matter of finding the associated VG. The command above won't > list a PV associated to the VG in step 1, it just complains the PV > cannot be read. > > # pvs -o pv_name,pv_uuid --config='devices/filter = ["a|.*|"]' > /dev/mapper/36001405063455cf7cd74c20bc06e9304 > /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at > offset 2198927383040 > Couldn't read volume group metadata from > /dev/mapper/36001405063455cf7cd74c20bc06e9304. > Metadata location on /dev/mapper/36001405063455cf7cd74c20bc06e9304 > at 2198927383040 has invalid summary for VG. > Failed to read metadata summary from > /dev/mapper/36001405063455cf7cd74c20bc06e9304 > Failed to scan VG from /dev/mapper/36001405063455cf7cd74c20bc06e9304 > No physical volume label read from > /dev/mapper/36001405063455cf7cd74c20bc06e9304. > > So, associated PV ID is: 36001405063455cf7cd74c20bc06e9304 > >> 3) Create a new PV on a different partition or disk (/dev/sdX) using >> the UUID found in previous step and restorefile option: >> >> #pvcreate --uuid <ID_OF_BROKEN_PV> --restorefile <PATH_TO_BACKUP_FILE> >> <EMPTY_DISK> >> > > I have a question here. As I understand it, pvcreate will restore the > correct metadata on <EMPTY_DISK>. Then how do you restore that > metadata on the broken storage domain, so other hosts can see the > right information as well? Or is this just a step to recover data on > <EMPTY_DISK> and then reattach the disks on the affected VMs? > > Thanks so much. > >> 4) Try to display the VG: >> >> # vgdisplay Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp >> >> -Roman >> >> On Thu, Sep 16, 2021 at 1:47 PM <nicolas(a)devels.es&gt; wrote: >> >>> I can also see... >>> >>> kvmr03:~# lvs | grep 927f423a-6689-4ddb-8fda-b3375c3bbca3 >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at >>> offset 2198927383040 >>> Couldn't read volume group metadata from >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304. >>> Metadata location on >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304 at >>> 2198927383040 has invalid summary for VG. >>> Failed to read metadata summary from >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304 >>> Failed to scan VG from >>> /dev/mapper/36001405063455cf7cd74c20bc06e9304 >>> >>> Seems to me like metadata from that VG has been corrupted. Is there >>> a >>> way to recover? >>> >>> El 2021-09-16 11:19, nicolas(a)devels.es escribió: >>>> The most relevant log snippet I have found is the following. I >>> assume >>>> it cannot scan the Storage Domain, but I'm unsure why, as the >>> storage >>>> domain backend is up and running. >>>> >>>> 021-09-16 11:16:58,884+0100 WARN (monitor/219fa16) [storage.LVM] >>>> Command ['/usr/sbin/lvm', 'vgs', '--config', 'devices { >>>> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 >>>> write_cache_state=0 disable_after_error_count=3 >>>> >>> >> > filter=["a|^/dev/mapper/36001405063455cf7cd74c20bc06e9304$|^/dev/mapper/360014056481868b09dd4d05bee5b4185$|^/dev/mapper/360014057d9d4bc57df046888b8d8b6eb$|^/dev/mapper/360014057e612d2079b649d5b539e5f6a$|^/dev/mapper/360014059b49883b502a4fa9b81add3e4$|^/dev/mapper/36001405acece27e83b547e3a873b19e2$|^/dev/mapper/36001405dc03f6be1b8c42219e8912fbd$|^/dev/mapper/36001405f3ab584afde347d3a8855baf0$|^/dev/mapper/3600c0ff00052a0fe013ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe033ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe1b40c65f01000000$|^/dev/mapper/3600c0ff00052a0fe2294c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2394c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2494c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2594c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2694c75f01000000$|^/dev/mapper/3600c0ff00052a0fee293c75f01000000$|^/dev/mapper/3600c0ff00052a0fee493c75f01000000$|^/dev/mapper/3600c0ff00064835b628d306101000000$|^/dev/mapper/3600c0ff00064835b628d306103000000$|^/dev/mapper/3600c0ff000648 >>>> >>> >> > 35b628d306105000000$|^/dev/mapper/3600c0ff00064835b638d306101000000$|^/dev/mapper/3600c0ff00064835b638d306103000000$|^/dev/mapper/3600c0ff00064835b638d306105000000$|^/dev/mapper/3600c0ff00064835b638d306107000000$|^/dev/mapper/3600c0ff00064835b638d306109000000$|^/dev/mapper/3600c0ff00064835b638d30610b000000$|^/dev/mapper/3600c0ff00064835cb98f306101000000$|^/dev/mapper/3600c0ff00064835cb98f306103000000$|^/dev/mapper/3600c0ff00064835cb98f306105000000$|^/dev/mapper/3600c0ff00064835cb98f306107000000$|^/dev/mapper/3600c0ff00064835cb98f306109000000$|^/dev/mapper/3600c0ff00064835cba8f306101000000$|^/dev/mapper/3600c0ff00064835cba8f306103000000$|^/dev/mapper/3600c0ff00064835cba8f306105000000$|^/dev/mapper/3600c0ff00064835cba8f306107000000$|^/dev/mapper/3634b35410019574796dcb0e300000007$|^/dev/mapper/3634b35410019574796dcdffc00000008$|^/dev/mapper/3634b354100195747999c2dc500000003$|^/dev/mapper/3634b354100195747999c3c4a00000004$|^/dev/mapper/3634b3541001957479c2b9c6400000001$|^/dev/mapper/3634 >>>> b3541001957479c2baba500000002$|", "r|.*|"] } global { >>> locking_type=4 >>>> prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup >>> { >>>> retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', >>>> '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', >>>> >>> >> 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', >>>> '--select', 'vg_name = 219fa16f-13c9-44e4-a07d-a40c0a7fe206'] >>>> succeeded with warnings: [' >>>> /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at >>>> offset 2198927383040', " Couldn't read volume group metadata from >>>> /dev/mapper/36001405063455cf7cd74c20bc06e9304.", ' Metadata >>> location >>>> on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 >>> has >>>> invalid summary for VG.', ' Failed to read metadata summary from >>>> /dev/mapper/36001405063455cf7cd74c20bc06e9304', ' Failed to scan >>> VG >>>> from /dev/mapper/36001405063455cf7cd74c20bc06e9304'] (lvm:462) >>>> 2021-09-16 11:16:58,909+0100 ERROR (monitor/219fa16) >>> [storage.Monitor] >>>> Setting up monitor for 219fa16f-13c9-44e4-a07d-a40c0a7fe206 failed >>>> (monitor:330) >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>>> line 327, in _setupLoop >>>> self._setupMonitor() >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>>> line 349, in _setupMonitor >>>> self._produceDomain() >>>> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 159, >>> in >>>> wrapper >>>> value = meth(self, *a, **kw) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", >>>> line 367, in _produceDomain >>>> self.domain = sdCache.produce(self.sdUUID) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >>> line >>>> 110, in produce >>>> domain.getRealDomain() >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >>> line >>>> 51, in getRealDomain >>>> return self._cache._realProduce(self._sdUUID) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >>> line >>>> 134, in _realProduce >>>> domain = self._findDomain(sdUUID) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >>> line >>>> 151, in _findDomain >>>> return findMethod(sdUUID) >>>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", >>> line >>>> 176, in _findUnfetchedDomain >>>> raise se.StorageDomainDoesNotExist(sdUUID) >>>> StorageDomainDoesNotExist: Storage domain does not exist: >>>> (u'219fa16f-13c9-44e4-a07d-a40c0a7fe206',) >>>> >>>> >>>> El 2021-09-16 08:28, Vojtech Juranek escribió: >>>>> On Wednesday, 15 September 2021 14:52:27 CEST nicolas(a)devels.es >>> wrote: >>>>>> Hi, >>>>>> >>>>>> We're running oVirt 4.3.8 and we recently had a oVirt crash >>> after >>>>>> moving >>>>>> too much disks between storage domains. >>>>>> >>>>>> Concretely, one of the Storage Domains reports status "Unknown", >>>>>> "Total/Free/Guaranteed free spaces" are "[N/A]". >>>>>> >>>>>> After trying to activate it in the Domain Center we see messages >>> like >>>>>> these from all of the hosts: >>>>>> >>>>>> VDSM hostX command GetVGInfoVDS failed: Volume Group does >>> not >>>>>> exist: >>>>>> (u'vg_uuid: Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp',) >>>>>> >>>>>> I tried putting the Storage Domain in maintenance and it fails >>> with >>>>>> messages like: >>>>>> >>>>>> Storage Domain iaasb13 (Data Center KVMRojo) was >>> deactivated by >>>>>> system because it's not visible by any of the hosts. >>>>>> Failed to update OVF disks >>> 8661acd1-d1c4-44a0-a4d4-ddee834844e9, >>>>>> OVF >>>>>> data isn't updated on those OVF stores (Data Center KVMRojo, >>> Storage >>>>>> Domain iaasb13). >>>>>> Failed to update VMs/Templates OVF data for Storage Domain >>>>>> iaasb13 >>>>>> in Data Center KVMRojo. >>>>>> >>>>>> I'm sure the storage domain backend is up and running, and the >>> LUN >>>>>> being >>>>>> exported. >>>>>> >>>>>> Any hints how can I debug this problem and restore the Storage >>>>>> Domain? >>>>> >>>>> I'd suggest to ssh to any of the host from given data center and >>>>> investigate >>>>> manually, if the device is visible to the host (e.g. using lsblk) >>> and >>>>> eventually check /var/log/messages to determine where the problem >>> >>>>> could be. >>>>> >>>>> >>>>>> Thanks. >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ List >>> >>>>>> Archives: >>>>>> >>> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UNXKR7HRCRDTT >>>>>> WLEYO6FFM4WOLD6YATW/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list -- users(a)ovirt.org >>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> >>> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y6BBLM6M5Z5... >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> >>> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/5ZXFLCBM3UJ... >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/NCZEXH7NMTN...