[ovirt-users] Re: Cannot activate a Storage Domain after an oVirt crash

I can also see... kvmr03:~# lvs | grep 927f423a-6689-4ddb-8fda-b3375c3bbca3 /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at offset 2198927383040 Couldn't read volume group metadata from /dev/mapper/36001405063455cf7cd74c20bc06e9304. Metadata location on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 has invalid summary for VG. Failed to read metadata summary from /dev/mapper/36001405063455cf7cd74c20bc06e9304 Failed to scan VG from /dev/mapper/36001405063455cf7cd74c20bc06e9304 Seems to me like metadata from that VG has been corrupted. Is there a way to recover? El 2021-09-16 11:19, nicolas(a)devels.es escribió: filter=["a|^/dev/mapper/36001405063455cf7cd74c20bc06e9304$|^/dev/mapper/360014056481868b09dd4d05bee5b4185$|^/dev/mapper/360014057d9d4bc57df046888b8d8b6eb$|^/dev/mapper/360014057e612d2079b649d5b539e5f6a$|^/dev/mapper/360014059b49883b502a4fa9b81add3e4$|^/dev/mapper/36001405acece27e83b547e3a873b19e2$|^/dev/mapper/36001405dc03f6be1b8c42219e8912fbd$|^/dev/mapper/36001405f3ab584afde347d3a8855baf0$|^/dev/mapper/3600c0ff00052a0fe013ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe033ec65f01000000$|^/dev/mapper/3600c0ff00052a0fe1b40c65f01000000$|^/dev/mapper/3600c0ff00052a0fe2294c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2394c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2494c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2594c75f01000000$|^/dev/mapper/3600c0ff00052a0fe2694c75f01000000$|^/dev/mapper/3600c0ff00052a0fee293c75f01000000$|^/dev/mapper/3600c0ff00052a0fee493c75f01000000$|^/dev/mapper/3600c0ff00064835b628d306101000000$|^/dev/mapper/3600c0ff00064835b628d306103000000$|^/dev/mapper/3600c0ff000648 35b628d306105000000$|^/dev/mapper/3600c0ff00064835b638d306101000000$|^/dev/mapper/3600c0ff00064835b638d306103000000$|^/dev/mapper/3600c0ff00064835b638d306105000000$|^/dev/mapper/3600c0ff00064835b638d306107000000$|^/dev/mapper/3600c0ff00064835b638d306109000000$|^/dev/mapper/3600c0ff00064835b638d30610b000000$|^/dev/mapper/3600c0ff00064835cb98f306101000000$|^/dev/mapper/3600c0ff00064835cb98f306103000000$|^/dev/mapper/3600c0ff00064835cb98f306105000000$|^/dev/mapper/3600c0ff00064835cb98f306107000000$|^/dev/mapper/3600c0ff00064835cb98f306109000000$|^/dev/mapper/3600c0ff00064835cba8f306101000000$|^/dev/mapper/3600c0ff00064835cba8f306103000000$|^/dev/mapper/3600c0ff00064835cba8f306105000000$|^/dev/mapper/3600c0ff00064835cba8f306107000000$|^/dev/mapper/3634b35410019574796dcb0e300000007$|^/dev/mapper/3634b35410019574796dcdffc00000008$|^/dev/mapper/3634b354100195747999c2dc500000003$|^/dev/mapper/3634b354100195747999c3c4a00000004$|^/dev/mapper/3634b3541001957479c2b9c6400000001$|^/dev/mapper/3634 > b3541001957479c2baba500000002$|", "r|.*|"] } global { locking_type=4 > prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { > retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', > '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', > 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', > '--select', 'vg_name = 219fa16f-13c9-44e4-a07d-a40c0a7fe206'] > succeeded with warnings: [' > /dev/mapper/36001405063455cf7cd74c20bc06e9304: Checksum error at > offset 2198927383040', " Couldn't read volume group metadata from > /dev/mapper/36001405063455cf7cd74c20bc06e9304.", ' Metadata location > on /dev/mapper/36001405063455cf7cd74c20bc06e9304 at 2198927383040 has > invalid summary for VG.', ' Failed to read metadata summary from > /dev/mapper/36001405063455cf7cd74c20bc06e9304', ' Failed to scan VG > from /dev/mapper/36001405063455cf7cd74c20bc06e9304'] (lvm:462) > 2021-09-16 11:16:58,909+0100 ERROR (monitor/219fa16) [storage.Monitor] > Setting up monitor for 219fa16f-13c9-44e4-a07d-a40c0a7fe206 failed > (monitor:330) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 327, in _setupLoop > self._setupMonitor() > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 349, in _setupMonitor > self._produceDomain() > File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 159, in > wrapper > value = meth(self, *a, **kw) > File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", > line 367, in _produceDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 110, in produce > domain.getRealDomain() > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 134, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 151, in _findDomain > return findMethod(sdUUID) > File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line > 176, in _findUnfetchedDomain > raise se.StorageDomainDoesNotExist(sdUUID) > StorageDomainDoesNotExist: Storage domain does not exist: > (u'219fa16f-13c9-44e4-a07d-a40c0a7fe206',) > > > El 2021-09-16 08:28, Vojtech Juranek escribió: >> On Wednesday, 15 September 2021 14:52:27 CEST nicolas(a)devels.es wrote: >>> Hi, >>> >>> We're running oVirt 4.3.8 and we recently had a oVirt crash after >>> moving >>> too much disks between storage domains. >>> >>> Concretely, one of the Storage Domains reports status "Unknown", >>> "Total/Free/Guaranteed free spaces" are "[N/A]". >>> >>> After trying to activate it in the Domain Center we see messages like >>> these from all of the hosts: >>> >>> VDSM hostX command GetVGInfoVDS failed: Volume Group does not >>> exist: >>> (u'vg_uuid: Usi3y8-S4eq-EXtl-FA58-MA3K-b4vE-4d9SCp',) >>> >>> I tried putting the Storage Domain in maintenance and it fails with >>> messages like: >>> >>> Storage Domain iaasb13 (Data Center KVMRojo) was deactivated by >>> system because it's not visible by any of the hosts. >>> Failed to update OVF disks 8661acd1-d1c4-44a0-a4d4-ddee834844e9, >>> OVF >>> data isn't updated on those OVF stores (Data Center KVMRojo, Storage >>> Domain iaasb13). >>> Failed to update VMs/Templates OVF data for Storage Domain >>> iaasb13 >>> in Data Center KVMRojo. >>> >>> I'm sure the storage domain backend is up and running, and the LUN >>> being >>> exported. >>> >>> Any hints how can I debug this problem and restore the Storage >>> Domain? >> >> I'd suggest to ssh to any of the host from given data center and >> investigate >> manually, if the device is visible to the host (e.g. using lsblk) and >> eventually check /var/log/messages to determine where the problem >> could be. >> >> >>> Thanks. >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ List >>> Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UNXKR7HRCRDTT >>> WLEYO6FFM4WOLD6YATW/ >> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y6BBLM6M5Z5... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5ZXFLCBM3UJ...