[ovirt-users] Storage Problem After Host Update (v3.6)

Charles Tassell charles at islandadmin.ca
Wed Jan 4 20:22:36 UTC 2017


Hello,

   One of our admins did a package update on one of our 3.6 hosts 
through the web GUI and then rebooted the host.  Since then, it hasn't 
been able to properly mount storage and thus we can't move any VMs to 
it.  I rebooted it myself this afternoon and it took our whole cluster 
down for about a half hour, which was kind of concerning.

   We use multipath fibre channel for storage and when I do a vgscan or 
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate 
of PV ..." warning messages.  I also get the log entries below 
constantly repeating in vdsm.log.  Any ideas what's going on and how I 
fix it?

CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General 
Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation 
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use 
\\\'--activationmode partial\\\' to 
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04 
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': 
{'message': 'Cannot activate Logical Volumes: (\'General Storage 
Exception: (\\\'5 [] [\\\\\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing 
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  
Use \\\\\\\'--activationmode partial\\\\\\\' to 
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)', 
'code': 570}}
Thread-101::ERROR::2017-01-04 
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error 
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
     self._produceDomain()
   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in 
wrapper
     value = meth(self, *a, **kw)
   File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
     self.domain = sdCache.produce(self.sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
     domain.getRealDomain()
   File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
     return self._cache._realProduce(self._sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
     domain = self._findDomain(sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
     dom = findMethod(sdUUID)
   File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
     return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
   File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
     manifest = self.manifestClass(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
     metadata = selectMetadata(self.sdUUID)
   File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
     if len(mdProvider) > 0:
   File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
     return len(self.keys())
   File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
     return list(self.__iter__())
   File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
     self._dict.__iter__())
   File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
     with self._accessWrapper():
   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
     return self.gen.next()
   File "/usr/share/vdsm/storage/persistentDict.py", line 152, in 
_accessWrapper
     self.refresh()
   File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
     lines = self._metaRW.readlines()
   File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
     lvm.activateLVs(self._vgName, self._lvName)
   File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
     _setLVAvailability(vgName, toActivate, "y")
   File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
     raise error(str(e))

LVM output:
[root at ovirt-01 mapper]# vgchange -a y
   WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used 
from both devices /dev/sdac and /dev/sdk
   Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk 
not /dev/sdac
   Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
   Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev 
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 
65:177
   Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev 
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 
8:145
   Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev 
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 
65:225
   Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev 
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 
8:193
   Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev 
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device 
8:241
   Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev 
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device 
66:17
   Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev 
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device 
65:241
   Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev 
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device 
8:209
   Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev 
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device 
66:49
   Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev 
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device 
65:17
   Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev 
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device 
8:225
   Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev 
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device 
66:1
   Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev 
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device 
66:33
   Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev 
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
  on unknown device 65:1
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master.  Use '--activationmode 
partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3. 
Use '--activationmode partial' to override.
  Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481. 
Use '--activationmode partial' to override.
   Refusing activation of partial LV 
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf. 
Use '--activationmode partial' to override.
   24 logical volume(s) in volume group 
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
   3 logical volume(s) in volume group "centos_ovirt-01" now active
   WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used 
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
   WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used 
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
   9 logical volume(s) in volume group 
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active

[root at ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976 
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172 
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6 
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71 
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55 
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64 
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058 
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c 
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7



More information about the Users mailing list