[ovirt-users] Storage Problem After Host Update (v3.6)
Charles Tassell
charles at islandadmin.ca
Wed Jan 4 20:22:36 UTC 2017
Hello,
One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host. Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it. I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.
We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages. I also get the log entries below
constantly repeating in vdsm.log. Any ideas what's going on and how I
fix it?
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\\\\\'--activationmode partial\\\\\\\' to
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
value = meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
manifest = self.manifestClass(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata = selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
raise error(str(e))
LVM output:
[root at ovirt-01 mapper]# vgchange -a y
WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
from both devices /dev/sdac and /dev/sdk
Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
not /dev/sdac
Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
65:177
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
8:145
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
65:225
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
8:193
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
8:241
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
66:17
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
65:241
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
8:209
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
66:49
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
65:17
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
8:225
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
66:1
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
66:33
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
on unknown device 65:1
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
Use '--activationmode partial' to override.
24 logical volume(s) in volume group
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
3 logical volume(s) in volume group "centos_ovirt-01" now active
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
9 logical volume(s) in volume group
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
[root at ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
More information about the Users
mailing list