12:22 p.m.
Hello,
One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host. Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it. I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.
We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages. I also get the log entries below
constantly repeating in vdsm.log. Any ideas what's going on and how I
fix it?
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\\\\\'--activationmode partial\\\\\\\' to
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
value = meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
manifest = self.manifestClass(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata = selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
raise error(str(e))
LVM output:
[root@ovirt-01 mapper]# vgchange -a y
WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
from both devices /dev/sdac and /dev/sdk
Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
not /dev/sdac
Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
65:177
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
8:145
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
65:225
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
8:193
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
8:241
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
66:17
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
65:241
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
8:209
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
66:49
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
65:17
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
8:225
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
66:1
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
66:33
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
on unknown device 65:1
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
Use '--activationmode partial' to override.
24 logical volume(s) in volume group
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
3 logical volume(s) in volume group "centos_ovirt-01" now active
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
9 logical volume(s) in volume group
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
2:13 p.m.
https://bugzilla.redhat.com/1363734
https://bugzilla.redhat.com/1364339
https://bugzilla.redhat.com/1366176
Downgrade the lvm2 package.
Or upgrade vdsm to 4.17.35
-----Original Message-----
From: <users-bounces(a)ovirt.org> on behalf of Charles Tassell
<charles(a)islandadmin.ca>
Date: Wednesday 4 January 2017 at 23:22
To: "users(a)ovirt.org" <users(a)ovirt.org>
Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
Hello,
One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host. Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it. I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.
We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages. I also get the log entries below
constantly repeating in vdsm.log. Any ideas what's going on and how I
fix it?
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\\\\\'--activationmode partial\\\\\\\' to
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
value = meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
manifest = self.manifestClass(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata = selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
raise error(str(e))
LVM output:
[root@ovirt-01 mapper]# vgchange -a y
WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
from both devices /dev/sdac and /dev/sdk
Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
not /dev/sdac
Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
65:177
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
8:145
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
65:225
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
8:193
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
8:241
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
66:17
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
65:241
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
8:209
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
66:49
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
65:17
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
8:225
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
66:1
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
66:33
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
on unknown device 65:1
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
Use '--activationmode partial' to override.
24 logical volume(s) in volume group
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
3 logical volume(s) in volume group "centos_ovirt-01" now active
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
9 logical volume(s) in volume group
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:33 p.m.
Hi Pavel,
Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!
On 2017-01-04 06:13 PM, Pavel Gashev wrote:
https://bugzilla.redhat.com/1363734
https://bugzilla.redhat.com/1364339
https://bugzilla.redhat.com/1366176
Downgrade the lvm2 package.
Or upgrade vdsm to 4.17.35
-----Original Message-----
From: <users-bounces(a)ovirt.org> on behalf of Charles Tassell
<charles(a)islandadmin.ca>
Date: Wednesday 4 January 2017 at 23:22
To: "users(a)ovirt.org" <users(a)ovirt.org>
Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
Hello,
One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host. Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it. I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.
We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages. I also get the log entries below
constantly repeating in vdsm.log. Any ideas what's going on and how I
fix it?
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\\\\\'--activationmode partial\\\\\\\' to
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
value = meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
manifest = self.manifestClass(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata = selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
raise error(str(e))
LVM output:
[root@ovirt-01 mapper]# vgchange -a y
WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
from both devices /dev/sdac and /dev/sdk
Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
not /dev/sdac
Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
65:177
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
8:145
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
65:225
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
8:193
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
8:241
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
66:17
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
65:241
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
8:209
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
66:49
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
65:17
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
8:225
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
66:1
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
66:33
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
on unknown device 65:1
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
Use '--activationmode partial' to override.
24 logical volume(s) in volume group
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
3 logical volume(s) in volume group "centos_ovirt-01" now active
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
9 logical volume(s) in volume group
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
11:05 p.m.
--_000_37039a0e2fbb4f879854bcb23b7c275cacroniscom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Charles,
Please note the warning in logs about lvmetad. It's better to disable this =
service if you invoke LVM utilitities manually. It caches some LVM metadata=
but VDSM doesn't use it. So technically VDSM has different view on PVs.
________________________________
From: Charles Tassell <charles(a)islandadmin.ca Sent: Jan
5, 2017 7:33 AM
To: Pavel Gashev; users(a)ovirt.org
Subject: Re: [ovirt-users] Storage Problem After Host Update (v3.6)
Hi Pavel,
Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!
On 2017-01-04 06:13 PM, Pavel Gashev wrote:
<head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Exchange Server" <!-- converted from text
--><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style </head
<body <style
type=3D"text/css" <!--
-- </style
<div <div
style=3D"font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.=
0pt; color:#1F497D" <p
dir=3D"ltr">Charles,</p <p
dir=3D"ltr">Please note the warning in logs about lvmetad. It's better
t=
o disable this service if you invoke LVM utilitities manually. It caches so=
me LVM metadata but VDSM doesn't use it. So technically VDSM has different =
view on PVs.<br </p <div
id=3D"x_signature-x" style=3D"font-family:Calibri,Arial,Helvetica,sans=
-serif; font-size:12.0pt; color:#1F497D"
<br </div </div <div
id=3D"x_quoted_header" style=3D"clear:both" <hr style=3D"border:none; height:1px; color:#E1E1E1;
background-color:#E1E1=
E1" <div style=3D"border:none;
padding:3.0pt 0cm 0cm 0cm"><span style=3D"font-s=
ize:11.0pt;
font-family:'Calibri','sans-serif'"><b>From:</b>
Charles Tassel=
l &lt;charles(a)islandadmin.ca&gt;<br
<b>Sent:</b> Jan 5, 2017 7:33 AM<br
<b>To:</b> Pavel Gashev; users(a)ovirt.org<br <b>Subject:</b> Re: [ovirt-users] Storage Problem
After Host Update (v3.6)<=
br </span></div
</div <br
type=3D"attribution" </div <font size=3D"2"><span
style=3D"font-size:10pt;" <div
class=3D"PlainText">Hi Pavel,<br
<br Thanks, I added
the 3.6-snapshot repo to upgrade to vdsm 4.17.=
36 and <br then diagnosed a bunch of LVM
problems (PV's set as missing in pvs -v) <br and that
got us fixed back up!<br <br
<br On 2017-01-04 06:13 PM, Pavel Gashev
wrote:<br > <a
href=3D"https://bugzilla.redhat.com/1363734">https://bugzill...
t.com/1363734</a><br > <a
href=3D"https://bugzilla.redhat.com/1364339">https://bugzill...
t.com/1364339</a><br > <a
href=3D"https://bugzilla.redhat.com/1366176">https://bugzill...
t.com/1366176</a><br ><br > Downgrade the lvm2 package.<br > Or upgrade vdsm to 4.17.35<br ><br >
-----Original Message-----<br > From:
&lt;users-bounces(a)ovirt.org&gt; on behalf of Charles Tassell <=
;charles(a)islandadmin.ca&gt;<br >
Date: Wednesday 4 January 2017 at 23:22<br >
To: &quot;users(a)ovirt.org&quot; &lt;users(a)ovirt.org&gt;<br > Subject: [ovirt-users] Storage Problem After Host
Update (v3.6)<br ><br > Hello,<br
><br
> One of our admins did a package
update on one =
of our 3.6 hosts<br > through the web GUI and
then rebooted the host. Since then, it h=
asn't<br > been able to properly mount
storage and thus we can't move any VMs to<=
br > it. I rebooted it myself this afternoon and
it took our whole cl=
uster<br > down for about a half hour,
which was kind of concerning.<br
><br
> We use multipath fibre channel for
storage and=
when I do a vgscan or<br > vgdisplay -s everything
shows up correctly but I get a lot of "Du=
plicate<br > of PV ..." warning
messages. I also get the log entries bel=
ow<br > constantly repeating in
vdsm.log. Any ideas what's going on and =
how I<br > fix it?<br ><br >
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('Gener=
al<br > Storage Exception: (\'5
[] [\\\' WARNING: lvmetad is running but=
<br > disabled. Restart lvmetad before enabling it!\\\',
" Refusing act=
ivation<br > of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use=
<br > \\\'--activationmode partial\\\' to<br >
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\=
\\']\',)',)<br >
Thread-1844::ERROR::2017-01-04<br >
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':=
<br > {'message': 'Cannot activate Logical
Volumes: (\'General Storage<br >
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but<b=
r > disabled. Restart lvmetad before enabling
it!\\\\\\\', " Refusing=
<br > activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata=
.<br > Use \\\\\\\'--activationmode partial\\\\\\\'
to<br >
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'met=
adata\\\\\\\']\\\',)\',)',<br >
'code': 570}}<br >
Thread-101::ERROR::2017-01-04<br >
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error<br=
> monitoring domain
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b<br >
Traceback (most recent call last):<br
> File
"/usr/share/vdsm/storage/monitor.py&=
quot;, line 264, in _monitorDomain<br
>
self._produceDomain()<br
> File
"/usr/lib/python2.7/site-packages/vd=
sm/utils.py", line 769, in<br >
wrapper<br
> value =3D
meth(self, *a, **kw)<br
> File
"/usr/share/vdsm/storage/monitor.py&=
quot;, line 323, in _produceDomain<br
> self.domain =3D
sdCache.produce(se=
lf.sdUUID)<br
> File
"/usr/share/vdsm/storage/sdc.py"=
;, line 100, in produce<br
>
domain.getRealDomain()<br
> File
"/usr/share/vdsm/storage/sdc.py"=
;, line 52, in getRealDomain<br
> return
self._cache._realProduce(se=
lf._sdUUID)<br
> File
"/usr/share/vdsm/storage/sdc.py"=
;, line 124, in _realProduce<br
> domain =3D
self._findDomain(sdUUID=
)<br > File
"/usr/share/vdsm/storage/sdc.py"=
;, line 143, in _findDomain<br
> dom =3D
findMethod(sdUUID)<br
> File
"/usr/share/vdsm/storage/blockSD.py&=
quot;, line 1402, in findDomain<br
> return
BlockStorageDomain(BlockSto=
rageDomain.findDomainPath(sdUUID))<br
> File
"/usr/share/vdsm/storage/blockSD.py&=
quot;, line 455, in __init__<br
> manifest =3D
self.manifestClass(sd=
UUID)<br
> File
"/usr/share/vdsm/storage/blockSD.py",=
line 407, in __init__<br
> metadata =3D
selectMetadata(self.s=
dUUID)<br
> File
"/usr/share/vdsm/storage/blockSD.py&=
quot;, line 375, in selectMetadata<br
> if
len(mdProvider) > 0:<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 65, in __len__<br
> return
len(self.keys())<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 110, in keys<br
> return
list(self.__iter__())<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 107, in __iter__<br
>
self._dict.__iter__())<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 225, in __iter__<br
> with
self._accessWrapper():<br
> File
"/usr/lib64/python2.7/contextlib.py&=
quot;, line 17, in __enter__<br
> return
self.gen.next()<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 152, in<br >
_accessWrapper<br
>
self.refresh()<br
> File
"/usr/share/vdsm/storage/persistentD=
ict.py", line 230, in refresh<br
> lines =3D
self._metaRW.readlines()=
<br > File
"/usr/share/vdsm/storage/blockSD.py&=
quot;, line 335, in readlines<br
>
lvm.activateLVs(self._vgName, self=
._lvName)<br
> File
"/usr/share/vdsm/storage/lvm.py"=
;, line 1191, in activateLVs<br
>
_setLVAvailability(vgName, toActiv=
ate, "y")<br
> File
"/usr/share/vdsm/storage/lvm.py"=
;, line 826, in _setLVAvailability<br
> raise
error(str(e))<br ><br > LVM output:<br >
[root@ovirt-01 mapper]# vgchange -a y<br
> WARNING: duplicate PV
1oDZGSqoQ4ueh0XEMzILNleR=
oNY8ZDAt is being used<br > from both devices /dev/sdac
and /dev/sdk<br
> Found duplicate PV
1oDZGSqoQ4ueh0XEMzILNleRoNY=
8ZDAt: using /dev/sdk<br > not /dev/sdac<br > Using
duplicate PV /dev/sdk which is last seen=
, replacing /dev/sdac<br
> Duplicate of PV
EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcI=
F-kvIfXx dev<br >
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown devi=
ce<br > 65:177<br > Duplicate of
PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcI=
F-kvIfXx dev<br >
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown devi=
ce<br > 8:145<br > Duplicate of
PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HK=
U-Oh3F8w dev<br >
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown devi=
ce<br > 65:225<br > Duplicate of
PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HK=
U-Oh3F8w dev<br >
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown devi=
ce<br > 8:193<br > Duplicate of
PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5O=
k-p1zPDd dev<br >
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown devi=
ce<br > 8:241<br > Duplicate of
PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5O=
k-p1zPDd dev<br >
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown devi=
ce<br > 66:17<br > Duplicate of
PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxj=
N-ZU6dJL dev<br >
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown devi=
ce<br > 65:241<br > Duplicate of
PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxj=
N-ZU6dJL dev<br >
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown devi=
ce<br > 8:209<br > Duplicate of
PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3=
P-idn2zW dev<br >
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown devi=
ce<br > 66:49<br > Duplicate of
PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3=
P-idn2zW dev<br >
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown devi=
ce<br > 65:17<br > Duplicate of
PV lwe5su-6xQw-MiJu-Dh86-HNVz-Tsn=
G-YLoSG9 dev<br >
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown devi=
ce<br > 8:225<br > Duplicate of
PV lwe5su-6xQw-MiJu-Dh86-HNVz-Tsn=
G-YLoSG9 dev<br >
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown devi=
ce<br > 66:1<br > Duplicate of
PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BF=
k-lRPeE0 dev<br >
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown devi=
ce<br > 66:33<br > Duplicate of
PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BF=
k-lRPeE0 dev<br >
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists<br
> on unknown device 65:1<br > Refusing
activation of partial LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activation=
mode<br > partial' to
override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmo=
de<br > partial' to
override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmo=
de<br > partial' to
override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode<=
br > partial' to override.<br > Refusing
activation of partial LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmod=
e<br > partial' to override.<br > Refusing
activation of partial LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmo=
de<br > partial' to
override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a78908=
4c0.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee=
77b.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa9=
57e.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fe=
faa.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f=
6c0.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c0=
2b2.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b=
100.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad=
93c.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaff=
cb3.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a293=
8c5.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083=
a2b.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1=
847.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66c=
ef1.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369e=
ab2.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5=
ff6.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8=
c6f.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a5=
2a5.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca378=
0f7.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d=
0ab.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422=
ac7.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23=
481.<br > Use '--activationmode
partial' to override.<br
> Refusing activation of partial
LV<br >
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3=
adf.<br > Use '--activationmode
partial' to override.<br
> 24 logical volume(s) in volume
group<br >
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active<br > 3 logical
volume(s) in volume group "cent=
os_ovirt-01" now active<br
> WARNING: duplicate PV
TgIsrjy7YlXlBrJJK1s30xs7=
AyY7CwmZ is being used<br > from both devices
/dev/mapper/360060160d5732300f01ea0bed593e511 and /d=
ev/sdu<br
> WARNING: duplicate PV
TgIsrjy7YlXlBrJJK1s30xs7=
AyY7CwmZ is being used<br > from both devices
/dev/mapper/360060160d5732300f01ea0bed593e511 and /d=
ev/sdc<br
> 9 logical volume(s) in volume
group<br >
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active<br ><br >
[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/<=
br > 04c87c50-1eb2-4453-bae2-fe705e567976<br > 596c3099-2e41-4434-9902-aa2aeab1fb85
bea2af77-301e-4f5e-942b-43e12c30a=
33c<br >
1c77f2e1-d649-49e7-87c1-52b7b2a4f172<br >
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631f=
ea1<br >
230da8da-373e-4dcc-bb79-bd7c666d8ca6<br >
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c=
40d<br >
2a4923c8-730b-474e-886a-e75404739d71<br >
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092=
e62<br >
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55<br >
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448a=
a37<br >
33c1634c-cace-4c70-a59c-51ff5ea0fe64<br >
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366=
421<br >
39c29ad6-bb02-409c-9cc4-6b2960812058<br >
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08=
bc2<br >
4924d2cc-8f98-4fa6-8169-82931b05d22c<br >
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc545=
6f7<br ><br >
_______________________________________________<br >
Users mailing list<br > Users(a)ovirt.org<br > <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.ovirt.org/mailman/listinfo/users</a><br
><br ><br <br </div </span></font
</body </html
--_000_37039a0e2fbb4f879854bcb23b7c275cacroniscom_--
https://bugzilla.redhat.com/1363734
https://bugzilla.redhat.com/1364339
https://bugzilla.redhat.com/1366176
Downgrade the lvm2 package.
Or upgrade vdsm to 4.17.35
-----Original Message-----
From: <users-bounces(a)ovirt.org> on behalf of Charles Tassell <charles@isl=
andadmin.ca Date: Wednesday
4 January 2017 at 23:22
To: "users(a)ovirt.org" <users(a)ovirt.org Subject:
[ovirt-users] Storage Problem After Host Update (v3.6)
Hello,
One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host. Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it. I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.
We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages. I also get the log entries below
constantly repeating in vdsm.log. Any ideas what's going on and how I
fix it?
CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)=
',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\\\\\'--activationmode partial\\\\\\\' to
override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\=
\\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomai=
n
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
value =3D meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomai=
n
self.domain =3D sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)=
)
File "/usr/share/vdsm/storage/blockSD.py", line 455, in
__init__
manifest =3D self.manifestClass(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata =3D selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadat=
a
if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter=
__
self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter=
__
with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refres=
h
lines =3D self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailabilit=
y
raise error(str(e))
LVM output:
[root@ovirt-01 mapper]# vgchange -a y
WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
from both devices /dev/sdac and /dev/sdk
Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
not /dev/sdac
Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
65:177
Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
8:145
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
65:225
Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
8:193
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
8:241
Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
/dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
66:17
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
65:241
Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
/dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
8:209
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
66:49
Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
/dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
65:17
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
8:225
Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
/dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
66:1
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
66:33
Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
/dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
on unknown device 65:1
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481=
.
Use '--activationmode partial' to override.
Refusing activation of partial LV
8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf=
.
Use '--activationmode partial' to override.
24 logical volume(s) in volume group
"8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
3 logical volume(s) in volume group "centos_ovirt-01" now active
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/=
sdu
WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being
used
from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/=
sdc
9 logical volume(s) in volume group
"383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
04c87c50-1eb2-4453-bae2-fe705e567976
596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
1c77f2e1-d649-49e7-87c1-52b7b2a4f172
6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
230da8da-373e-4dcc-bb79-bd7c666d8ca6
71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
2a4923c8-730b-474e-886a-e75404739d71
821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
33c1634c-cace-4c70-a59c-51ff5ea0fe64
893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
39c29ad6-bb02-409c-9cc4-6b2960812058
8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
4924d2cc-8f98-4fa6-8169-82931b05d22c
b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_37039a0e2fbb4f879854bcb23b7c275cacroniscom_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html
2668
days inactive
2669
days old
3 comments
2 participants
participants (2)
-
Charles Tassell -
Pavel Gashev