Charles,

Please note the warning in logs about lvmetad. It's better to disable this service if you invoke LVM utilitities manually. It caches some LVM metadata but VDSM doesn't use it. So technically VDSM has different view on PVs.

---

From: Charles Tassell <charles@islandadmin.ca>
Sent: Jan 5, 2017 7:33 AM
To: Pavel Gashev; users@ovirt.org
Subject: Re: [ovirt-users] Storage Problem After Host Update (v3.6)

Hi Pavel,

   Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!


On 2017-01-04 06:13 PM, Pavel Gashev wrote:
> https://bugzilla.redhat.com/1363734
> https://bugzilla.redhat.com/1364339
> https://bugzilla.redhat.com/1366176
>
> Downgrade the lvm2 package.
> Or upgrade vdsm to 4.17.35
>
> -----Original Message-----
> From: <users-bounces@ovirt.org> on behalf of Charles Tassell <charles@islandadmin.ca>
> Date: Wednesday 4 January 2017 at 23:22
> To: "users@ovirt.org" <users@ovirt.org>
> Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
>
> Hello,
>
>     One of our admins did a package update on one of our 3.6 hosts
> through the web GUI and then rebooted the host.  Since then, it hasn't
> been able to properly mount storage and thus we can't move any VMs to
> it.  I rebooted it myself this afternoon and it took our whole cluster
> down for about a half hour, which was kind of concerning.
>
>     We use multipath fibre channel for storage and when I do a vgscan or
> vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
> of PV ..." warning messages.  I also get the log entries below
> constantly repeating in vdsm.log.  Any ideas what's going on and how I
> fix it?
>
> CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
> Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
> of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use
> \\\'--activationmode partial\\\' to
> override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
> Thread-1844::ERROR::2017-01-04
> 16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
> {'message': 'Cannot activate Logical Volumes: (\'General Storage
> Exception: (\\\'5 [] [\\\\\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
> activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
> Use \\\\\\\'--activationmode partial\\\\\\\' to
> override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
> 'code': 570}}
> Thread-101::ERROR::2017-01-04
> 16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
> monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
> Traceback (most recent call last):
>     File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
>       self._produceDomain()
>     File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
> wrapper
>       value = meth(self, *a, **kw)
>     File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
>       self.domain = sdCache.produce(self.sdUUID)
>     File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
>       domain.getRealDomain()
>     File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>       return self._cache._realProduce(self._sdUUID)
>     File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
>       domain = self._findDomain(sdUUID)
>     File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
>       dom = findMethod(sdUUID)
>     File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
>       return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
>     File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
>       manifest = self.manifestClass(sdUUID)
>    File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
>       metadata = selectMetadata(self.sdUUID)
>     File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
>       if len(mdProvider) > 0:
>     File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
>       return len(self.keys())
>     File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
>       return list(self.__iter__())
>     File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
>       self._dict.__iter__())
>     File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
>       with self._accessWrapper():
>     File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>       return self.gen.next()
>     File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
> _accessWrapper
>       self.refresh()
>     File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
>       lines = self._metaRW.readlines()
>     File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
>       lvm.activateLVs(self._vgName, self._lvName)
>     File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
>       _setLVAvailability(vgName, toActivate, "y")
>     File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
>       raise error(str(e))
>
> LVM output:
> [root@ovirt-01 mapper]# vgchange -a y
>     WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
> from both devices /dev/sdac and /dev/sdk
>     Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
> not /dev/sdac
>     Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
>     Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
> /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
> 65:177
>     Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
> /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
> 8:145
>     Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
> /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
> 65:225
>     Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
> /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
> 8:193
>     Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
> /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
> 8:241
>     Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
> /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
> 66:17
>     Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
> /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
> 65:241
>     Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
> /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
> 8:209
>     Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
> /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
> 66:49
>     Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
> /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
> 65:17
>     Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
> /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
> 8:225
>     Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
> /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
> 66:1
>     Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
> /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
> 66:33
>     Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
> /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
>    on unknown device 65:1
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master.  Use '--activationmode
> partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
> Use '--activationmode partial' to override.
>    Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
> Use '--activationmode partial' to override.
>     Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
> Use '--activationmode partial' to override.
>     24 logical volume(s) in volume group
> "8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
>     3 logical volume(s) in volume group "centos_ovirt-01" now active
>     WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
> from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
>     WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
> from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
>     9 logical volume(s) in volume group
> "383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
>
> [root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
> 04c87c50-1eb2-4453-bae2-fe705e567976
> 596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
> 1c77f2e1-d649-49e7-87c1-52b7b2a4f172
> 6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
> 230da8da-373e-4dcc-bb79-bd7c666d8ca6
> 71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
> 2a4923c8-730b-474e-886a-e75404739d71
> 821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
> 2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
> 830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
> 33c1634c-cace-4c70-a59c-51ff5ea0fe64
> 893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
> 39c29ad6-bb02-409c-9cc4-6b2960812058
> 8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
> 4924d2cc-8f98-4fa6-8169-82931b05d22c
> b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
>
> _______________________________________________
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users