[ovirt-users] Storage Problem After Host Update (v3.6)
Charles Tassell
charles at islandadmin.ca
Thu Jan 5 04:33:40 UTC 2017
Hi Pavel,
Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!
On 2017-01-04 06:13 PM, Pavel Gashev wrote:
> https://bugzilla.redhat.com/1363734
> https://bugzilla.redhat.com/1364339
> https://bugzilla.redhat.com/1366176
>
> Downgrade the lvm2 package.
> Or upgrade vdsm to 4.17.35
>
> -----Original Message-----
> From: <users-bounces at ovirt.org> on behalf of Charles Tassell <charles at islandadmin.ca>
> Date: Wednesday 4 January 2017 at 23:22
> To: "users at ovirt.org" <users at ovirt.org>
> Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
>
> Hello,
>
> One of our admins did a package update on one of our 3.6 hosts
> through the web GUI and then rebooted the host. Since then, it hasn't
> been able to properly mount storage and thus we can't move any VMs to
> it. I rebooted it myself this afternoon and it took our whole cluster
> down for about a half hour, which was kind of concerning.
>
> We use multipath fibre channel for storage and when I do a vgscan or
> vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
> of PV ..." warning messages. I also get the log entries below
> constantly repeating in vdsm.log. Any ideas what's going on and how I
> fix it?
>
> CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
> Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
> of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use
> \\\'--activationmode partial\\\' to
> override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
> Thread-1844::ERROR::2017-01-04
> 16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
> {'message': 'Cannot activate Logical Volumes: (\'General Storage
> Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing
> activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
> Use \\\\\\\'--activationmode partial\\\\\\\' to
> override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\\'metadata\\\\\\\']\\\',)\',)',
> 'code': 570}}
> Thread-101::ERROR::2017-01-04
> 16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
> monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
> self._produceDomain()
> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
> wrapper
> value = meth(self, *a, **kw)
> File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
> self.domain = sdCache.produce(self.sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
> domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
> return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
> domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
> dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
> return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
> manifest = self.manifestClass(sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
> metadata = selectMetadata(self.sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
> if len(mdProvider) > 0:
> File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
> return len(self.keys())
> File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
> return list(self.__iter__())
> File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
> self._dict.__iter__())
> File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
> with self._accessWrapper():
> File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> return self.gen.next()
> File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
> _accessWrapper
> self.refresh()
> File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
> lines = self._metaRW.readlines()
> File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
> lvm.activateLVs(self._vgName, self._lvName)
> File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
> _setLVAvailability(vgName, toActivate, "y")
> File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
> raise error(str(e))
>
> LVM output:
> [root at ovirt-01 mapper]# vgchange -a y
> WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used
> from both devices /dev/sdac and /dev/sdk
> Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk
> not /dev/sdac
> Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
> Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
> /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
> 65:177
> Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev
> /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device
> 8:145
> Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
> /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
> 65:225
> Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev
> /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device
> 8:193
> Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
> /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
> 8:241
> Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev
> /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device
> 66:17
> Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
> /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
> 65:241
> Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev
> /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device
> 8:209
> Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
> /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
> 66:49
> Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev
> /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device
> 65:17
> Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
> /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
> 8:225
> Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev
> /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device
> 66:1
> Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
> /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device
> 66:33
> Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev
> /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists
> on unknown device 65:1
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode
> partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481.
> Use '--activationmode partial' to override.
> Refusing activation of partial LV
> 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf.
> Use '--activationmode partial' to override.
> 24 logical volume(s) in volume group
> "8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active
> 3 logical volume(s) in volume group "centos_ovirt-01" now active
> WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
> from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu
> WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used
> from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc
> 9 logical volume(s) in volume group
> "383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active
>
> [root at ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/
> 04c87c50-1eb2-4453-bae2-fe705e567976
> 596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c
> 1c77f2e1-d649-49e7-87c1-52b7b2a4f172
> 6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1
> 230da8da-373e-4dcc-bb79-bd7c666d8ca6
> 71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d
> 2a4923c8-730b-474e-886a-e75404739d71
> 821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62
> 2bd2cb44-017c-4cb7-b710-9c2b3fedbf55
> 830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37
> 33c1634c-cace-4c70-a59c-51ff5ea0fe64
> 893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421
> 39c29ad6-bb02-409c-9cc4-6b2960812058
> 8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2
> 4924d2cc-8f98-4fa6-8169-82931b05d22c
> b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
More information about the Users
mailing list