https://bugzilla.redhat.com/1363734 https://bugzilla.redhat.com/1364339 https://bugzilla.redhat.com/1366176

Downgrade the lvm2 package. Or upgrade vdsm to 4.17.35

-----Original Message----- From: <users-bounces@ovirt.org> on behalf of Charles Tassell < charles@islandadmin.ca> Date: Wednesday 4 January 2017 at 23:22 To: "users@ovirt.org" <users@ovirt.org> Subject: [ovirt-users] Storage Problem After Host Update (v3.6)

Hello,

One of our admins did a package update on one of our 3.6 hosts through the web GUI and then rebooted the host. Since then, it hasn't been able to properly mount storage and thus we can't move any VMs to it. I rebooted it myself this afternoon and it took our whole cluster down for about a half hour, which was kind of concerning.

We use multipath fibre channel for storage and when I do a vgscan or vgdisplay -s everything shows up correctly but I get a lot of "Duplicate of PV ..." warning messages. I also get the log entries below constantly repeating in vdsm.log. Any ideas what's going on and how I fix it?

CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General Storage Exception: (\'5 [] [\\\' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\\\', " Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use \\\'--activationmode partial\\\' to override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'

On Jan 5, 2017 9:05 AM, "Pavel Gashev" <Pax@acronis.com> wrote: Charles, Please note the warning in logs about lvmetad. It's better to disable this service if you invoke LVM utilitities manually. It caches some LVM metadata but VDSM doesn't use it. So technically VDSM has different view on PVs. +1. We are going to disable it entirely in coming version. Y. ------------------------------ *From:* Charles Tassell <charles@islandadmin.ca> *Sent:* Jan 5, 2017 7:33 AM *To:* Pavel Gashev; users@ovirt.org *Subject:* Re: [ovirt-users] Storage Problem After Host Update (v3.6) Hi Pavel, Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v) and that got us fixed back up! On 2017-01-04 06:13 PM, Pavel Gashev wrote: metadata\\\']\',)',)

Thread-1844::ERROR::2017-01-04 16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Cannot activate Logical Volumes: (\'General Storage Exception: (\\\'5 [] [\\\\\\\' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\\\\\\\', " Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use \\\\\\\'--activationmode partial\\\\\\\' to override."]\\\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\\\\ \'metadata\\\\\\\']\\\',)\',)', 'code': 570}} Thread-101::ERROR::2017-01-04 16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in wrapper value = meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 100, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__ manifest = self.manifestClass(sdUUID) File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__ metadata = selectMetadata(self.sdUUID) File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata if len(mdProvider) > 0: File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__ return len(self.keys()) File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys return list(self.__iter__()) File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__ self._dict.__iter__()) File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__ with self._accessWrapper(): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper self.refresh() File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh lines = self._metaRW.readlines() File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines lvm.activateLVs(self._vgName, self._lvName) File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs _setLVAvailability(vgName, toActivate, "y") File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability raise error(str(e))

LVM output: [root@ovirt-01 mapper]# vgchange -a y WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used from both devices /dev/sdac and /dev/sdk Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk not /dev/sdac Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 65:177 Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev /dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 8:145 Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 65:225 Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev /dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 8:193 Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device 8:241 Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev /dev/mapper/360060160ec7323001e5bcfa3026fe111p1 exists on unknown device 66:17 Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device 65:241 Duplicate of PV N2B92Z-jpvT-RZVW-ot5M-4J9x-wxjN-ZU6dJL dev /dev/mapper/360060160ec73230036b603a1db6ee111p1 exists on unknown device 8:209 Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device 66:49 Duplicate of PV UabEG9-XpDC-aE0j-gES1-SieC-Ys3P-idn2zW dev /dev/mapper/360060160ec73230058c1b232906fe111p1 exists on unknown device 65:17 Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device 8:225 Duplicate of PV lwe5su-6xQw-MiJu-Dh86-HNVz-TsnG-YLoSG9 dev /dev/mapper/360060160ec732300be469e57ea6ee111p1 exists on unknown device 66:1 Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device 66:33 Duplicate of PV tZsNnJ-LbSu-xOC2-uyVO-ATmH-0BFk-lRPeE0 dev /dev/mapper/360060160ec732300c40d7ae05d6fe111p1 exists on unknown device 65:1 Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/outbox. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/leases. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ids. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/inbox. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/master. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/389cceba-d9a4-4452-b9a2-fd5a789084c0. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/873a161b-0933-41ef-ad14-950a19dee77b. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/31dc9c3d-d316-42ef-83c2-ed1bc6aa957e. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/d7c018c4-6488-4a44-8ef5-42a4723fefaa. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/944a179a-35b3-4bfe-ad21-43b1e902f6c0. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ecfba68b-4fb4-41d9-bd79-a570b08c02b2. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/c3b6fb85-1611-453b-88c4-6d222089b100. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/6e1a0348-5305-4d07-87cf-6ba74a9ad93c. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/121922ab-aa31-4681-b268-61834eaffcb3. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/1ed17401-ba4f-42de-8b47-86ea9a2938c5. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/36a0f2a4-25c2-4201-b578-d67897083a2b. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f20a2045-1d13-4141-a544-75d4404c1847. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/7dc84aad-f4bb-4f7a-8dce-72c5ec66cef1. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/68d74d9b-b8c1-433c-8be7-735ab369eab2. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/59bcadd0-4fd9-494b-a137-205588ba5ff6. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f14f9a94-515c-468e-b9aa-17acd1cf8c6f. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/4aa24cd6-e218-4d31-be1f-d646fb5a52a5. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/23300e46-e1aa-4b72-ba62-5833ca3780f7. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/5992ba58-6fb0-421f-816c-04c162b6d0ab. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/9830bb60-960a-432b-ad7c-279b37422ac7. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/3c4199fe-8ec3-4b0f-83d6-78ec0ec23481. Use '--activationmode partial' to override. Refusing activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/f409d03f-c1db-420f-a8a3-5f445fab3adf. Use '--activationmode partial' to override. 24 logical volume(s) in volume group "8dfcc83b-64d7-4ff7-9800-02ba3430ea7b" now active 3 logical volume(s) in volume group "centos_ovirt-01" now active WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdu WARNING: duplicate PV TgIsrjy7YlXlBrJJK1s30xs7AyY7CwmZ is being used from both devices /dev/mapper/360060160d5732300f01ea0bed593e511 and /dev/sdc 9 logical volume(s) in volume group "383a1e7d-8b86-4a50-8bc3-e4374c57a3e5" now active

[root@ovirt-01 mapper]# ls /dev/8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/ 04c87c50-1eb2-4453-bae2-fe705e567976 596c3099-2e41-4434-9902-aa2aeab1fb85 bea2af77-301e-4f5e-942b-43e12c30a33c 1c77f2e1-d649-49e7-87c1-52b7b2a4f172 6324cfd7-0369-4b29-aca8-0f83bf0593c8 cb155486-08f7-42a4-b46c-48d98631fea1 230da8da-373e-4dcc-bb79-bd7c666d8ca6 71bcdd9b-db35-46a6-9ba5-94a7b4b5216d d1e9cefd-fb24-4cc1-af7c-01debe42c40d 2a4923c8-730b-474e-886a-e75404739d71 821862ea-0a09-4178-9625-7c1e65021325 d638feb1-e3e4-4103-a47f-5dc771092e62 2bd2cb44-017c-4cb7-b710-9c2b3fedbf55 830c7f9d-6fd8-4feb-a70e-00f4af25177f dec27ebe-2491-4a40-9c02-82627448aa37 33c1634c-cace-4c70-a59c-51ff5ea0fe64 893d55d1-032a-4c7d-90d1-d2e2dcafd2b4 ea5471af-f1ab-43a0-a646-787354366421 39c29ad6-bb02-409c-9cc4-6b2960812058 8bbef2d3-2c27-41f8-82b2-21b7e1ba17dd f425d6e0-8641-4207-84eb-bb4d39d08bc2 4924d2cc-8f98-4fa6-8169-82931b05d22c b09b84b7-875b-4a60-9f70-8b0b8a57cc53 f904d381-7c0d-4850-8c92-cf72bc5456f7

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users