[Users] SD Disk's Logical Volume not visible/activated on some nodes
Boyan Tabakov
blade at alslayer.net
Mon Mar 3 07:51:05 UTC 2014
On 28.2.2014, 20:05, Nir Soffer wrote:
> ----- Original Message -----
>> From: "Boyan Tabakov" <blade at alslayer.net>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: users at ovirt.org
>> Sent: Tuesday, February 25, 2014 11:53:45 AM
>> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on some nodes
>>
>> Hello,
>>
>> On 22.2.2014, 22:19, Nir Soffer wrote:
>>> ----- Original Message -----
>>>> From: "Boyan Tabakov" <blade at alslayer.net>
>>>> To: "Nir Soffer" <nsoffer at redhat.com>
>>>> Cc: users at ovirt.org
>>>> Sent: Wednesday, February 19, 2014 7:18:36 PM
>>>> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on
>>>> some nodes
>>>>
>>>> Hello,
>>>>
>>>> On 19.2.2014, 17:09, Nir Soffer wrote:
>>>>> ----- Original Message -----
>>>>>> From: "Boyan Tabakov" <blade at alslayer.net>
>>>>>> To: users at ovirt.org
>>>>>> Sent: Tuesday, February 18, 2014 3:34:49 PM
>>>>>> Subject: [Users] SD Disk's Logical Volume not visible/activated on some
>>>>>> nodes
>>>>>
>>>>>> Consequently, when creating/booting
>>>>>> a VM with the said disk attached, the VM fails to start on host2,
>>>>>> because host2 can't see the LV. Similarly, if the VM is started on
>>>>>> host1, it fails to migrate to host2. Extract from host2 log is in the
>>>>>> end. The LV in question is 6b35673e-7062-4716-a6c8-d5bf72fe3280.
>>>>>>
>>>>>> As far as I could track quickly the vdsm code, there is only call to lvs
>>>>>> and not to lvscan or lvchange so the host2 LVM doesn't fully refresh.
>
> lvs should see any change on the shared storage.
>
>>>>>> The only workaround so far has been to restart VDSM on host2, which
>>>>>> makes it refresh all LVM data properly.
>
> When vdsm starts, it calls multipath -r, which ensure that we see all physical volumes.
>
>>>>>>
>>>>>> When is host2 supposed to pick up any newly created LVs in the SD VG?
>>>>>> Any suggestions where the problem might be?
>>>>>
>>>>> When you create a new lv on the shared storage, the new lv should be
>>>>> visible on the other host. Lets start by verifying that you do see
>>>>> the new lv after a disk was created.
>>>>>
>>>>> Try this:
>>>>>
>>>>> 1. Create a new disk, and check the disk uuid in the engine ui
>>>>> 2. On another machine, run this command:
>>>>>
>>>>> lvs -o vg_name,lv_name,tags
>>>>>
>>>>> You can identify the new lv using tags, which should contain the new disk
>>>>> uuid.
>>>>>
>>>>> If you don't see the new lv from the other host, please provide
>>>>> /var/log/messages
>>>>> and /var/log/sanlock.log.
>>>>
>>>> Just tried that. The disk is not visible on the non-SPM node.
>>>
>>> This means that storage is not accessible from this host.
>>
>> Generally, the storage seems accessible ok. For example, if I restart
>> the vdsmd, all volumes get picked up correctly (become visible in lvs
>> output and VMs can be started with them).
>
> Lests repeat this test, but now, if you do not see the new lv, please
> run:
>
> multipath -r
>
> And report the results.
>
Running multipath -r helped and the disk was properly picked up by the
second host.
Is running multipath -r safe while host is not in maintenance mode? If
yes, as a temporary workaround I can patch vdsmd to run multipath -r
when e.g. monitoring the storage domain.
>>>> Here's the full
>>>> sanlock.log for that host:
>> ...
>>>> 0x7fc37c0008c0:0x7fc37c0008d0:0x7fc391f5f000 ioto 10 to_count 1
>>>> 2014-02-06 05:24:10+0200 563065 [31453]: s1 delta_renew read rv -202
>>>> offset 0 /dev/3307f6fa-dd58-43db-ab23-b1fb299006c7/ids
>>>
>>> Sanlock cannot write to the ids lockspace
>>
>> Which line shows that sanlock can't write? The messages are not very
>> "human readable".
>
> The one above my comment at 2014-02-06 05:24:10+0200
>
> I suggest to set sanlock debug level on the sanlock log to get more detailed output.
>
> Edit /etc/sysconfig/sanlock and add:
>
> # -L 7: use debug level logging to sanlock log file
> SANLOCKOPTS="$SANLOCKOPTS -L 7"
>
>>>>
>>>> Last entry is from yesterday, while I just created a new disk.
>>>
>>> What was the status of this host in the engine from 2014-02-06
>>> 05:24:10+0200 to 2014-02-18 14:22:16?
>>>
>>> vdsm.log and engine.log for this time frame will make it more clear.
>>
>> Host was up and running. The vdsm and engine logs are quite large, as we
>> were running some VM migrations between the hosts. Any pointers at what
>> to look for? For example, I noticed many entries in engine.log like this:
>
> It will be hard to make any progress without the logs.
>
>>
>> One warning that I keep seeing in vdsm logs on both nodes is this:
>>
>> Thread-1617881::WARNING::2014-02-24
>> 16:57:50,627::sp::1553::Storage.StoragePool::(getInfo) VG
>> 3307f6fa-dd58-43db-ab23-b1fb299006c7's metadata size exceeded
>> critical size: mdasize=134217728 mdafree=0
>
> Can you share the output of the command bellow?
>
> lvs -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
Here's the output for both hosts.
host1:
[root at host1 ~]# lvs -o
uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count
LV UUID LV
Attr LSize VFree Ext #Ext Free LV Tags
VMdaSize VMdaFree #LV #PV
jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL
3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-ao--- 2.00g 114.62g 128.00m
1596 917
IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-ac73492ac465
128.00m 0 13 2
ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC
3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi------ 7.00g 114.62g 128.00m
1596 917
IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-000000000000
128.00m 0 13 2
YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n
57668a89-c5ca-4231-b18a-dd0607e459a5 -wi------ 10.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-f02d683f07f8
128.00m 0 13 2
FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf
637bdaf7-61cb-4feb-838d-24355367dcec -wi------ 7.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2feb223,MD_9
128.00m 0 13 2
9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE
65a711a2-d08a-496a-93de-e47aec19cfb1 -wi------ 12.00g 114.62g 128.00m
1596 917
IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000000000,MD_8
128.00m 0 13 2
Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd
f4231952-76c5-4764-9c8b-ac73492ac465 -wi-ao--- 3.00g 114.62g 128.00m
1596 917
MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d42e7b760036
128.00m 0 13 2
u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0
f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-ao--- 40.00g 114.62g 128.00m
1596 917
MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-eae7f136d7e1
128.00m 0 13 2
h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids
-wi-ao--- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases
-wi-a---- 2.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master
-wi-ao--- 1.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata
-wi-a---- 512.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
JTDXAM-vMQI-UY0o-Axy9-UQKv-7Hl7-M9XSwQ mysql
-wi-a---- 40.00g 0 4.00m 10241 0
188.00k 0 1 1
nQHcrF-KWhE-s8pY-90bc-jPSN-3qZR-pxHe0O iso-domain
-wi-ao--- 10.00g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
qElr6v-lPNE-Kvc5-0g9W-I41J-dsBD-7QfZLv root
-wi-ao--- 19.53g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
FSaRPB-WiZo-Ddqo-hPsB-jUe9-bf4v-eJBTJW swap
-wi-ao--- 7.81g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
host2:
[root at host2 log]# lvs -o
uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count
LV UUID LV
Attr LSize VFree Ext #Ext Free LV Tags
VMdaSize
VMdaFree #LV #PV
jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL
3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-a---- 2.00g 114.62g 128.00m
1596 917
IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-ac73492ac465
128.00m 0 13 2
ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC
3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi-ao--- 7.00g 114.62g 128.00m
1596 917
IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-000000000000
128.00m 0 13 2
YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n
57668a89-c5ca-4231-b18a-dd0607e459a5 -wi-a---- 10.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-f02d683f07f8
128.00m 0 13 2
FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf
637bdaf7-61cb-4feb-838d-24355367dcec -wi-ao--- 7.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2feb223,MD_9
128.00m 0 13 2
9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE
65a711a2-d08a-496a-93de-e47aec19cfb1 -wi-ao--- 12.00g 114.62g 128.00m
1596 917
IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000000000,MD_8
128.00m 0 13 2
Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd
f4231952-76c5-4764-9c8b-ac73492ac465 -wi-a---- 3.00g 114.62g 128.00m
1596 917
MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d42e7b760036
128.00m 0 13 2
u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0
f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-a---- 40.00g 114.62g 128.00m
1596 917
MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-eae7f136d7e1
128.00m 0 13 2
h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids
-wi-ao--- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases
-wi-a---- 2.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master
-wi-a---- 1.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata
-wi-a---- 512.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
G2zjlF-QO3X-tznc-BZE8-mY63-8I9n-gZQ7k5 root
-wi-ao--- 19.53g 4.00m 4.00m 7001 1
1020.00k 0 2 1
vtuPP3-itfy-DtFX-j2gJ-wZBI-3RpF-fsWBJc swap
-wi-ao--- 7.81g 4.00m 4.00m 7001 1
1020.00k 0 2 1
>
> I suggest that you open a bug and attach there engine.log, /var/log/messages, vdsm.log and sanlock.log.
>
> Please also give detailed info on the host os, vdsm version etc.
>
> Nir
>
I'll gather logs and will create bug report.
Thanks!
Boyan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 268 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140303/b17c1a64/attachment-0001.sig>
More information about the Users
mailing list