2:51 a.m.
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--LREaOVJqkWp8iv9aCReT65xDKv7JE33We
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
On 28.2.2014, 20:05, Nir Soffer wrote:
----- Original Message -----
> From: "Boyan Tabakov" <blade(a)alslayer.net>
> To: "Nir Soffer" <nsoffer(a)redhat.com>
> Cc: users(a)ovirt.org
> Sent: Tuesday, February 25, 2014 11:53:45 AM
> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on=
some nodes
>
> Hello,
>
> On 22.2.2014, 22:19, Nir Soffer wrote:
>> ----- Original Message -----
>>> From: "Boyan Tabakov" <blade(a)alslayer.net>
>>> To: "Nir Soffer" <nsoffer(a)redhat.com>
>>> Cc: users(a)ovirt.org
>>> Sent: Wednesday, February 19, 2014 7:18:36 PM
>>> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated =
on
>>> some nodes
>>>
>>> Hello,
>>>
>>> On 19.2.2014, 17:09, Nir Soffer wrote:
>>>> ----- Original Message -----
>>>>> From: "Boyan Tabakov" <blade(a)alslayer.net>
>>>>> To: users(a)ovirt.org
>>>>> Sent: Tuesday, February 18, 2014 3:34:49 PM
>>>>> Subject: [Users] SD Disk's Logical Volume not visible/activated
on=
some
>>>>> nodes
>>>>
>>>>> Consequently, when creating/booting
>>>>> a VM with the said disk attached, the VM fails to start on host2,
>>>>> because host2 can't see the LV. Similarly, if the VM is started
on=
>>>>> host1, it fails to migrate to host2. Extract from
host2 log is in =
the
>>>>> end. The LV in question is
6b35673e-7062-4716-a6c8-d5bf72fe3280.
>>>>>
>>>>> As far as I could track quickly the vdsm code, there is only call =
to lvs
>>>>> and not to lvscan or lvchange so the host2 LVM
doesn't fully refre=
sh.
=20
lvs should see any change on the shared storage.
=20
>>>>> The only workaround so far has been to restart VDSM on host2, whic=
h
>>>>> makes it refresh all LVM data properly.
=20
When vdsm starts, it calls multipath -r, which ensure that we see all p=
hysical
volumes.
=20
>>>>>
>>>>> When is host2 supposed to pick up any newly created LVs in the SD =
VG?
>>>>> Any suggestions where the problem might be?
>>>>
>>>> When you create a new lv on the shared storage, the new lv should b=
e
>>>> visible on the other host. Lets start by verifying
that you do see
>>>> the new lv after a disk was created.
>>>>
>>>> Try this:
>>>>
>>>> 1. Create a new disk, and check the disk uuid in the engine ui
>>>> 2. On another machine, run this command:
>>>>
>>>> lvs -o vg_name,lv_name,tags
>>>>
>>>> You can identify the new lv using tags, which should contain the ne=
w disk
>>>> uuid.
>>>>
>>>> If you don't see the new lv from the other host, please provide
>>>> /var/log/messages
>>>> and /var/log/sanlock.log.
>>>
>>> Just tried that. The disk is not visible on the non-SPM node.
>>
>> This means that storage is not accessible from this host.
>
> Generally, the storage seems accessible ok. For example, if I restart
> the vdsmd, all volumes get picked up correctly (become visible in lvs
> output and VMs can be started with them).
=20
Lests repeat this test, but now, if you do not see the new lv, please=20
run:
=20
multipath -r
=20
And report the results.
=20
Running multipath -r helped and the disk was properly picked up by the
second host.
Is running multipath -r safe while host is not in maintenance mode? If
yes, as a temporary workaround I can patch vdsmd to run multipath -r
when e.g. monitoring the storage domain.
>>> Here's the full
>>> sanlock.log for that host:
> ...
>>> 0x7fc37c0008c0:0x7fc37c0008d0:0x7fc391f5f000 ioto 10 to_count 1
>>> 2014-02-06 05:24:10+0200 563065 [31453]: s1 delta_renew read rv -202=
>>> offset 0 /dev/3307f6fa-dd58-43db-ab23-b1fb299006c7/ids
>>
>> Sanlock cannot write to the ids lockspace
>
> Which line shows that sanlock can't write? The messages are not very
> "human readable".
=20
The one above my comment at 2014-02-06 05:24:10+0200
=20
I suggest to set sanlock debug level on the sanlock log to get more det=
ailed
output.
=20
Edit /etc/sysconfig/sanlock and add:
=20
# -L 7: use debug level logging to sanlock log file
SANLOCKOPTS=3D"$SANLOCKOPTS -L 7"
=20
>>>
>>> Last entry is from yesterday, while I just created a new disk.
>>
>> What was the status of this host in the engine from 2014-02-06
>> 05:24:10+0200 to 2014-02-18 14:22:16?
>>
>> vdsm.log and engine.log for this time frame will make it more clear.
>
> Host was up and running. The vdsm and engine logs are quite large, as =
we
> were running some VM migrations between the hosts. Any pointers
at wha=
t
> to look for? For example, I noticed many entries in engine.log
like th=
is:
=20
It will be hard to make any progress without the logs.
=20
>
> One warning that I keep seeing in vdsm logs on both nodes is this:
>
> Thread-1617881::WARNING::2014-02-24
> 16:57:50,627::sp::1553::Storage.StoragePool::(getInfo) VG
> 3307f6fa-dd58-43db-ab23-b1fb299006c7's metadata size exceeded
> critical size: mdasize=3D134217728 mdafree=3D0
=20
Can you share the output of the command bellow?
=20
lvs -o uuid,name,attr,size,free,extent_size,extent_count,free_count=
,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
Here's the output for both hosts.
host1:
[root@host1 ~]# lvs -o
uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,=
tags,vg_mda_size,vg_mda_free,lv_count,pv_count
LV UUID LV
Attr LSize VFree Ext #Ext Free LV Tags
VMdaSize VMdaFree #LV #PV
jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL
3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-ao--- 2.00g 114.62g 128.00m
1596 917
IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-a=
c73492ac465
128.00m 0 13 2
ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC
3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi------ 7.00g 114.62g 128.00m
1596 917
IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-0=
00000000000
128.00m 0 13 2
YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n
57668a89-c5ca-4231-b18a-dd0607e459a5 -wi------ 10.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-=
f02d683f07f8
128.00m 0 13 2
FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf
637bdaf7-61cb-4feb-838d-24355367dcec -wi------ 7.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2=
feb223,MD_9
128.00m 0 13 2
9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE
65a711a2-d08a-496a-93de-e47aec19cfb1 -wi------ 12.00g 114.62g 128.00m
1596 917
IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000=
000000,MD_8
128.00m 0 13 2
Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd
f4231952-76c5-4764-9c8b-ac73492ac465 -wi-ao--- 3.00g 114.62g 128.00m
1596 917
MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d=
42e7b760036
128.00m 0 13 2
u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0
f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-ao--- 40.00g 114.62g 128.00m
1596 917
MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-e=
ae7f136d7e1
128.00m 0 13 2
h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids
-wi-ao--- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases
-wi-a---- 2.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master
-wi-ao--- 1.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata
-wi-a---- 512.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
JTDXAM-vMQI-UY0o-Axy9-UQKv-7Hl7-M9XSwQ mysql
-wi-a---- 40.00g 0 4.00m 10241 0
188.00k 0 1 1
nQHcrF-KWhE-s8pY-90bc-jPSN-3qZR-pxHe0O iso-domain
-wi-ao--- 10.00g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
qElr6v-lPNE-Kvc5-0g9W-I41J-dsBD-7QfZLv root
-wi-ao--- 19.53g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
FSaRPB-WiZo-Ddqo-hPsB-jUe9-bf4v-eJBTJW swap
-wi-ao--- 7.81g 97.80g 4.00m 34596 25036
1020.00k 0 3 2
host2:
[root@host2 log]# lvs -o
uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,=
tags,vg_mda_size,vg_mda_free,lv_count,pv_count
LV UUID LV
Attr LSize VFree Ext #Ext Free LV Tags
VMdaSize
VMdaFree #LV #PV
jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL
3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-a---- 2.00g 114.62g 128.00m
1596 917
IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-a=
c73492ac465
128.00m 0 13 2
ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC
3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi-ao--- 7.00g 114.62g 128.00m
1596 917
IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-0=
00000000000
128.00m 0 13 2
YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n
57668a89-c5ca-4231-b18a-dd0607e459a5 -wi-a---- 10.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-=
f02d683f07f8
128.00m 0 13 2
FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf
637bdaf7-61cb-4feb-838d-24355367dcec -wi-ao--- 7.00g 114.62g 128.00m
1596 917
PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2=
feb223,MD_9
128.00m 0 13 2
9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE
65a711a2-d08a-496a-93de-e47aec19cfb1 -wi-ao--- 12.00g 114.62g 128.00m
1596 917
IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000=
000000,MD_8
128.00m 0 13 2
Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd
f4231952-76c5-4764-9c8b-ac73492ac465 -wi-a---- 3.00g 114.62g 128.00m
1596 917
MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d=
42e7b760036
128.00m 0 13 2
u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0
f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-a---- 40.00g 114.62g 128.00m
1596 917
MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-e=
ae7f136d7e1
128.00m 0 13 2
h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids
-wi-ao--- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases
-wi-a---- 2.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master
-wi-a---- 1.00g 114.62g 128.00m 1596 917
128.00m 0 13 2
2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata
-wi-a---- 512.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox
-wi-a---- 128.00m 114.62g 128.00m 1596 917
128.00m 0 13 2
G2zjlF-QO3X-tznc-BZE8-mY63-8I9n-gZQ7k5 root
-wi-ao--- 19.53g 4.00m 4.00m 7001 1
1020.00k 0 2 1
vtuPP3-itfy-DtFX-j2gJ-wZBI-3RpF-fsWBJc swap
-wi-ao--- 7.81g 4.00m 4.00m 7001 1
1020.00k 0 2 1
=20
I suggest that you open a bug and attach there engine.log, /var/log/me=
ssages,
vdsm.log and sanlock.log.
=20
Please also give detailed info on the host os, vdsm version etc.
=20
Nir
=20
I'll gather logs and will create bug report.
Thanks!
Boyan
--LREaOVJqkWp8iv9aCReT65xDKv7JE33We
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMUNG4ACgkQXOXFG4fgV76cLQCfRgVxYUJGwyREbC+tBawsveje
z5YAn1oMSGckv6tDa7zxL4rK+oEp31/5
=t4kg
-----END PGP SIGNATURE-----
--LREaOVJqkWp8iv9aCReT65xDKv7JE33We--