This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --LREaOVJqkWp8iv9aCReT65xDKv7JE33We Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 28.2.2014, 20:05, Nir Soffer wrote:
----- Original Message -----
From: "Boyan Tabakov" <blade@alslayer.net> To: "Nir Soffer" <nsoffer@redhat.com> Cc: users@ovirt.org Sent: Tuesday, February 25, 2014 11:53:45 AM Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on= some nodes
Hello,
On 22.2.2014, 22:19, Nir Soffer wrote:
----- Original Message -----
From: "Boyan Tabakov" <blade@alslayer.net> To: "Nir Soffer" <nsoffer@redhat.com> Cc: users@ovirt.org Sent: Wednesday, February 19, 2014 7:18:36 PM Subject: Re: [Users] SD Disk's Logical Volume not visible/activated = on some nodes
Hello,
On 19.2.2014, 17:09, Nir Soffer wrote:
----- Original Message -----
From: "Boyan Tabakov" <blade@alslayer.net> To: users@ovirt.org Sent: Tuesday, February 18, 2014 3:34:49 PM Subject: [Users] SD Disk's Logical Volume not visible/activated on= some nodes
Consequently, when creating/booting a VM with the said disk attached, the VM fails to start on host2, because host2 can't see the LV. Similarly, if the VM is started on=
host1, it fails to migrate to host2. Extract from host2 log is in = the end. The LV in question is 6b35673e-7062-4716-a6c8-d5bf72fe3280.
As far as I could track quickly the vdsm code, there is only call = to lvs and not to lvscan or lvchange so the host2 LVM doesn't fully refre= sh. =20 lvs should see any change on the shared storage. =20 The only workaround so far has been to restart VDSM on host2, whic= h makes it refresh all LVM data properly. =20 When vdsm starts, it calls multipath -r, which ensure that we see all p= hysical volumes. =20
When is host2 supposed to pick up any newly created LVs in the SD = VG? Any suggestions where the problem might be?
When you create a new lv on the shared storage, the new lv should b= e visible on the other host. Lets start by verifying that you do see the new lv after a disk was created.
Try this:
1. Create a new disk, and check the disk uuid in the engine ui 2. On another machine, run this command:
lvs -o vg_name,lv_name,tags
You can identify the new lv using tags, which should contain the ne= w disk uuid.
If you don't see the new lv from the other host, please provide /var/log/messages and /var/log/sanlock.log.
Just tried that. The disk is not visible on the non-SPM node.
This means that storage is not accessible from this host.
Generally, the storage seems accessible ok. For example, if I restart the vdsmd, all volumes get picked up correctly (become visible in lvs output and VMs can be started with them). =20 Lests repeat this test, but now, if you do not see the new lv, please=20 run: =20 multipath -r =20 And report the results. =20
Running multipath -r helped and the disk was properly picked up by the second host. Is running multipath -r safe while host is not in maintenance mode? If yes, as a temporary workaround I can patch vdsmd to run multipath -r when e.g. monitoring the storage domain.
Here's the full sanlock.log for that host: ... 0x7fc37c0008c0:0x7fc37c0008d0:0x7fc391f5f000 ioto 10 to_count 1 2014-02-06 05:24:10+0200 563065 [31453]: s1 delta_renew read rv -202=
offset 0 /dev/3307f6fa-dd58-43db-ab23-b1fb299006c7/ids
Sanlock cannot write to the ids lockspace
Which line shows that sanlock can't write? The messages are not very "human readable". =20 The one above my comment at 2014-02-06 05:24:10+0200 =20 I suggest to set sanlock debug level on the sanlock log to get more det= ailed output. =20 Edit /etc/sysconfig/sanlock and add: =20 # -L 7: use debug level logging to sanlock log file SANLOCKOPTS=3D"$SANLOCKOPTS -L 7" =20
Last entry is from yesterday, while I just created a new disk.
What was the status of this host in the engine from 2014-02-06 05:24:10+0200 to 2014-02-18 14:22:16?
vdsm.log and engine.log for this time frame will make it more clear.
Host was up and running. The vdsm and engine logs are quite large, as = we were running some VM migrations between the hosts. Any pointers at wha= t to look for? For example, I noticed many entries in engine.log like th= is: =20 It will be hard to make any progress without the logs. =20
One warning that I keep seeing in vdsm logs on both nodes is this:
Thread-1617881::WARNING::2014-02-24 16:57:50,627::sp::1553::Storage.StoragePool::(getInfo) VG 3307f6fa-dd58-43db-ab23-b1fb299006c7's metadata size exceeded critical size: mdasize=3D134217728 mdafree=3D0 =20 Can you share the output of the command bellow? =20 lvs -o uuid,name,attr,size,free,extent_size,extent_count,free_count= ,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
Here's the output for both hosts. host1: [root@host1 ~]# lvs -o uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,= tags,vg_mda_size,vg_mda_free,lv_count,pv_count LV UUID LV Attr LSize VFree Ext #Ext Free LV Tags VMdaSize VMdaFree #LV #PV jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL 3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-ao--- 2.00g 114.62g 128.00m 1596 917 IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-a= c73492ac465 128.00m 0 13 2 ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC 3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi------ 7.00g 114.62g 128.00m 1596 917 IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-0= 00000000000 128.00m 0 13 2 YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n 57668a89-c5ca-4231-b18a-dd0607e459a5 -wi------ 10.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-= f02d683f07f8 128.00m 0 13 2 FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf 637bdaf7-61cb-4feb-838d-24355367dcec -wi------ 7.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2= feb223,MD_9 128.00m 0 13 2 9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE 65a711a2-d08a-496a-93de-e47aec19cfb1 -wi------ 12.00g 114.62g 128.00m 1596 917 IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000= 000000,MD_8 128.00m 0 13 2 Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd f4231952-76c5-4764-9c8b-ac73492ac465 -wi-ao--- 3.00g 114.62g 128.00m 1596 917 MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d= 42e7b760036 128.00m 0 13 2 u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0 f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-ao--- 40.00g 114.62g 128.00m 1596 917 MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-e= ae7f136d7e1 128.00m 0 13 2 h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids -wi-ao--- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases -wi-a---- 2.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master -wi-ao--- 1.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata -wi-a---- 512.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 JTDXAM-vMQI-UY0o-Axy9-UQKv-7Hl7-M9XSwQ mysql -wi-a---- 40.00g 0 4.00m 10241 0 188.00k 0 1 1 nQHcrF-KWhE-s8pY-90bc-jPSN-3qZR-pxHe0O iso-domain -wi-ao--- 10.00g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 qElr6v-lPNE-Kvc5-0g9W-I41J-dsBD-7QfZLv root -wi-ao--- 19.53g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 FSaRPB-WiZo-Ddqo-hPsB-jUe9-bf4v-eJBTJW swap -wi-ao--- 7.81g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 host2: [root@host2 log]# lvs -o uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,= tags,vg_mda_size,vg_mda_free,lv_count,pv_count LV UUID LV Attr LSize VFree Ext #Ext Free LV Tags VMdaSize VMdaFree #LV #PV jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL 3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-a---- 2.00g 114.62g 128.00m 1596 917 IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-a= c73492ac465 128.00m 0 13 2 ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC 3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi-ao--- 7.00g 114.62g 128.00m 1596 917 IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-0= 00000000000 128.00m 0 13 2 YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n 57668a89-c5ca-4231-b18a-dd0607e459a5 -wi-a---- 10.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-= f02d683f07f8 128.00m 0 13 2 FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf 637bdaf7-61cb-4feb-838d-24355367dcec -wi-ao--- 7.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2= feb223,MD_9 128.00m 0 13 2 9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE 65a711a2-d08a-496a-93de-e47aec19cfb1 -wi-ao--- 12.00g 114.62g 128.00m 1596 917 IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000= 000000,MD_8 128.00m 0 13 2 Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd f4231952-76c5-4764-9c8b-ac73492ac465 -wi-a---- 3.00g 114.62g 128.00m 1596 917 MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d= 42e7b760036 128.00m 0 13 2 u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0 f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-a---- 40.00g 114.62g 128.00m 1596 917 MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-e= ae7f136d7e1 128.00m 0 13 2 h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids -wi-ao--- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases -wi-a---- 2.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master -wi-a---- 1.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata -wi-a---- 512.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 G2zjlF-QO3X-tznc-BZE8-mY63-8I9n-gZQ7k5 root -wi-ao--- 19.53g 4.00m 4.00m 7001 1 1020.00k 0 2 1 vtuPP3-itfy-DtFX-j2gJ-wZBI-3RpF-fsWBJc swap -wi-ao--- 7.81g 4.00m 4.00m 7001 1 1020.00k 0 2 1
=20 I suggest that you open a bug and attach there engine.log, /var/log/me= ssages, vdsm.log and sanlock.log. =20 Please also give detailed info on the host os, vdsm version etc. =20 Nir =20
I'll gather logs and will create bug report. Thanks! Boyan --LREaOVJqkWp8iv9aCReT65xDKv7JE33We Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMUNG4ACgkQXOXFG4fgV76cLQCfRgVxYUJGwyREbC+tBawsveje z5YAn1oMSGckv6tDa7zxL4rK+oEp31/5 =t4kg -----END PGP SIGNATURE----- --LREaOVJqkWp8iv9aCReT65xDKv7JE33We--