[Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors. Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G
Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors. Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G
Hello, When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors. Here is some information about the system: [wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled [wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64 [wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3: Thank you in advance. Limor G
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Hello,
When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors.
Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') vdsm.log should add more details about this exception Thanks
Here is some information about the system:
[wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled
[wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64
[wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3:
Thank you in advance. Limor G
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Thank you very much for your reply. I attached the vdsm.log On Wed, Mar 20, 2013 at 11:52 AM, Eli Mesika <emesika@redhat.com> wrote:
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Hello,
When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors.
Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception')
vdsm.log should add more details about this exception
Thanks
Here is some information about the system:
[wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled
[wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64
[wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3:
Thank you in advance. Limor G
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Eli Mesika" <emesika@redhat.com> Cc: users@ovirt.org, "Yuval M" <yuvalme@gmail.com>, "Nezer Zaidenberg" <nzaidenberg@mac.com> Sent: Wednesday, March 20, 2013 12:47:49 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your reply. I attached the vdsm.log
thanks, I am CC people from our storage team. Ayal, Eduardo, Allon this is failing in clusterlock.py::acquireHostId Any idea on the problem origin? Thanks
On Wed, Mar 20, 2013 at 11:52 AM, Eli Mesika < emesika@redhat.com > wrote:
----- Original Message -----
From: "Limor Gavish" < lgavish@gmail.com > To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Hello,
When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors.
Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception')
vdsm.log should add more details about this exception
Thanks
Here is some information about the system:
[wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled
[wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64
[wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3:
Thank you in advance. Limor G
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Also adding Federico to the thread, could be a san lock issue: SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception') Regards, Maor On 03/20/2013 01:50 PM, Eli Mesika wrote:
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Eli Mesika" <emesika@redhat.com> Cc: users@ovirt.org, "Yuval M" <yuvalme@gmail.com>, "Nezer Zaidenberg" <nzaidenberg@mac.com> Sent: Wednesday, March 20, 2013 12:47:49 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your reply. I attached the vdsm.log
thanks, I am CC people from our storage team.
Ayal, Eduardo, Allon this is failing in clusterlock.py::acquireHostId Any idea on the problem origin? Thanks
On Wed, Mar 20, 2013 at 11:52 AM, Eli Mesika < emesika@redhat.com > wrote:
----- Original Message -----
From: "Limor Gavish" < lgavish@gmail.com > To: users@ovirt.org Sent: Wednesday, March 20, 2013 9:10:00 AM Subject: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Hello,
When trying to attach the first NFS data storage domain to the default data center it does not get attached. It appears and disappears after a few seconds from the "storage domains" table in the "Data Centers" tab in the GUI. In engine.log (attached) I see many errors.
Hן Limor Can you please add vdsm.log (/var/log/vdsm/vdsm.log) I see this in the engine.log SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception')
vdsm.log should add more details about this exception
Thanks
Here is some information about the system:
[wil@bufferoverflow ovirt-engine]$ cat /etc/issue Fedora release 18 (Spherical Cow) Kernel \r on an \m (\l) [wil@bufferoverflow ovirt-engine]$ sestatus SELinux status: disabled
[wil@bufferoverflow ovirt-engine]$ rpm -q vdsm vdsm-4.10.3-7.fc18.x86_64
[wil@bufferoverflow ovirt-engine]$ nfsstat | grep -i nfs Server nfs v3: Client nfs v3:
Thank you in advance. Limor G
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Eli Mesika" <emesika@redhat.com> Cc: "Yuval M" <yuvalme@gmail.com>, users@ovirt.org, "Nezer Zaidenberg" <nzaidenberg@mac.com> Sent: Wednesday, March 20, 2013 11:47:49 AM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your reply. I attached the vdsm.log
Hi Limor, can you please inspect the status of the NFS mount? # mkdir /mnt/tmp # mount -t nfs <your_nfs_share> /mnt/tmp # cd /mnt/tmp/1902354b-4c39-4707-ac6c-3637aaf1943b/dom_md And please report the output of: # ls -l # sanlock direct dump ids Can you also include more vdsm logs? More specifically the ones where the NFS domain has been created? (createStorageDomain with sdUUID='1902354b-4c39-4707-ac6c-3637aaf1943b') Thanks, -- Federico
Thank you very much for your response. Attached VDSM logs as you requested (The VDSM logs where the NFS domain was created were missing so we had to recreate the NFS domain, therefore the sdUUID has changed). Here is the rest of the commands you asked: [root@bufferoverflow wil]# mount -t nfs bufferoverflow:/home/BO_Ovirt_Storage /mnt/tmp [root@bufferoverflow wil]# cd /mnt/tmp/1083422e-a5db-41b6-b667-b9ef1ef244f0/dom_md/ [root@bufferoverflow dom_md]# ls -l total 2052 -rw-rw---- 1 vdsm kvm 1048576 Mar 20 21:46 ids -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 inbox -rw-rw---- 1 vdsm kvm 2097152 Mar 20 21:45 leases -rw-r--r-- 1 vdsm kvm 311 Mar 20 21:45 metadata -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 outbox [root@bufferoverflow dom_md]# sanlock direct dump ids [root@bufferoverflow dom_md]# Please note, the VDSM is running as a system service (it was installed from a package) while ovirt-engine was built from sources and thus is not running as root. Is this an issue? On Wed, Mar 20, 2013 at 8:07 PM, Federico Simoncelli <fsimonce@redhat.com>wrote:
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Eli Mesika" <emesika@redhat.com> Cc: "Yuval M" <yuvalme@gmail.com>, users@ovirt.org, "Nezer Zaidenberg" < nzaidenberg@mac.com> Sent: Wednesday, March 20, 2013 11:47:49 AM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your reply. I attached the vdsm.log
Hi Limor, can you please inspect the status of the NFS mount?
# mkdir /mnt/tmp # mount -t nfs <your_nfs_share> /mnt/tmp # cd /mnt/tmp/1902354b-4c39-4707-ac6c-3637aaf1943b/dom_md
And please report the output of:
# ls -l
# sanlock direct dump ids
Can you also include more vdsm logs? More specifically the ones where the NFS domain has been created? (createStorageDomain with sdUUID='1902354b-4c39-4707-ac6c-3637aaf1943b')
Thanks, -- Federico
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Yuval M" <yuvalme@gmail.com>, users@ovirt.org, "Nezer Zaidenberg" <nzaidenberg@mac.com>, "Eli Mesika" <emesika@redhat.com>, "Maor Lipchuk" <mlipchuk@redhat.com> Sent: Wednesday, March 20, 2013 9:02:35 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your response.
Attached VDSM logs as you requested (The VDSM logs where the NFS domain was created were missing so we had to recreate the NFS domain, therefore the sdUUID has changed). Here is the rest of the commands you asked:
[root@bufferoverflow wil]# mount -t nfs bufferoverflow:/home/BO_Ovirt_Storage /mnt/tmp [root@bufferoverflow wil]# cd /mnt/tmp/1083422e-a5db-41b6-b667-b9ef1ef244f0/dom_md/ [root@bufferoverflow dom_md]# ls -l total 2052 -rw-rw---- 1 vdsm kvm 1048576 Mar 20 21:46 ids -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 inbox -rw-rw---- 1 vdsm kvm 2097152 Mar 20 21:45 leases -rw-r--r-- 1 vdsm kvm 311 Mar 20 21:45 metadata -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 outbox [root@bufferoverflow dom_md]# sanlock direct dump ids
Sorry I should have mentioned that if you use root_squash for your nfs share you have to switch to the vdsm user: (root)# su -s /bin/sh vdsm (vdsm)$ cd /mnt/tmp/<sduuid>/dom_md/ (vdsm)$ sanlock direct dump ids (and now you should be able to see the output) If the output is still empty then used hexdump -C to inspect it (and eventually post it here compressed). Another important thing that you should check is: # ps fax | grep sanlock If the output doesn't look like the following: 1966 ? SLs 0:00 wdmd -G sanlock 2036 ? SLsl 0:00 sanlock daemon -U sanlock -G sanlock 2037 ? S 0:00 \_ sanlock daemon -U sanlock -G sanlock Then I suggest you to update sanlock to the latest build: http://koji.fedoraproject.org/koji/buildinfo?buildID=377815 (sanlock-2.6-7.fc18) And eventually if after rebooting the problem persists, please post also the sanlock log (/var/log/sanlock.log)
Please note, the VDSM is running as a system service (it was installed from a package) while ovirt-engine was built from sources and thus is not running as root. Is this an issue?
It shouldn't be. -- Federico
Updating from sanlock-2.6-4.fc18 to sanlock-2.6-7.fc18 fixed it. Huge thanks! On Mar 20, 2013 10:24 PM, "Federico Simoncelli" <fsimonce@redhat.com> wrote:
----- Original Message -----
From: "Limor Gavish" <lgavish@gmail.com> To: "Federico Simoncelli" <fsimonce@redhat.com> Cc: "Yuval M" <yuvalme@gmail.com>, users@ovirt.org, "Nezer Zaidenberg" < nzaidenberg@mac.com>, "Eli Mesika" <emesika@redhat.com>, "Maor Lipchuk" <mlipchuk@redhat.com> Sent: Wednesday, March 20, 2013 9:02:35 PM Subject: Re: [Users] Failing to attach NFS data storage domain (Ovirt 3.2)
Thank you very much for your response.
Attached VDSM logs as you requested (The VDSM logs where the NFS domain was created were missing so we had to recreate the NFS domain, therefore the sdUUID has changed). Here is the rest of the commands you asked:
[root@bufferoverflow wil]# mount -t nfs bufferoverflow:/home/BO_Ovirt_Storage /mnt/tmp [root@bufferoverflow wil]# cd /mnt/tmp/1083422e-a5db-41b6-b667-b9ef1ef244f0/dom_md/ [root@bufferoverflow dom_md]# ls -l total 2052 -rw-rw---- 1 vdsm kvm 1048576 Mar 20 21:46 ids -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 inbox -rw-rw---- 1 vdsm kvm 2097152 Mar 20 21:45 leases -rw-r--r-- 1 vdsm kvm 311 Mar 20 21:45 metadata -rw-rw---- 1 vdsm kvm 0 Mar 20 21:45 outbox [root@bufferoverflow dom_md]# sanlock direct dump ids
Sorry I should have mentioned that if you use root_squash for your nfs share you have to switch to the vdsm user:
(root)# su -s /bin/sh vdsm (vdsm)$ cd /mnt/tmp/<sduuid>/dom_md/
(vdsm)$ sanlock direct dump ids (and now you should be able to see the output)
If the output is still empty then used hexdump -C to inspect it (and eventually post it here compressed).
Another important thing that you should check is:
# ps fax | grep sanlock
If the output doesn't look like the following:
1966 ? SLs 0:00 wdmd -G sanlock 2036 ? SLsl 0:00 sanlock daemon -U sanlock -G sanlock 2037 ? S 0:00 \_ sanlock daemon -U sanlock -G sanlock
Then I suggest you to update sanlock to the latest build:
http://koji.fedoraproject.org/koji/buildinfo?buildID=377815 (sanlock-2.6-7.fc18)
And eventually if after rebooting the problem persists, please post also the sanlock log (/var/log/sanlock.log)
Please note, the VDSM is running as a system service (it was installed from a package) while ovirt-engine was built from sources and thus is not running as root. Is this an issue?
It shouldn't be.
-- Federico
participants (4)
-
Eli Mesika -
Federico Simoncelli -
Limor Gavish -
Maor Lipchuk