On 02/23/2012 02:21 PM, Terry Phelps wrote: > Thanks for the quick reply. > > My one hypervisor already had the ISO domain mounted (without any > explicit action by me): This is to be expected. VDSM needs the mount. I suggested that command just in case it wasn't mounted for some odd reason. > mount | grep iso > > oravm3.acbl.net:/isodomain/ on > /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 > (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) > > Using this mount (I didn't do exactly what you said, if that matters), Nope, you're fine. > I did the tests you asked for. > Yes, I can touch a new file. > Yes, I can read the ISO file > > Here is what I saw: > I'm assuming you were "vdsm" when you executed these commands, right? > bash-4.2$ ls > OracleLinux-R6-U2-Server-x86_64-dvd.iso > bash-4.2$ touch me > bash-4.2$ ls > me OracleLinux-R6-U2-Server-x86_64-dvd.iso > bash-4.2$ strings Orac* |head -2 > CD001 > LINUX OL6.2 x86_64 Disc 1 20111212 > > > Funny, though. When I typed "su - vdsm" by mistake, from root, it said > "This account is currently not available." (Is that relevant?) But > what you said to do did work fine. By default vdsm is given a nologin shell for security reasons. The "-s /bin/bash" overrides that when switching users. > Other ideas/ Not at the moment. I think you've done a fairly good job of demonstrating that VDSM would not have any permission problems reading or writing to the NFS export.

On 02/24/2012 09:19 AM, Terry Phelps wrote: > On 2/23/12, Keith Robertson<kroberts(a)redhat.com> wrote: >> On 02/23/2012 02:21 PM, Terry Phelps wrote: >>> Thanks for the quick reply. >>> >>> My one hypervisor already had the ISO domain mounted (without any >>> explicit action by me): >> This is to be expected. VDSM needs the mount. I suggested that command >> just in case it wasn't mounted for some odd reason. >>> mount | grep iso >>> >>> oravm3.acbl.net:/isodomain/ on >>> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 >>> (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) >>> >>> Using this mount (I didn't do exactly what you said, if that matters), >> Nope, you're fine. >>> I did the tests you asked for. >>> Yes, I can touch a new file. >>> Yes, I can read the ISO file >>> >>> Here is what I saw: >>> >> I'm assuming you were "vdsm" when you executed these commands, right? >>> bash-4.2$ ls >>> OracleLinux-R6-U2-Server-x86_64-dvd.iso >>> bash-4.2$ touch me >>> bash-4.2$ ls >>> me OracleLinux-R6-U2-Server-x86_64-dvd.iso >>> bash-4.2$ strings Orac* |head -2 >>> CD001 >>> LINUX OL6.2 x86_64 Disc 1 20111212 >>> >>> >>> Funny, though. When I typed "su - vdsm" by mistake, from root, it said >>> "This account is currently not available." (Is that relevant?) But >>> what you said to do did work fine. >> By default vdsm is given a nologin shell for security reasons. The "-s >> /bin/bash" overrides that when switching users. >>> Other ideas/ >> Not at the moment. I think you've done a fairly good job of >> demonstrating that VDSM would not have any permission problems reading >> or writing to the NFS export. > Just to gather more information, I re-ran engine-iso-uploader to > upload my ISO. It complained that the ISO was already there, which it > IS. I used the "--force" option to make him do it again. He did. Yup, standard behavior. > It still doesn't show up in the admin portal. > > Is there something else I can do to help find the problem? Well you've demonstrated that the user "vdsm" can r/w the NFS export from the hypervisor. This is a common source of problems as things like selinux and UID/GID mismatches can cause all sorts blockages preventing VDSM's ability to r/w the NFS export. Let's see what VDSM thinks. From a hypervisor do this... 1. Type "mount" 2. Look for your ISO domain in the returned list. 3. Note the local path to the ISO domain. It might look something like this... /rhev/data-center/mnt/oravm3.acbl.net:_isodomain 4. List the directories in it: ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain 5. Notice the returned UUID directory name: [root@node ~]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain 92cf90c2-3698-48b5-84fd-d8e4f8684549 6. Supply that to the vdsClient command as follows: vdsClient -s 0 getFileList 92cf90c2-3698-48b5-84fd-d8e4f8684549 What happens?

OK, so VDSM looks fine. Let's see what the REST API and by extension ovirt-engine thinks about it... From the host upon which ovirt-engine running do this: wget -q -O - --no-check-certificate --user=admin@internal --password='password here' https://localhost:8443/api/storagedomains/48a5390f-2f86-485c-8537-b6bc9dd... Do you see the files?

Cheers, Keith On 02/24/2012 10:28 AM, Terry Phelps wrote: > It looks like you were doing this as root, so I did, too. In any case, > the result looks good to me: > > # mount | grep iso > > oravm3.acbl.net:/isodomain/ on > /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 > (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) > > ]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain > > 48a5390f-2f86-485c-8537-b6bc9dd71796 vdsmTest > > [root@oravm2 ~]# vdsClient -s 0 getFileList > 48a5390f-2f86-485c-8537-b6bc9dd71796 > > file: OracleLinux-R6-U2-Server-x86_64-dvd.iso status: {'status': > 469, 'ctime': '1330092866.03', 'size': '3591360512'} > > > NOTE: That "vdsmTest" file you see has appeared there since yesterday, > I think. I didn't put it there. Have no idea what that is. > On 2/24/12, Keith Robertson<kroberts(a)redhat.com> wrote: >> On 02/24/2012 09:19 AM, Terry Phelps wrote: >>> On 2/23/12, Keith Robertson<kroberts(a)redhat.com> wrote: >>>> On 02/23/2012 02:21 PM, Terry Phelps wrote: >>>>> Thanks for the quick reply. >>>>> >>>>> My one hypervisor already had the ISO domain mounted (without any >>>>> explicit action by me): >>>> This is to be expected. VDSM needs the mount. I suggested that command >>>> just in case it wasn't mounted for some odd reason. >>>>> mount | grep iso >>>>> >>>>> oravm3.acbl.net:/isodomain/ on >>>>> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 >>>>> (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) >>>>> >>>>> Using this mount (I didn't do exactly what you said, if that matters), >>>> Nope, you're fine. >>>>> I did the tests you asked for. >>>>> Yes, I can touch a new file. >>>>> Yes, I can read the ISO file >>>>> >>>>> Here is what I saw: >>>>> >>>> I'm assuming you were "vdsm" when you executed these commands, right? >>>>> bash-4.2$ ls >>>>> OracleLinux-R6-U2-Server-x86_64-dvd.iso >>>>> bash-4.2$ touch me >>>>> bash-4.2$ ls >>>>> me OracleLinux-R6-U2-Server-x86_64-dvd.iso >>>>> bash-4.2$ strings Orac* |head -2 >>>>> CD001 >>>>> LINUX OL6.2 x86_64 Disc 1 20111212 >>>>> >>>>> >>>>> Funny, though. When I typed "su - vdsm" by mistake, from root, it said >>>>> "This account is currently not available." (Is that relevant?) But >>>>> what you said to do did work fine. >>>> By default vdsm is given a nologin shell for security reasons. The "-s >>>> /bin/bash" overrides that when switching users. >>>>> Other ideas/ >>>> Not at the moment. I think you've done a fairly good job of >>>> demonstrating that VDSM would not have any permission problems reading >>>> or writing to the NFS export. >>> Just to gather more information, I re-ran engine-iso-uploader to >>> upload my ISO. It complained that the ISO was already there, which it >>> IS. I used the "--force" option to make him do it again. He did. >> Yup, standard behavior. >>> It still doesn't show up in the admin portal. >>> >>> Is there something else I can do to help find the problem? >> Well you've demonstrated that the user "vdsm" can r/w the NFS export >> from the hypervisor. This is a common source of problems as things like >> selinux and UID/GID mismatches can cause all sorts blockages preventing >> VDSM's ability to r/w the NFS export. >> >> Let's see what VDSM thinks. From a hypervisor do this... >> 1. Type "mount" >> 2. Look for your ISO domain in the returned list. >> 3. Note the local path to the ISO domain. It might look something like >> this... >> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >> 4. List the directories in it: >> ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >> 5. Notice the returned UUID directory name: >> [root@node ~]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >> 92cf90c2-3698-48b5-84fd-d8e4f8684549 >> 6. Supply that to the vdsClient command as follows: >> vdsClient -s 0 getFileList 92cf90c2-3698-48b5-84fd-d8e4f8684549 >> >> What happens? >> >> >> >> >>

Is your storage domain active? Just run the "list" command on the iso uploader (or look in the UI). Example: [root@rhevm ~]# rhevm-iso-uploader list Please provide the REST API password for RHEV-M (CTRL+D to abort): ISO Storage Domain Name | Datacenter | ISO Domain Status iso1 | Default | active iso1 | iSCSI | inactive iso1 | NFS | active On 02/24/2012 01:04 PM, Terry Phelps wrote: > On 2/24/12, Keith Robertson&lt;kroberts(a)redhat.com&gt; wrote: >> OK, so VDSM looks fine. Let's see what the REST API and by extension >> ovirt-engine thinks about it... >> >> From the host upon which ovirt-engine running do this: >> wget -q -O - --no-check-certificate --user=admin@internal >> --password='password here' >> https://localhost:8443/api/storagedomains/48a5390f-2f86-485c-8537-b6bc9dd... >> >> Do you see the files? > NOPE: > > # cat doit > wget -q -O - --no-check-certificate --user=admin@internal \ > --password=****** > https://localhost:8443/api/storagedomains/48a5390f-2f86-485c-8537-b6bc9dd... > > [root@oravm3 ~]# sh doit > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > <files/> > > Maybe this has gotten us closer to the problem. > > > > >> Cheers, >> Keith >> >> On 02/24/2012 10:28 AM, Terry Phelps wrote: >>> It looks like you were doing this as root, so I did, too. In any case, >>> the result looks good to me: >>> >>> # mount | grep iso >>> >>> oravm3.acbl.net:/isodomain/ on >>> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 >>> (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) >>> >>> ]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >>> >>> 48a5390f-2f86-485c-8537-b6bc9dd71796 vdsmTest >>> >>> [root@oravm2 ~]# vdsClient -s 0 getFileList >>> 48a5390f-2f86-485c-8537-b6bc9dd71796 >>> >>> file: OracleLinux-R6-U2-Server-x86_64-dvd.iso status: {'status': >>> 469, 'ctime': '1330092866.03', 'size': '3591360512'} >>> >>> >>> NOTE: That "vdsmTest" file you see has appeared there since yesterday, >>> I think. I didn't put it there. >> Have no idea what that is. >>> On 2/24/12, Keith Robertson&lt;kroberts(a)redhat.com&gt; wrote: >>>> On 02/24/2012 09:19 AM, Terry Phelps wrote: >>>>> On 2/23/12, Keith Robertson&lt;kroberts(a)redhat.com&gt; wrote: >>>>>> On 02/23/2012 02:21 PM, Terry Phelps wrote: >>>>>>> Thanks for the quick reply. >>>>>>> >>>>>>> My one hypervisor already had the ISO domain mounted (without any >>>>>>> explicit action by me): >>>>>> This is to be expected. VDSM needs the mount. I suggested that >>>>>> command >>>>>> just in case it wasn't mounted for some odd reason. >>>>>>> mount | grep iso >>>>>>> >>>>>>> oravm3.acbl.net:/isodomain/ on >>>>>>> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 >>>>>>> (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) >>>>>>> >>>>>>> Using this mount (I didn't do exactly what you said, if that >>>>>>> matters), >>>>>> Nope, you're fine. >>>>>>> I did the tests you asked for. >>>>>>> Yes, I can touch a new file. >>>>>>> Yes, I can read the ISO file >>>>>>> >>>>>>> Here is what I saw: >>>>>>> >>>>>> I'm assuming you were "vdsm" when you executed these commands, right? >>>>>>> bash-4.2$ ls >>>>>>> OracleLinux-R6-U2-Server-x86_64-dvd.iso >>>>>>> bash-4.2$ touch me >>>>>>> bash-4.2$ ls >>>>>>> me OracleLinux-R6-U2-Server-x86_64-dvd.iso >>>>>>> bash-4.2$ strings Orac* |head -2 >>>>>>> CD001 >>>>>>> LINUX OL6.2 x86_64 Disc 1 20111212 >>>>>>> >>>>>>> >>>>>>> Funny, though. When I typed "su - vdsm" by mistake, from root, it >>>>>>> said >>>>>>> "This account is currently not available." (Is that relevant?) But >>>>>>> what you said to do did work fine. >>>>>> By default vdsm is given a nologin shell for security reasons. The >>>>>> "-s >>>>>> /bin/bash" overrides that when switching users. >>>>>>> Other ideas/ >>>>>> Not at the moment. I think you've done a fairly good job of >>>>>> demonstrating that VDSM would not have any permission problems >>>>>> reading >>>>>> or writing to the NFS export. >>>>> Just to gather more information, I re-ran engine-iso-uploader to >>>>> upload my ISO. It complained that the ISO was already there, which it >>>>> IS. I used the "--force" option to make him do it again. He did. >>>> Yup, standard behavior. >>>>> It still doesn't show up in the admin portal. >>>>> >>>>> Is there something else I can do to help find the problem? >>>> Well you've demonstrated that the user "vdsm" can r/w the NFS export >>>> from the hypervisor. This is a common source of problems as things >>>> like >>>> selinux and UID/GID mismatches can cause all sorts blockages preventing >>>> VDSM's ability to r/w the NFS export. >>>> >>>> Let's see what VDSM thinks. From a hypervisor do this... >>>> 1. Type "mount" >>>> 2. Look for your ISO domain in the returned list. >>>> 3. Note the local path to the ISO domain. It might look something like >>>> this... >>>> /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >>>> 4. List the directories in it: >>>> ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >>>> 5. Notice the returned UUID directory name: >>>> [root@node ~]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain >>>> 92cf90c2-3698-48b5-84fd-d8e4f8684549 >>>> 6. Supply that to the vdsClient command as follows: >>>> vdsClient -s 0 getFileList 92cf90c2-3698-48b5-84fd-d8e4f8684549 >>>> >>>> What happens? >>>> >>>> >>>> >>>> >>>> >>

> Hi Terry, The engine.log should contain logs regarding ISO files, can you please attach it to the mail, maybe we can find some clues there. Regards, Maor

On 02/24/2012 10:28 AM, Terry Phelps wrote: > It looks like you were doing this as root, so I did, too. In any case, > the result looks good to me: > > # mount | grep iso > > oravm3.acbl.net:/isodomain/ on > /rhev/data-center/mnt/oravm3.acbl.net:_isodomain type nfs4 > (rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=172.16.2.52,minorversion=0,local_lock=none,addr=192.168.118.10) > > > ]# ls /rhev/data-center/mnt/oravm3.acbl.net:_isodomain > > 48a5390f-2f86-485c-8537-b6bc9dd71796 vdsmTest > > [root@oravm2 ~]# vdsClient -s 0 getFileList > 48a5390f-2f86-485c-8537-b6bc9dd71796 > > file: OracleLinux-R6-U2-Server-x86_64-dvd.iso status: {'status': > 469, 'ctime': '1330092866.03', 'size': '3591360512'} > > > NOTE: That "vdsmTest" file you see has appeared there since yesterday, > I think. I didn't put it there. You didn't, this file can be removed, yesterday the nfs-check couldn't complete the test (remove the file) as you answered me (below) and it's still there. > # python nfs-check.py oravm3.acbl.net:/isodomain > Current hostname: oravm2.acbl.net - IP addr 127.0.0.1 > Trying to /bin/mount -t nfs oravm3.acbl.net:/isodomain... > Executing NFS tests.. > Removing vdsmTest file.. > Traceback (most recent call last): > File "nfs-check.py", line 268, in<module> > os.removedirs(LOCALPATH) > File "/usr/lib64/python2.7/os.py", line 170, in removedirs > OSError: [Errno 16] Device or resource busy: '/tmp/tmpV9KEh5'

On 02/24/2012 09:37 PM, Terry Phelps wrote: > > > Hi Terry, > The engine.log should contain logs regarding ISO files, > can you please attach it to the mail, maybe we can find some clues > there. > > Regards, > Maor > > > I am attaching what I think you're asking for: > /var/log/ovirt-engine/engine.log > > Near the end of it, I do see this: > > 2012-02-24 13:49:28,795 INFO > [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-5-thread-48) > Finished automatic refresh process for Unknown file type with success, > for storage domain id 48a5390f-2f86-485c-8537-b6bc9dd71796. > > Perhaps that's relevant. > Hi Terry, you right, that is the log, although it could be helpful if the debug mode was enabled there (It can be done by editing the configuration file $JBOSS_HOME/standalone/configuration/standalone.xml and change the xml entry:<logger category="org.ovirt.engine.core.bll"> from INFO to DEBUG) From what I see, there are no files fetched from VDSM 2012-02-23 15:06:43,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (http--0.0.0.0-8080-1) FINISH, HsmGetIsoListVDSCommand, return: [] The return list is empty. Im not sure if it was already discussed in this mail thread, but maybe it is also better to check the VDSM log in the hypervisor, or check what does getIsoList verb returns. Regarding the log that you were indicating, I am guessing, some how, there is an unknown file in the DB, that the files in the DB are somehow with file type Unknown, (I can not see the reason for it in the log). but I'm not sure its relevant. _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

** Hi, On 02/24/2012 03:58 PM, Terry Phelps wrote: On Fri, Feb 24, 2012 at 3:46 PM, Keith Robertson <kroberts(a)redhat.com&gt;wrote: > Terry, > > Like this... > [root@node ~]# vdsClient -s 0 getConnectedStoragePoolsList > 9775f154-7578-4e22-ae44-4664b298a8cc > > [root@node ~]# vdsClient -s 0 getIsoList > 9775f154-7578-4e22-ae44-4664b298a8cc > ------ ISO list with proper permissions only ------- > rhel-server-6.2.x86_64-dvd.iso > > Aha! That doesn't show anything: [root@oravm2 ~]# vdsClient -s 0 getConnectedStoragePoolsList f465251e-5679-11e1-ba81-97917332892e [root@oravm2 ~]# vdsClient -s 0 getIsoList f465251e-5679-11e1-ba81-97917332892e ------ ISO list with proper permissions only ------- [root@oravm2 ~]# As Maor suggested, can you please attach the /var/log/vdsm/vdsm.log ? You got it: