Just so you know I have solved the issue.  So it is file permissions on the openfiler server.  This is openfiler 2.3.  For some reason on the openfiler server &quot;node&quot;  the file permissions where still ofguest:ofguest.  When ovirt connects to a data node that is NFS is it suppose to change the permissions on that server automatically?  If it is it might be a bug.  This again is a NFS Domain datacenter.  Using hardware based Openfiler nodes.  I will try adding another openfiler node to see if the same thing happens.  Also has anyone else experienced this also?<div>
<br></div><div>Let me know if you have any thoughts,</div><div><br></div><div>Dominic</div><div><br><div class="gmail_quote">On Tue, Mar 13, 2012 at 5:35 AM, Moran Goldboim <span dir="ltr">&lt;<a href="mailto:mgoldboi@redhat.com">mgoldboi@redhat.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>

  
    
  
  <div text="#000000" bgcolor="#ffffff"><div class="im">
    <br>
    On 03/12/2012 07:47 PM, Dominic Kaiser wrote:
    <blockquote type="cite">I noticed permisions on files in the that folder are
      saying group # 96 what is the 96 group and should it be kvm?  On
      the /mnt folder should the files read owner:vdsm group:kvm on all
      files?
      <div><br>
      </div>
    </blockquote>
    <br></div>
    Yes, user and group should be vdsm:kvm (36:36) on all &quot;/rhev&quot; tree.<span class="HOEnZb"><font color="#888888"><br>
    Moran.</font></span><div><div class="h5"><br>
    <br>
    <blockquote type="cite">
      <div>Dominic<br>
        <br>
        <div class="gmail_quote">On Mon, Mar 12, 2012 at 10:49 AM,
          Dominic Kaiser <span dir="ltr">&lt;<a href="mailto:dominic@bostonvineyard.org" target="_blank">dominic@bostonvineyard.org</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
            Hello Moran,
            <div><br>
            </div>
            <div>Thanks for responding.  </div>
            <div><br>
            </div>
            <div>Here is output you asked for and I have selinux
              disabled on this host:</div>
            <div><br>
            </div>
            <div>
              <div>ls -lZh c10c25e0-526a-11e1-a5e4-87fa674ffb5d/</div>
              <div>lrwxrwxrwx vdsm kvm ?                              
                 095fcd9d-6203-496f-b254-7a9eafaa8ac2 -&gt;
/rhev/data-center/mnt/10.3.0.248:_mnt_mainstorage_mainstorage_MainStorage_ISO/095fcd9d-6203-496f-b254-7a9eafaa8ac2</div>
              <div>lrwxrwxrwx vdsm kvm ?                              
                 14308dd8-1fd1-452c-a288-2f975ef31de1 -&gt;
/rhev/data-center/mnt/_var_lib_vdsm_storage/14308dd8-1fd1-452c-a288-2f975ef31de1</div>
              <div>lrwxrwxrwx vdsm kvm ?                              
                 340dad49-2ec7-475e-b6c6-668504bdb5b1 -&gt;
/rhev/data-center/mnt/10.3.0.248:_mnt_mainstorage_mainstorage_MainStorage_MS/340dad49-2ec7-475e-b6c6-668504bdb5b1</div>
              <div><font color="#ff0000">lrwxrwxrwx vdsm kvm ?          
                                     
                   651cb693-3ffa-43b6-ab49-43cfb03d179b -&gt;
/rhev/data-center/mnt/ruth.bostonvineyard.org:_mnt_ruthstorage1_mainstorage_data/651cb693-3ffa-43b6-ab49-43cfb03d179b</font></div>
              <div>lrwxrwxrwx vdsm kvm ?                              
                 bcc2d564-2daa-45ae-990b-0d27a17e47e0 -&gt;
/rhev/data-center/mnt/10.3.0.248:_mnt_mainstorage_mainstorage_MainStorage_Export/bcc2d564-2daa-45ae-990b-0d27a17e47e0</div>
              <div>lrwxrwxrwx vdsm kvm ?                              
                 mastersd -&gt;
/rhev/data-center/mnt/10.3.0.248:_mnt_mainstorage_mainstorage_MainStorage_MS/340dad49-2ec7-475e-b6c6-668504bdb5b1</div>
              <div><br>
              </div>
              The one I am having the problem with is the one in red.
               It is an openfiler server called ruth.  I can add it just
              fine create disks but cannot start a VM.  It is weird.
               Again it is a NFS datacenter this is a data domain not
              master data domain.</div>
            <div><br>
            </div>
            <div>Thanks,</div>
            <div><br>
            </div>
            <div>Dominic</div>
            <div>
              <div>
                <div><br>
                </div>
                <div><br>
                  <div class="gmail_quote">On Sun, Mar 11, 2012 at 4:20
                    AM, Moran Goldboim <span dir="ltr">&lt;<a href="mailto:mgoldboi@redhat.com" target="_blank">mgoldboi@redhat.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                      <div text="#000000" bgcolor="#ffffff">
                        <div>
                          <div> On 03/10/2012 01:26 AM, Dominic Kaiser
                            wrote: </div>
                        </div>
                        <blockquote type="cite">
                          <div>
                            <div> Here is the error I am getting:<br>
                              <br>
                              <div title="VM fs1 is down. Exit message
                                internal error process exited while
                                connecting to monitor: qemu-kvm: -drive
                                file=/rhev/data-center/c10c25e0-526a-11e1-a5e4-87fa674ffb5d/651cb693-3ffa-43b6-ab49-43cfb03d179b/images/7fa8a2e7-214e-4081-9f1f-f78c88b2f96e/49c199b6-cbda-4aa0-9b50-e3dc28341e09,if=none,id=drive-virtio-disk0,format=raw,serial=81-9f1f-f78c88b2f96e,cache=none,werror=stop,rerror=stop,aio=threads:

                                could not open disk image
                                /rhev/data-center/c10c25e0-526a-11e1-a5e4-87fa674ffb5d/651cb693-3ffa-43b6-ab49-43cfb03d179b/images/7fa8a2e7-214e-4081-9f1f-f78c88b2f96e/49c199b6-cbda-4aa0-9b50-e3dc28341e09:

                                Permission denied .">
                                <div>VM fs1 is down. Exit message
                                  internal error process exited while
                                  connecting to monitor: qemu-kvm:
                                  -drive
                                  file=/rhev/data-center/c10c25e0-526a-11e1-a5e4-87fa674ffb5d/651cb693-3ffa-43b6-ab49-43cfb03d179b/images/7fa8a2e7-214e-4081-9f1f-f78c88b2f96e/49c199b6-cbda-4aa0-9b50-e3dc28341e09,if=none,id=drive-virtio-disk0,format=raw,serial=81-9f1f-f78c88b2f96e,cache=none,werror=stop,rerror=stop,aio=threads:


                                  could not open disk image
                                  /rhev/data-center/c10c25e0-526a-11e1-a5e4-87fa674ffb5d/651cb693-3ffa-43b6-ab49-43cfb03d179b/images/7fa8a2e7-214e-4081-9f1f-f78c88b2f96e/49c199b6-cbda-4aa0-9b50-e3dc28341e09:


                                  Permission denied .</div>
                              </div>
                              <br>
                              I can add the domain fine.  it is not the
                              master domain and it is another node. I
                              can create virtual disks but cannot start
                              vm&#39;s.  I saw a post on google that
                              mentions nfs=0 not nfs=1 i think it was in
                              /usr/sbin/senforce maybe?  Does anyone
                              recollect this problem and can help me? 
                              Again this is on a separate node and is a
                              standard data domain  not master.<br>
                              <br>
                              Thanks,<br>
                              <br>
                              -- <br>
                              Dominic Kaiser<br>
                              Greater Boston Vineyard<br>
                              Director of Operations<br>
                              <br>
                              cell: <a href="tel:617-230-1412" value="+16172301412" target="_blank">617-230-1412</a><br>
                              fax: <a href="tel:617-252-0238" value="+16172520238" target="_blank">617-252-0238</a><br>
                              email: <a href="mailto:dominic@bostonvineyard.org" target="_blank">dominic@bostonvineyard.org</a><br>
                              <br>
                              <br>
                            </div>
                          </div>
                          <pre><fieldset></fieldset>
_______________________________________________
Users mailing list
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
                        </blockquote>
                        Dominic, sounds like you have some permission
                        problem with the image, can you try with selinux
                        in permissive mode - &quot;/usr/sbin/setenforce 0&quot;<br>
                        can you please add the output of &quot;ls -lZh
/rhev/data-center/c10c25e0-526a-11e1-a5e4-87fa674ffb5d/651cb693-3ffa-43b6-ab49-43cfb03d179b/images/7fa8a2e7-214e-4081-9f1f-f78c88b2f96e/49c199b6-cbda-4aa0-9b50-e3dc28341e09&quot;<span><font color="#888888"><br>
                            <br>
                            Moran.<br>
                          </font></span></div>
                    </blockquote>
                  </div>
                  <br>
                  <br clear="all">
                  <div><br>
                  </div>
                  -- <br>
                  Dominic Kaiser<br>
                  Greater Boston Vineyard<br>
                  Director of Operations<br>
                  <br>
                  cell: <a href="tel:617-230-1412" value="+16172301412" target="_blank">617-230-1412</a><br>
                  fax: <a href="tel:617-252-0238" value="+16172520238" target="_blank">617-252-0238</a><br>
                  email: <a href="mailto:dominic@bostonvineyard.org" target="_blank">dominic@bostonvineyard.org</a><br>
                  <br>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        Dominic Kaiser<br>
        Greater Boston Vineyard<br>
        Director of Operations<br>
        <br>
        cell: <a href="tel:617-230-1412" value="+16172301412" target="_blank">617-230-1412</a><br>
        fax: <a href="tel:617-252-0238" value="+16172520238" target="_blank">617-252-0238</a><br>
        email: <a href="mailto:dominic@bostonvineyard.org" target="_blank">dominic@bostonvineyard.org</a><br>
        <br>
        <br>
      </div>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br>Dominic Kaiser<br>Greater Boston Vineyard<br>Director of Operations<br><br>cell: 617-230-1412<br>fax: 617-252-0238<br>email: <a href="mailto:dominic@bostonvineyard.org">dominic@bostonvineyard.org</a><br>
<br><br>
</div>