This is a multi-part message in MIME format. --------------050400000307010503020608 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"
Maor?
On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On 01/28/2015 03:10 AM, Mikola Rose wrote:
Hi there,
I seem to have run into a problem. my hosted engine vm is failing to start;
vdsm.log;
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied
I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.
My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0
drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
the host machine seems to mount the shares fine....
drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
and as I said above I can create files in any one of those mounts
Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.
please echo the output of
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
the failure is to create this file.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831> This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
--------------050400000307010503020608 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 01/28/2015 11:04 AM, Mikola Rose wrote:<br> </div> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__</div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> </blockquote> please attach some logs. <br> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Also may be related...</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class="">root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso</div> <div style="margin: 0px;" class="">Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): </div> <div style="margin: 0px;" class="">Uploading, please wait...</div> <div style="margin: 0px;" class="">INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso </div> <div style="margin: 0px;" class="">ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso.</div> <div style="margin: 0px;" class="">ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"</div> </div> </blockquote> <br> Maor? <br> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class=""><br class=""> </div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div class=""><br class=""> </div> <br class=""> <div> <blockquote type="cite" class=""> <div class="">On Jan 28, 2015, at 12:53 AM, Roy Golan <<a moz-do-not-send="true" href="mailto:rgolan@redhat.com" class="">rgolan@redhat.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <div bgcolor="#FFFFFF" text="#000000" class=""> <div class="moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<br class=""> </div> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite" class=""> Hi there, <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">I seem to have run into a problem. my hosted engine vm is failing to start;</div> <div class=""><br class=""> </div> <div class="">vdsm.log;</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Traceback (most recent call last):</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.domain = sdCache.produce(self.sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain.getRealDomain()</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return self._cache._realProduce(self._sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain = self._findDomain(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> dom = findMethod(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> ioproc.touch(path, flags, mode)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.timeout)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> raise OSError(errcode, errstr)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">OSError: [Errno 13] Permission denied</div> </div> <div class=""><br class=""> </div> <div class="">I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.</div> <div class=""><br class=""> </div> <div class="">My test bed is usign an old OS X NFS via xraid adn the export is ;</div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">/Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0</div> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class="">the host machine seems to mount the shares fine....</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> .</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> ..</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style="margin: 0px;" class=""><br class=""> </div> </div> </div> </div> <div class="">and as I said above I can create files in any one of those mounts</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.</div> <div class=""><br class=""> </div> </blockquote> <br class=""> please echo the output of <br class=""> <br class=""> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la<br class=""> <br class=""> the failure is to create this file. <br class=""> <br class=""> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite" class=""> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <br class=""> <fieldset class="mimeAttachmentHeader"></fieldset> <br class=""> <pre class="" wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br class=""> </div> </div> </blockquote> </div> <br class=""> <div class=""> <div class=""> <table class="MsoNormalTable" style="margin-left: 1.65pt; border-collapse: collapse;" border="0" cellpadding="0" cellspacing="0"> <tbody class=""> <tr style="height: 26.25pt;" class=""> <td style="orphans: 2; text-align: -webkit-auto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" class="" valign="top" width="421"> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class="">Mik Rose | Manager, IT - Support Services | </span><b class=""><span style="font-size: 9pt; font-family: Arial, sans-serif; color: rgb(0, 102, 0);" class="" lang="EN-CA">PowerSoft Development Corp<o:p class=""></o:p></span></b></div> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 8pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class="">1 (250) 642-0295 x23 <a moz-do-not-send="true" href="http://www.power-soft.com/" style="color: purple;" class=""><span style="color: black;" class="">http://www.power-soft.com</span></a> <a moz-do-not-send="true" href="https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831" target="_blank" style="color: purple;" class=""><span style="color: black;" class="">Live Support</span></a></span><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div> <div style="margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div> <div style="margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">This e-mail may be privileged and/or confidential, and the sender does not waive<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">any related rights and obligations. Any distribution, use or copying of this e-mail or the information<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">it contains by other than an intended recipient is unauthorized.<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.</span></div> </div> </td> <td style="orphans: 2; text-align: -webkit-auto; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class="" valign="top" width="69"> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="color: rgb(31, 73, 125);" class=""><br class="Apple-interchange-newline" style="text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=""><br class=""> </div> </div> <div class=""><br class=""> </div> <br class="Apple-interchange-newline"> </div> <br class=""> </blockquote> <br> </body> </html> --------------050400000307010503020608--