Live Support<https://secure.logmeinrescue.com/Customer/Download.aspx?En=
--_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Found the cause of this issue... the __DIRECT_IO_TEST__ on the mount was inaccessible due to permisions. removed the file and it was recreated On Jan 27, 2015, at 5:10 PM, Mikola Rose <mrose@power-soft.com<mailto:mrose= @power-soft.com>> wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831> This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <89D3C6EB8F745442B89AE7855D74CEBE@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDom= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Dom= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Traceback (most recent call last):</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/domainMonitor.py", line 221= , in _monitorDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.domain =3D sdCache.produce(self.sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 98, in produc= e</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain.getRealDomain()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 52, in getRea= lDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return self._cache._realProduce(self._sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 122, in _real= Produce</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain =3D self._findDomain(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 141, in _find= Domain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dom =3D findMethod(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/nfsSD.py", line 122, in fin= dDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUU= ID))</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 159, in __= init__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 88, in val= idateFileSystemFeatures</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/outOfProcess.py", line 320,= in directTouch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ioproc.touch(path, flags, mode)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 507, in touch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.timeout)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 391, in _sendCommand</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= raise OSError(errcode, errstr)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= OSError: [Errno 13] Permission denied</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, which I could.</=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> <div class=3D"">Found the cause of this issue...</div> <div class=3D""><br class=3D""> </div> <div class=3D"">the <font face=3D"Menlo" class=3D""><span style=3D"fon= t-size: 11px;" class=3D"">__DIRECT_IO_TEST__ on the mount was inaccess= ible due to permisions.</span></font></div> <div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D""><br class=3D""> </span></div> <div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D"">removed the file and it was recreated</span></div> <div class=3D""><br class=3D""> </div> <br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Jan 27, 2015, at 5:10 PM, Mikola Rose <<a href=3D"mai= lto:mrose@power-soft.com" class=3D"">mrose@power-soft.com</a>> wrote:</d= iv> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> Hi there, <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I seem to have run into a problem. my hosted engine v= m is failing to start;</div> <div class=3D""><br class=3D""> </div> <div class=3D"">vdsm.log;</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= div> <div class=3D""><br class=3D""> </div> <div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
/Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.25= 5.0</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= drwxr-xr-x 6 vdsm kvm 204 Jan 27 1= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 12:32 data</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 00:18 iso</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm = ; kvm 102 Jan 26 09:43 vm</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> .</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> ..</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> </div> </div> <div class=3D"">and as I said above I can create files in any one of those = mounts</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Is there a place I can look at to find the offending file? = If that is the issue... Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> </div> _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br class= =3D""> http://lists.ovirt.org/mailman/listinfo/users<br class=3D""> </div> </blockquote> </div> <br class=3D""> <div class=3D""> <div class=3D""> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"> <tbody class=3D""> <tr style=3D"height: 26.25pt;" class=3D""> <td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23 <a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a> = ; <a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D""> </span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div> </div> </td> <td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=3D""><br class=3D""> </div> </div> <div class=3D""><br class=3D""> </div> <br class=3D"Apple-interchange-newline"> </div> <br class=3D""> </body> </html> --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_--