Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start

--_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Found the cause of this issue... the __DIRECT_IO_TEST__ on the mount was inaccessible due to permisions. removed the file and it was recreated On Jan 27, 2015, at 5:10 PM, Mikola Rose <mrose@power-soft.com<mailto:mrose= @power-soft.com>> wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831 This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <89D3C6EB8F745442B89AE7855D74CEBE(a)power-soft.netContent-Transfer-Encoding: quoted-printable <html<head<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= </head<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""<div class=3D"">Found the cause of this issue...</div<div class=3D""><br class=3D""</div<div class=3D"">the&nbsp;<font face=3D"Menlo" class=3D""><span style=3D"fon= t-size: 11px;" class=3D"">__DIRECT_IO_TEST__ on the mount was&nbsp;inaccess= ible&nbsp;due to permisions.</span></font></div<div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D""><br class=3D""</span></div<div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D"">removed the file and it was recreated</span></div<div class=3D""><br class=3D""</div<br class=3D""<div<blockquote type=3D"cite" class=3D""<div class=3D"">On Jan 27, 2015, at 5:10 PM, Mikola Rose &lt;<a href=3D"mai= lto:mrose@power-soft.com" class=3D&quot;&quot;&gt;mrose(a)power-soft.com&lt;/a&gt;&amp;gt; wrote:</d= iv<br class=3D"Apple-interchange-newline"<div class=3D""<div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""Hi there, <div class=3D""><br class=3D""</div<div class=3D""><br class=3D""</div<div class=3D"">I seem to have run into a problem. &nbsp;my hosted engine v= m is failing to start;</div<div class=3D""><br class=3D""</div<div class=3D"">vdsm.log;</div<div class=3D""><br class=3D""</div<div class=3D""><br class=3D""</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >Traceback (most recent call last):</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= , in _monitorDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; self.domain =3D sdCache.produce(self.sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= e</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; domain.getRealDomain()</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= lDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; return self._cache._realProduce(self._sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Produce</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; domain =3D self._findDomain(sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Domain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; dom =3D findMethod(sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ID))</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= init__</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; validateFileSystemFeatures(sdUUID, self.mountpoint)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= idateFileSystemFeatures</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; oop.getProcessPool(sdUUID).directTouch(testFilePath)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= in directTouch</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; ioproc.touch(path, flags, mode)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= uot;, line 507, in touch</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; self.timeout)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= uot;, line 391, in _sendCommand</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; raise OSError(errcode, errstr)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >OSError: [Errno 13] Permission denied</div</div<div class=3D""><br class=3D""</div<div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, &nbsp;which I could.</= div<div class=3D""><br class=3D""</div<div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= 5.0</div</div<div class=3D""><br class=3D""</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 4 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 136 Jan 27 12:32 data</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 4 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 136 Jan 27 00:18 iso</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 3 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 102 Jan 26 09:43 vm</div<div style=3D"margin: 0px;" class=3D""><br class=3D""</div<div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div<div style=3D"margin: 0px;" class=3D""><br class=3D""</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D"".</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""..</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm&nbsp; 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1_data</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1_iso</span></div</div<div style=3D"margin: 0px;" class=3D""><br class=3D""</div</div</div</div<div class=3D"">and as I said above I can create files in any one of those = mounts</div<div class=3D""><br class=3D""</div<div class=3D""><br class=3D""</div<div class=3D"">Is there a place I can look at to find the offending file? = &nbsp;If that is the issue... &nbsp;Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div<div class=3D""><br class=3D""</div<div class=3D""><br class=3D""</div<div class=3D""><br class=3D""</div</div_______________________________________________<br class=3D""Users mailing list<br class=3D""<a href=3D"mailto:Users@ovirt.org" class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br class= =3D""http://lists.ovirt.org/mailman/listinfo/users<br class=3D""</div</blockquote</div<br class=3D""<div class=3D""<div class=3D""<table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"<tbody class=3D""<tr style=3D"height: 26.25pt;" class=3D""<td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23&nbsp;&nbsp;&nbsp;<a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a>&nbsp= ;&nbsp;&nbsp;<a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div<div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""<span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">&nbsp;</span></div<div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div</div</td<td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"</span></div</td</tr</tbody</table<div class=3D""><br class=3D""</div</div<div class=3D""><br class=3D""</div<br class=3D"Apple-interchange-newline"</div<br class=3D""</body</html --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_--