oVirt 3.5.1 - VM "hostedengine" Failing to start
--_000_DDFE403B96084E3CA029EFC39908A6C9powersoftcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. --_000_DDFE403B96084E3CA029EFC39908A6C9powersoftcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <24D10F8AA975974E9791EA01D2A22C0E@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDom= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Dom= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Traceback (most recent call last):</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/domainMonitor.py", line 221= , in _monitorDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.domain =3D sdCache.produce(self.sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 98, in produc= e</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain.getRealDomain()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 52, in getRea= lDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return self._cache._realProduce(self._sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 122, in _real= Produce</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain =3D self._findDomain(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 141, in _find= Domain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dom =3D findMethod(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/nfsSD.py", line 122, in fin= dDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUU= ID))</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 159, in __= init__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 88, in val= idateFileSystemFeatures</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/outOfProcess.py", line 320,= in directTouch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ioproc.touch(path, flags, mode)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 507, in touch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.timeout)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 391, in _sendCommand</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= raise OSError(errcode, errstr)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= OSError: [Errno 13] Permission denied</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, which I could.</=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> Hi there, <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I seem to have run into a problem. my hosted engine v= m is failing to start;</div> <div class=3D""><br class=3D""> </div> <div class=3D"">vdsm.log;</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= div> <div class=3D""><br class=3D""> </div> <div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
/Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.25= 5.0</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= drwxr-xr-x 6 vdsm kvm 204 Jan 27 1= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 12:32 data</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 00:18 iso</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm = ; kvm 102 Jan 26 09:43 vm</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> .</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> ..</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> </div> </div> <div class=3D"">and as I said above I can create files in any one of those = mounts</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Is there a place I can look at to find the offending file? = If that is the issue... Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> </body> </html> --_000_DDFE403B96084E3CA029EFC39908A6C9powersoftcom_--
Live Support<https://secure.logmeinrescue.com/Customer/Download.aspx?En=
--_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Found the cause of this issue... the __DIRECT_IO_TEST__ on the mount was inaccessible due to permisions. removed the file and it was recreated On Jan 27, 2015, at 5:10 PM, Mikola Rose <mrose@power-soft.com<mailto:mrose= @power-soft.com>> wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831> This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <89D3C6EB8F745442B89AE7855D74CEBE@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDom= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Dom= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Traceback (most recent call last):</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/domainMonitor.py", line 221= , in _monitorDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.domain =3D sdCache.produce(self.sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 98, in produc= e</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain.getRealDomain()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 52, in getRea= lDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return self._cache._realProduce(self._sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 122, in _real= Produce</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain =3D self._findDomain(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 141, in _find= Domain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dom =3D findMethod(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/nfsSD.py", line 122, in fin= dDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUU= ID))</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 159, in __= init__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 88, in val= idateFileSystemFeatures</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/outOfProcess.py", line 320,= in directTouch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ioproc.touch(path, flags, mode)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 507, in touch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.timeout)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 391, in _sendCommand</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= raise OSError(errcode, errstr)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= OSError: [Errno 13] Permission denied</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, which I could.</=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> <div class=3D"">Found the cause of this issue...</div> <div class=3D""><br class=3D""> </div> <div class=3D"">the <font face=3D"Menlo" class=3D""><span style=3D"fon= t-size: 11px;" class=3D"">__DIRECT_IO_TEST__ on the mount was inaccess= ible due to permisions.</span></font></div> <div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D""><br class=3D""> </span></div> <div class=3D""><span style=3D"font-family: Menlo; font-size: 11px;" class= =3D"">removed the file and it was recreated</span></div> <div class=3D""><br class=3D""> </div> <br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Jan 27, 2015, at 5:10 PM, Mikola Rose <<a href=3D"mai= lto:mrose@power-soft.com" class=3D"">mrose@power-soft.com</a>> wrote:</d= iv> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> Hi there, <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I seem to have run into a problem. my hosted engine v= m is failing to start;</div> <div class=3D""><br class=3D""> </div> <div class=3D"">vdsm.log;</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= div> <div class=3D""><br class=3D""> </div> <div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
/Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.25= 5.0</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= drwxr-xr-x 6 vdsm kvm 204 Jan 27 1= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 12:32 data</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 00:18 iso</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm = ; kvm 102 Jan 26 09:43 vm</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> .</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> ..</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color= : #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> </div> </div> <div class=3D"">and as I said above I can create files in any one of those = mounts</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Is there a place I can look at to find the offending file? = If that is the issue... Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> </div> _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br class= =3D""> http://lists.ovirt.org/mailman/listinfo/users<br class=3D""> </div> </blockquote> </div> <br class=3D""> <div class=3D""> <div class=3D""> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"> <tbody class=3D""> <tr style=3D"height: 26.25pt;" class=3D""> <td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23 <a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a> = ; <a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D""> </span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div> </div> </td> <td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=3D""><br class=3D""> </div> </div> <div class=3D""><br class=3D""> </div> <br class=3D"Apple-interchange-newline"> </div> <br class=3D""> </body> </html> --_000_798D39F7986E4B24AC2233FB67954DCEpowersoftcom_--
This is a multi-part message in MIME format. --------------020803060003090906080202 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 01/28/2015 03:10 AM, Mikola Rose wrote:
Hi there,
I seem to have run into a problem. my hosted engine vm is failing to start;
vdsm.log;
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied
I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.
My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0
drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
the host machine seems to mount the shares fine....
drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
and as I said above I can create files in any one of those mounts
Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.
please echo the output of find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la the failure is to create this file.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------020803060003090906080202 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<br> </div> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> Hi there, <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">I seem to have run into a problem. my hosted engine vm is failing to start;</div> <div class=""><br class=""> </div> <div class="">vdsm.log;</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Traceback (most recent call last):</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.domain = sdCache.produce(self.sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain.getRealDomain()</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return self._cache._realProduce(self._sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain = self._findDomain(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> dom = findMethod(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> ioproc.touch(path, flags, mode)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.timeout)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> raise OSError(errcode, errstr)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">OSError: [Errno 13] Permission denied</div> </div> <div class=""><br class=""> </div> <div class="">I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.</div> <div class=""><br class=""> </div> <div class="">My test bed is usign an old OS X NFS via xraid adn the export is ;</div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">/Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0</div> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class="">the host machine seems to mount the shares fine....</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> .</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> ..</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style="margin: 0px;" class=""><br class=""> </div> </div> </div> </div> <div class="">and as I said above I can create files in any one of those mounts</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.</div> <div class=""><br class=""> </div> </blockquote> <br> please echo the output of <br> <br> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> <meta http-equiv="content-type" content="text/html; charset=windows-1252"> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la<br> <br> the failure is to create this file. <br> <br> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite"> <div class=""> </div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------020803060003090906080202--
Live Support<https://secure.logmeinrescue.com/Customer/Download.aspx?En=
--_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TE= ST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TES= T__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST= __ Note, since removing the file the hosted_engine1 is starting up and everyt= hing seems fine but then all of a sudden it restarts. Also may be related... root@powerhost1 ~]# engine-iso-uploader --ssh-user=3D36 upload -i iso rhel-= server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine us= er (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain = on iso. ERROR: Error message is "unable to test the available space on /Volumes/Rai= d1/iso" On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com<mailto:rgolan@re= dhat.com>> wrote: On 01/28/2015 03:10 AM, Mikola Rose wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. please echo the output of find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la the failure is to create this file. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831> This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <D1D444DCFA731E41BB9F92E07530435D@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__D= IRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/dat= a/__DIRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso= /__DIRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/= __DIRECT_IO_TEST__</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Note, since removing the file the hosted_engine1 is starting up and = everything seems fine but then all of a sudden it restarts.</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Also may be related...</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">root@powerhost1 ~]# engine-iso-uploa= der --ssh-user=3D36 upload -i iso rhel-server-6.6-x86_64-dvd.iso</div> <div style=3D"margin: 0px;" class=3D"">Please provide the REST API password= for the admin@internal oVirt Engine user (CTRL+D to abort): </div=
<br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div class=3D""><br class=3D""> </div> <br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Jan 28, 2015, at 12:53 AM, Roy Golan <<a href=3D"mail= to:rgolan@redhat.com" class=3D"">rgolan@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""> <div class=3D"moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<b= r class=3D""> </div> <blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""> Hi there, <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I seem to have run into a problem. my hosted engine v= m is failing to start;</div> <div class=3D""><br class=3D""> </div> <div class=3D"">vdsm.log;</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDom= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Dom= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Traceback (most recent call last):</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/domainMonitor.py", line 221= , in _monitorDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.domain =3D sdCache.produce(self.sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 98, in produc= e</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain.getRealDomain()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 52, in getRea= lDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return self._cache._realProduce(self._sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 122, in _real= Produce</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain =3D self._findDomain(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 141, in _find= Domain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dom =3D findMethod(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/nfsSD.py", line 122, in fin= dDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUU= ID))</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 159, in __= init__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 88, in val= idateFileSystemFeatures</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/outOfProcess.py", line 320,= in directTouch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ioproc.touch(path, flags, mode)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 507, in touch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.timeout)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 391, in _sendCommand</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= raise OSError(errcode, errstr)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= OSError: [Errno 13] Permission denied</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, which I could.</=
<div style=3D"margin: 0px;" class=3D"">Uploading, please wait...</div> <div style=3D"margin: 0px;" class=3D"">INFO: Start uploading rhel-server-6.= 6-x86_64-dvd.iso </div> <div style=3D"margin: 0px;" class=3D"">ERROR: Unable to copy rhel-server-6.= 6-x86_64-dvd.iso to ISO storage domain on iso.</div> <div style=3D"margin: 0px;" class=3D"">ERROR: Error message is "unable= to test the available space on /Volumes/Raid1/iso"</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= div> <div class=3D""><br class=3D""> </div> <div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
/Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.25= 5.0</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= drwxr-xr-x 6 vdsm kvm 204 Jan 27 1= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 12:32 data</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 00:18 iso</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm = ; kvm 102 Jan 26 09:43 vm</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> .</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> ..</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> </div> </div> <div class=3D"">and as I said above I can create files in any one of those = mounts</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Is there a place I can look at to find the offending file? = If that is the issue... Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div> <div class=3D""><br class=3D""> </div> </blockquote> <br class=3D""> please echo the output of <br class=3D""> <br class=3D""> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -= la<br class=3D""> <br class=3D""> the failure is to create this file. <br class=3D""> <br class=3D""> <blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""> <div class=3D""></div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <br class=3D""> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br class=3D""> <pre wrap=3D"" class=3D"">_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Users= @ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> <div class=3D""> <div class=3D""> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"> <tbody class=3D""> <tr style=3D"height: 26.25pt;" class=3D""> <td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23 <a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a> = ; <a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D""> </span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div> </div> </td> <td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=3D""><br class=3D""> </div> </div> <div class=3D""><br class=3D""> </div> <br class=3D"Apple-interchange-newline"> </div> <br class=3D""> </body> </html> --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_--
This is a multi-part message in MIME format. --------------050400000307010503020608 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"
Maor?
On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On 01/28/2015 03:10 AM, Mikola Rose wrote:
Hi there,
I seem to have run into a problem. my hosted engine vm is failing to start;
vdsm.log;
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied
I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.
My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0
drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
the host machine seems to mount the shares fine....
drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
and as I said above I can create files in any one of those mounts
Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.
please echo the output of
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
the failure is to create this file.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831> This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
--------------050400000307010503020608 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 01/28/2015 11:04 AM, Mikola Rose wrote:<br> </div> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__</div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> </blockquote> please attach some logs. <br> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Also may be related...</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class="">root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso</div> <div style="margin: 0px;" class="">Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): </div> <div style="margin: 0px;" class="">Uploading, please wait...</div> <div style="margin: 0px;" class="">INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso </div> <div style="margin: 0px;" class="">ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso.</div> <div style="margin: 0px;" class="">ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"</div> </div> </blockquote> <br> Maor? <br> <blockquote cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com" type="cite"> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class=""><br class=""> </div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""><br class=""> </div> <div class=""><br class=""> </div> <br class=""> <div> <blockquote type="cite" class=""> <div class="">On Jan 28, 2015, at 12:53 AM, Roy Golan <<a moz-do-not-send="true" href="mailto:rgolan@redhat.com" class="">rgolan@redhat.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <div bgcolor="#FFFFFF" text="#000000" class=""> <div class="moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<br class=""> </div> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite" class=""> Hi there, <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">I seem to have run into a problem. my hosted engine vm is failing to start;</div> <div class=""><br class=""> </div> <div class="">vdsm.log;</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">Traceback (most recent call last):</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.domain = sdCache.produce(self.sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain.getRealDomain()</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return self._cache._realProduce(self._sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> domain = self._findDomain(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> dom = findMethod(sdUUID)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> ioproc.touch(path, flags, mode)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> self.timeout)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> raise OSError(errcode, errstr)</div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">OSError: [Errno 13] Permission denied</div> </div> <div class=""><br class=""> </div> <div class="">I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.</div> <div class=""><br class=""> </div> <div class="">My test bed is usign an old OS X NFS via xraid adn the export is ;</div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">/Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0</div> </div> <div class=""><br class=""> </div> <div class=""> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class="">drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style="margin: 0px; font-size: 11px; font-family: Menlo;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso</div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class="">the host machine seems to mount the shares fine....</div> <div style="margin: 0px;" class=""><br class=""> </div> <div style="margin: 0px;" class=""> <div style="margin: 0px;" class="">drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> .</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> ..</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style="margin: 0px;" class="">drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 <span style="font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style="margin: 0px;" class=""><br class=""> </div> </div> </div> </div> <div class="">and as I said above I can create files in any one of those mounts</div> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <div class="">Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.</div> <div class=""><br class=""> </div> </blockquote> <br class=""> please echo the output of <br class=""> <br class=""> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la<br class=""> <br class=""> the failure is to create this file. <br class=""> <br class=""> <blockquote cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com" type="cite" class=""> <div class=""><br class=""> </div> <div class=""><br class=""> </div> <br class=""> <fieldset class="mimeAttachmentHeader"></fieldset> <br class=""> <pre class="" wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br class=""> </div> </div> </blockquote> </div> <br class=""> <div class=""> <div class=""> <table class="MsoNormalTable" style="margin-left: 1.65pt; border-collapse: collapse;" border="0" cellpadding="0" cellspacing="0"> <tbody class=""> <tr style="height: 26.25pt;" class=""> <td style="orphans: 2; text-align: -webkit-auto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" class="" valign="top" width="421"> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class="">Mik Rose | Manager, IT - Support Services | </span><b class=""><span style="font-size: 9pt; font-family: Arial, sans-serif; color: rgb(0, 102, 0);" class="" lang="EN-CA">PowerSoft Development Corp<o:p class=""></o:p></span></b></div> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 8pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class="">1 (250) 642-0295 x23 <a moz-do-not-send="true" href="http://www.power-soft.com/" style="color: purple;" class=""><span style="color: black;" class="">http://www.power-soft.com</span></a> <a moz-do-not-send="true" href="https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831" target="_blank" style="color: purple;" class=""><span style="color: black;" class="">Live Support</span></a></span><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div> <div style="margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div> <div style="margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">This e-mail may be privileged and/or confidential, and the sender does not waive<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">any related rights and obligations. Any distribution, use or copying of this e-mail or the information<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">it contains by other than an intended recipient is unauthorized.<o:p class=""></o:p></span></div> <div style="font-size: 11pt; margin: 0in 0in 0.0001pt;" class=""><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73, 125);" class="">If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.</span></div> </div> </td> <td style="orphans: 2; text-align: -webkit-auto; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class="" valign="top" width="69"> <div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <span style="color: rgb(31, 73, 125);" class=""><br class="Apple-interchange-newline" style="text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=""><br class=""> </div> </div> <div class=""><br class=""> </div> <br class="Apple-interchange-newline"> </div> <br class=""> </blockquote> <br> </body> </html> --------------050400000307010503020608--
----- Original Message -----
From: "Roy Golan" <rgolan@redhat.com> To: "Mikola Rose" <mrose@power-soft.com>, "Maor Lipchuk" <mlipchuk@redhat.com> Cc: users@ovirt.org Sent: Wednesday, January 28, 2015 12:02:50 PM Subject: Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start
On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"
Maor?
Hi, I think this is related to the engine-iso-uploader tool, Sandro, do you have any insights about this? Regards, Maor
On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On 01/28/2015 03:10 AM, Mikola Rose wrote:
Hi there,
I seem to have run into a problem. my hosted engine vm is failing to start;
vdsm.log;
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied
I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.
My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0
drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
the host machine seems to mount the shares fine....
drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
and as I said above I can create files in any one of those mounts
Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.
please echo the output of
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
the failure is to create this file.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831> This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
Il 28/01/2015 16:05, Maor Lipchuk ha scritto:
----- Original Message -----
From: "Roy Golan" <rgolan@redhat.com> To: "Mikola Rose" <mrose@power-soft.com>, "Maor Lipchuk" <mlipchuk@redhat.com> Cc: users@ovirt.org Sent: Wednesday, January 28, 2015 12:02:50 PM Subject: Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start
On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"
Maor?
Hi,
I think this is related to the engine-iso-uploader tool, Sandro, do you have any insights about this?
Please don't use UIDs like 36 as user. a command line like: ovirt-iso-uploader --ssh-user=root upload -i iso rhel-server-6.6-x86_64-dvd.iso or like ovirt-iso-uploader --ssh-user=your_user_in_kvm_group upload -i iso rhel-server-6.6-x86_64-dvd.iso should work.
Regards, Maor
On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com>> wrote:
On 01/28/2015 03:10 AM, Mikola Rose wrote:
Hi there,
I seem to have run into a problem. my hosted engine vm is failing to start;
vdsm.log;
Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied
I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could.
My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask 255.255.255.0
drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
the host machine seems to mount the shares fine....
drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
and as I said above I can create files in any one of those mounts
Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered.
please echo the output of
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
the failure is to create this file.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831> This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
--_000_BA0B24167CC842B29720515DA71C5FF5powersoftcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Sandro I was was trying both Here is the debug version... [root@powerhost1 ~]# engine-iso-uploader -v upload -i iso /tmp/rhel-server= -6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine us= er (CTRL+D to abort): DEBUG: API Vendor(ovirt.org<http://ovirt.org>) API Version(3.5.0) DEBUG: id=3D1328bbfe-f05a-4779-ac34-560b213ece11 address=3D192.168.1.32 pat= h=3D/Volumes/Raid1/iso Uploading, please wait... DEBUG: local NFS mount point is /tmp/tmpCjZhA_ DEBUG: NFS mount command (/bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/V= olumes/Raid1/iso /tmp/tmpCjZhA_) DEBUG: /bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/Volumes/Raid1/iso /t= mp/tmpCjZhA_ DEBUG: _cmds(['/bin/mount', '-t', 'nfs', '-o', 'rw,sync,soft', '192.168.1.3= 2:/Volumes/Raid1/iso', '/tmp/tmpCjZhA_']) DEBUG: returncode(32) DEBUG: STDOUT() DEBUG: STDERR(mount.nfs: Connection timed out ) ERROR: mount.nfs: Connection timed out DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_ DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_ DEBUG: _cmds(['/bin/umount', '-t', 'nfs', '-f', '/tmp/tmpCjZhA_']) DEBUG: returncode(1) DEBUG: STDOUT() DEBUG: STDERR(umount2: Invalid argument umount: /tmp/tmpCjZhA_: not mounted ) DEBUG: umount2: Invalid argument umount: /tmp/tmpCjZhA_: not mounted Could this be associated with the auto negotiation of the nfs version? Is = there a way I can specify nfs3 on the mount or on the export(os x) On Jan 28, 2015, at 6:11 AM, Sandro Bonazzola <sbonazzo@redhat.com<mailto:s= bonazzo@redhat.com>> wrote: Il 28/01/2015 16:05, Maor Lipchuk ha scritto: ----- Original Message ----- From: "Roy Golan" <rgolan@redhat.com<mailto:rgolan@redhat.com>> To: "Mikola Rose" <mrose@power-soft.com<mailto:mrose@power-soft.com>>, "Mao= r Lipchuk" <mlipchuk@redhat.com<mailto:mlipchuk@redhat.com>> Cc: users@ovirt.org<mailto:users@ovirt.org> Sent: Wednesday, January 28, 2015 12:02:50 PM Subject: Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start On 01/28/2015 11:04 AM, Mikola Rose wrote: -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__ Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts. please attach some logs. Also may be related... root@powerhost1 ~]# engine-iso-uploader --ssh-user=3D36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso" Maor? Hi, I think this is related to the engine-iso-uploader tool, Sandro, do you have any insights about this? Please don't use UIDs like 36 as user. a command line like: ovirt-iso-uploader --ssh-user=3Droot upload -i iso rhel-server-6.6-x86_64-d= vd.iso or like ovirt-iso-uploader --ssh-user=3Dyour_user_in_kvm_group upload -i iso rhel-s= erver-6.6-x86_64-dvd.iso should work. Regards, Maor On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com<mailto:rgolan@re= dhat.com> <mailto:rgolan@redhat.com>> wrote: On 01/28/2015 03:10 AM, Mikola Rose wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-5= 1384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDom= ain) Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDomain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemFeatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255.0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the issue... Oddly enough everything worked until i rebooted, so I must have either changed something or something is buggered. please echo the output of find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la the failure is to create this file. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=3D15095831=
Live Support<https://secure.logmeinrescue.com/Customer/Download.aspx?En=
This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately. -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com<http://redhat.com/> Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831> This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_BA0B24167CC842B29720515DA71C5FF5powersoftcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <FB974CA6F6E8BE4BA07611E34E3D1BA1@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
[root@powerhost1 ~]# engine-iso-uploader -v upload -i iso /tmp/rhel-= server-6.6-x86_64-dvd.iso</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Please provide the REST API password for the admin@internal oVirt Engine u= ser (CTRL+D to abort): </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: API Vendor(<a href=3D"http://ovirt.org" class=3D"">ovirt.org</a>)<s=
DEBUG: id=3D1328bbfe-f05a-4779-ac34-560b213ece11 address=3D192.168.1.32 pa=
Uploading, please wait...</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: local NFS mount point is /tmp/tmpCjZhA_</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: NFS mount command (/bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/= Volumes/Raid1/iso /tmp/tmpCjZhA_)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: /bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/Volumes/Raid1/iso /= tmp/tmpCjZhA_</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: _cmds(['/bin/mount', '-t', 'nfs', '-o', 'rw,sync,soft', '192.168.1.= 32:/Volumes/Raid1/iso', '/tmp/tmpCjZhA_'])</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: returncode(32)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: STDOUT()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: STDERR(mount.nfs: Connection timed out</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= )</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ERROR: mount.nfs: Connection timed out</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo; min-height:= 13px;" class=3D""> <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: _cmds(['/bin/umount', '-t', 'nfs', '-f', '/tmp/tmpCjZhA_'])</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: returncode(1)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: STDOUT()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: STDERR(umount2: Invalid argument</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= umount: /tmp/tmpCjZhA_: not mounted</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= )</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= DEBUG: umount2: Invalid argument</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= umount: /tmp/tmpCjZhA_: not mounted</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Could this be associated with the auto negotiation of the nfs version? &nb= sp;Is there a way I can specify nfs3 on the mount or on the export(os x)</d= iv> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div class=3D""><br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Jan 28, 2015, at 6:11 AM, Sandro Bonazzola <<a href= =3D"mailto:sbonazzo@redhat.com" class=3D"">sbonazzo@redhat.com</a>> wrot= e:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""><span style=3D"font-family: Helvetica; font-size: 12px; fon= t-style: normal; font-variant: normal; font-weight: normal; letter-spacing:= normal; line-height: normal; orphans: auto; text-align: start; text-indent= : 0px; text-transform: none; white-space: normal; widows: auto; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !impor= tant;" class=3D"">Il 28/01/2015 16:05, Maor Lipchuk ha scritto:</span><br style=3D"font-family:= Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font= -weight: normal; letter-spacing: normal; line-height: normal; orphans: auto= ; text-align: start; text-indent: 0px; text-transform: none; white-space: n= ormal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" cl= ass=3D""> <blockquote type=3D"cite" style=3D"font-family: Helvetica; font-size: 12px;= font-style: normal; font-variant: normal; font-weight: normal; letter-spac= ing: normal; line-height: normal; orphans: auto; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: auto; word-sp= acing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> ----- Original Message -----<br class=3D""> <blockquote type=3D"cite" class=3D"">From: "Roy Golan" <<a hre= f=3D"mailto:rgolan@redhat.com" class=3D"">rgolan@redhat.com</a>><br clas= s=3D""> To: "Mikola Rose" <<a href=3D"mailto:mrose@power-soft.com" cla= ss=3D"">mrose@power-soft.com</a>>, "Maor Lipchuk" <<a href= =3D"mailto:mlipchuk@redhat.com" class=3D"">mlipchuk@redhat.com</a>><br c= lass=3D""> Cc: <a href=3D"mailto:users@ovirt.org" class=3D"">users@ovirt.org</a><br cl= ass=3D""> Sent: Wednesday, January 28, 2015 12:02:50 PM<br class=3D""> Subject: Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failin= g to start<br class=3D""> <br class=3D""> On 01/28/2015 11:04 AM, Mikola Rose wrote:<br class=3D""> <blockquote type=3D"cite" class=3D"">-rwxr-xr-x 1 vdsm kv= m 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__<br class=3D""> -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55<br class=3D""> /Volumes/Raid1/data/__DIRECT_IO_TEST__<br class=3D""> -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55<br class=3D""> /Volumes/Raid1/iso/__DIRECT_IO_TEST__<br class=3D""> -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43<br class=3D""> /Volumes/Raid1/vm/__DIRECT_IO_TEST__<br class=3D""> <br class=3D""> Note, since removing the file the hosted_engine1 is starting up and<b= r class=3D""> everything seems fine but then all of a sudden it restarts.<br class=3D""> <br class=3D""> </blockquote>
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> Hi Sandro <div class=3D""><br class=3D""> </div> <div class=3D"">I was was trying both</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Here is the debug version...</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= pan class=3D"Apple-tab-span" style=3D"white-space:pre"> </span>API Version(3.5.0)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= th=3D/Volumes/Raid1/iso</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= please attach some logs.<br class=3D""> <blockquote type=3D"cite" class=3D"">Also may be related...<br class=3D""> <br class=3D""> root@powerhost1 ~]# engine-iso-uploader --ssh-user=3D36 upload -i iso<br cl= ass=3D""> rhel-server-6.6-x86_64-dvd.iso<br class=3D""> Please provide the REST API password for the admin@internal oVirt<br class= =3D""> Engine user (CTRL+D to abort):<br class=3D""> Uploading, please wait...<br class=3D""> INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso<br class=3D""> ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage<br clas= s=3D""> domain on iso.<br class=3D""> ERROR: Error message is "unable to test the available space on<br clas= s=3D""> /Volumes/Raid1/iso"<br class=3D""> </blockquote> <br class=3D""> Maor?<br class=3D""> </blockquote> <br class=3D""> <br class=3D""> Hi,<br class=3D""> <br class=3D""> I think this is related to the engine-iso-uploader tool,<br class=3D""> Sandro, do you have any insights about this?<br class=3D""> </blockquote> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">Please don't use UIDs like 36 as user.</span><br style=3D"font-family: Helvetica;= font-size: 12px; font-style: normal; font-variant: normal; font-weight: no= rmal; letter-spacing: normal; line-height: normal; orphans: auto; text-alig= n: start; text-indent: 0px; text-transform: none; white-space: normal; wido= ws: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">a command line like:</span><br style=3D"font-family: Helvetica; font-size: 1= 2px; font-style: normal; font-variant: normal; font-weight: normal; letter-= spacing: normal; line-height: normal; orphans: auto; text-align: start; tex= t-indent: 0px; text-transform: none; white-space: normal; widows: auto; wor= d-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">ovirt-iso-uploader --ssh-user=3Droot upload -i iso rhel-server-6.6-x86_64-dvd.iso</span><br s= tyle=3D"font-family: Helvetica; font-size: 12px; font-style: normal; font-v= ariant: normal; font-weight: normal; letter-spacing: normal; line-height: n= ormal; orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-st= roke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">or like</span><br style=3D"font-family: Helvetica; font-size: 12px; font-styl= e: normal; font-variant: normal; font-weight: normal; letter-spacing: norma= l; line-height: normal; orphans: auto; text-align: start; text-indent: 0px;= text-transform: none; white-space: normal; widows: auto; word-spacing: 0px= ; -webkit-text-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">ovirt-iso-uploader --ssh-user=3Dyour_user_in_kvm_group upload -i iso rhel-server-6.6-x86_64-d= vd.iso</span><br style=3D"font-family: Helvetica; font-size: 12px; font-sty= le: normal; font-variant: normal; font-weight: normal; letter-spacing: norm= al; line-height: normal; orphans: auto; text-align: start; text-indent: 0px= ; text-transform: none; white-space: normal; widows: auto; word-spacing: 0p= x; -webkit-text-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">should work.</span><br style=3D"font-family: Helvetica; font-size: 12px; font-sty= le: normal; font-variant: normal; font-weight: normal; letter-spacing: norm= al; line-height: normal; orphans: auto; text-align: start; text-indent: 0px= ; text-transform: none; white-space: normal; widows: auto; word-spacing: 0p= x; -webkit-text-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <blockquote type=3D"cite" style=3D"font-family: Helvetica; font-size: 12px;= font-style: normal; font-variant: normal; font-weight: normal; letter-spac= ing: normal; line-height: normal; orphans: auto; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: auto; word-sp= acing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> <br class=3D""> Regards,<br class=3D""> Maor<br class=3D""> <br class=3D""> <blockquote type=3D"cite" class=3D""> <blockquote type=3D"cite" class=3D""><br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <blockquote type=3D"cite" class=3D"">On Jan 28, 2015, at 12:53 AM, Roy Gola= n <<a href=3D"mailto:rgolan@redhat.com" class=3D"">rgolan@redhat.com</a>= <br class=3D""> <<a href=3D"mailto:rgolan@redhat.com" class=3D"">mailto:rgolan@redhat.co= m</a>>> wrote:<br class=3D""> <br class=3D""> On 01/28/2015 03:10 AM, Mikola Rose wrote:<br class=3D""> <blockquote type=3D"cite" class=3D"">Hi there,<br class=3D""> <br class=3D""> <br class=3D""> I seem to have run into a problem. my hosted engine vm is failing<br = class=3D""> to start;<br class=3D""> <br class=3D""> vdsm.log;<br class=3D""> <br class=3D""> <br class=3D""> Thread-20::DEBUG::2015-01-27<br class=3D""> 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading<br cla= ss=3D""> domain in path<br class=3D""> /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-5= 1384e67a0af<br class=3D""> Thread-20::ERROR::2015-01-27<br class=3D""> 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDom= ain)<br class=3D""> Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af<br class= =3D""> monitoring information<br class=3D""> Traceback (most recent call last):<br class=3D""> File "/usr/share/vdsm/storage/domainMonitor.py", line 221, = in<br class=3D""> _monitorDomain<br class=3D""> self.domain =3D sdCache.produce(self.sdUUID)<br class=3D"= "> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce<= br class=3D""> domain.getRealDomain()<br class=3D""> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealD= omain<br class=3D""> return self._cache._realProduce(self._sdUUID)<br class=3D= ""> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realPr= oduce<br class=3D""> domain =3D self._findDomain(sdUUID)<br class=3D""> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDo= main<br class=3D""> dom =3D findMethod(sdUUID)<br class=3D""> File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findD= omain<br class=3D""> return NfsStorageDomain(NfsStorageDomain.findDomainPath(s= dUUID))<br class=3D""> File "/usr/share/vdsm/storage/fileSD.py", line 159, in __in= it__<br class=3D""> validateFileSystemFeatures(sdUUID, self.mountpoint)<br class=3D""> File "/usr/share/vdsm/storage/fileSD.py", line 88, in<br cl= ass=3D""> validateFileSystemFeatures<br class=3D""> oop.getProcessPool(sdUUID).directTouch(testFilePath)<br class=3D""> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, i= n<br class=3D""> directTouch<br class=3D""> ioproc.touch(path, flags, mode)<br class=3D""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&quo= t;,<br class=3D""> line 507, in touch<br class=3D""> self.timeout)<br class=3D""> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&quo= t;,<br class=3D""> line 391, in _sendCommand<br class=3D""> raise OSError(errcode, errstr)<br class=3D""> OSError: [Errno 13] Permission denied<br class=3D""> <br class=3D""> I assume this is an nfs issue so I checked to see if I could create<br clas= s=3D""> a file in the mounts from the host machine, which I could.<br class= =3D""> <br class=3D""> My test bed is usign an old OS X NFS via xraid adn the export is ;<br class= =3D""> /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask<br class=3D= ""> 255.255.255.0<br class=3D""> <br class=3D""> drwxr-xr-x 6 vdsm kvm = ; 204 Jan 27 16:30<br class=3D""> 443b4931-667f-441f-98d8-51384e67a0af<br class=3D""> drwxr-xr-x 4 vdsm kvm = ; 136 Jan 27 12:32 data<br class=3D""> drwxr-xr-x 4 vdsm kvm = ; 136 Jan 27 00:18 iso<br class=3D""> drwxr-xr-x 3 vdsm kvm = ; 102 Jan 26 09:43 vm<br class=3D""> <br class=3D""> the host machine seems to mount the shares fine....<br class=3D""> <br class=3D""> drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 .<br class=3D""> drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 ..<br class=3D""> drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1<b= r class=3D""> drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12<br class=3D""> 192.168.1.32:_Volumes_Raid1_data<br class=3D""> drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_i= so<br class=3D""> <br class=3D""> and as I said above I can create files in any one of those mounts<br class= =3D""> <br class=3D""> <br class=3D""> Is there a place I can look at to find the offending file? If that<br= class=3D""> is the issue... Oddly enough everything worked until i rebooted, so<b= r class=3D""> I must have either changed something or something is buggered.<br class=3D"= "> <br class=3D""> </blockquote> <br class=3D""> please echo the output of<br class=3D""> <br class=3D""> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -= la<br class=3D""> <br class=3D""> the failure is to create this file.<br class=3D""> <br class=3D""> <blockquote type=3D"cite" class=3D""><br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br class= =3D""> http://lists.ovirt.org/mailman/listinfo/users<br class=3D""> </blockquote> <br class=3D""> </blockquote> <br class=3D""> Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp*<br= class=3D""> 1 (250) 642-0295 x23 <a href=3D"http://www.power-soft.com" class=3D"">http:= //www.power-soft.com</a><br class=3D""> <<a href=3D"http://www.power-soft.com/" class=3D"">http://www.power-soft= .com/</a>> Live Support<br class=3D""> <<a href=3D"https://secure.logmeinrescue.com/Customer/Download.aspx?Entr= yID=3D15095831" class=3D"">https://secure.logmeinrescue.com/Customer/Downlo= ad.aspx?EntryID=3D15095831</a>><br class=3D""> This e-mail may be privileged and/or confidential, and the sender does<br c= lass=3D""> not waive<br class=3D""> any related rights and obligations. Any distribution, use or copying<br cla= ss=3D""> of this e-mail or the information<br class=3D""> it contains by other than an intended recipient is unauthorized.<br class= =3D""> If you received this e-mail in error, please advise me (by return<br class= =3D""> e-mail or otherwise) immediately.<br class=3D""> <span class=3D"Apple-tab-span" style=3D"white-space: pre;"></span><br class= =3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> </blockquote> <br class=3D""> <br class=3D""> </blockquote> </blockquote> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; f= ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig= ht: normal; orphans: auto; text-align: start; text-indent: 0px; text-transf= orm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-te= xt-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">--<span class=3D"Apple-converted-space"> </span></span><br style=3D"= font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: = normal; font-weight: normal; letter-spacing: normal; line-height: normal; o= rphans: auto; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-wid= th: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">Sandro Bonazzola</span><br style=3D"font-family: Helvetica; font-size: 12px; font= -style: normal; font-variant: normal; font-weight: normal; letter-spacing: = normal; line-height: normal; orphans: auto; text-align: start; text-indent:= 0px; text-transform: none; white-space: normal; widows: auto; word-spacing= : 0px; -webkit-text-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">Better technology. Faster innovation. Powered by community collaboration.</span><= br style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; fo= nt-variant: normal; font-weight: normal; letter-spacing: normal; line-heigh= t: normal; orphans: auto; text-align: start; text-indent: 0px; text-transfo= rm: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-tex= t-stroke-width: 0px;" class=3D""> <span style=3D"font-family: Helvetica; font-size: 12px; font-style: normal;= font-variant: normal; font-weight: normal; letter-spacing: normal; line-he= ight: normal; orphans: auto; text-align: start; text-indent: 0px; text-tran= sform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-= text-stroke-width: 0px; float: none; display: inline !important;" class=3D"= ">See how it works at<span class=3D"Apple-converted-space"> </span></span><= a href=3D"http://redhat.com/" style=3D"font-family: Helvetica; font-size: 1= 2px; font-style: normal; font-variant: normal; font-weight: normal; letter-= spacing: normal; line-height: normal; orphans: auto; text-align: start; tex= t-indent: 0px; text-transform: none; white-space: normal; widows: auto; wor= d-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D"">redhat.com</a><= /div> </blockquote> </div> <br class=3D""> <div class=3D""> <div class=3D""> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"> <tbody class=3D""> <tr style=3D"height: 26.25pt;" class=3D""> <td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23 <a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a> = ; <a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D""> </span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div> </div> </td> <td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=3D""><br class=3D""> </div> </div> <div class=3D""><br class=3D""> </div> <br class=3D"Apple-interchange-newline"> </div> <br class=3D""> </div> </body> </html> --_000_BA0B24167CC842B29720515DA71C5FF5powersoftcom_--
Il 28/01/2015 18:39, Mikola Rose ha scritto:
Hi Sandro
I was was trying both
Here is the debug version...
[root@powerhost1 ~]# engine-iso-uploader -v upload -i iso /tmp/rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): DEBUG: API Vendor(ovirt.org <http://ovirt.org>)API Version(3.5.0) DEBUG: id=1328bbfe-f05a-4779-ac34-560b213ece11 address=192.168.1.32 path=/Volumes/Raid1/iso Uploading, please wait... DEBUG: local NFS mount point is /tmp/tmpCjZhA_ DEBUG: NFS mount command (/bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/Volumes/Raid1/iso /tmp/tmpCjZhA_) DEBUG: /bin/mount -t nfs -o rw,sync,soft 192.168.1.32:/Volumes/Raid1/iso /tmp/tmpCjZhA_ DEBUG: _cmds(['/bin/mount', '-t', 'nfs', '-o', 'rw,sync,soft', '192.168.1.32:/Volumes/Raid1/iso', '/tmp/tmpCjZhA_']) DEBUG: returncode(32) DEBUG: STDOUT() DEBUG: STDERR(mount.nfs: Connection timed out ) ERROR: mount.nfs: Connection timed out
DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_ DEBUG: /bin/umount -t nfs -f /tmp/tmpCjZhA_ DEBUG: _cmds(['/bin/umount', '-t', 'nfs', '-f', '/tmp/tmpCjZhA_']) DEBUG: returncode(1) DEBUG: STDOUT() DEBUG: STDERR(umount2: Invalid argument umount: /tmp/tmpCjZhA_: not mounted ) DEBUG: umount2: Invalid argument umount: /tmp/tmpCjZhA_: not mounted
The NFS connection timed out may depend on several factors. CCing storage guys that may assist you better than me on this.
Could this be associated with the auto negotiation of the nfs version? Is there a way I can specify nfs3 on the mount or on the export(os x)
On Jan 28, 2015, at 6:11 AM, Sandro Bonazzola <sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>> wrote:
Il 28/01/2015 16:05, Maor Lipchuk ha scritto:
----- Original Message -----
From: "Roy Golan" <rgolan@redhat.com <mailto:rgolan@redhat.com>> To: "Mikola Rose" <mrose@power-soft.com <mailto:mrose@power-soft.com>>, "Maor Lipchuk" <mlipchuk@redhat.com <mailto:mlipchuk@redhat.com>> Cc: users@ovirt.org <mailto:users@ovirt.org> Sent: Wednesday, January 28, 2015 12:02:50 PM Subject: Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start
On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TEST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso rhel-server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso. ERROR: Error message is "unable to test the available space on /Volumes/Raid1/iso"
Maor?
Hi,
I think this is related to the engine-iso-uploader tool, Sandro, do you have any insights about this?
Please don't use UIDs like 36 as user.
a command line like:
ovirt-iso-uploader --ssh-user=root upload -i iso rhel-server-6.6-x86_64-dvd.iso
or like
ovirt-iso-uploader --ssh-user=your_user_in_kvm_group upload -i iso rhel-server-6.6-x86_64-dvd.iso
should work.
Regards, Maor
On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com <mailto:rgolan@redhat.com> <mailto:rgolan@redhat.com>> wrote:
On 01/28/2015 03:10 AM, Mikola Rose wrote: > Hi there, > > > I seem to have run into a problem. my hosted engine vm is failing > to start; > > vdsm.log; > > > Thread-20::DEBUG::2015-01-27 > 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading > domain in path > /rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af > Thread-20::ERROR::2015-01-27 > 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) > Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af > monitoring information > Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in > _monitorDomain > self.domain = sdCache.produce(self.sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 98, in produce > domain.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ > validateFileSystemFeatures(sdUUID, self.mountpoint) > File "/usr/share/vdsm/storage/fileSD.py", line 88, in > validateFileSystemFeatures > oop.getProcessPool(sdUUID).directTouch(testFilePath) > File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in > directTouch > ioproc.touch(path, flags, mode) > File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", > line 507, in touch > self.timeout) > File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", > line 391, in _sendCommand > raise OSError(errcode, errstr) > OSError: [Errno 13] Permission denied > > I assume this is an nfs issue so I checked to see if I could create > a file in the mounts from the host machine, which I could. > > My test bed is usign an old OS X NFS via xraid adn the export is ; > /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask > 255.255.255.0 > > drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 > 443b4931-667f-441f-98d8-51384e67a0af > drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data > drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso > drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm > > the host machine seems to mount the shares fine.... > > drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . > drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. > drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 > drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 > 192.168.1.32:_Volumes_Raid1_data > drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso > > and as I said above I can create files in any one of those mounts > > > Is there a place I can look at to find the offending file? If that > is the issue... Oddly enough everything worked until i rebooted, so > I must have either changed something or something is buggered. >
please echo the output of
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
the failure is to create this file.
> > > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831> This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com <http://redhat.com/>
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp* 1 (250) 642-0295 x23 http://www.power-soft.com <http://www.power-soft.com/> Live Support <https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831>
This e-mail may be privileged and/or confidential, and the sender does not waive any related rights and obligations. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or otherwise) immediately.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
participants (4)
-
Maor Lipchuk -
Mikola Rose -
Roy Golan -
Sandro Bonazzola