Re: [ovirt-users] oVirt 3.5.1 - VM "hostedengine" Failing to start

--_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TE= ST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TES= T__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST= __ Note, since removing the file the hosted_engine1 is starting up and everyt= hing seems fine but then all of a sudden it restarts. Also may be related... root@powerhost1 ~]# engine-iso-uploader --ssh-user=3D36 upload -i iso rhel-= server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine us= er (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain = on iso. ERROR: Error message is "unable to test the available space on /Volumes/Rai= d1/iso" On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com<mailto:rgolan@re= dhat.com>> wrote: On 01/28/2015 03:10 AM, Mikola Rose wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. please echo the output of find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la the failure is to create this file. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831 This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <D1D444DCFA731E41BB9F92E07530435D(a)power-soft.netContent-Transfer-Encoding: quoted-printable <html<head<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"</head<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= IRECT_IO_TEST__</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= a/__DIRECT_IO_TEST__</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= /__DIRECT_IO_TEST__</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= __DIRECT_IO_TEST__</div</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= </div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= everything seems fine but then all of a sudden it restarts.</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= </div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= </div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <div style=3D"margin: 0px;" class=3D"">root@powerhost1 ~]# engine-iso-uploa= der --ssh-user=3D36 upload -i iso rhel-server-6.6-x86_64-dvd.iso</div<div style=3D"margin: 0px;" class=3D"">Please provide the REST API password= for the admin@internal oVirt Engine user (CTRL&#43;D to abort):&nbsp;</div= <div style=3D"margin: 0px;" class=3D"">Uploading, please wait...</div<div style=3D"margin: 0px;" class=3D"">INFO: Start uploading rhel-server-6.= 6-x86_64-dvd.iso&nbsp;</div<div style=3D"margin: 0px;" class=3D"">ERROR: Unable to copy rhel-server-6.= 6-x86_64-dvd.iso to ISO storage domain on iso.</div<div style=3D"margin: 0px;" class=3D"">ERROR: Error message is &quot;unable= to test the available space on /Volumes/Raid1/iso&quot;</div<div style=3D"margin: 0px;" class=3D""</div<div style=3D"margin: 0px;" class=3D""</div</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= </div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= </div<div class=3D""</div<br class=3D""<div<blockquote type=3D"cite" class=3D""<div class=3D"">On Jan 28, 2015, at 12:53 AM, Roy Golan &lt;<a href=3D"mail= to:rgolan@redhat.com" class=3D&quot;&quot;&gt;rgolan(a)redhat.com&lt;/a&gt;&amp;gt; wrote:</div<br class=3D"Apple-interchange-newline"<div class=3D""<div bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""<div class=3D"moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<b= r class=3D""</div<blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""Hi there, <div class=3D""</div<div class=3D""</div<div class=3D"">I seem to have run into a problem. &nbsp;my hosted engine v= m is failing to start;</div<div class=3D""</div<div class=3D"">vdsm.log;</div<div class=3D""</div<div class=3D""</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >Traceback (most recent call last):</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= , in _monitorDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; self.domain =3D sdCache.produce(self.sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= e</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; domain.getRealDomain()</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= lDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; return self._cache._realProduce(self._sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Produce</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; domain =3D self._findDomain(sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Domain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; dom =3D findMethod(sdUUID)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dDomain</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ID))</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= init__</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; validateFileSystemFeatures(sdUUID, self.mountpoint)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= idateFileSystemFeatures</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; oop.getProcessPool(sdUUID).directTouch(testFilePath)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= in directTouch</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; ioproc.touch(path, flags, mode)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= uot;, line 507, in touch</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; self.timeout)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= uot;, line 391, in _sendCommand</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >&nbsp; &nbsp; raise OSError(errcode, errstr)</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= >OSError: [Errno 13] Permission denied</div</div<div class=3D""</div<div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, &nbsp;which I could.</= div<div class=3D""</div<div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= 5.0</div</div<div class=3D""</div<div class=3D""<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div</div<div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 4 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 136 Jan 27 12:32 data</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 4 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 136 Jan 27 00:18 iso</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; &nbsp; 3 vdsm &nbsp= ; kvm&nbsp; &nbsp; &nbsp; 102 Jan 26 09:43 vm</div<div style=3D"margin: 0px;" class=3D""</div<div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div<div style=3D"margin: 0px;" class=3D""</div<div style=3D"margin: 0px;" class=3D""<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D"".</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""..</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm&nbsp; 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1_data</span></div<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x&nbsp; 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""192.168.1.32:_Volumes_Raid1_iso</span></div</div<div style=3D"margin: 0px;" class=3D""</div</div</div</div<div class=3D"">and as I said above I can create files in any one of those = mounts</div<div class=3D""</div<div class=3D""</div<div class=3D"">Is there a place I can look at to find the offending file? = &nbsp;If that is the issue... &nbsp;Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div<div class=3D""</div</blockquote<br class=3D""please echo the output of <br class=3D""<br class=3D""find&nbsp; /Volumes/Raid1 -name &quot;__DIRECT_IO_TEST__&quot; | xargs ls -= la<br class=3D""<br class=3D""the failure is to create this file. <br class=3D""<br class=3D""<blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""<div class=3D""></div<div class=3D""</div<div class=3D""</div<br class=3D""<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br class=3D""<pre wrap=3D"" class=3D"">_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Users= @ovirt.org</a<a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a</pre</blockquote<br class=3D""</div</div</blockquote</div<br class=3D""<div class=3D""<div class=3D""<table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"<tbody class=3D""<tr style=3D"height: 26.25pt;" class=3D""<td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23&nbsp;&nbsp;&nbsp;<a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a>&nbsp= ;&nbsp;&nbsp;<a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div<div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""<span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">&nbsp;</span></div<div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div<div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div</div</td<td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""<div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""<span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"</span></div</td</tr</tbody</table<div class=3D""</div</div<div class=3D""</div<br class=3D"Apple-interchange-newline"</div<br class=3D""</body</html --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_--