Live Support<https://secure.logmeinrescue.com/Customer/Download.aspx?En=
--_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__DIRECT_IO_TEST__ -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/data/__DIRECT_IO_TE= ST__ -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso/__DIRECT_IO_TES= T__ -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/__DIRECT_IO_TEST= __ Note, since removing the file the hosted_engine1 is starting up and everyt= hing seems fine but then all of a sudden it restarts. Also may be related... root@powerhost1 ~]# engine-iso-uploader --ssh-user=3D36 upload -i iso rhel-= server-6.6-x86_64-dvd.iso Please provide the REST API password for the admin@internal oVirt Engine us= er (CTRL+D to abort): Uploading, please wait... INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage domain = on iso. ERROR: Error message is "unable to test the available space on /Volumes/Rai= d1/iso" On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan@redhat.com<mailto:rgolan@re= dhat.com>> wrote: On 01/28/2015 03:10 AM, Mikola Rose wrote: Hi there, I seem to have run into a problem. my hosted engine vm is failing to start= ; vdsm.log; Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDoma= in::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_V= olumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Doma= inMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-66= 7f-441f-98d8-51384e67a0af monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in _monitorDom= ain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__ validateFileSystemFeatures(sdUUID, self.mountpoint) File "/usr/share/vdsm/storage/fileSD.py", line 88, in validateFileSystemF= eatures oop.getProcessPool(sdUUID).directTouch(testFilePath) File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch ioproc.touch(path, flags, mode) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 507, = in touch self.timeout) File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 391, = in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 13] Permission denied I assume this is an nfs issue so I checked to see if I could create a file = in the mounts from the host machine, which I could. My test bed is usign an old OS X NFS via xraid adn the export is ; /Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.255= .0 drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30 443b4931-667f-441f-98d8-51= 384e67a0af drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm the host machine seems to mount the shares fine.... drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 . drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 .. drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1 drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12 192.168.1.32:_Volumes_Raid1_data drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso and as I said above I can create files in any one of those mounts Is there a place I can look at to find the offending file? If that is the = issue... Oddly enough everything worked until i rebooted, so I must have e= ither changed something or something is buggered. please echo the output of find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la the failure is to create this file. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users Mik Rose | Manager, IT - Support Services | PowerSoft Development Corp 1 (250) 642-0295 x23 http://www.power-soft.com<http://www.power-soft.com/= tryID=3D15095831> This e-mail may be privileged and/or confidential, and the sender does not = waive any related rights and obligations. Any distribution, use or copying of thi= s e-mail or the information it contains by other than an intended recipient is unauthorized. If you received this e-mail in error, please advise me (by return e-mail or= otherwise) immediately. --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <D1D444DCFA731E41BB9F92E07530435D@power-soft.net> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> </head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space;" class=3D""> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59 /Volumes/Raid1/__D= IRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/dat= a/__DIRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55 /Volumes/Raid1/iso= /__DIRECT_IO_TEST__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= -rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43 /Volumes/Raid1/vm/= __DIRECT_IO_TEST__</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Note, since removing the file the hosted_engine1 is starting up and = everything seems fine but then all of a sudden it restarts.</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Also may be related...</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">root@powerhost1 ~]# engine-iso-uploa= der --ssh-user=3D36 upload -i iso rhel-server-6.6-x86_64-dvd.iso</div> <div style=3D"margin: 0px;" class=3D"">Please provide the REST API password= for the admin@internal oVirt Engine user (CTRL+D to abort): </div=
<br class=3D""> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= <br class=3D""> </div> <div class=3D""><br class=3D""> </div> <br class=3D""> <div> <blockquote type=3D"cite" class=3D""> <div class=3D"">On Jan 28, 2015, at 12:53 AM, Roy Golan <<a href=3D"mail= to:rgolan@redhat.com" class=3D"">rgolan@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""> <div class=3D"moz-cite-prefix">On 01/28/2015 03:10 AM, Mikola Rose wrote:<b= r class=3D""> </div> <blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""> Hi there, <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I seem to have run into a problem. my hosted engine v= m is failing to start;</div> <div class=3D""><br class=3D""> </div> <div class=3D"">vdsm.log;</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::DEBUG::2015-01-27 16:53:37,999::fileSD::152::Storage.StorageDom= ain::(__init__) Reading domain in path /rhev/data-center/mnt/192.168.1.32:_= Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Thread-20::ERROR::2015-01-27 16:53:38,000::domainMonitor::256::Storage.Dom= ainMonitorThread::(_monitorDomain) Error while collecting domain 443b4931-6= 67f-441f-98d8-51384e67a0af monitoring information</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= Traceback (most recent call last):</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/domainMonitor.py", line 221= , in _monitorDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.domain =3D sdCache.produce(self.sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 98, in produc= e</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain.getRealDomain()</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 52, in getRea= lDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return self._cache._realProduce(self._sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 122, in _real= Produce</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= domain =3D self._findDomain(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/sdc.py", line 141, in _find= Domain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= dom =3D findMethod(sdUUID)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/nfsSD.py", line 122, in fin= dDomain</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUU= ID))</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 159, in __= init__</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= validateFileSystemFeatures(sdUUID, self.mountpoint)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/fileSD.py", line 88, in val= idateFileSystemFeatures</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= oop.getProcessPool(sdUUID).directTouch(testFilePath)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/share/vdsm/storage/outOfProcess.py", line 320,= in directTouch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= ioproc.touch(path, flags, mode)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 507, in touch</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= self.timeout)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py&q= uot;, line 391, in _sendCommand</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= raise OSError(errcode, errstr)</div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= OSError: [Errno 13] Permission denied</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">I assume this is an nfs issue so I checked to see if I coul= d create a file in the mounts from the host machine, which I could.</=
<div style=3D"margin: 0px;" class=3D"">Uploading, please wait...</div> <div style=3D"margin: 0px;" class=3D"">INFO: Start uploading rhel-server-6.= 6-x86_64-dvd.iso </div> <div style=3D"margin: 0px;" class=3D"">ERROR: Unable to copy rhel-server-6.= 6-x86_64-dvd.iso to ISO storage domain on iso.</div> <div style=3D"margin: 0px;" class=3D"">ERROR: Error message is "unable= to test the available space on /Volumes/Raid1/iso"</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= div> <div class=3D""><br class=3D""> </div> <div class=3D"">My test bed is usign an old OS X NFS via xraid adn the expo= rt is ;</div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
/Volumes/Raid1 -maproot=3Droot:wheel -network 192.168.1.0 -mask 255.255.25= 5.0</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""= drwxr-xr-x 6 vdsm kvm 204 Jan 27 1= 6:30 443b4931-667f-441f-98d8-51384e67a0af</div> </div> <div style=3D"margin: 0px; font-size: 11px; font-family: Menlo;" class=3D""=
<div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 12:32 data</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 4 vdsm = ; kvm 136 Jan 27 00:18 iso</div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm = ; kvm 102 Jan 26 09:43 vm</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D"">the host machine seems to mount the = shares fine....</div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> <div style=3D"margin: 0px;" class=3D""> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 5 vdsm kvm 4096 Jan= 27 12:28 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> .</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 3 vdsm kvm 4096 Jan= 27 10:04 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> ..</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 13 vdsm kvm 544 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:12 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_data</span></div> <div style=3D"margin: 0px;" class=3D"">drwxr-xr-x 2 vdsm kvm 4096 Jan= 27 00:24 <span style=3D"font-variant-ligatures: no-common-ligatures; color: #4c7aff" class=3D""> 192.168.1.32:_Volumes_Raid1_iso</span></div> </div> <div style=3D"margin: 0px;" class=3D""><br class=3D""> </div> </div> </div> </div> <div class=3D"">and as I said above I can create files in any one of those = mounts</div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">Is there a place I can look at to find the offending file? = If that is the issue... Oddly enough everything worked until i = rebooted, so I must have either changed something or something is buggered.= </div> <div class=3D""><br class=3D""> </div> </blockquote> <br class=3D""> please echo the output of <br class=3D""> <br class=3D""> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -= la<br class=3D""> <br class=3D""> the failure is to create this file. <br class=3D""> <br class=3D""> <blockquote cite=3D"mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com= " type=3D"cite" class=3D""> <div class=3D""></div> <div class=3D""><br class=3D""> </div> <div class=3D""><br class=3D""> </div> <br class=3D""> <fieldset class=3D"mimeAttachmentHeader"></fieldset> <br class=3D""> <pre wrap=3D"" class=3D"">_______________________________________________ Users mailing list <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Users= @ovirt.org</a> <a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman/l= istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> <div class=3D""> <div class=3D""> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"margin-left: 1.65pt; border-collapse: collapse;"> <tbody class=3D""> <tr style=3D"height: 26.25pt;" class=3D""> <td width=3D"421" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-a= uto; widows: 2; width: 315.75pt; padding: 0in 5.4pt; height: 26.25pt;" clas= s=3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">Mik Rose | Manager, IT - Support Services |&nb= sp;</span><b class=3D""><span lang=3D"EN-CA" style=3D"font-size: 9pt; font-= family: Arial, sans-serif; color: rgb(0, 102, 0);" class=3D"">PowerSoft Development Corp<o:p class=3D""></o:p></span></b></div> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"font-size: 8pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D"">1 (250) 642-0295 x23 <a href= =3D"http://www.power-soft.com/" style=3D"color: purple;" class=3D""><span s= tyle=3D"color: black;" class=3D"">http://www.power-soft.com</span></a> = ; <a href=3D"https://secure.logmeinrescue.com/Customer/Download.= aspx?EntryID=3D15095831" target=3D"_blank" style=3D"color: purple;" class= =3D""><span style=3D"color: black;" class=3D"">Live Support</span></a></span><span style=3D"font-size: 9pt; font-family: Helve= tica, sans-serif; color: rgb(31, 73, 125);" class=3D""><o:p class=3D""></o:= p></span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <span style=3D"font-size: 9pt; font-family: Helvetica, sans-serif; color: r= gb(31, 73, 125);" class=3D""> </span></div> <div style=3D"margin: 0in 0in 0.0001pt 3.75pt; font-size: 11pt; font-family= : Calibri, sans-serif;" class=3D""> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">This e-mail may be privileged and/or confidential, and t= he sender does not waive<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">any related rights and obligations. Any distribution, us= e or copying of this e-mail or the information<o:p class=3D""></o:p></span>= </div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">it contains by other than an intended recipient is unaut= horized.<o:p class=3D""></o:p></span></div> <div style=3D"font-size: 11pt; margin: 0in 0in 0.0001pt;" class=3D""><span = style=3D"font-size: 7pt; font-family: Arial, sans-serif; color: rgb(31, 73,= 125);" class=3D"">If you received this e-mail in error, please advise me (= by return e-mail or otherwise) immediately.</span></div> </div> </td> <td width=3D"69" valign=3D"top" style=3D"orphans: 2; text-align: -webkit-au= to; widows: 2; width: 51.75pt; padding: 0in 5.4pt; height: 26.25pt;" class= =3D""> <div style=3D"margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D""> <span style=3D"color: rgb(31, 73, 125);" class=3D""><br class=3D"Apple-inte= rchange-newline" style=3D"text-align: -webkit-auto;"> </span></div> </td> </tr> </tbody> </table> <div class=3D""><br class=3D""> </div> </div> <div class=3D""><br class=3D""> </div> <br class=3D"Apple-interchange-newline"> </div> <br class=3D""> </body> </html> --_000_1A0F3EDCD10D4892AA686123E5BC7451powersoftcom_--