12:02 p.m.
This is a multi-part message in MIME format.
--------------050400000307010503020608
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
On 01/28/2015 11:04 AM, Mikola Rose wrote:
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59
/Volumes/Raid1/__DIRECT_IO_TEST__
-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55
/Volumes/Raid1/data/__DIRECT_IO_TEST__
-rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55
/Volumes/Raid1/iso/__DIRECT_IO_TEST__
-rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43
/Volumes/Raid1/vm/__DIRECT_IO_TEST__
Note, since removing the file the hosted_engine1 is starting up and
everything seems fine but then all of a sudden it restarts.
please attach some logs.
Also may be related...
root@powerhost1 ~]# engine-iso-uploader --ssh-user=36 upload -i iso
rhel-server-6.6-x86_64-dvd.iso
Please provide the REST API password for the admin@internal oVirt
Engine user (CTRL+D to abort):
Uploading, please wait...
INFO: Start uploading rhel-server-6.6-x86_64-dvd.iso
ERROR: Unable to copy rhel-server-6.6-x86_64-dvd.iso to ISO storage
domain on iso.
ERROR: Error message is "unable to test the available space on
/Volumes/Raid1/iso"
Maor?
> On Jan 28, 2015, at 12:53 AM, Roy Golan <rgolan(a)redhat.com
> <mailto:rgolan@redhat.com>> wrote:
>
> On 01/28/2015 03:10 AM, Mikola Rose wrote:
>> Hi there,
>>
>>
>> I seem to have run into a problem. my hosted engine vm is failing
>> to start;
>>
>> vdsm.log;
>>
>>
>> Thread-20::DEBUG::2015-01-27
>> 16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__) Reading
>> domain in path
>>
/rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af
>> Thread-20::ERROR::2015-01-27
>> 16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain)
>> Error while collecting domain 443b4931-667f-441f-98d8-51384e67a0af
>> monitoring information
>> Traceback (most recent call last):
>> File "/usr/share/vdsm/storage/domainMonitor.py", line 221, in
>> _monitorDomain
>> self.domain = sdCache.produce(self.sdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 98, in produce
>> domain.getRealDomain()
>> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>> return self._cache._realProduce(self._sdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
>> domain = self._findDomain(sdUUID)
>> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>> dom = findMethod(sdUUID)
>> File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain
>> return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
>> File "/usr/share/vdsm/storage/fileSD.py", line 159, in __init__
>> validateFileSystemFeatures(sdUUID, self.mountpoint)
>> File "/usr/share/vdsm/storage/fileSD.py", line 88, in
>> validateFileSystemFeatures
>> oop.getProcessPool(sdUUID).directTouch(testFilePath)
>> File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in
>> directTouch
>> ioproc.touch(path, flags, mode)
>> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py",
>> line 507, in touch
>> self.timeout)
>> File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py",
>> line 391, in _sendCommand
>> raise OSError(errcode, errstr)
>> OSError: [Errno 13] Permission denied
>>
>> I assume this is an nfs issue so I checked to see if I could create
>> a file in the mounts from the host machine, which I could.
>>
>> My test bed is usign an old OS X NFS via xraid adn the export is ;
>> /Volumes/Raid1 -maproot=root:wheel -network 192.168.1.0 -mask
>> 255.255.255.0
>>
>> drwxr-xr-x 6 vdsm kvm 204 Jan 27 16:30
>> 443b4931-667f-441f-98d8-51384e67a0af
>> drwxr-xr-x 4 vdsm kvm 136 Jan 27 12:32 data
>> drwxr-xr-x 4 vdsm kvm 136 Jan 27 00:18 iso
>> drwxr-xr-x 3 vdsm kvm 102 Jan 26 09:43 vm
>>
>> the host machine seems to mount the shares fine....
>>
>> drwxr-xr-x 5 vdsm kvm 4096 Jan 27 12:28 .
>> drwxr-xr-x 3 vdsm kvm 4096 Jan 27 10:04 ..
>> drwxr-xr-x 13 vdsm kvm 544 Jan 27 00:12 192.168.1.32:_Volumes_Raid1
>> drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:12
>> 192.168.1.32:_Volumes_Raid1_data
>> drwxr-xr-x 2 vdsm kvm 4096 Jan 27 00:24 192.168.1.32:_Volumes_Raid1_iso
>>
>> and as I said above I can create files in any one of those mounts
>>
>>
>> Is there a place I can look at to find the offending file? If that
>> is the issue... Oddly enough everything worked until i rebooted, so
>> I must have either changed something or something is buggered.
>>
>
> please echo the output of
>
> find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls -la
>
> the failure is to create this file.
>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
Mik Rose | Manager, IT - Support Services | *PowerSoft Development Corp*
1 (250) 642-0295 x23 http://www.power-soft.com
<http://www.power-soft.com/> Live Support
<https://secure.logmeinrescue.com/Customer/Download.aspx?EntryID=15095831>
This e-mail may be privileged and/or confidential, and the sender does
not waive
any related rights and obligations. Any distribution, use or copying
of this e-mail or the information
it contains by other than an intended recipient is unauthorized.
If you received this e-mail in error, please advise me (by return
e-mail or otherwise) immediately.
--------------050400000307010503020608
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 01/28/2015 11:04 AM, Mikola Rose
wrote:<br>
</div>
<blockquote
cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="">
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:59
/Volumes/Raid1/__DIRECT_IO_TEST__</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 28 00:55
/Volumes/Raid1/data/__DIRECT_IO_TEST__</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">-rwxrwxrwx 1 vdsm kvm 0 Jan 28 00:55
/Volumes/Raid1/iso/__DIRECT_IO_TEST__</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">-rwxr-xr-x 1 vdsm kvm 0 Jan 26 09:43
/Volumes/Raid1/vm/__DIRECT_IO_TEST__</div>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class=""><br class="">
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">Note, since removing the file the hosted_engine1 is
starting up and everything seems fine but then all of a sudden
it restarts.</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class=""><br class="">
</div>
</blockquote>
please attach some logs. <br>
<blockquote
cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com"
type="cite">
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">Also may be related...</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class=""><br class="">
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">
<div style="margin: 0px;" class="">root@powerhost1 ~]#
engine-iso-uploader --ssh-user=36 upload -i iso
rhel-server-6.6-x86_64-dvd.iso</div>
<div style="margin: 0px;" class="">Please provide the
REST API
password for the admin@internal oVirt Engine user (CTRL+D to
abort): </div>
<div style="margin: 0px;" class="">Uploading, please
wait...</div>
<div style="margin: 0px;" class="">INFO: Start
uploading
rhel-server-6.6-x86_64-dvd.iso </div>
<div style="margin: 0px;" class="">ERROR: Unable to
copy
rhel-server-6.6-x86_64-dvd.iso to ISO storage domain on iso.</div>
<div style="margin: 0px;" class="">ERROR: Error message
is
"unable to test the available space on
/Volumes/Raid1/iso"</div>
</div>
</blockquote>
<br>
Maor? <br>
<blockquote
cite="mid:1A0F3EDC-D10D-4892-AA68-6123E5BC7451@power-soft.com"
type="cite">
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class="">
<div style="margin: 0px;" class=""><br
class="">
</div>
<div style="margin: 0px;" class=""><br
class="">
</div>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class=""><br class="">
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"
class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jan 28, 2015, at 12:53 AM, Roy Golan
<<a
moz-do-not-send="true" href="mailto:rgolan@redhat.com"
class="">rgolan(a)redhat.com</a>&gt;
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000"
class="">
<div class="moz-cite-prefix">On 01/28/2015 03:10 AM,
Mikola Rose wrote:<br class="">
</div>
<blockquote
cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com"
type="cite" class="">
Hi there,
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I seem to have run into a problem. my
hosted engine vm is failing to start;</div>
<div class=""><br class="">
</div>
<div class="">vdsm.log;</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">Thread-20::DEBUG::2015-01-27
16:53:37,999::fileSD::152::Storage.StorageDomain::(__init__)
Reading domain in path
/rhev/data-center/mnt/192.168.1.32:_Volumes_Raid1/443b4931-667f-441f-98d8-51384e67a0af</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">Thread-20::ERROR::2015-01-27
16:53:38,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain
443b4931-667f-441f-98d8-51384e67a0af monitoring
information</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">Traceback (most recent call
last):</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/domainMonitor.py", line
221, in _monitorDomain</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> self.domain =
sdCache.produce(self.sdUUID)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/sdc.py", line 98, in
produce</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">
domain.getRealDomain()</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/sdc.py", line 52, in
getRealDomain</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> return
self._cache._realProduce(self._sdUUID)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/sdc.py", line 122, in
_realProduce</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> domain =
self._findDomain(sdUUID)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/sdc.py", line 141, in
_findDomain</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> dom =
findMethod(sdUUID)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/nfsSD.py", line 122, in
findDomain</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> return
NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/fileSD.py", line 159, in
__init__</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">
validateFileSystemFeatures(sdUUID, self.mountpoint)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/fileSD.py", line 88, in
validateFileSystemFeatures</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">
oop.getProcessPool(sdUUID).directTouch(testFilePath)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/share/vdsm/storage/outOfProcess.py", line 320,
in directTouch</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> ioproc.touch(path, flags,
mode)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py",
line 507, in touch</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> self.timeout)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> File
"/usr/lib/python2.6/site-packages/ioprocess/__init__.py",
line 391, in _sendCommand</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class=""> raise OSError(errcode,
errstr)</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">OSError: [Errno 13] Permission
denied</div>
</div>
<div class=""><br class="">
</div>
<div class="">I assume this is an nfs issue so I checked
to see if I could create a file in the mounts from the
host machine, which I could.</div>
<div class=""><br class="">
</div>
<div class="">My test bed is usign an old OS X NFS via
xraid adn the export is ;</div>
<div class="">
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">/Volumes/Raid1 -maproot=root:wheel
-network 192.168.1.0 -mask 255.255.255.0</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">drwxr-xr-x 6 vdsm kvm
204
Jan 27 16:30 443b4931-667f-441f-98d8-51384e67a0af</div>
</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo;" class="">
<div style="margin: 0px;" class="">drwxr-xr-x
4
vdsm kvm 136 Jan 27 12:32 data</div>
<div style="margin: 0px;" class="">
<div style="margin: 0px;"
class="">drwxr-xr-x 4
vdsm kvm 136 Jan 27 00:18 iso</div>
<div style="margin: 0px;" class="">
<div style="margin: 0px;"
class="">drwxr-xr-x 3
vdsm kvm 102 Jan 26 09:43 vm</div>
<div style="margin: 0px;" class=""><br
class="">
</div>
<div style="margin: 0px;" class="">the
host
machine seems to mount the shares fine....</div>
<div style="margin: 0px;" class=""><br
class="">
</div>
<div style="margin: 0px;" class="">
<div style="margin: 0px;"
class="">drwxr-xr-x 5
vdsm kvm 4096 Jan 27 12:28 <span
style="font-variant-ligatures:
no-common-ligatures; color: #4c7aff"
class="">
.</span></div>
<div style="margin: 0px;"
class="">drwxr-xr-x 3
vdsm kvm 4096 Jan 27 10:04 <span
style="font-variant-ligatures:
no-common-ligatures; color: #4c7aff"
class="">
..</span></div>
<div style="margin: 0px;"
class="">drwxr-xr-x 13
vdsm kvm 544 Jan 27 00:12 <span
style="font-variant-ligatures:
no-common-ligatures; color: #4c7aff"
class="">
192.168.1.32:_Volumes_Raid1</span></div>
<div style="margin: 0px;"
class="">drwxr-xr-x 2
vdsm kvm 4096 Jan 27 00:12 <span
style="font-variant-ligatures:
no-common-ligatures; color: #4c7aff"
class="">
192.168.1.32:_Volumes_Raid1_data</span></div>
<div style="margin: 0px;"
class="">drwxr-xr-x 2
vdsm kvm 4096 Jan 27 00:24 <span
style="font-variant-ligatures:
no-common-ligatures; color: #4c7aff"
class="">
192.168.1.32:_Volumes_Raid1_iso</span></div>
</div>
<div style="margin: 0px;" class=""><br
class="">
</div>
</div>
</div>
</div>
<div class="">and as I said above I can create files in
any one of those mounts</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Is there a place I can look at to find the
offending file? If that is the issue... Oddly enough
everything worked until i rebooted, so I must have
either changed something or something is buggered.</div>
<div class=""><br class="">
</div>
</blockquote>
<br class="">
please echo the output of <br class="">
<br class="">
find /Volumes/Raid1 -name "__DIRECT_IO_TEST__" | xargs ls
-la<br class="">
<br class="">
the failure is to create this file. <br class="">
<br class="">
<blockquote
cite="mid:DDFE403B-9608-4E3C-A029-EFC39908A6C9@power-soft.com"
type="cite" class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div class="">
<table class="MsoNormalTable" style="margin-left: 1.65pt;
border-collapse: collapse;" border="0"
cellpadding="0"
cellspacing="0">
<tbody class="">
<tr style="height: 26.25pt;" class="">
<td style="orphans: 2; text-align: -webkit-auto; widows:
2; width: 315.75pt; padding: 0in 5.4pt; height:
26.25pt;" class="" valign="top"
width="421">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt;
font-family: Calibri, sans-serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica,
sans-serif; color: rgb(31, 73, 125);"
class="">Mik
Rose | Manager, IT - Support Services | </span><b
class=""><span style="font-size: 9pt;
font-family:
Arial, sans-serif; color: rgb(0, 102, 0);"
class="" lang="EN-CA">PowerSoft Development
Corp<o:p
class=""></o:p></span></b></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt;
font-family: Calibri, sans-serif;" class="">
<span style="font-size: 8pt; font-family: Helvetica,
sans-serif; color: rgb(31, 73, 125);" class="">1
(250) 642-0295 x23 <a moz-do-not-send="true"
href="http://www.power-soft.com/" style="color:
purple;" class=""><span style="color:
black;"
class="">http://www.power-soft.com</span></a> <a
moz-do-not-send="true"
href="https://secure.logmeinrescue.com/Customer/Download.aspx?EntryI...
target="_blank" style="color: purple;"
class=""><span
style="color: black;" class="">Live
Support</span></a></span><span
style="font-size: 9pt; font-family: Helvetica,
sans-serif; color: rgb(31, 73, 125);"
class=""><o:p
class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 3.75pt;
font-size: 11pt; font-family: Calibri, sans-serif;"
class="">
<span style="font-size: 9pt; font-family: Helvetica,
sans-serif; color: rgb(31, 73, 125);"
class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt 3.75pt;
font-size: 11pt; font-family: Calibri, sans-serif;"
class="">
<div style="font-size: 11pt; margin: 0in 0in
0.0001pt;" class=""><span
style="font-size: 7pt;
font-family: Arial, sans-serif; color: rgb(31,
73, 125);" class="">This e-mail may be
privileged and/or confidential, and the sender
does not waive<o:p
class=""></o:p></span></div>
<div style="font-size: 11pt; margin: 0in 0in
0.0001pt;" class=""><span
style="font-size: 7pt;
font-family: Arial, sans-serif; color: rgb(31,
73, 125);" class="">any related rights and
obligations. Any distribution, use or copying of
this e-mail or the information<o:p
class=""></o:p></span></div>
<div style="font-size: 11pt; margin: 0in 0in
0.0001pt;" class=""><span
style="font-size: 7pt;
font-family: Arial, sans-serif; color: rgb(31,
73, 125);" class="">it contains by other than
an
intended recipient is unauthorized.<o:p
class=""></o:p></span></div>
<div style="font-size: 11pt; margin: 0in 0in
0.0001pt;" class=""><span
style="font-size: 7pt;
font-family: Arial, sans-serif; color: rgb(31,
73, 125);" class="">If you received this e-mail
in error, please advise me (by return e-mail or
otherwise) immediately.</span></div>
</div>
</td>
<td style="orphans: 2; text-align: -webkit-auto; widows:
2; width: 51.75pt; padding: 0in 5.4pt; height:
26.25pt;" class="" valign="top"
width="69">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt;
font-family: Calibri, sans-serif;" class="">
<span style="color: rgb(31, 73, 125);"
class=""><br
class="Apple-interchange-newline"
style="text-align: -webkit-auto;">
</span></div>
</td>
</tr>
</tbody>
</table>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</blockquote>
<br>
</body>
</html>
--------------050400000307010503020608--