<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 22, 2015 at 3:06 PM, Will Dennis <span dir="ltr"><<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">See attached for requested logs</span></p></div></div></blockquote><div><br></div><div><br></div><div>Thanks, the issue is here:</div><div><div>Dec 21 19:40:53 ovirt-node-03 etc-glusterfs-glusterd.vol[1079]: [2015-12-22 00:40:53.496109] C [MSGID: 106002] [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume engine. Stopping local bricks.</div><div>Dec 21 19:40:53 ovirt-node-03 etc-glusterfs-glusterd.vol[1079]: [2015-12-22 00:40:53.496410] C [MSGID: 106002] [glusterd-server-quorum.c:351:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume vmdata. Stopping local bricks.</div></div><div><br></div><div>So at that point gluster lost its quorum and the fail system got read-only.</div><div><br></div><div>On the getStorageDomainsList VDSM internally raises cause the file-system is read only:<br></div><div><br></div><div><div><div>Thread-141::DEBUG::2015-12-21 11:29:59,666::fileSD::157::Storage.StorageDomainManifest::(__init__) Reading domain in path /rhev/data-center/mnt/glusterSD/localhost:_engine/e89b6e64-bd7d-4846-b970-9af32a3295ee</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,666::__init__::320::IOProcessClient::(_run) Starting IOProcess...</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,680::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend</div><div>Thread-141::ERROR::2015-12-21 11:29:59,686::hsm::2898::Storage.HSM::(getStorageDomainsList) Unexpected error</div><div>Traceback (most recent call last):</div></div><div> File "/usr/share/vdsm/storage/hsm.py", line 2882, in getStorageDomainsList</div><div> dom = sdCache.produce(sdUUID=sdUUID)</div><div> File "/usr/share/vdsm/storage/sdc.py", line 100, in produce</div><div> domain.getRealDomain()</div><div> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain</div><div> return self._cache._realProduce(self._sdUUID)</div><div> File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce</div><div> domain = self._findDomain(sdUUID)</div><div> File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain</div><div> dom = findMethod(sdUUID)</div><div> File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain</div><div> return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))</div><div> File "/usr/share/vdsm/storage/fileSD.py", line 198, in __init__</div><div> validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint)</div><div> File "/usr/share/vdsm/storage/fileSD.py", line 93, in validateFileSystemFeatures</div><div> oop.getProcessPool(sdUUID).directTouch(testFilePath)</div><div> File "/usr/share/vdsm/storage/outOfProcess.py", line 350, in directTouch</div><div> ioproc.touch(path, flags, mode)</div><div> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in touch</div><div> self.timeout)</div><div> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 427, in _sendCommand</div><div> raise OSError(errcode, errstr)</div><div>OSError: [Errno 30] Read-only file system</div><div><br></div><div>But instead of reporting a failure to hosted-engine-setup, it reported a successfully execution where it wasn't able to find any storage domain there ( this one is a real bug, I'm going to open a bug on that, can I attach your logs there? ): <br></div><div><br></div><div><div>Thread-141::INFO::2015-12-21 11:29:59,702::logUtils::51::dispatcher::(wrapper) Run and protect: getStorageDomainsList, Return response: {'domlist': []}</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,702::task::1191::Storage.TaskManager.Task::(prepare) Task=`96a9ea03-dc13-483e-9b17-b55a759c9b44`::finished: {'domlist': []}</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,702::task::595::Storage.TaskManager.Task::(_updateState) Task=`96a9ea03-dc13-483e-9b17-b55a759c9b44`::moving from state preparing -> state finished</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,703::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,703::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}</div><div>Thread-141::DEBUG::2015-12-21 11:29:59,703::task::993::Storage.TaskManager.Task::(_decref) Task=`96a9ea03-dc13-483e-9b17-b55a759c9b44`::ref 0 aborting False</div><div>Thread-141::INFO::2015-12-21 11:29:59,704::xmlrpc::92::vds.XMLRPCServer::(_process_requests) Request handler for <a href="http://127.0.0.1:39718">127.0.0.1:39718</a> stopped</div></div></div><div><br></div><div>And so, cause VDSM doesn't report any existing storage domain, hosted-engine-setup assumes that you are going to deploy the first host and so your original issue.</div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> Simone Tiraboschi [mailto:<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 22, 2015 8:56 AM<br>
<b>To:</b> Will Dennis<br>
<b>Cc:</b> Sahina Bose; Yedidyah Bar David</span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [ovirt-users] Cannot retrieve answer file from 1st HE host when setting up 2nd host<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Dec 22, 2015 at 2:44 PM, Will Dennis <<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Which logs are needed?</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><br>
Let's start with vdsm.log and /var/log/messages<br>
Then it's quite strange that you have that amount of data in mom.log so also that one could be interesting. <u></u><u></u></p>
</div>
<div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">/var/log/vdsm</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">total 24M</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">drwxr-xr-x 3 vdsm kvm 4.0K Dec 18 20:10 .</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">drwxr-xr-x. 13 root root 4.0K Dec 20 03:15 ..</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">drwxr-xr-x 2 vdsm kvm 6 Dec 9 03:24 backup</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.5K Dec 21 11:29 connectivity.log</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 173K Dec 21 11:21 mom.log</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.0M Dec 17 10:09 mom.log.1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.0M Dec 17 04:06 mom.log.2</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.0M Dec 16 22:03 mom.log.3</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.0M Dec 16 16:00 mom.log.4</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 2.0M Dec 16 09:57 mom.log.5</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 root root 115K Dec 21 11:29 supervdsm.log</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 root root 2.7K Oct 16 11:38 upgrade.log</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-rw-r--r-- 1 vdsm kvm 13M Dec 22 08:44 vdsm.log</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> Simone Tiraboschi [mailto:<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 22, 2015 3:58 AM<br>
<b>To:</b> Will Dennis; Sahina Bose<br>
<b>Cc:</b> Yedidyah Bar David; users<br>
<b>Subject:</b> Re: [ovirt-users] Cannot retrieve answer file from 1st HE host when setting up 2nd host</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Dec 22, 2015 at 2:09 AM, Will Dennis <<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><a href="http://ur1.ca/ocstf" target="_blank">http://ur1.ca/ocstf</a><u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
2015-12-21 11:28:39 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND Please specify the full shared storage connection path to use (example: host:/path):<br>
2015-12-21 11:28:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:RECEIVE localhost:/engine<u></u><u></u></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">OK, so you are trying to deploy hosted-engine on GlusterFS in a hyper-converged way (using the same hosts for virtualization and for serving GlusterFS). Unfortunately I've to advise
you that this is not a supported configuration on oVirt 3.6 due to different open bugs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">So I'm glad you can help us testing it but I prefer to advise that today that schema is not production ready.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In your case it seams that VDSM correctly connects the GlusterFS volume seeing all the bricks<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
2015-12-21 11:28:55 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs plugin.execute:936 execute-output: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'engine', '--remote-host=localhost') stdout:<br>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><br>
<cliOutput><br>
<opRet>0</opRet><br>
<opErrno>0</opErrno><br>
<opErrstr/><br>
<volInfo><br>
<volumes><br>
<volume><br>
<name>engine</name><br>
<id>974c9da4-b236-4fc1-b26a-645f14601db8</id><br>
<status>1</status><br>
<statusStr>Started</statusStr><br>
<brickCount>6</brickCount><br>
<distCount>3</distCount> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">but then VDSM doesn't find any storage domain there:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New';color:black">otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._late_customization</span><br>
<span style="font-family:'Courier New';color:black">2015-12-21 11:29:58 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._getExistingDomain:476 _getExistingDomain</span><br>
<span style="font-family:'Courier New';color:black">2015-12-21 11:29:58 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storageServerConnection:638 connectStorageServer</span><br>
<span style="font-family:'Courier New';color:black">2015-12-21 11:29:58 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storageServerConnection:701 {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': '67ece152-dd66-444c-8d18-4249d1b8f488'}]}</span><br>
<span style="font-family:'Courier New';color:black">2015-12-21 11:29:58 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._getStorageDomainsList:595 getStorageDomainsList</span><br>
<span style="font-family:'Courier New';color:black">2015-12-21 11:29:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._getStorageDomainsList:598 {'status': {'message': 'OK', 'code': 0}, 'domlist': []}</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Can you please attach also the correspondent VDSM logs?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Adding Sahina here.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<p class="MsoNormal">On Dec 21, 2015, at 11:58 AM, Simone Tiraboschi <<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a><mailto:<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>>>
wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
On Mon, Dec 21, 2015 at 5:52 PM, Will Dennis <<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a><mailto:<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>>> wrote:<br>
<br>
However, when I went to the 3rd host and did the setup, I selected 'glusterfs' and gave the path of the engine volume, it came back and incorrectly identified it as the first host, instead of an additional host... How does setup determine that? I confirmed
that on this 3rd host that the engine volume is available and has the GUID subfolder of the hosted engine...<br>
<br>
<br>
Can you please attach a log of hosted-engine-setup also from there?<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div>