<div dir="ltr">Sorry, hit send by accident.<div><br></div><div>More details :</div><div><br></div><div>When I notice that the engine is down, if I type hosted-engine --vm-status on any hosts, it hangs and then writes a bunch of stuff saying it's down.</div><div>If I type hosted-engine --vm-start on one of the hosts (Any), it just starts and gets back to business.</div><div><br></div><div>hosted-engine --vm-status result :</div><div><br></div><div><div>ovirt_hosted_engine_ha.lib.exceptions.RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '3d67cf89-92de-428d-9714-e02aceae281e'}: Connection timed out</div></div><div><br></div><div><br></div><div>Here's some logs from vdsm.log :</div><div><br></div><div><div>Thread-98649::WARNING::2016-07-13 22:54:04,418::fileSD::749::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/engine.domain.com:_var_lib_exports_iso</div><div>Traceback (most recent call last):</div><div> File "/usr/share/vdsm/storage/fileSD.py", line 735, in collectMetaFiles</div><div> sd.DOMAIN_META_DATA))</div><div> File "/usr/share/vdsm/storage/outOfProcess.py", line 121, in glob</div><div> return self._iop.glob(pattern)</div><div> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 534, in glob</div><div> return self._sendCommand("glob", {"pattern": pattern}, self.timeout)</div><div> File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 419, in _sendCommand</div><div> raise Timeout(os.strerror(errno.ETIMEDOUT))</div><div>Timeout: Connection timed out</div><div>Thread-63::ERROR::2016-07-13 22:54:04,418::sdc::145::Storage.StorageDomainCache::(_findDomain) domain bd73cb0f-bb9c-432a-90ee-a32757a8bc10 not found</div></div><div><br></div><div><br></div><div><div>Thread-98498::ERROR::2016-07-13 22:50:33,895::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection timed out</div><div>Thread-98498::ERROR::2016-07-13 22:50:33,895::API::1871::vds::(_getHaInfo) failed to retrieve Hosted Engine HA info</div><div>Traceback (most recent call last):</div><div> File "/usr/share/vdsm/API.py", line 1851, in _getHaInfo</div><div> stats = instance.get_all_stats()</div><div> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats</div><div> self._configure_broker_conn(broker)</div><div> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn</div><div> dom_type=dom_type)</div><div> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain</div><div> .format(sd_type, options, e))</div><div>RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '3d67cf89-92de-428d-9714-e02aceae281e'}: Connection timed out</div></div><div><br></div><div><br></div><div>Thanks for your input, and even if it's a storage problem, if it's to happen, how can I force it to restart the engine?</div><div>At first I tought it was a split-brain issue so I added a 3rd host but I still have the same problem.</div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 13, 2016 at 11:13 PM, Mark Gagnon <span dir="ltr"><<a href="mailto:rhubarbe@gmail.com" target="_blank">rhubarbe@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div>We have a 3 hosted-engine nodes setup using 2 NFS3 shares on which the engine keeps crashing every few days.</div><div><br></div><div>Looking at VDSM logs, it looks like a storage problem but I'm wondering why don't they restart the engine?</div><div><br></div></div>
</blockquote></div><br></div></div>