<div dir="ltr"><div dir="auto">Good afternoon,<div dir="auto"><br></div><div dir="auto">I was able to get the engine to run again but there are still some issues.</div><div dir="auto"><br></div><div dir="auto">Following the steps detailed at <a href="http://www.ovirt.org/documentation/self-hosted/chap-Troubleshooting/">http://www.ovirt.org/documentation/self-hosted/chap-Troubleshooting/</a> , I was able to connect to the engine console. Form there I was able to see that the engine did not have an IP address.</div><div dir="auto"><br></div><div dir="auto">For some reason, the configuration for eth0 was gone. I re-created the /etc/sysconfig/network-scripts/ifcfg-eth0 file, restarted the network service. I had an IP address and could access the engine engine after that.</div><div dir="auto"><br></div><div>However, there are some issues still.</div><div><br></div><div>The data gluster volume is still down. I can't see any of the hosts even though I have three. Those issues force the data center to be set as non-operational. See captions below.</div><div><br></div><div>I think If I could get the data volume to mount properly, most of the issues would be resolved.</div><div><br></div><div>Any assistance would be greatly appreciated.</div><div><br></div><div>Thank you,</div><div><br></div><div>Joel</div><div><img src="cid:ii_15cc185016798edd" alt="Inline image 1" width="562" height="102"><br></div><div><img src="cid:ii_15cc1870903eb79d" alt="Inline image 2" width="562" height="65"><br></div><div><img src="cid:ii_15cc187f70ebb1a5" alt="Inline image 3" width="562" height="318"><br></div><div dir="auto"><br></div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Jun 19, 2017 10:59 AM, "Joel Diaz" <<a href="mailto:mrjoeldiaz@gmail.com" target="_blank">mrjoeldiaz@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Ok.<div dir="auto"><br></div><div dir="auto">Simone,</div><div dir="auto"><br></div><div dir="auto">Please let me know if I can provide any additional log files.</div><div dir="auto"><br></div><div dir="auto">Thanks for taking the time to look into this.</div><div dir="auto"><br></div><div dir="auto">Joel</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Jun 16, 2017 8:59 AM, "Sahina Bose" <<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I don't notice anything wrong on the gluster end.<br><br></div>Maybe Simone can help take a look at HE behaviour?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 16, 2017 at 6:14 PM, Joel Diaz <span dir="ltr"><<a href="mailto:mrjoeldiaz@gmail.com" target="_blank">mrjoeldiaz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Good morning,<div dir="auto"><br></div><div dir="auto">Info requested below.</div><div dir="auto"><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-02 ~]# hosted-engine --vm-start<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown):VM exists and its status is Up<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-02 ~]# ping engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">PING engine.example.lan (192.168.170.149) 56(84) bytes of data.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=1 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=2 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=3 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">>From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=4 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=5 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=6 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=7 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=8 Destination Host Unreachable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-02 ~]# gluster volume status engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status of volume: engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Gluster process <wbr> TCP Port RDMA Port Online Pid<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick 192.168.170.141:/gluster_brick<wbr>s/engin<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">e/engine <wbr> 49159 0 Y 1799<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick 192.168.170.143:/gluster_brick<wbr>s/engin<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">e/engine <wbr> 49159 0 Y 2900<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Self-heal Daemon on localhost N/A N/A Y 2914<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Self-heal Daemon on ovirt-hyp-01.example.lan N/A N/A Y 1854<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Task Status of Volume engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">There are no active volume tasks<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-02 ~]# gluster volume heal engine info<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick 192.168.170.141:/gluster_brick<wbr>s/engine/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status: Connected<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of entries: 0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick 192.168.170.143:/gluster_brick<wbr>s/engine/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status: Connected<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of entries: 0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick 192.168.170.147:/gluster_brick<wbr>s/engine/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status: Connected<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of entries: 0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-02 ~]# cat /var/log/glusterfs/rhev-data-c<wbr>enter-mnt-glusterSD-ovirt-hyp-<wbr>01.example.lan\:engine.log<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">[2017-06-15 13:37:02.009436] I [glusterfsd-mgmt.c:1600:mgmt_g<wbr>etspec_cbk] 0-glusterfs: No change in volfile, continuing<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Each of the three host sends out the following notifications about every 15 minutes.<u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hosted engine host: ovirt-hyp-01.example.lan changed state: EngineDown-EngineStart.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hosted engine host: ovirt-hyp-01.example.lan changed state: EngineStart-EngineStarting.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hosted engine host: ovirt-hyp-01.example.lan changed state: EngineStarting-EngineForceStop<wbr>.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hosted engine host: ovirt-hyp-01.example.lan changed state: EngineForceStop-EngineDown.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Please let me know if you need any additional information.</p><p style="font-family:sans-serif;font-size:13.696px">Thank you,</p><p style="font-family:sans-serif;font-size:13.696px">Joel</p><br></div><div dir="auto"><br></div></div><div class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395HOEnZb"><div class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395h5"><div class="gmail_extra"><br><div class="gmail_quote">On Jun 16, 2017 2:52 AM, "Sahina Bose" <<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>From the agent.log, <br>MainThread::INFO::2017-06-15 11:16:50,583::states::473::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine::(consu<wbr>me) Engine vm is running on host <a href="http://ovirt-hyp-02.reis.com" target="_blank">ovirt-hyp-02.reis.com</a> (id 2)<br><br></div>It looks like the HE VM was started successfully? Is it possible that the ovirt-engine service could not be started on the HE VM. Could you try to start the HE vm using below and then logging into the VM console.<br></div>#hosted-engine --vm-start<br><br></div>Also, please check<br></div># gluster volume status engine<br></div># gluster volume heal engine info<br><br></div>Please also check if there are errors in gluster mount logs - at /var/log/glusterfs/rhev-data-c<wbr>enter-mnt..<engine>.log<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 15, 2017 at 8:53 PM, Joel Diaz <span dir="ltr"><<a href="mailto:mrjoeldiaz@gmail.com" target="_blank">mrjoeldiaz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Sorry. I forgot to attached the requested logs in the previous email.<div dir="auto"><br></div><div dir="auto">Thanks,</div></div><div class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395m_2441799973223385412m_3956012596527723989HOEnZb"><div class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395m_2441799973223385412m_3956012596527723989h5"><div class="gmail_extra"><br><div class="gmail_quote">On Jun 15, 2017 9:38 AM, "Joel Diaz" <<a href="mailto:mrjoeldiaz@gmail.com" target="_blank">mrjoeldiaz@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395m_2441799973223385412m_3956012596527723989m_-2896857831599651719quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Good morning,<div dir="auto"><br></div><div dir="auto">Requested info below. Along with some additional info. </div><div dir="auto"><br></div><div dir="auto">You'll notice the data volume is not mounted.</div><div dir="auto"><br></div><div dir="auto">Any help in getting HE back running would be greatly appreciated.</div><div dir="auto"><br></div><div dir="auto">Thank you,</div><div dir="auto"><br></div><div dir="auto">Joel</div><div dir="auto"><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# hosted-engine --vm-status<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">--== Host 1 status ==--<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">conf_on_shared_storage <wbr> : True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status up-to-date : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hostname <wbr> : ovirt-hyp-01.example.lan<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host ID : 1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Engine status : unknown stale-data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Score <wbr> : 3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">stopped <wbr> : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Local maintenance : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">crc32 <wbr> : 5558a7d3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">local_conf_timestamp <wbr> : 20356<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host timestamp : 20341<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Extra metadata (valid at timestamp):<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_parse_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_feature_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> timestamp=20341 (Fri Jun 9 14:38:57 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> host-id=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> score=3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> vm_conf_refresh_time=20356 (Fri Jun 9 14:39:11 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> conf_on_shared_storage=True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> maintenance=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> state=EngineDown<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> stopped=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">--== Host 2 status ==--<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">conf_on_shared_storage <wbr> : True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status up-to-date : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hostname <wbr> : ovirt-hyp-02.example.lan<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host ID : 2<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Engine status : unknown stale-data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Score <wbr> : 3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">stopped <wbr> : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Local maintenance : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">crc32 <wbr> : 936d4cf3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">local_conf_timestamp <wbr> : 20351<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host timestamp : 20337<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Extra metadata (valid at timestamp):<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_parse_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_feature_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> timestamp=20337 (Fri Jun 9 14:39:03 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> host-id=2<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> score=3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> vm_conf_refresh_time=20351 (Fri Jun 9 14:39:17 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> conf_on_shared_storage=True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> maintenance=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> state=EngineDown<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> stopped=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">--== Host 3 status ==--<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">conf_on_shared_storage <wbr> : True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status up-to-date : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hostname <wbr> : ovirt-hyp-03.example.lan<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host ID : 3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Engine status : unknown stale-data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Score <wbr> : 3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">stopped <wbr> : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Local maintenance : False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">crc32 <wbr> : f646334e<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">local_conf_timestamp <wbr> : 20391<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Host timestamp : 20377<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Extra metadata (valid at timestamp):<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_parse_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> metadata_feature_version=1<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> timestamp=20377 (Fri Jun 9 14:39:37 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> host-id=3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> score=3400<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> vm_conf_refresh_time=20391 (Fri Jun 9 14:39:51 2017)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> conf_on_shared_storage=True<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> maintenance=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> state=EngineStop<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> stopped=False<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> timeout=Thu Jan 1 00:43:08 1970<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# gluster peer status<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of Peers: 2<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hostname: 192.168.170.143<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Uuid: b2b30d05-cf91-4567-92fd-022575<wbr>e082f5<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">State: Peer in Cluster (Connected)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Other names:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">10.0.0.2<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hostname: 192.168.170.147<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Uuid: 4e50acc4-f3cb-422d-b499-fb5796<wbr>a53529<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">State: Peer in Cluster (Connected)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Other names:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">10.0.0.3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# gluster volume info all<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Volume Name: data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Type: Replicate<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Volume ID: 1d6bb110-9be4-4630-ae91-36ec1c<wbr>f6cc02<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status: Started<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Snapshot Count: 0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of Bricks: 1 x (2 + 1) = 3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Transport-type: tcp<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Bricks:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick1: 192.168.170.141:/gluster_brick<wbr>s/data/data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick2: 192.168.170.143:/gluster_brick<wbr>s/data/data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick3: 192.168.170.147:/gluster_brick<wbr>s/data/data (arbiter)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Options Reconfigured:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">nfs.disable: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.readdir-ahead: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">transport.address-family: inet<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.quick-read: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.read-ahead: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.io-cache: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.stat-prefetch: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.low-prio-threads: 32<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">network.remote-dio: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.eager-lock: enable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.quorum-type: auto<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.server-quorum-type: server<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.data-self-heal-algorit<wbr>hm: full<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.locking-scheme: granular<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.shd-max-threads: 8<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.shd-wait-qlength: 10000<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">features.shard: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">user.cifs: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">storage.owner-uid: 36<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">storage.owner-gid: 36<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">network.ping-timeout: 30<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.strict-o-direct: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.granular-entry-heal: enable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Volume Name: engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Type: Replicate<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Volume ID: b160f0b2-8bd3-4ff2-a07c-134cab<wbr>1519dd<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Status: Started<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Snapshot Count: 0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Number of Bricks: 1 x (2 + 1) = 3<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Transport-type: tcp<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Bricks:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick1: 192.168.170.141:/gluster_brick<wbr>s/engine/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick2: 192.168.170.143:/gluster_brick<wbr>s/engine/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Brick3: 192.168.170.147:/gluster_brick<wbr>s/engine/engine (arbiter)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Options Reconfigured:<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">nfs.disable: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.readdir-ahead: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">transport.address-family: inet<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.quick-read: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.read-ahead: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.io-cache: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.stat-prefetch: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.low-prio-threads: 32<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">network.remote-dio: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.eager-lock: enable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.quorum-type: auto<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.server-quorum-type: server<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.data-self-heal-algorit<wbr>hm: full<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.locking-scheme: granular<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.shd-max-threads: 8<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.shd-wait-qlength: 10000<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">features.shard: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">user.cifs: off<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">storage.owner-uid: 36<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">storage.owner-gid: 36<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">network.ping-timeout: 30<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">performance.strict-o-direct: on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">cluster.granular-entry-heal: enable<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# df -h<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Filesystem <wbr> Size Used Avail Use% Mounted on<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">/dev/mapper/centos_ovirt--hyp-<wbr>-01-root 50G 4.1G 46G 9% /<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">devtmpfs <wbr> 7.7G 0 7.7G 0% /dev<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">tmpfs <wbr> 7.8G 0 7.8G 0% /dev/shm<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">tmpfs <wbr> 7.8G 8.7M 7.7G 1% /run<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">/dev/mapper/centos_ovirt--hyp-<wbr>-01-home 61G 33M 61G 1% /home<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">/dev/mapper/gluster_vg_sdb-glu<wbr>ster_lv_engine 50G 7.6G 43G 16% /gluster_bricks/engine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">/dev/mapper/gluster_vg_sdb-glu<wbr>ster_lv_data 730G 157G 574G 22% /gluster_bricks/data<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">/dev/sda1 <wbr> 497M 173M 325M 35% /boot<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">ovirt-hyp-01.example.lan:engin<wbr>e 50G 7.6G 43G 16% /rhev/data-center/mnt/glusterS<wbr>D/ovirt-hyp-01.example.lan:eng<wbr>ine<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">tmpfs <wbr> 1.6G 0 1.6G 0% /run/user/0<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# systemctl list-unit-files|grep ovirt<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">ovirt-ha-agent.service <wbr> enabled<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">ovirt-ha-broker.service <wbr> enabled<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">ovirt-imageio-daemon.service <wbr> disabled<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">ovirt-vmconsole-host-sshd.serv<wbr>ice enabled<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ~]# systemctl status ovirt-ha-agent.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Loaded: loaded (/usr/lib/systemd/system/ovirt<wbr>-ha-agent.service; enabled; vendor preset: disabled)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Active: active (running) since Thu 2017-06-15 08:56:15 EDT; 21min ago<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Main PID: 3150 (ovirt-ha-agent)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> CGroup: /system.slice/ovirt-ha-agent.s<wbr>ervice<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> └─3150 /usr/bin/python /usr/share/ovirt-hosted-engine<wbr>-ha/ovirt-ha-agent --no-daemon<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Started oVirt Hosted Engine High Availability Monitoring Agent.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt Hosted Engine High Availability Monitoring Agent...<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 09:17:18 ovirt-hyp-01.example.lan ovirt-ha-agent[3150]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.h<wbr>osted_engine.HostedEngine ERROR Engine VM stopped on localhost<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Loaded: loaded (/usr/lib/systemd/system/ovirt<wbr>-ha-broker.service; enabled; vendor preset: disabled)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Active: active (running) since Thu 2017-06-15 08:54:06 EDT; 24min ago<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Main PID: 968 (ovirt-ha-broker)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> CGroup: /system.slice/ovirt-ha-broker.<wbr>service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> └─968 /usr/bin/python /usr/share/ovirt-hosted-engine<wbr>-ha/ovirt-ha-broker --no-daemon<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker...<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 08:56:16 ovirt-hyp-01.example.lan ovirt-ha-broker[968]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.<wbr>listener.ConnectionHandler ERROR Error handling request, data: '...1b55bcf76'<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> <wbr> Traceback (most recent call last):<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> <wbr> <wbr> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirt...<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Hint: Some lines were ellipsized, use -l to show in full.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-agent.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-agent.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Loaded: loaded (/usr/lib/systemd/system/ovirt<wbr>-ha-agent.service; enabled; vendor preset: disabled)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Active: active (running) since Thu 2017-06-15 09:19:21 EDT; 26s ago<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Main PID: 8563 (ovirt-ha-agent)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> CGroup: /system.slice/ovirt-ha-agent.s<wbr>ervice<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> └─8563 /usr/bin/python /usr/share/ovirt-hosted-engine<wbr>-ha/ovirt-ha-agent --no-daemon<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Started oVirt Hosted Engine High Availability Monitoring Agent.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt Hosted Engine High Availability Monitoring Agent...<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-broker.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">[root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Loaded: loaded (/usr/lib/systemd/system/ovirt<wbr>-ha-broker.service; enabled; vendor preset: disabled)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> Active: active (running) since Thu 2017-06-15 09:20:59 EDT; 28s ago<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Main PID: 8844 (ovirt-ha-broker)<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> CGroup: /system.slice/ovirt-ha-broker.<wbr>service<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"> └─8844 /usr/bin/python /usr/share/ovirt-hosted-engine<wbr>-ha/ovirt-ha-broker --no-daemon<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px"><u></u> <u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker.<u></u><u></u></p><p style="font-family:sans-serif;font-size:13.696px">Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker...</p></div><div dir="auto"><br></div></div><div class="gmail-m_-665670002394786067m_3667191607201549233m_-6368583041289807395m_2441799973223385412m_3956012596527723989m_-2896857831599651719elided-text"><div class="gmail_extra"><br><div class="gmail_quote">On Jun 14, 2017 4:45 AM, "Sahina Bose" <<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>What's the output of "hosted-engine --vm-status" and "gluster volume status engine" tell you? Are all the bricks running as per gluster vol status?<br><br></div>Can you try to restart the ovirt-ha-agent and ovirt-ha-broker services?<br><br></div>If HE still has issues powering up, please provide agent.log and broker.log from /var/log/ovirt-hosted-engine-h<wbr>a and gluster mount logs from /var/log/glusterfs/rhev-data-c<wbr>enter-mnt <engine>.log<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 8, 2017 at 6:57 PM, Joel Diaz <span dir="ltr"><<a href="mailto:mrjoeldiaz@gmail.com" target="_blank">mrjoeldiaz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Good morning oVirt community,<div dir="auto"><br></div><div dir="auto">I'm running a three host gluster environment with hosted engine.</div><div dir="auto"><br></div><div dir="auto">Yesterday the engine went down and has not been able to come up properly. It tries to start on all three host.</div><div dir="auto"><br></div><div dir="auto">I have two gluster volumes, data and engne. The data storage domian volume is no longer mounted but the engine volume is up. I've restarted the gluster service and make sure both volumes were running. The data volume will not mount.</div><div dir="auto"><br></div><div dir="auto">How can I get the engine running properly again?</div><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto"><br></div><div dir="auto">Joel</div></div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div>
</blockquote></div></div>
</div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div></div>
</div></div></blockquote></div><br></div>
</blockquote></div></div>
</blockquote></div></div>
</div>