Re: [ovirt-users] Hosted engine

Ok. Simone, Please let me know if I can provide any additional log files. Thanks for taking the time to look into this. Joel On Jun 16, 2017 8:59 AM, "Sahina Bose" <sabose(a)redhat.com&gt; wrote: > I don't notice anything wrong on the gluster end. > > Maybe Simone can help take a look at HE behaviour? > > On Fri, Jun 16, 2017 at 6:14 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; wrote: > >> Good morning, >> >> Info requested below. >> >> [root@ovirt-hyp-02 ~]# hosted-engine --vm-start >> >> Exception in thread Client localhost:54321 (most likely raised during >> interpreter shutdown):VM exists and its status is Up >> >> >> >> [root@ovirt-hyp-02 ~]# ping engine >> >> PING engine.example.lan (192.168.170.149) 56(84) bytes of data. >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=1 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=2 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=3 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=4 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=5 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=6 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=7 Destination >> Host Unreachable >> >> From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=8 Destination >> Host Unreachable >> >> >> >> >> >> [root@ovirt-hyp-02 ~]# gluster volume status engine >> >> Status of volume: engine >> >> Gluster process TCP Port RDMA Port >> Online Pid >> >> ------------------------------------------------------------ >> ------------------ >> >> Brick 192.168.170.141:/gluster_bricks/engin >> >> e/engine 49159 0 >> Y 1799 >> >> Brick 192.168.170.143:/gluster_bricks/engin >> >> e/engine 49159 0 >> Y 2900 >> >> Self-heal Daemon on localhost N/A N/A Y >> 2914 >> >> Self-heal Daemon on ovirt-hyp-01.example.lan N/A N/A >> Y 1854 >> >> >> >> Task Status of Volume engine >> >> ------------------------------------------------------------ >> ------------------ >> >> There are no active volume tasks >> >> >> >> [root@ovirt-hyp-02 ~]# gluster volume heal engine info >> >> Brick 192.168.170.141:/gluster_bricks/engine/engine >> >> Status: Connected >> >> Number of entries: 0 >> >> >> >> Brick 192.168.170.143:/gluster_bricks/engine/engine >> >> Status: Connected >> >> Number of entries: 0 >> >> >> >> Brick 192.168.170.147:/gluster_bricks/engine/engine >> >> Status: Connected >> >> Number of entries: 0 >> >> >> >> [root@ovirt-hyp-02 ~]# cat /var/log/glusterfs/rhev-data-c >> enter-mnt-glusterSD-ovirt-hyp-01.example.lan\:engine.log >> >> [2017-06-15 13:37:02.009436] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] >> 0-glusterfs: No change in volfile, continuing >> >> >> >> >> >> Each of the three host sends out the following notifications about every >> 15 minutes. >> >> Hosted engine host: ovirt-hyp-01.example.lan changed state: >> EngineDown-EngineStart. >> >> Hosted engine host: ovirt-hyp-01.example.lan changed state: >> EngineStart-EngineStarting. >> >> Hosted engine host: ovirt-hyp-01.example.lan changed state: >> EngineStarting-EngineForceStop. >> >> Hosted engine host: ovirt-hyp-01.example.lan changed state: >> EngineForceStop-EngineDown. >> >> Please let me know if you need any additional information. >> >> Thank you, >> >> Joel >> >> >> >> On Jun 16, 2017 2:52 AM, "Sahina Bose" <sabose(a)redhat.com&gt; wrote: >> >>> From the agent.log, >>> MainThread::INFO::2017-06-15 11:16:50,583::states::473::ovi >>> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine >>> vm is running on host ovirt-hyp-02.reis.com (id 2) >>> >>> It looks like the HE VM was started successfully? Is it possible that >>> the ovirt-engine service could not be started on the HE VM. Could you try >>> to start the HE vm using below and then logging into the VM console. >>> #hosted-engine --vm-start >>> >>> Also, please check >>> # gluster volume status engine >>> # gluster volume heal engine info >>> >>> Please also check if there are errors in gluster mount logs - at >>> /var/log/glusterfs/rhev-data-center-mnt..<engine>.log >>> >>> >>> On Thu, Jun 15, 2017 at 8:53 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; >>> wrote: >>> >>>> Sorry. I forgot to attached the requested logs in the previous email. >>>> >>>> Thanks, >>>> >>>> On Jun 15, 2017 9:38 AM, "Joel Diaz" <mrjoeldiaz(a)gmail.com&gt; wrote: >>>> >>>> Good morning, >>>> >>>> Requested info below. Along with some additional info. >>>> >>>> You'll notice the data volume is not mounted. >>>> >>>> Any help in getting HE back running would be greatly appreciated. >>>> >>>> Thank you, >>>> >>>> Joel >>>> >>>> [root@ovirt-hyp-01 ~]# hosted-engine --vm-status >>>> >>>> >>>> >>>> >>>> >>>> --== Host 1 status ==-- >>>> >>>> >>>> >>>> conf_on_shared_storage : True >>>> >>>> Status up-to-date : False >>>> >>>> Hostname : ovirt-hyp-01.example.lan >>>> >>>> Host ID : 1 >>>> >>>> Engine status : unknown stale-data >>>> >>>> Score : 3400 >>>> >>>> stopped : False >>>> >>>> Local maintenance : False >>>> >>>> crc32 : 5558a7d3 >>>> >>>> local_conf_timestamp : 20356 >>>> >>>> Host timestamp : 20341 >>>> >>>> Extra metadata (valid at timestamp): >>>> >>>> metadata_parse_version=1 >>>> >>>> metadata_feature_version=1 >>>> >>>> timestamp=20341 (Fri Jun 9 14:38:57 2017) >>>> >>>> host-id=1 >>>> >>>> score=3400 >>>> >>>> vm_conf_refresh_time=20356 (Fri Jun 9 14:39:11 2017) >>>> >>>> conf_on_shared_storage=True >>>> >>>> maintenance=False >>>> >>>> state=EngineDown >>>> >>>> stopped=False >>>> >>>> >>>> >>>> >>>> >>>> --== Host 2 status ==-- >>>> >>>> >>>> >>>> conf_on_shared_storage : True >>>> >>>> Status up-to-date : False >>>> >>>> Hostname : ovirt-hyp-02.example.lan >>>> >>>> Host ID : 2 >>>> >>>> Engine status : unknown stale-data >>>> >>>> Score : 3400 >>>> >>>> stopped : False >>>> >>>> Local maintenance : False >>>> >>>> crc32 : 936d4cf3 >>>> >>>> local_conf_timestamp : 20351 >>>> >>>> Host timestamp : 20337 >>>> >>>> Extra metadata (valid at timestamp): >>>> >>>> metadata_parse_version=1 >>>> >>>> metadata_feature_version=1 >>>> >>>> timestamp=20337 (Fri Jun 9 14:39:03 2017) >>>> >>>> host-id=2 >>>> >>>> score=3400 >>>> >>>> vm_conf_refresh_time=20351 (Fri Jun 9 14:39:17 2017) >>>> >>>> conf_on_shared_storage=True >>>> >>>> maintenance=False >>>> >>>> state=EngineDown >>>> >>>> stopped=False >>>> >>>> >>>> >>>> >>>> >>>> --== Host 3 status ==-- >>>> >>>> >>>> >>>> conf_on_shared_storage : True >>>> >>>> Status up-to-date : False >>>> >>>> Hostname : ovirt-hyp-03.example.lan >>>> >>>> Host ID : 3 >>>> >>>> Engine status : unknown stale-data >>>> >>>> Score : 3400 >>>> >>>> stopped : False >>>> >>>> Local maintenance : False >>>> >>>> crc32 : f646334e >>>> >>>> local_conf_timestamp : 20391 >>>> >>>> Host timestamp : 20377 >>>> >>>> Extra metadata (valid at timestamp): >>>> >>>> metadata_parse_version=1 >>>> >>>> metadata_feature_version=1 >>>> >>>> timestamp=20377 (Fri Jun 9 14:39:37 2017) >>>> >>>> host-id=3 >>>> >>>> score=3400 >>>> >>>> vm_conf_refresh_time=20391 (Fri Jun 9 14:39:51 2017) >>>> >>>> conf_on_shared_storage=True >>>> >>>> maintenance=False >>>> >>>> state=EngineStop >>>> >>>> stopped=False >>>> >>>> timeout=Thu Jan 1 00:43:08 1970 >>>> >>>> >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ~]# gluster peer status >>>> >>>> Number of Peers: 2 >>>> >>>> >>>> >>>> Hostname: 192.168.170.143 >>>> >>>> Uuid: b2b30d05-cf91-4567-92fd-022575e082f5 >>>> >>>> State: Peer in Cluster (Connected) >>>> >>>> Other names: >>>> >>>> 10.0.0.2 >>>> >>>> >>>> >>>> Hostname: 192.168.170.147 >>>> >>>> Uuid: 4e50acc4-f3cb-422d-b499-fb5796a53529 >>>> >>>> State: Peer in Cluster (Connected) >>>> >>>> Other names: >>>> >>>> 10.0.0.3 >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ~]# gluster volume info all >>>> >>>> >>>> >>>> Volume Name: data >>>> >>>> Type: Replicate >>>> >>>> Volume ID: 1d6bb110-9be4-4630-ae91-36ec1cf6cc02 >>>> >>>> Status: Started >>>> >>>> Snapshot Count: 0 >>>> >>>> Number of Bricks: 1 x (2 + 1) = 3 >>>> >>>> Transport-type: tcp >>>> >>>> Bricks: >>>> >>>> Brick1: 192.168.170.141:/gluster_bricks/data/data >>>> >>>> Brick2: 192.168.170.143:/gluster_bricks/data/data >>>> >>>> Brick3: 192.168.170.147:/gluster_bricks/data/data (arbiter) >>>> >>>> Options Reconfigured: >>>> >>>> nfs.disable: on >>>> >>>> performance.readdir-ahead: on >>>> >>>> transport.address-family: inet >>>> >>>> performance.quick-read: off >>>> >>>> performance.read-ahead: off >>>> >>>> performance.io-cache: off >>>> >>>> performance.stat-prefetch: off >>>> >>>> performance.low-prio-threads: 32 >>>> >>>> network.remote-dio: off >>>> >>>> cluster.eager-lock: enable >>>> >>>> cluster.quorum-type: auto >>>> >>>> cluster.server-quorum-type: server >>>> >>>> cluster.data-self-heal-algorithm: full >>>> >>>> cluster.locking-scheme: granular >>>> >>>> cluster.shd-max-threads: 8 >>>> >>>> cluster.shd-wait-qlength: 10000 >>>> >>>> features.shard: on >>>> >>>> user.cifs: off >>>> >>>> storage.owner-uid: 36 >>>> >>>> storage.owner-gid: 36 >>>> >>>> network.ping-timeout: 30 >>>> >>>> performance.strict-o-direct: on >>>> >>>> cluster.granular-entry-heal: enable >>>> >>>> >>>> >>>> Volume Name: engine >>>> >>>> Type: Replicate >>>> >>>> Volume ID: b160f0b2-8bd3-4ff2-a07c-134cab1519dd >>>> >>>> Status: Started >>>> >>>> Snapshot Count: 0 >>>> >>>> Number of Bricks: 1 x (2 + 1) = 3 >>>> >>>> Transport-type: tcp >>>> >>>> Bricks: >>>> >>>> Brick1: 192.168.170.141:/gluster_bricks/engine/engine >>>> >>>> Brick2: 192.168.170.143:/gluster_bricks/engine/engine >>>> >>>> Brick3: 192.168.170.147:/gluster_bricks/engine/engine (arbiter) >>>> >>>> Options Reconfigured: >>>> >>>> nfs.disable: on >>>> >>>> performance.readdir-ahead: on >>>> >>>> transport.address-family: inet >>>> >>>> performance.quick-read: off >>>> >>>> performance.read-ahead: off >>>> >>>> performance.io-cache: off >>>> >>>> performance.stat-prefetch: off >>>> >>>> performance.low-prio-threads: 32 >>>> >>>> network.remote-dio: off >>>> >>>> cluster.eager-lock: enable >>>> >>>> cluster.quorum-type: auto >>>> >>>> cluster.server-quorum-type: server >>>> >>>> cluster.data-self-heal-algorithm: full >>>> >>>> cluster.locking-scheme: granular >>>> >>>> cluster.shd-max-threads: 8 >>>> >>>> cluster.shd-wait-qlength: 10000 >>>> >>>> features.shard: on >>>> >>>> user.cifs: off >>>> >>>> storage.owner-uid: 36 >>>> >>>> storage.owner-gid: 36 >>>> >>>> network.ping-timeout: 30 >>>> >>>> performance.strict-o-direct: on >>>> >>>> cluster.granular-entry-heal: enable >>>> >>>> >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ~]# df -h >>>> >>>> Filesystem Size Used Avail Use% >>>> Mounted on >>>> >>>> /dev/mapper/centos_ovirt--hyp--01-root 50G 4.1G 46G 9% / >>>> >>>> devtmpfs 7.7G 0 7.7G 0% >>>> /dev >>>> >>>> tmpfs 7.8G 0 7.8G 0% >>>> /dev/shm >>>> >>>> tmpfs 7.8G 8.7M 7.7G 1% >>>> /run >>>> >>>> tmpfs 7.8G 0 7.8G 0% >>>> /sys/fs/cgroup >>>> >>>> /dev/mapper/centos_ovirt--hyp--01-home 61G 33M 61G 1% >>>> /home >>>> >>>> /dev/mapper/gluster_vg_sdb-gluster_lv_engine 50G 7.6G 43G 16% >>>> /gluster_bricks/engine >>>> >>>> /dev/mapper/gluster_vg_sdb-gluster_lv_data 730G 157G 574G 22% >>>> /gluster_bricks/data >>>> >>>> /dev/sda1 497M 173M 325M 35% >>>> /boot >>>> >>>> ovirt-hyp-01.example.lan:engine 50G 7.6G 43G >>>> 16% /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.example.lan:engine >>>> >>>> tmpfs 1.6G 0 1.6G 0% >>>> /run/user/0 >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ~]# systemctl list-unit-files|grep ovirt >>>> >>>> ovirt-ha-agent.service enabled >>>> >>>> ovirt-ha-broker.service enabled >>>> >>>> ovirt-imageio-daemon.service disabled >>>> >>>> ovirt-vmconsole-host-sshd.service enabled >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ~]# systemctl status ovirt-ha-agent.service >>>> >>>> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability >>>> Monitoring Agent >>>> >>>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; >>>> enabled; vendor preset: disabled) >>>> >>>> Active: active (running) since Thu 2017-06-15 08:56:15 EDT; 21min >>>> ago >>>> >>>> Main PID: 3150 (ovirt-ha-agent) >>>> >>>> CGroup: /system.slice/ovirt-ha-agent.service >>>> >>>> └─3150 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent >>>> --no-daemon >>>> >>>> >>>> >>>> Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>>> Hosted Engine High Availability Monitoring Agent. >>>> >>>> Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>>> Hosted Engine High Availability Monitoring Agent... >>>> >>>> Jun 15 09:17:18 ovirt-hyp-01.example.lan ovirt-ha-agent[3150]: >>>> ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine >>>> ERROR Engine VM stopped on localhost >>>> >>>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service >>>> >>>> ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability >>>> Communications Broker >>>> >>>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; >>>> enabled; vendor preset: disabled) >>>> >>>> Active: active (running) since Thu 2017-06-15 08:54:06 EDT; 24min >>>> ago >>>> >>>> Main PID: 968 (ovirt-ha-broker) >>>> >>>> CGroup: /system.slice/ovirt-ha-broker.service >>>> >>>> └─968 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>>> --no-daemon >>>> >>>> >>>> >>>> Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>>> Hosted Engine High Availability Communications Broker. >>>> >>>> Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>>> Hosted Engine High Availability Communications Broker... >>>> >>>> Jun 15 08:56:16 ovirt-hyp-01.example.lan ovirt-ha-broker[968]: >>>> ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler >>>> ERROR Error handling request, data: '...1b55bcf76' >>>> >>>> Traceback >>>> (most recent call last): >>>> >>>> File >>>> "/usr/lib/python2.7/site-packages/ovirt... >>>> >>>> Hint: Some lines were ellipsized, use -l to show in full. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> [root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-agent.service >>>> >>>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-agent.service >>>> >>>> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability >>>> Monitoring Agent >>>> >>>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; >>>> enabled; vendor preset: disabled) >>>> >>>> Active: active (running) since Thu 2017-06-15 09:19:21 EDT; 26s ago >>>> >>>> Main PID: 8563 (ovirt-ha-agent) >>>> >>>> CGroup: /system.slice/ovirt-ha-agent.service >>>> >>>> └─8563 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent >>>> --no-daemon >>>> >>>> >>>> >>>> Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>>> Hosted Engine High Availability Monitoring Agent. >>>> >>>> Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>>> Hosted Engine High Availability Monitoring Agent... >>>> >>>> [root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-broker.service >>>> >>>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service >>>> >>>> ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability >>>> Communications Broker >>>> >>>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; >>>> enabled; vendor preset: disabled) >>>> >>>> Active: active (running) since Thu 2017-06-15 09:20:59 EDT; 28s ago >>>> >>>> Main PID: 8844 (ovirt-ha-broker) >>>> >>>> CGroup: /system.slice/ovirt-ha-broker.service >>>> >>>> └─8844 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>>> --no-daemon >>>> >>>> >>>> >>>> Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>>> Hosted Engine High Availability Communications Broker. >>>> >>>> Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>>> Hosted Engine High Availability Communications Broker... >>>> >>>> >>>> On Jun 14, 2017 4:45 AM, "Sahina Bose" <sabose(a)redhat.com&gt; wrote: >>>> >>>>> What's the output of "hosted-engine --vm-status" and "gluster volume >>>>> status engine" tell you? Are all the bricks running as per gluster vol >>>>> status? >>>>> >>>>> Can you try to restart the ovirt-ha-agent and ovirt-ha-broker >>>>> services? >>>>> >>>>> If HE still has issues powering up, please provide agent.log and >>>>> broker.log from /var/log/ovirt-hosted-engine-ha and gluster mount >>>>> logs from /var/log/glusterfs/rhev-data-center-mnt <engine>.log >>>>> >>>>> On Thu, Jun 8, 2017 at 6:57 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Good morning oVirt community, >>>>>> >>>>>> I'm running a three host gluster environment with hosted engine. >>>>>> >>>>>> Yesterday the engine went down and has not been able to come up >>>>>> properly. It tries to start on all three host. >>>>>> >>>>>> I have two gluster volumes, data and engne. The data storage domian >>>>>> volume is no longer mounted but the engine volume is up. I've restarted the >>>>>> gluster service and make sure both volumes were running. The data volume >>>>>> will not mount. >>>>>> >>>>>> How can I get the engine running properly again? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Joel >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>>> >>>>> >>>> >>> >