Re: [ovirt-users] Hosted engine

I don't notice anything wrong on the gluster end. Maybe Simone can help take a look at HE behaviour? On Fri, Jun 16, 2017 at 6:14 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; wrote: > Good morning, > > Info requested below. > > [root@ovirt-hyp-02 ~]# hosted-engine --vm-start > > Exception in thread Client localhost:54321 (most likely raised during > interpreter shutdown):VM exists and its status is Up > > > > [root@ovirt-hyp-02 ~]# ping engine > > PING engine.example.lan (192.168.170.149) 56(84) bytes of data. > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=1 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=2 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=3 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=4 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=5 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=6 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=7 Destination > Host Unreachable > > From ovirt-hyp-02.example.lan (192.168.170.143) icmp_seq=8 Destination > Host Unreachable > > > > > > [root@ovirt-hyp-02 ~]# gluster volume status engine > > Status of volume: engine > > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------ > ------------------ > > Brick 192.168.170.141:/gluster_bricks/engin > > e/engine 49159 0 Y > 1799 > > Brick 192.168.170.143:/gluster_bricks/engin > > e/engine 49159 0 Y > 2900 > > Self-heal Daemon on localhost N/A N/A Y > 2914 > > Self-heal Daemon on ovirt-hyp-01.example.lan N/A N/A > Y 1854 > > > > Task Status of Volume engine > > ------------------------------------------------------------ > ------------------ > > There are no active volume tasks > > > > [root@ovirt-hyp-02 ~]# gluster volume heal engine info > > Brick 192.168.170.141:/gluster_bricks/engine/engine > > Status: Connected > > Number of entries: 0 > > > > Brick 192.168.170.143:/gluster_bricks/engine/engine > > Status: Connected > > Number of entries: 0 > > > > Brick 192.168.170.147:/gluster_bricks/engine/engine > > Status: Connected > > Number of entries: 0 > > > > [root@ovirt-hyp-02 ~]# cat /var/log/glusterfs/rhev-data-c > enter-mnt-glusterSD-ovirt-hyp-01.example.lan\:engine.log > > [2017-06-15 13:37:02.009436] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] > 0-glusterfs: No change in volfile, continuing > > > > > > Each of the three host sends out the following notifications about every > 15 minutes. > > Hosted engine host: ovirt-hyp-01.example.lan changed state: > EngineDown-EngineStart. > > Hosted engine host: ovirt-hyp-01.example.lan changed state: > EngineStart-EngineStarting. > > Hosted engine host: ovirt-hyp-01.example.lan changed state: > EngineStarting-EngineForceStop. > > Hosted engine host: ovirt-hyp-01.example.lan changed state: > EngineForceStop-EngineDown. > > Please let me know if you need any additional information. > > Thank you, > > Joel > > > > On Jun 16, 2017 2:52 AM, "Sahina Bose" <sabose(a)redhat.com&gt; wrote: > >> From the agent.log, >> MainThread::INFO::2017-06-15 11:16:50,583::states::473::ovi >> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine >> vm is running on host ovirt-hyp-02.reis.com (id 2) >> >> It looks like the HE VM was started successfully? Is it possible that >> the ovirt-engine service could not be started on the HE VM. Could you try >> to start the HE vm using below and then logging into the VM console. >> #hosted-engine --vm-start >> >> Also, please check >> # gluster volume status engine >> # gluster volume heal engine info >> >> Please also check if there are errors in gluster mount logs - at >> /var/log/glusterfs/rhev-data-center-mnt..<engine>.log >> >> >> On Thu, Jun 15, 2017 at 8:53 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; wrote: >> >>> Sorry. I forgot to attached the requested logs in the previous email. >>> >>> Thanks, >>> >>> On Jun 15, 2017 9:38 AM, "Joel Diaz" <mrjoeldiaz(a)gmail.com&gt; wrote: >>> >>> Good morning, >>> >>> Requested info below. Along with some additional info. >>> >>> You'll notice the data volume is not mounted. >>> >>> Any help in getting HE back running would be greatly appreciated. >>> >>> Thank you, >>> >>> Joel >>> >>> [root@ovirt-hyp-01 ~]# hosted-engine --vm-status >>> >>> >>> >>> >>> >>> --== Host 1 status ==-- >>> >>> >>> >>> conf_on_shared_storage : True >>> >>> Status up-to-date : False >>> >>> Hostname : ovirt-hyp-01.example.lan >>> >>> Host ID : 1 >>> >>> Engine status : unknown stale-data >>> >>> Score : 3400 >>> >>> stopped : False >>> >>> Local maintenance : False >>> >>> crc32 : 5558a7d3 >>> >>> local_conf_timestamp : 20356 >>> >>> Host timestamp : 20341 >>> >>> Extra metadata (valid at timestamp): >>> >>> metadata_parse_version=1 >>> >>> metadata_feature_version=1 >>> >>> timestamp=20341 (Fri Jun 9 14:38:57 2017) >>> >>> host-id=1 >>> >>> score=3400 >>> >>> vm_conf_refresh_time=20356 (Fri Jun 9 14:39:11 2017) >>> >>> conf_on_shared_storage=True >>> >>> maintenance=False >>> >>> state=EngineDown >>> >>> stopped=False >>> >>> >>> >>> >>> >>> --== Host 2 status ==-- >>> >>> >>> >>> conf_on_shared_storage : True >>> >>> Status up-to-date : False >>> >>> Hostname : ovirt-hyp-02.example.lan >>> >>> Host ID : 2 >>> >>> Engine status : unknown stale-data >>> >>> Score : 3400 >>> >>> stopped : False >>> >>> Local maintenance : False >>> >>> crc32 : 936d4cf3 >>> >>> local_conf_timestamp : 20351 >>> >>> Host timestamp : 20337 >>> >>> Extra metadata (valid at timestamp): >>> >>> metadata_parse_version=1 >>> >>> metadata_feature_version=1 >>> >>> timestamp=20337 (Fri Jun 9 14:39:03 2017) >>> >>> host-id=2 >>> >>> score=3400 >>> >>> vm_conf_refresh_time=20351 (Fri Jun 9 14:39:17 2017) >>> >>> conf_on_shared_storage=True >>> >>> maintenance=False >>> >>> state=EngineDown >>> >>> stopped=False >>> >>> >>> >>> >>> >>> --== Host 3 status ==-- >>> >>> >>> >>> conf_on_shared_storage : True >>> >>> Status up-to-date : False >>> >>> Hostname : ovirt-hyp-03.example.lan >>> >>> Host ID : 3 >>> >>> Engine status : unknown stale-data >>> >>> Score : 3400 >>> >>> stopped : False >>> >>> Local maintenance : False >>> >>> crc32 : f646334e >>> >>> local_conf_timestamp : 20391 >>> >>> Host timestamp : 20377 >>> >>> Extra metadata (valid at timestamp): >>> >>> metadata_parse_version=1 >>> >>> metadata_feature_version=1 >>> >>> timestamp=20377 (Fri Jun 9 14:39:37 2017) >>> >>> host-id=3 >>> >>> score=3400 >>> >>> vm_conf_refresh_time=20391 (Fri Jun 9 14:39:51 2017) >>> >>> conf_on_shared_storage=True >>> >>> maintenance=False >>> >>> state=EngineStop >>> >>> stopped=False >>> >>> timeout=Thu Jan 1 00:43:08 1970 >>> >>> >>> >>> >>> >>> [root@ovirt-hyp-01 ~]# gluster peer status >>> >>> Number of Peers: 2 >>> >>> >>> >>> Hostname: 192.168.170.143 >>> >>> Uuid: b2b30d05-cf91-4567-92fd-022575e082f5 >>> >>> State: Peer in Cluster (Connected) >>> >>> Other names: >>> >>> 10.0.0.2 >>> >>> >>> >>> Hostname: 192.168.170.147 >>> >>> Uuid: 4e50acc4-f3cb-422d-b499-fb5796a53529 >>> >>> State: Peer in Cluster (Connected) >>> >>> Other names: >>> >>> 10.0.0.3 >>> >>> >>> >>> [root@ovirt-hyp-01 ~]# gluster volume info all >>> >>> >>> >>> Volume Name: data >>> >>> Type: Replicate >>> >>> Volume ID: 1d6bb110-9be4-4630-ae91-36ec1cf6cc02 >>> >>> Status: Started >>> >>> Snapshot Count: 0 >>> >>> Number of Bricks: 1 x (2 + 1) = 3 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: 192.168.170.141:/gluster_bricks/data/data >>> >>> Brick2: 192.168.170.143:/gluster_bricks/data/data >>> >>> Brick3: 192.168.170.147:/gluster_bricks/data/data (arbiter) >>> >>> Options Reconfigured: >>> >>> nfs.disable: on >>> >>> performance.readdir-ahead: on >>> >>> transport.address-family: inet >>> >>> performance.quick-read: off >>> >>> performance.read-ahead: off >>> >>> performance.io-cache: off >>> >>> performance.stat-prefetch: off >>> >>> performance.low-prio-threads: 32 >>> >>> network.remote-dio: off >>> >>> cluster.eager-lock: enable >>> >>> cluster.quorum-type: auto >>> >>> cluster.server-quorum-type: server >>> >>> cluster.data-self-heal-algorithm: full >>> >>> cluster.locking-scheme: granular >>> >>> cluster.shd-max-threads: 8 >>> >>> cluster.shd-wait-qlength: 10000 >>> >>> features.shard: on >>> >>> user.cifs: off >>> >>> storage.owner-uid: 36 >>> >>> storage.owner-gid: 36 >>> >>> network.ping-timeout: 30 >>> >>> performance.strict-o-direct: on >>> >>> cluster.granular-entry-heal: enable >>> >>> >>> >>> Volume Name: engine >>> >>> Type: Replicate >>> >>> Volume ID: b160f0b2-8bd3-4ff2-a07c-134cab1519dd >>> >>> Status: Started >>> >>> Snapshot Count: 0 >>> >>> Number of Bricks: 1 x (2 + 1) = 3 >>> >>> Transport-type: tcp >>> >>> Bricks: >>> >>> Brick1: 192.168.170.141:/gluster_bricks/engine/engine >>> >>> Brick2: 192.168.170.143:/gluster_bricks/engine/engine >>> >>> Brick3: 192.168.170.147:/gluster_bricks/engine/engine (arbiter) >>> >>> Options Reconfigured: >>> >>> nfs.disable: on >>> >>> performance.readdir-ahead: on >>> >>> transport.address-family: inet >>> >>> performance.quick-read: off >>> >>> performance.read-ahead: off >>> >>> performance.io-cache: off >>> >>> performance.stat-prefetch: off >>> >>> performance.low-prio-threads: 32 >>> >>> network.remote-dio: off >>> >>> cluster.eager-lock: enable >>> >>> cluster.quorum-type: auto >>> >>> cluster.server-quorum-type: server >>> >>> cluster.data-self-heal-algorithm: full >>> >>> cluster.locking-scheme: granular >>> >>> cluster.shd-max-threads: 8 >>> >>> cluster.shd-wait-qlength: 10000 >>> >>> features.shard: on >>> >>> user.cifs: off >>> >>> storage.owner-uid: 36 >>> >>> storage.owner-gid: 36 >>> >>> network.ping-timeout: 30 >>> >>> performance.strict-o-direct: on >>> >>> cluster.granular-entry-heal: enable >>> >>> >>> >>> >>> >>> [root@ovirt-hyp-01 ~]# df -h >>> >>> Filesystem Size Used Avail Use% >>> Mounted on >>> >>> /dev/mapper/centos_ovirt--hyp--01-root 50G 4.1G 46G 9% / >>> >>> devtmpfs 7.7G 0 7.7G 0% >>> /dev >>> >>> tmpfs 7.8G 0 7.8G 0% >>> /dev/shm >>> >>> tmpfs 7.8G 8.7M 7.7G 1% >>> /run >>> >>> tmpfs 7.8G 0 7.8G 0% >>> /sys/fs/cgroup >>> >>> /dev/mapper/centos_ovirt--hyp--01-home 61G 33M 61G 1% >>> /home >>> >>> /dev/mapper/gluster_vg_sdb-gluster_lv_engine 50G 7.6G 43G 16% >>> /gluster_bricks/engine >>> >>> /dev/mapper/gluster_vg_sdb-gluster_lv_data 730G 157G 574G 22% >>> /gluster_bricks/data >>> >>> /dev/sda1 497M 173M 325M 35% >>> /boot >>> >>> ovirt-hyp-01.example.lan:engine 50G 7.6G 43G 16% >>> /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.example.lan:engine >>> >>> tmpfs 1.6G 0 1.6G 0% >>> /run/user/0 >>> >>> >>> >>> [root@ovirt-hyp-01 ~]# systemctl list-unit-files|grep ovirt >>> >>> ovirt-ha-agent.service enabled >>> >>> ovirt-ha-broker.service enabled >>> >>> ovirt-imageio-daemon.service disabled >>> >>> ovirt-vmconsole-host-sshd.service enabled >>> >>> >>> >>> [root@ovirt-hyp-01 ~]# systemctl status ovirt-ha-agent.service >>> >>> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability >>> Monitoring Agent >>> >>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; >>> enabled; vendor preset: disabled) >>> >>> Active: active (running) since Thu 2017-06-15 08:56:15 EDT; 21min ago >>> >>> Main PID: 3150 (ovirt-ha-agent) >>> >>> CGroup: /system.slice/ovirt-ha-agent.service >>> >>> └─3150 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent >>> --no-daemon >>> >>> >>> >>> Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>> Hosted Engine High Availability Monitoring Agent. >>> >>> Jun 15 08:56:15 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>> Hosted Engine High Availability Monitoring Agent... >>> >>> Jun 15 09:17:18 ovirt-hyp-01.example.lan ovirt-ha-agent[3150]: >>> ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine >>> ERROR Engine VM stopped on localhost >>> >>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service >>> >>> ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability >>> Communications Broker >>> >>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; >>> enabled; vendor preset: disabled) >>> >>> Active: active (running) since Thu 2017-06-15 08:54:06 EDT; 24min ago >>> >>> Main PID: 968 (ovirt-ha-broker) >>> >>> CGroup: /system.slice/ovirt-ha-broker.service >>> >>> └─968 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>> --no-daemon >>> >>> >>> >>> Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>> Hosted Engine High Availability Communications Broker. >>> >>> Jun 15 08:54:06 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>> Hosted Engine High Availability Communications Broker... >>> >>> Jun 15 08:56:16 ovirt-hyp-01.example.lan ovirt-ha-broker[968]: >>> ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler >>> ERROR Error handling request, data: '...1b55bcf76' >>> >>> Traceback >>> (most recent call last): >>> >>> File >>> "/usr/lib/python2.7/site-packages/ovirt... >>> >>> Hint: Some lines were ellipsized, use -l to show in full. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> [root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-agent.service >>> >>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-agent.service >>> >>> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability >>> Monitoring Agent >>> >>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; >>> enabled; vendor preset: disabled) >>> >>> Active: active (running) since Thu 2017-06-15 09:19:21 EDT; 26s ago >>> >>> Main PID: 8563 (ovirt-ha-agent) >>> >>> CGroup: /system.slice/ovirt-ha-agent.service >>> >>> └─8563 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent >>> --no-daemon >>> >>> >>> >>> Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>> Hosted Engine High Availability Monitoring Agent. >>> >>> Jun 15 09:19:21 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>> Hosted Engine High Availability Monitoring Agent... >>> >>> [root@ovirt-hyp-01 ‾]# systemctl restart ovirt-ha-broker.service >>> >>> [root@ovirt-hyp-01 ‾]# systemctl status ovirt-ha-broker.service >>> >>> ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability >>> Communications Broker >>> >>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; >>> enabled; vendor preset: disabled) >>> >>> Active: active (running) since Thu 2017-06-15 09:20:59 EDT; 28s ago >>> >>> Main PID: 8844 (ovirt-ha-broker) >>> >>> CGroup: /system.slice/ovirt-ha-broker.service >>> >>> └─8844 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker >>> --no-daemon >>> >>> >>> >>> Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Started oVirt >>> Hosted Engine High Availability Communications Broker. >>> >>> Jun 15 09:20:59 ovirt-hyp-01.example.lan systemd[1]: Starting oVirt >>> Hosted Engine High Availability Communications Broker... >>> >>> >>> On Jun 14, 2017 4:45 AM, "Sahina Bose" <sabose(a)redhat.com&gt; wrote: >>> >>>> What's the output of "hosted-engine --vm-status" and "gluster volume >>>> status engine" tell you? Are all the bricks running as per gluster vol >>>> status? >>>> >>>> Can you try to restart the ovirt-ha-agent and ovirt-ha-broker services? >>>> >>>> If HE still has issues powering up, please provide agent.log and >>>> broker.log from /var/log/ovirt-hosted-engine-ha and gluster mount >>>> logs from /var/log/glusterfs/rhev-data-center-mnt <engine>.log >>>> >>>> On Thu, Jun 8, 2017 at 6:57 PM, Joel Diaz <mrjoeldiaz(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Good morning oVirt community, >>>>> >>>>> I'm running a three host gluster environment with hosted engine. >>>>> >>>>> Yesterday the engine went down and has not been able to come up >>>>> properly. It tries to start on all three host. >>>>> >>>>> I have two gluster volumes, data and engne. The data storage domian >>>>> volume is no longer mounted but the engine volume is up. I've restarted the >>>>> gluster service and make sure both volumes were running. The data volume >>>>> will not mount. >>>>> >>>>> How can I get the engine running properly again? >>>>> >>>>> Thanks, >>>>> >>>>> Joel >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>> >>