[ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth
Will Dennis
wdennis at nec-labs.com
Tue Dec 22 15:48:41 UTC 2015
Ah OK, my ignorance strikes again... I flushed all the rules on IPtables on hosts -01 and -02, and now Gluster seems up and happy...
I saw messages after I flushed IPtables that the engine state was changing, and eventually it landed on status: ReinitializeFSM-LocalMaintenance
...but I could not log into the admin website.
I then queried the engine VM state at the CLI, and saw it was thus:
[root at ovirt-node-01 ~]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True
Hostname : ovirt-node-01
Host ID : 1
Engine status : {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "paused"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 538868a0
Host timestamp : 214954
--== Host 2 status ==--
Status up-to-date : True
Hostname : ovirt-node-02
Host ID : 2
Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score : 0
stopped : False
Local maintenance : True
crc32 : 419a0c6d
Host timestamp : 53528
So, I issued the command “hosted-engine --vm-shutdown” on host -01, and it eventually came down (had to be force-killed per status emails) and then HA restarted it :)
Looks like I’m back to being good now... Thanks everyone for the assist, and talk to you soon, I’m sure ;)
-Will
From: Simone Tiraboschi [mailto:stirabos at redhat.com]
Sent: Tuesday, December 22, 2015 10:23 AM
To: Will Dennis; users
Cc: Sahina Bose; Dan Kenigsberg
Subject: Re: [ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth
hosted-engine-setup asks:
iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]:
You have just to say no here.
If you say no it's completely up to you to configure it opening the required ports or everything disabling it if you don't care.
The issue with gluster ports is that hosted-engine-setup simply configure iptables for what it knows you'll need and on 3.6 it's always assuming that the gluster volume is served by external hosts.
From: Sahina Bose [mailto:sabose at redhat.com<mailto:sabose at redhat.com>]
Sent: Tuesday, December 22, 2015 9:19 AM
To: Will Dennis; Simone Tiraboschi; Dan Kenigsberg
Subject: Re: [ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth
On 12/22/2015 07:47 PM, Sahina Bose wrote:
On 12/22/2015 07:28 PM, Will Dennis wrote:
See attached for requested log files
From gluster logs
[2015-12-22 00:40:53.501341] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-engine-replicate-1: Client-quorum is not met
[2015-12-22 00:40:53.502288] W [socket.c:588:__socket_rwv] 0-engine-client-2: readv on 138.15.200.93:49217<http://138.15.200.93:49217> failed (No data available)
[2015-12-22 00:41:17.667302] W [fuse-bridge.c:2292:fuse_writev_cbk] 0-glusterfs-fuse: 3875597: WRITE => -1 (Read-only file system)
Could you check if the gluster ports are open on all nodes?
It's possible you ran into this ? - https://bugzilla.redhat.com/show_bug.cgi?id=1288979
From: Sahina Bose [mailto:sabose at redhat.com]
Sent: Tuesday, December 22, 2015 4:59 AM
To: Simone Tiraboschi; Will Dennis; Dan Kenigsberg
Cc: users
Subject: Re: [ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth
On 12/22/2015 02:38 PM, Simone Tiraboschi wrote:
On Tue, Dec 22, 2015 at 2:31 AM, Will Dennis <wdennis at nec-labs.com<mailto:wdennis at nec-labs.com>> wrote:
OK, another problem :(
I was having the same problem with my second oVirt host that I had with my first one, where when I ran “hosted-engine —deploy” on it, after it completed successfully, then I was experiencing a ~50sec lag when SSH’ing into the node…
vpnp71:~ will$ time ssh root at ovirt-node-02 uptime
19:36:06 up 4 days, 8:31, 0 users, load average: 0.68, 0.70, 0.67
real 0m50.540s
user 0m0.025s
sys 0m0.008s
So, in the oVirt web admin console, I put the "ovirt-node-02” node into Maintenance mode, then SSH’d to the server and rebooted it. Sure enough, after the server came back up, SSH was fine (no delay), which again was the same experience I had had with the first oVirt host. So, I went back to the web console, and choose the “Confirm host has been rebooted” option, which I thought would be the right action to take after a reboot. The system opened a dialog box with a spinner, which never stopped spinning… So finally, I closed the dialog box with the upper right (X) symbol, and then for this same host choose “Activate” from the menu. It was then I noticed I had recieved a state transition email notifying me that "EngineUp-EngineUpBadHealth” and sure enough, the web UI was then unresponsive. I checked on the first oVirt host, the VM with the name “HostedEngine” is still running, but obviously isn’t working…
So, looks like I need to restart the HostedEngine VM or take whatever action is needed to return oVirt to operation… Hate to keep asking this question, but what’s the correct action at this point?
ovirt-ha-agent should always restart it for you after a few minutes but the point is that the network configuration seams to be not that stable.
I know from another thread that you are trying to deploy hosted-engine over GlusterFS in an hyperconverged way and this, as I said, is currently not supported.
I think that it can also requires some specific configuration on network side.
For hyperconverged gluster+engine , it should work without any specific configuration on network side. However if the network is flaky, it is possible that there are errors with gluster volume access. Could you provide the ovirt-ha-agent logs as well as gluster mount logs?
Adding Sahina and Dan here.
Thanks, again,
Will
_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20151222/288ab485/attachment-0001.html>
More information about the Users
mailing list