I've never managed to get a connection to the engine via VNC/Spice (works fine for my other hypervisors...)

As I said, the network setup is super simple. All three nodes have 1 interface each (eth0). They are all set with static IP's, with matching DHCP reservations on the DHCP server, with matching DNS. All nodes have entries in /etc/hosts on each machine. IP's are 192.168.1.195-7, and the engine VM gets 192.168.1.198. During the engine deployment, the VM does come up on 198. I can ping it and ssh into it, but at some point, the connection drops. 
So I'm not relying on DHCP or DNS at all. VM comes up where expected, for a while, then it goes to reboot to get transferred to the gluster_engine storage, and that's where it drops offline and never comes back.

I did another round of deployment tests last night and discovered that the nodes all fail to boot immediately after the gluster deployment (not after VM deployment as I mistakenly stated earlier). So the nodes get in a bad state during gluster deployment. They stay online just fine and gluster works perfect, until the node tries to reboot (which it fails to do).

Also, the networking I'm using is identical to my ovirt 4.2 setup. I'm using the same MAC addresses, IP's, and hostnames (4.2 cluster is offline when I'm trying 4.3). They are identical configurations other than the version of ovirt-node.

On Thu, Feb 7, 2019 at 12:15 AM Simone Tiraboschi <stirabos@redhat.com> wrote:


On Wed, Feb 6, 2019 at 11:07 PM feral <blistovmhz@gmail.com> wrote:
I have no idea what's wrong at this point. Very vanilla install of 3 nodes. Run the Hyperconverged wizard, completes fine. Run the engine deployment, takes hours, eventually fails with :

[ INFO ] TASK [oVirt.hosted-engine-setup : Check engine VM health]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.340985", "end": "2019-02-06 11:44:48.836431", "rc": 0, "start": "2019-02-06 11:44:48.495446", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=12994 (Wed Feb 6 11:44:44 2019)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=12995 (Wed Feb 6 11:44:44 2019)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\n\", \"hostname\": \"ovirt-431.localdomain\", \"host-id\": 1, \"engine-status\": {\"reason\": \"failed liveliness check\", \"health\": \"bad\", \"vm\": \"up\", \"detail\": \"Up\"}, \"score\": 3400, \"stopped\": false, \"maintenance\": false, \"crc32\": \"5474927a\", \"local_conf_timestamp\": 12995, \"host-ts\": 12994}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=12994 (Wed Feb 6 11:44:44 2019)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=12995 (Wed Feb 6 11:44:44 2019)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\n\", \"hostname\": \"ovirt-431.localdomain\", \"host-id\": 1, \"engine-status\": {\"reason\": \"failed liveliness check\", \"health\": \"bad\", \"vm\": \"up\", \"detail\": \"Up\"}, \"score\": 3400, \"stopped\": false, \"maintenance\": false, \"crc32\": \"5474927a\", \"local_conf_timestamp\": 12995, \"host-ts\": 12994}, \"global_maintenance\": false}"]}
[ INFO ] TASK [oVirt.hosted-engine-setup : Check VM status at virt level]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : debug]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Fail if engine VM is not running]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Get target engine VM IP address]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Get VDSM's target engine VM stats]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Convert stats to JSON format]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Get target engine VM IP address from VDSM stats]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : debug]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Fail if Engine IP is different from engine's he_fqdn resolved IP]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Fail is for any other reason the engine didn't started]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The engine failed to start inside the engine VM; please check engine.log."}

---------------------------------------------------

I can't check the engine.log as I can't connect to the VM once this failure occurs. I can ssh in prior to the VM being moved to gluster storage, but as soon as it starts doing so, the VM never comes back online.

{\"reason\": \"failed liveliness check\", \"health\": \"bad\", \"vm\": \"up\", \"detail\": \"Up\"}

means that the engine VM is up at virt level but we cannot reach the engine over http for a liveness check.
A network issue, a wrong DHCP reservation, a bad name resolution or something live that could be the reason.
I suggest to try connecting to the engine VM via VNC or serial console to check what's wrong there.

 


--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LAM3TX64UQOOO7A2WBMGZUEF4TVFHXJA/


--
_____
Fact:
1. Ninjas are mammals.
2. Ninjas fight ALL the time.
3. The purpose of the ninja is to flip out and kill people.