[ovirt-users] Re: hosted-engine --deploy fails on Ovirt-Node-NG 4.2.7

On Mon, Dec 3, 2018 at 2:07 PM Ralf Schenk <rs(a)databay.de <mailto:rs@databay.de>> wrote: Hello, I try to deploy hosted-engine to a NFS Share accessible by (currently) two hosts. The host is running latest ovirt-node-ng 4.2.7. hosted-engine --deploy fails constantly in late stage when trying to run engine from NFS. It already ran as "HostedEngineLocal" and I think is then migrated to NFS storage. Engine seems to be deployed to NFS already: [root@epycdphv02 ~]# ls -al /rhev/data-center/mnt/storage01.office.databay.de:_ovirt_engine total 23 drwxrwxrwx 3 vdsm kvm    4 Dec  3 13:01 . drwxr-xr-x 3 vdsm kvm 4096 Dec  1 17:11 .. drwxr-xr-x 6 vdsm kvm    6 Dec  3 13:09 1dacf1ea-0934-4840-bed4-e9d023572f59 -rwxr-xr-x 1 vdsm kvm    0 Dec  3 13:42 __DIRECT_IO_TEST__ NFS Mount: storage01.office.databay.de:/ovirt/engine on /rhev/data-center/mnt/storage01.office.databay.de:_ovirt_engine type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.1.121,local_lock=none,addr=192.168.1.3) Libvirt quemu states an error: Could not open '/var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08': Permission denied Even permissions of mentioned file seem to be ok. SELINUX is disabled since I had a lots of problems with earlier versions trying to deploy hosted-engine. You can keep it on without any know issue.   [root@epycdphv02 ~]# ls -al '/var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08' -rw-rw---- 1 vdsm kvm 53687091200 Dec  3 13:09 /var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08 hosted-engine --deploy ends with error. Logfile is attached. [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.218320", "end": "2018-12-03 13:20:19.139919", "rc": 0, "start": "2018-12-03 13:20:18.921599", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=156443 (Mon Dec  3 13:20:16 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=156443 (Mon Dec  3 13:20:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Fri Jan  2 20:29:01 1970\\n\", \"hostname\": \"epycdphv02.office.databay.de <http://epycdphv02.office.databay.de>\";, \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"d3355c40\", \"local_conf_timestamp\": 156443, \"host-ts\": 156443}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=156443 (Mon Dec  3 13:20:16 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=156443 (Mon Dec  3 13:20:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Fri Jan  2 20:29:01 1970\\n\", \"hostname\": \"epycdphv02.office.databay.de <http://epycdphv02.office.databay.de>\";, \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"d3355c40\", \"local_conf_timestamp\": 156443, \"host-ts\": 156443}, \"global_maintenance\": false}"]} state=EngineUnexpectedlyDown means that the host tried to start the engine VM from the shared storage but it failed due to other reasons. Could you please attach /var/log/messages and /var/log/libvirt/qemu/HostedEngine.log ?   [ INFO  ] TASK [Check VM status at virt level] [ INFO  ] TASK [Fail if engine VM is not running] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO  ] Stage: Clean up [ INFO  ] Cleaning temporary resources [ INFO  ] TASK [Gathering Facts] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Fetch logs from the engine VM] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Set destination directory path] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Create destination directory] [ INFO  ] changed: [localhost] [ INFO  ] TASK [include_tasks] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Find the local appliance image] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Set local_vm_disk_path] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Give the vm time to flush dirty buffers] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Copy engine logs] [ INFO  ] TASK [include_tasks] [ INFO  ] ok: [localhost] [ INFO  ] TASK [Remove local vm dir] [ INFO  ] changed: [localhost] [ INFO  ] TASK [Remove temporary entry in /etc/hosts for the local VM] [ INFO  ] ok: [localhost] [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20181203132110.conf' [ INFO  ] Stage: Pre-termination [ INFO  ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.           Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20181203124703-45t4ja.log -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de>   *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------ _______________________________________________ Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NK2TNNPA7TF...