Hello, attached the qemu Log. This ist the problem: Could not open '/var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08': Permission denied When I do "su - vdsm -s /bin/bash" I can hexdump the file ! -bash-4.2$ id uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) -bash-4.2$ hexdump -Cn 512 /var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08 00000000 eb 63 90 10 8e d0 bc 00 b0 b8 00 00 8e d8 8e c0 |.c..............| 00000010 fb be 00 7c bf 00 06 b9 00 02 f3 a4 ea 21 06 00 |...|.........!..| 00000020 00 be be 07 38 04 75 0b 83 c6 10 81 fe fe 07 75 |....8.u........u| 00000030 f3 eb 16 b4 02 b0 01 bb 00 7c b2 80 8a 74 01 8b |.........|...t..| 00000040 4c 02 cd 13 ea 00 7c 00 00 eb fe 00 00 00 00 00 |L.....|.........| 00000050 00 00 00 00 00 00 00 00 00 00 00 80 01 00 00 00 |................| 00000060 00 00 00 00 ff fa 90 90 f6 c2 80 74 05 f6 c2 70 |...........t...p| 00000070 74 02 b2 80 ea 79 7c 00 00 31 c0 8e d8 8e d0 bc |t....y|..1......| 00000080 00 20 fb a0 64 7c 3c ff 74 02 88 c2 52 be 05 7c |. ..d|<.t...R..|| 00000090 b4 41 bb aa 55 cd 13 5a 52 72 3d 81 fb 55 aa 75 |.A..U..ZRr=..U.u| 000000a0 37 83 e1 01 74 32 31 c0 89 44 04 40 88 44 ff 89 |7...t21..D.@.D..| 000000b0 44 02 c7 04 10 00 66 8b 1e 5c 7c 66 89 5c 08 66 |D.....f..\|f.\.f| 000000c0 8b 1e 60 7c 66 89 5c 0c c7 44 06 00 70 b4 42 cd |..`|f.\..D..p.B.| 000000d0 13 72 05 bb 00 70 eb 76 b4 08 cd 13 73 0d 5a 84 |.r...p.v....s.Z.| 000000e0 d2 0f 83 de 00 be 85 7d e9 82 00 66 0f b6 c6 88 |.......}...f....| 000000f0 64 ff 40 66 89 44 04 0f b6 d1 c1 e2 02 88 e8 88 |d.@f.D..........| 00000100 f4 40 89 44 08 0f b6 c2 c0 e8 02 66 89 04 66 a1 |.@.D.......f..f.| 00000110 60 7c 66 09 c0 75 4e 66 a1 5c 7c 66 31 d2 66 f7 |`|f..uNf.\|f1.f.| 00000120 34 88 d1 31 d2 66 f7 74 04 3b 44 08 7d 37 fe c1 |4..1.f.t.;D.}7..| 00000130 88 c5 30 c0 c1 e8 02 08 c1 88 d0 5a 88 c6 bb 00 |..0........Z....| 00000140 70 8e c3 31 db b8 01 02 cd 13 72 1e 8c c3 60 1e |p..1......r...`.| 00000150 b9 00 01 8e db 31 f6 bf 00 80 8e c6 fc f3 a5 1f |.....1..........| 00000160 61 ff 26 5a 7c be 80 7d eb 03 be 8f 7d e8 34 00 |a.&Z|..}....}.4.| 00000170 be 94 7d e8 2e 00 cd 18 eb fe 47 52 55 42 20 00 |..}.......GRUB .| 00000180 47 65 6f 6d 00 48 61 72 64 20 44 69 73 6b 00 52 |Geom.Hard Disk.R| 00000190 65 61 64 00 20 45 72 72 6f 72 0d 0a 00 bb 01 00 |ead. Error......| 000001a0 b4 0e cd 10 ac 3c 00 75 f4 c3 00 00 00 00 00 00 |.....<.u........| 000001b0 00 00 00 00 00 00 00 00 4f ee 04 00 00 00 80 20 |........O...... | 000001c0 21 00 83 aa 28 82 00 08 00 00 00 00 20 00 00 aa |!...(....... ...| 000001d0 29 82 8e fe ff ff 00 08 20 00 00 f8 5b 05 00 fe |)....... ...[...| 000001e0 ff ff 83 fe ff ff 00 00 7c 05 00 f8 c3 00 00 00 |........|.......| 000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.| 00000200 Am 03.12.2018 um 16:43 schrieb Simone Tiraboschi:
On Mon, Dec 3, 2018 at 2:07 PM Ralf Schenk <rs@databay.de <mailto:rs@databay.de>> wrote:
Hello,
I try to deploy hosted-engine to a NFS Share accessible by (currently) two hosts. The host is running latest ovirt-node-ng 4.2.7.
hosted-engine --deploy fails constantly in late stage when trying to run engine from NFS. It already ran as "HostedEngineLocal" and I think is then migrated to NFS storage.
Engine seems to be deployed to NFS already:
[root@epycdphv02 ~]# ls -al /rhev/data-center/mnt/storage01.office.databay.de:_ovirt_engine total 23 drwxrwxrwx 3 vdsm kvm 4 Dec 3 13:01 . drwxr-xr-x 3 vdsm kvm 4096 Dec 1 17:11 .. drwxr-xr-x 6 vdsm kvm 6 Dec 3 13:09 1dacf1ea-0934-4840-bed4-e9d023572f59 -rwxr-xr-x 1 vdsm kvm 0 Dec 3 13:42 __DIRECT_IO_TEST__
NFS Mount:
storage01.office.databay.de:/ovirt/engine on /rhev/data-center/mnt/storage01.office.databay.de:_ovirt_engine type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.1.121,local_lock=none,addr=192.168.1.3)
Libvirt quemu states an error:
Could not open '/var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08': Permission denied
Even permissions of mentioned file seem to be ok. SELINUX is disabled since I had a lots of problems with earlier versions trying to deploy hosted-engine.
You can keep it on without any know issue.
[root@epycdphv02 ~]# ls -al '/var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08' -rw-rw---- 1 vdsm kvm 53687091200 Dec 3 13:09 /var/run/vdsm/storage/1dacf1ea-0934-4840-bed4-e9d023572f59/2b1332f6-3bb6-495b-87fe-c5b85e0ac495/39d45b33-5f29-430b-8b58-14a8ea20fb08
hosted-engine --deploy ends with error. Logfile is attached.
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.218320", "end": "2018-12-03 13:20:19.139919", "rc": 0, "start": "2018-12-03 13:20:18.921599", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=156443 (Mon Dec 3 13:20:16 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=156443 (Mon Dec 3 13:20:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Fri Jan 2 20:29:01 1970\\n\", \"hostname\": \"epycdphv02.office.databay.de <http://epycdphv02.office.databay.de>\", \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"d3355c40\", \"local_conf_timestamp\": 156443, \"host-ts\": 156443}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"conf_on_shared_storage\": true, \"live-data\": true, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=156443 (Mon Dec 3 13:20:16 2018)\\nhost-id=1\\nscore=0\\nvm_conf_refresh_time=156443 (Mon Dec 3 13:20:16 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUnexpectedlyDown\\nstopped=False\\ntimeout=Fri Jan 2 20:29:01 1970\\n\", \"hostname\": \"epycdphv02.office.databay.de <http://epycdphv02.office.databay.de>\", \"host-id\": 1, \"engine-status\": {\"reason\": \"bad vm status\", \"health\": \"bad\", \"vm\": \"down_unexpected\", \"detail\": \"Down\"}, \"score\": 0, \"stopped\": false, \"maintenance\": false, \"crc32\": \"d3355c40\", \"local_conf_timestamp\": 156443, \"host-ts\": 156443}, \"global_maintenance\": false}"]}
state=EngineUnexpectedlyDown means that the host tried to start the engine VM from the shared storage but it failed due to other reasons. Could you please attach /var/log/messages and /var/log/libvirt/qemu/HostedEngine.log ?
[ INFO ] TASK [Check VM status at virt level] [ INFO ] TASK [Fail if engine VM is not running] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM is not running, please check vdsm logs"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [Set local_vm_disk_path] [ INFO ] ok: [localhost] [ INFO ] TASK [Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [Copy engine logs] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Remove local vm dir] [ INFO ] changed: [localhost] [ INFO ] TASK [Remove temporary entry in /etc/hosts for the local VM] [ INFO ] ok: [localhost] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20181203132110.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20181203124703-45t4ja.log --
*Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------ _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NK2TNNPA7TFLPK...
-- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------
