[ovirt-users] Re: Engine Setup Error

Hi All, The host deploy logs are showing the below errors: [root@garlic engine-logs-2018-09-05T08:48:22Z]# cat /var/log/ovirt-hosted-engine-setup/engine-logs-2018-09-05T08\:34\:55Z/ovirt-engine/host-deploy/ovirt-host-deploy-20180905103605-garlic.sanren.ac.za-543b536b.log | grep -i error 2018-09-05 10:35:46,909+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'False' 2018-09-05 10:35:47,116 [ERROR] __main__.py:8011:MainThread @identity.py:145 - Reload of consumer identity cert /etc/pki/consumer/cert.pem raised an exception with msg: [Errno 2] No such file or directory: '/etc/pki/consumer/key.pem' 2018-09-05 10:35:47,383+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'False' 2018-09-05 10:35:47,593 [ERROR] __main__.py:8011:MainThread @identity.py:145 - Reload of consumer identity cert /etc/pki/consumer/cert.pem raised an exception with msg: [Errno 2] No such file or directory: '/etc/pki/consumer/key.pem' 2018-09-05 10:35:48,245+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'False' Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details. RuntimeError: Failed to start service 'ovirt-imageio-daemon' 2018-09-05 10:36:05,098+0200 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon' 2018-09-05 10:36:05,099+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'True' 2018-09-05 10:36:05,099+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError("Failed to start service 'ovirt-imageio-daemon'",), <traceback object at 0x7f4de25ff320>)]' 2018-09-05 10:36:05,106+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/error=bool:'True' 2018-09-05 10:36:05,106+0200 DEBUG otopi.context context.dumpEnvironment:869 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError("Failed to start service 'ovirt-imageio-daemon'",), <traceback object at 0x7f4de25ff320>)]' I coudn't find anything helpful from the internet.

On Tue, Sep 4, 2018 at 6:46 PM, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Tue, Sep 4, 2018 at 6:07 PM Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; wrote: > >> Hi Sahina, >> >> I am sorry I can't reproduce the error nor access the logs since I did a >> fresh installed pn nodes. However now I can't even react that far because >> the engine deployment fails to start the host up: >> >> >> [ INFO ] TASK [Wait for the host to be up] >> [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": >> {"ovirt_hosts": [{"address": "goku.sanren.ac.za", "affinity_labels": >> [], "auto_numa_status": "unknown", "certificate": {"organization": " >> sanren.ac.za", "subject": "O=sanren.ac.za,CN=goku.sanren.ac.za"}, >> "cluster": {"href": "/ovirt-engine/api/clusters/1ca368cc-b052-11e8-b7de-00163e008187", >> "id": "1ca368cc-b052-11e8-b7de-00163e008187"}, "comment": "", "cpu": >> {"speed": 0.0, "topology": {}}, "device_passthrough": {"enabled": false}, >> "devices": [], "external_network_provider_configurations": [], >> "external_status": "ok", "hardware_information": {"supported_rng_sources": >> []}, "hooks": [], "href": "/ovirt-engine/api/hosts/ >> 1c575995-70b1-43f7-b348-4a9788e070cd", "id": "1c575995-70b1-43f7-b348-4a9788e070cd", >> "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, >> "max_scheduling_memory": 0, "memory": 0, "name": "goku.sanren.ac.za", >> "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": >> false, "os": {"custom_kernel_cmdline": ""}, "permissions": [], "port": >> 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, >> "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", >> "se_linux": {}, "spm": {"priority": 5, "status": "none"}, "ssh": >> {"fingerprint": "SHA256:B3/PDH551EFid93fm6PoRryi6/cXuVE8yNgiiiROh84", >> "port": 22}, "statistics": [], "status": "install_failed", >> "storage_connection_extensions": [], "summary": {"total": 0}, "tags": >> [], "transparent_huge_pages": {"enabled": false}, "type": "ovirt_node", >> "unmanaged_networks": [], "update_available": false}]}, "attempts": 120, >> "changed": false} >> >> "status": "install_failed" > > You have to check host-deploy logs to get a details error message. > > >> >> Please help. >> >> On Mon, Sep 3, 2018 at 1:34 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: >> >>> >>> >>> On Wed, Aug 29, 2018 at 8:39 PM, Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; >>> wrote: >>> >>>> Hi, >>>> >>>> I am sorry to bother you again. >>>> >>>> I am trying to deploy an oVirt engine for oVirtNode-4.2.5.1. I get the >>>> same error I encountered before: >>>> >>>> [ INFO ] TASK [Add glusterfs storage domain] >>>> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is >>>> "[Problem while trying to mount target]". HTTP response code is 400. >>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>> "Fault reason is \"Operation Failed\". Fault detail is \"[Problem >>>> while trying to mount target]\". HTTP response code is 400."} >>>> Please specify the storage you would like to use (glusterfs, >>>> iscsi, fc, nfs)[nfs]: >>>> >>>> The glusterd daemon is running. >>>> >>> >>> mounting 172.16.4.18:/engine at >>> /rhev/data-center/mnt/glusterSD/172.16.4.18:_engine (mount:204) >>> 2018-08-29 16:47:28,846+0200 ERROR (jsonrpc/3) [storage.HSM] Could not >>> connect to storageServer (hsm:2398) >>> >>> Can you try to see if you are able to mount 172.16.4.18:/engine on the >>> server you're deploying Hosted Engine using "mount -t glusterfs 172.16.4.18:/engine >>> /mnt/test" >>> >>> >>>> During the deployment of the engine it sets the engine entry in the >>>> /etc/hosts file with the IP Address of 192.168.124.* which it gets form the >>>> virbr0 bridge interface. I stopped the bridge and deleted it, but still >>>> giving the same error. Not sure what causes it to use that interface. >>>> Please help! >>>> >>>> But I give the engine an IP of 192.168.1.10 same subnet as my gateway >>>> and my ovirtmgmt bridge. Attached is the ifconfig output of my Node, >>>> engine.log and vdsm.log. >>>> >>>> Your assistance is always appreciated >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jul 11, 2018 at 11:47 AM, Sahina Bose <sabose(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Is glusterd running on the server: goku.sanren.** >>>>> There's an error >>>>> Failed to get volume info: Command execution failed >>>>> error: Connection failed. Please check if gluster daemon is >>>>> operational >>>>> >>>>> Please check the volume status using "gluster volume status engine" >>>>> >>>>> and if all looks ok, attach the mount logs from /var/log/glusterfs >>>>> >>>>> On Wed, Jul 11, 2018 at 1:57 PM, Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have managed to fix the error by enabling the DMA Virtualisation >>>>>> in BIOS. I am now hit with a new error: It's failing to add a glusterfs >>>>>> storage domain: >>>>>> >>>>>> [ INFO ] TASK [Add glusterfs storage domain] >>>>>> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is >>>>>> "[Problem while trying to mount target]". HTTP response code is 400. >>>>>> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >>>>>> "Fault reason is \"Operation Failed\". Fault detail is \"[Problem while >>>>>> trying to mount target]\". HTTP response code is 400."} >>>>>> Please specify the storage you would like to use >>>>>> (glusterfs, iscsi, fc, nfs)[nfs]: >>>>>> >>>>>> Attached are vdsm and engine log files. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jul 11, 2018 at 9:57 AM, Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 11, 2018 at 9:33 AM, Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Below are the versions of packages installed. Please find the logs >>>>>>>> attached. >>>>>>>> Qemu: >>>>>>>> ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch >>>>>>>> libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64 >>>>>>>> qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64 >>>>>>>> qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64 >>>>>>>> qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64 >>>>>>>> >>>>>>>> Libvirt installed packages: >>>>>>>> libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-config-nwfilter-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-network-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-libs-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-secret-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-python-3.9.0-1.el7.x86_64 >>>>>>>> libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-config-network-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-client-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-kvm-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-interface-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-lock-sanlock-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-lxc-3.9.0-14.el7_5.6.x86_64 >>>>>>>> libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.6.x86_64 >>>>>>>> >>>>>>>> Virt-manager: >>>>>>>> virt-manager-common-1.4.3-3.el7.noarch >>>>>>>> >>>>>>>> oVirt: >>>>>>>> [root@localhost network-scripts]# rpm -qa | grep ovirt >>>>>>>> ovirt-setup-lib-1.1.4-1.el7.centos.noarch >>>>>>>> cockpit-ovirt-dashboard-0.11.28-1.el7.noarch >>>>>>>> ovirt-imageio-common-1.3.1.2-0.el7.centos.noarch >>>>>>>> ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch >>>>>>>> ovirt-host-dependencies-4.2.3-1.el7.x86_64 >>>>>>>> ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch >>>>>>>> ovirt-imageio-daemon-1.3.1.2-0.el7.centos.noarch >>>>>>>> ovirt-host-4.2.3-1.el7.x86_64 >>>>>>>> python-ovirt-engine-sdk4-4.2.7-2.el7.x86_64 >>>>>>>> ovirt-host-deploy-1.7.4-1.el7.noarch >>>>>>>> cockpit-machines-ovirt-169-1.el7.noarch >>>>>>>> ovirt-hosted-engine-ha-2.2.14-1.el7.noarch >>>>>>>> ovirt-vmconsole-1.0.5-4.el7.centos.noarch >>>>>>>> ovirt-provider-ovn-driver-1.2.11-1.el7.noarch >>>>>>>> ovirt-engine-appliance-4.2-20180626.1.el7.noarch >>>>>>>> ovirt-release42-4.2.4-1.el7.noarch >>>>>>>> ovirt-hosted-engine-setup-2.2.22.1-1.el7.noarch >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 11, 2018 at 6:48 AM, Yedidyah Bar David < >>>>>>>> didi(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> On Tue, Jul 10, 2018 at 11:32 PM, Sakhi Hadebe < >>>>>>>>> sakhi(a)sanren.ac.za&gt; wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I did not select any CPU architecture. It doenst gove me the >>>>>>>>>> option to select one. It only states the number of virtual CPUs and the >>>>>>>>>> memory for the engine VM. >>>>>>>>>> >>>>>>>>>> Looking at the documentation of installing >>>>>>>>>> ovirt-release36.rpm....it does allow you to select te CPU, but not when >>>>>>>>>> installing ovirt-release42.rpm >>>>>>>>>> >>>>>>>>>> On Tuesday, July 10, 2018, Alastair Neil <ajneil.tech(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> what did you select as your CPU architecture when you created >>>>>>>>>>> the cluster? It looks like the VM is trying to use a CPU type of "Custom", >>>>>>>>>>> how many nodes in your cluster? I suggest you specify the lowest common >>>>>>>>>>> denominator of CPU architecture (e.g. Sandybridge) of the nodes as the CPU >>>>>>>>>>> architecture of the cluster.. >>>>>>>>>>> >>>>>>>>>>> On Tue, 10 Jul 2018 at 12:01, Sakhi Hadebe <sakhi(a)sanren.ac.za&gt; >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I have just re-installed centOS 7 in 3 servers and have >>>>>>>>>>>> configured gluster volumes following this documentation: >>>>>>>>>>>> https://www.ovirt.org/blog/2016/03/up-and-running-with-ovirt-3-6/, >>>>>>>>>>>> But I have installed >>>>>>>>>>>> >>>>>>>>>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm >>>>>>>>>>>> >>>>>>>>>>>> package. >>>>>>>>>>>> Hosted-engine --deploy is failing with this error: >>>>>>>>>>>> >>>>>>>>>>>> "rhel7", "--virt-type", "kvm", "--memory", "16384", >>>>>>>>>>>> "--vcpus", "4", "--network", >>>>>>>>>>>> "network=default,mac=00:16:3e:09:5e:5d,model=virtio", "--disk", >>>>>>>>>>>> "/var/tmp/localvm0nnJH9/images/eacac30d-0304-4c77-8753-6965e4b8c2e7/d494577e-027a-4209-895b-6132e6fc6b9a", >>>>>>>>>>>> "--import", "--disk", "path=/var/tmp/localvm0nnJH9/seed.iso,device=cdrom", >>>>>>>>>>>> "--noautoconsole", "--rng", "/dev/random", "--graphics", "vnc", "--video", >>>>>>>>>>>> "vga", "--sound", "none", "--controller", "usb,model=none", "--memballoon", >>>>>>>>>>>> "none", "--boot", "hd,menu=off", "--clock", "kvmclock_present=yes"], >>>>>>>>>>>> "delta": "0:00:00.979003", "end": "2018-07-10 17:55:11.308555", "msg": >>>>>>>>>>>> "non-zero return code", "rc": 1, "start": "2018-07-10 17:55:10.329552", >>>>>>>>>>>> "stderr": "ERROR unsupported configuration: CPU mode 'custom' for x86_64 >>>>>>>>>>>> kvm domain on x86_64 host is not supported by hypervisor\nDomain >>>>>>>>>>>> installation does not appear to have been successful.\nIf it was, you can >>>>>>>>>>>> restart your domain by running:\n virsh --connect qemu:///system start >>>>>>>>>>>> HostedEngineLocal\notherwise, please restart your installation.", >>>>>>>>>>>> "stderr_lines": ["ERROR unsupported configuration: CPU mode 'custom' for >>>>>>>>>>>> x86_64 kvm domain on x86_64 host is not supported by hypervisor", "Domain >>>>>>>>>>>> installation does not appear to have been successful.", "If it was, you can >>>>>>>>>>>> restart your domain by running:", " virsh --connect qemu:///system start >>>>>>>>>>>> HostedEngineLocal", "otherwise, please restart your installation."], >>>>>>>>>>>> "stdout": "\nStarting install...", "stdout_lines": ["", "Starting >>>>>>>>>>>> install..."]} >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> This seems to be in the phase where we create a local vm for the >>>>>>>>> engine. We do this with plain virt-install, nothing fancy. Searching the >>>>>>>>> net for "unsupported configuration: CPU mode 'custom'" finds other relevant >>>>>>>>> reports, you might want to check them. You can see the command in >>>>>>>>> bootstrap_local_vm.yml . >>>>>>>>> >>>>>>>>> Please check/share versions of relevant packages (libvirt*, >>>>>>>>> qemu*, etc) and relevant logs (libvirt). >>>>>>>>> >>>>>>>>> Also updating the subject line and adding Simone. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> -- >>>>>>>>> Didi >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Sakhi Hadebe >>>>>>>> >>>>>>>> Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR >>>>>>>> >>>>>>>> Tel: +27 12 841 2308 <+27128414213> >>>>>>>> Fax: +27 12 841 4223 <+27128414223> >>>>>>>> Cell: +27 71 331 9622 <+27823034657> >>>>>>>> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt; >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Sakhi Hadebe >>>>>>> >>>>>>> Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR >>>>>>> >>>>>>> Tel: +27 12 841 2308 <+27128414213> >>>>>>> Fax: +27 12 841 4223 <+27128414223> >>>>>>> Cell: +27 71 331 9622 <+27823034657> >>>>>>> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt; >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Sakhi Hadebe >>>>>> >>>>>> Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR >>>>>> >>>>>> Tel: +27 12 841 2308 <+27128414213> >>>>>> Fax: +27 12 841 4223 <+27128414223> >>>>>> Cell: +27 71 331 9622 <+27823034657> >>>>>> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt; >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/YHKUKW22QLR... >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Sakhi Hadebe >>>> >>>> Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR >>>> >>>> Tel: +27 12 841 2308 <+27128414213> >>>> Fax: +27 12 841 4223 <+27128414223> >>>> Cell: +27 71 331 9622 <+27823034657> >>>> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt; >>>> >>>> >>> >> >> >> -- >> Regards, >> Sakhi Hadebe >> >> Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR >> >> Tel: +27 12 841 2308 <+27128414213> >> Fax: +27 12 841 4223 <+27128414223> >> Cell: +27 71 331 9622 <+27823034657> >> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt; >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/DS7O33ZSGV5... >> > -- Regards, Sakhi Hadebe Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR Tel: +27 12 841 2308 <+27128414213> Fax: +27 12 841 4223 <+27128414223> Cell: +27 71 331 9622 <+27823034657> Email: sakhi(a)sanren.ac.za <shadebe(a)csir.co.za&gt;