I'm just testing deployment of HCI single host using oVirt Node NG CentOS 7 iso
I was able to complete the gluster setup via cockpit with these modifications:
1) I wanted to check via ssh and found that files *key under /etc/ssh/ had too weak config so that ssh daemon didn't started after installation of node from iso
changing to 600 and restarting the service was ok
2) I used a single disk configured as jbod, so I choose that option instead of the default proposed RAID6
But the playook failed with
. . .
PLAY [gluster_servers] *********************************************************
TASK [Create LVs with specified size for the VGs] ******************************
changed: [192.168.124.211] => (item={u'lv': u'gluster_thinpool_sdb', u'size': u'50GB', u'extent': u'100%FREE', u'vg': u'gluster_vg_sdb'})
PLAY RECAP *********************************************************************
192.168.124.211 : ok=1 changed=1 unreachable=0 failed=0
Ignoring errors...
Error: Section diskcount not found in the configuration file
Reading inside the playbooks involved here:
/usr/share/gdeploy/playbooks/auto_lvcreate_for_gluster.yml
/usr/share/gdeploy/playbooks/vgcreate.yml
and the snippet
- name: Convert the logical volume
lv: action=convert thinpool={{
item.vg }}/{{item.pool }}
poolmetadata={{
item.vg }}/'metadata' poolmetadataspare=n
vgname={{
item.vg }} disktype="{{disktype}}"
diskcount="{{ diskcount }}"
stripesize="{{stripesize}}"
chunksize="{{ chunksize | default('') }}"
snapshot_reserve="{{ snapshot_reserve }}"
with_items: "{{ lvpools }}"
ignore_errors: yes
I simply edited the gdeploy.conf from the gui button adding this section under the [disktype] one
"
[diskcount]
1
"
then clean lv/vg/pv and the gdeploy step completed successfully
3) at first stage of ansible deploy I have this failed command that seems not to prevent from completion but that I have not understood..
PLAY [gluster_servers] *********************************************************
TASK [Run a command in the shell] **********************************************
failed: [192.168.124.211] (item=vdsm-tool configure --force) => {"changed": true, "cmd": "vdsm-tool configure --force", "delta": "0:00:01.475528", "end": "2019-01-11 10:59:55.147601", "item": "vdsm-tool configure --force", "msg": "non-zero return code", "rc": 1, "start": "2019-01-11 10:59:53.672073", "stderr": "Traceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 220, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python2.7/site-packages/vdsm/tool/__init__.py\", line 40, in wrapper\n func(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py\", line 143, in configure\n _configure(c)\n File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py\", line 90, in _configure\n getattr(module, 'configure', lambda: None)()\n File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurators/bond_defaults.py\", line 39, in configure\n sysfs_options_mapper.dump_bonding_options()\n File \"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py\", line 48, in dump_bonding_options\n with open(sysfs_options.BONDING_DEFAULTS, 'w') as f:\nIOError: [Errno 2] No such file or directory: '/var/run/vdsm/bonding-defaults.json'", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/vdsm-tool\", line 220, in main", " return tool_command[cmd][\"command\"](*args)", " File \"/usr/lib/python2.7/site-packages/vdsm/tool/__init__.py\", line 40, in wrapper", " func(*args, **kwargs)", " File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py\", line 143, in configure", " _configure(c)", " File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurator.py\", line 90, in _configure", " getattr(module, 'configure', lambda: None)()", " File \"/usr/lib/python2.7/site-packages/vdsm/tool/configurators/bond_defaults.py\", line 39, in configure", " sysfs_options_mapper.dump_bonding_options()", " File \"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py\", line 48, in dump_bonding_options", " with open(sysfs_options.BONDING_DEFAULTS, 'w') as f:", "IOError: [Errno 2] No such file or directory: '/var/run/vdsm/bonding-defaults.json'"], "stdout": "\nChecking configuration status...\n\nabrt is already configured for vdsm\nlvm is configured for vdsm\nlibvirt is already configured for vdsm\nSUCCESS: ssl configured to true. No conflicts\nManual override for multipath.conf detected - preserving current configuration\nThis manual override for multipath.conf was based on downrevved template. You are strongly advised to contact your support representatives\n\nRunning configure...\nReconfiguration of abrt is done.\nReconfiguration of passwd is done.\nReconfiguration of libvirt is done.", "stdout_lines": ["", "Checking configuration status...", "", "abrt is already configured for vdsm", "lvm is configured for vdsm", "libvirt is already configured for vdsm", "SUCCESS: ssl configured to true. No conflicts", "Manual override for multipath.conf detected - preserving current configuration", "This manual override for multipath.conf was based on downrevved template. You are strongly advised to contact your support representatives", "", "Running configure...", "Reconfiguration of abrt is done.", "Reconfiguration of passwd is done.", "Reconfiguration of libvirt is done."]}
to retry, use: --limit @/tmp/tmpQXe2el/shell_cmd.retry
PLAY RECAP *********************************************************************
192.168.124.211 : ok=0 changed=0 unreachable=0 failed=1
Would it be possible to save in some way the ansible playbook log even if it completes ok, without going directly to the "successful" page?
Or is anyway stored in some location on disk of host?
I then proceeded with Hosted Engine install/setup and
4) it fails here at final stages of the local vm engine setup during host activation:
[ INFO ] TASK [oVirt.hosted-engine-setup : Set Engine public key as authorized key without validating the TLS/SSL certificates]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Obtain SSO token using username/password credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Ensure that the target datacenter is present]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Ensure that the target cluster is present in the target datacenter]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Enable GlusterFS at cluster level]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Set VLAN ID at datacenter level]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Force host-deploy in offline mode]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Add host]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Wait for the host to be up]
then after several minutes:
[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": [{"address": "ov4301.localdomain.local", "affinity_labels": [], "auto_numa_status": "unknown", "certificate": {"organization": "localdomain.local", "subject": "O=localdomain.local,CN=ov4301.localdomain.local"}, "cluster": {"href": "/ovirt-engine/api/clusters/5e8fea14-158b-11e9-b2f0-00163e29b9f2", "id": "5e8fea14-158b-11e9-b2f0-00163e29b9f2"}, "comment": "", "cpu": {"speed": 0.0, "topology": {}}, "device_passthrough": {"enabled": false}, "devices": [], "external_network_provider_configurations": [], "external_status": "ok", "hardware_information": {"supported_rng_sources": []}, "hooks": [], "href": "/ovirt-engine/api/hosts/4202de75-75d3-4dcb-b128-2c4a1d257a15", "id": "4202de75-75d3-4dcb-b128-2c4a1d257a15", "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, "max_scheduling_memory": 0, "memory": 0, "name": "ov4301.localdomain.local", "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": false, "os": {"custom_kernel_cmdline": ""}, "permissions": [], "port": 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", "se_linux": {}, "spm": {"priority": 5, "status": "none"}, "ssh": {"fingerprint": "SHA256:iqeQjdWCm15+xe74xEnswrgRJF7JBAWrvsjO/RaW8q8", "port": 22}, "statistics": [], "status": "install_failed", "storage_connection_extensions": [], "summary": {"total": 0}, "tags": [], "transparent_huge_pages": {"enabled": false}, "type": "ovirt_node", "unmanaged_networks": [], "update_available": false}]}, "attempts": 120, "changed": false}
[ INFO ] TASK [oVirt.hosted-engine-setup : Fetch logs from the engine VM]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Set destination directory path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Create destination directory]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Find the local appliance image]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : debug]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Set local_vm_disk_path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Give the vm time to flush dirty buffers]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Copy engine logs]
[ INFO ] TASK [oVirt.hosted-engine-setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Remove local vm dir]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : debug]
[ INFO ] ok: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Remove temporary entry in /etc/hosts for the local VM]
[ INFO ] changed: [localhost]
[ INFO ] TASK [oVirt.hosted-engine-setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
Going to see the log /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-20190111113227-ov4301.localdomain.local-5d387e0d.log
it seems the error is about ovirt-imageio-daemon
2019-01-11 11:32:26,893+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service'), rc=1
2019-01-11 11:32:26,894+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stdout:
2019-01-11 11:32:26,895+0100 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'ovirt-imageio-daemon.service') stderr:
Job for ovirt-imageio-daemon.service failed because the control process exited with error code. See "systemctl status ovirt-imageio-daemon.service" and "journalctl -xe" for details.
2019-01-11 11:32:26,896+0100 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
File "/tmp/ovirt-PBFI2dyoDO/pythonlib/otopi/context.py", line 133, in _executeMethod
method['method']()
File "/tmp/ovirt-PBFI2dyoDO/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 175, in _start
self.services.state('ovirt-imageio-daemon', True)
File "/tmp/ovirt-PBFI2dyoDO/otopi-plugins/otopi/services/systemd.py", line 141, in state
service=name,
RuntimeError: Failed to start service 'ovirt-imageio-daemon'
2019-01-11 11:32:26,898+0100 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio-daemon'
2019-01-11 11:32:26,899+0100 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventEnd STAGE closeup METHOD otopi.plugins.ovirt_host_deploy.vdsm.packages.Plugin._start (odeploycons.packages.vdsm.started)
The reason:
[root@ov4301 ~]# systemctl status ovirt-imageio-daemon -l
● ovirt-imageio-daemon.service - oVirt ImageIO Daemon
Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-daemon.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2019-01-11 11:32:29 CET; 27min ago
Process: 11625 ExecStart=/usr/bin/ovirt-imageio-daemon (code=exited, status=1/FAILURE)
Main PID: 11625 (code=exited, status=1/FAILURE)
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: ovirt-imageio-daemon.service: main process exited, code=exited, status=1/FAILURE
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: Failed to start oVirt ImageIO Daemon.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: Unit ovirt-imageio-daemon.service entered failed state.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: ovirt-imageio-daemon.service failed.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: ovirt-imageio-daemon.service holdoff time over, scheduling restart.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: Stopped oVirt ImageIO Daemon.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: start request repeated too quickly for ovirt-imageio-daemon.service
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: Failed to start oVirt ImageIO Daemon.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: Unit ovirt-imageio-daemon.service entered failed state.
Jan 11 11:32:29 ov4301.localdomain.local systemd[1]: ovirt-imageio-daemon.service failed.
[root@ov4301 ~]#
The file /var/log/ovirt-imageio-daemon/daemon.log contains
2019-01-11 10:28:30,191 INFO (MainThread) [server] Starting (pid=3702, version=1.4.6)
2019-01-11 10:28:30,229 ERROR (MainThread) [server] Service failed (remote_service=<ovirt_imageio_daemon.server.RemoteService object at 0x7fea9dc88050>, local_service=<ovirt_imageio_daemon.server.LocalService object at 0x7fea9ca24850>, control_service=None, running=True)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 58, in main
start(config)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 99, in start
control_service = ControlService(config)
File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/server.py", line 206, in __init__
config.tickets.socket, uhttp.UnixWSGIRequestHandler)
File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
self.server_bind()
File "/usr/lib/python2.7/site-packages/ovirt_imageio_daemon/uhttp.py", line 79, in server_bind
self.socket.bind(self.server_address)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
One potential problem I noticed is that on this host I setup eth0 with 192.168.122.x (for ovirtmgmt) and eth1 with 192.168.124.y (for gluster, even if only one host, but aiming at adding other 2 hosts in second step) and the libvirt network temporarily created for the local engine vm is also on 192.168.124.0 network.....
4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:b8:6b:3c brd ff:ff:ff:ff:ff:ff
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:b8:6b:3c brd ff:ff:ff:ff:ff:ff
I can change my gluster network of this env and re-test, but would it be possible to have the libvirt network configurable? It seems risky to have a fixed one...
Can I go ahead from this failed hosted engine after understanding reason of ovirt-imageio-daemon failure or am I forced to scratch?
Supposing I go to power down and then power on again this host, how can I retry without scratching?
Gianluca