12:45 p.m.
Hi everybody!
I am having a hard time getting oVirt 4.4 to work. We want to update our
4.3 Cluster and i am trying to set up a fresh 4.4 Cluster (and restore
the backup later on) in order to update to 4.5. It fails at the end of
the engine deloyment, when the Gluster Storage Domain should be added.
I installed oVirt Node 4.4.10 on an old PC and made the following
modifications to the engine deployment process:
- altered defaults in
/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/defaults/main.yml:
- "he_pause_before_engine_setup: true" (in this pause before engine
setup, i ssh into the engine and exclude the package postgresql-jdbc
from update, which otherwise breaks the deployment [1])
- "he_remove_appliance_rpm: false" (to avoid the large download every
single try, i tried a lot)
- "he_force_ip4: true" (to avoid problems with IPv6, see below)
- in
/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/fetch_host_ip.yml
i added after "- name: Get host address resolution:" the following lines
(to avoid a problem with an "invalid" IPv6-Adress, which otherwise
breaks the deployment [2]):
- name: Get host IP addresses
ansible.builtin.command: hostname -I
register: hostname_addresses_output
changed_when: true
Most times, i started deployment via shell but tried via webinterface of
the node as well. It fails at the task "Add glusterfs storage domain"
with the following message:
"[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is "[Failed to fetch Gluster Volume List]". HTTP response code is
400." (See also [3])
When the setup asks for storage, i tried different answers
(gluster.local:/volume, gluster.local:/path/to/brick/volume,
192.168.8.51:/volume ...), no mount options.
I added firewall rules for glusterfs at node and engine. Even tried
disabling the firewall. No firewall on the gluster servers running. On
the Node, i also tested setting SELinux to permissive.
Recorded the traffic at different interfaces ("ovirtmgmt" and "virbr0"
on the node and "eth0" on the engine) and i can see the node and the
gluster server talking: Node gets the volume with options (which are,
btw, compliant to the docs, "storage.owner-gid: 36" "storage.owner-uid:
36" etc) but thats it, no further packets to mount the volume.
I noticed some ARP packets as well, the node asks the IP from the engine
(the configured static IP, which is not yet active). And the engine
sends a dns request for the gluster server to the node (via interface
virbr0), but doesnt connect to the gluster server. At least, thats what
i can see, most of the traffic is TLS, which i couldnt decrypt yet. I
appreciate any hint where to find the right keys.
Anyway, i can ssh from the engine to the gluster server and mount the
gluster volume manually on the node (mount -t glusterfs
gluster.local:/volume /local/path), so there seem no connectivity
issues.
Since the engine deployment log is around 30MB i attached a log summary
with findings i found relevant. I'll provide more logs if needed.
I really wanna put this huge timesink to an end. Can anyone help me or
point me in the right direction?
Many thanks in advance :)
Regards,
Niko
[1] This was the error message i got:
"[ ERROR ] fatal: [localhost -> 192.168.222.195]: FAILED! =>
{"attempts": 30, "changed": false, "connection":
"close", "content":
"Error500 - Internal Server Error", "content_encoding":
"identity",
"content_length": "86", "content_type": "text/html;
charset=UTF-8",
"date": "Wed, 17 May 2023 22:42:27 GMT", "elapsed": 0,
"msg": "Status
code was 500 and not [200]: HTTP Error 500: Internal Server Error",
"redirected": false, "server": "Apache/2.4.37 (centos)
OpenSSL/1.1.1k
mod_auth_gssapi/1.6.1 mod_wsgi/4.6.4 Python/3.6", "status": 500,
"url":
"http://localhost/ovirt-engine/services/health"}"
[2] This was the error message i got:
"VDSM ovirt.martinwi.local command HostSetupNetworksVDS failed: Internal
JSON-RPC error: {'reason': "Invalid IP address:
'fe80::ea3f:67ff:fe7f:a029%ovirtmgmt' does not appear to be an IPv4 or
IPv6 address"}"
[3]
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230718140628-rryscj.log:
2023-07-18 16:32:35,877+0200 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:106 {'msg': 'Fault reason is "Operation
Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP
response code is 400.', 'exception': 'Traceback (most recent call
last):\n File
"/tmp/ansible_ovirt_storage_domain_payload_b4ofbzxa/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py",
line 804, in main\n File
"/tmp/ansible_ovirt_storage_domain_payload_b4ofbzxa/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py",
line 674, in create\n **kwargs\n File
"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py", line 26258,
in add\n return self._internal_add(storage_domain, headers, query,
wait)\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py",
line 232, in _internal_add\n return future.wait() if wait else
future\n File
"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 55, in
wait\n return self._code(response)\n File
"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 229, in
callback\n self._check_fault(response)\n File
"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 132, in
_check_fault\n self._raise_error(response, body)\n File
"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 118, in
_raise_error\n raise error\novirtsdk4.Error: Fault reason is
"Operation Failed". Fault detail is "[Failed to fetch Gluster Volume
List]". HTTP response code is 400.\n', 'invocation':
{'module_args':
{'state': 'unattached', 'name': 'hosted_storage',
'host':
'ovirt.martinwi.local', 'data_center': 'Default', 'wait':
True,
'glusterfs': {'address': 'gluster1.martinwi.local',
'path': '/gv3',
'mount_options': ''}, 'timeout': 180, 'poll_interval': 3,
'fetch_nested': False, 'nested_attributes': [], 'domain_function':
'data', 'id': None, 'description': None, 'comment': None,
'localfs':
None, 'nfs': None, 'iscsi': None, 'managed_block_storage': None,
'posixfs': None, 'fcp': None, 'wipe_after_delete': None,
'backup': None,
'critical_space_action_blocker': None, 'warning_low_space': None,
'destroy': None, 'format': None, 'discard_after_delete': None}},
'_ansible_no_log': False, 'changed': False}