10:57 p.m.
Hello,
We are attempting to move a hosted engine from an NFS to an iSCSI storage domain,
following the normal backup / restore procedure.
With a little intervention in the new hosted engine VM, everything seems to be working
with the new engine and the process gets to the point of trying to create the new iSCSI
hosted_storage domain. At this point, it fails with "Storage domain cannot be
reached. Please ensure it is accessible from the host(s).”
The host itself does see the storage, and everything looks fine using iscsiadm and
multipath commands. However, if I look at the new hosted_storage domain in the new
engine, it’s stuck “unattached” and the “hosted-engine --deploy --restore-from-file …”
command loops at the "Please specify the storage you would like to use” step.
Please see below for an excerpt of the output and logs, and let me know what additional
information I can provide.
Many thanks,
Devin
Here’s the output from the “hosted-engine --deploy --restore-from-file …” command.
———
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : iSCSI login]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get iSCSI LUNs]
[ INFO ] ok: [localhost]
The following luns have been found on the requested target:
[1] mpathm 1024.0GiB IFT DS 3000 Series
status: free, paths: 8 active
Please select the destination LUN (1) [1]:
[ INFO ] iSCSI discard after delete is disabled
[ INFO ] Creating Storage Domain
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage interface to be
up]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir existence]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using username/password
credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add iSCSI storage domain]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add Fibre Channel storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get storage domain details]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Find the appliance OVF]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get ovf data]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get disk size from ovf data]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get required size]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Remove unsuitable storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check storage domain free space]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
"[Storage domain cannot be reached. Please ensure it is accessible from the
host(s).]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"Fault reason is \"Operation Failed\". Fault detail is \"[Storage
domain cannot be reached. Please ensure it is accessible from the host(s).]\". HTTP
response code is 400.”}
———
Here’s an excerpt from the
"ovirt-hosted-engine-setup-ansible-create_storage_domain-20240911081524-c7yrkf.log”:
———
2024-09-11 08:16:12,354-0400 INFO ansible task start {'status': 'OK',
'ansible_type': 'task', 'ansible_playbook':
'/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml',
'ansible_task': 'ovirt.ovirt.hosted_engine_setup : Activate storage
domain'}
2024-09-11 08:16:12,354-0400 DEBUG ansible on_any args TASK:
ovirt.ovirt.hosted_engine_setup : Activate storage domain kwargs is_conditional:False
2024-09-11 08:16:12,355-0400 DEBUG ansible on_any args localhost TASK:
ovirt.ovirt.hosted_engine_setup : Activate storage domain kwargs 2024-09-11
08:16:25,550-0400 DEBUG var changed: host "localhost" var
"ansible_play_hosts" type "<class 'list'>" value:
"[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var
"ansible_play_batch" type "<class 'list'>" value:
"[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var
"play_hosts" type "<class 'list'>" value:
"[]"
2024-09-11 08:16:25,551-0400 ERROR ansible failed {
"ansible_host": "localhost",
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
"ansible_result": {
"_ansible_no_log": false,
"changed": false,
"exception": "Traceback (most recent call last):\n File
\"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 811, in main\n File
\"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 660, in post_create_check\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py\", line 3647, in
add\n return self._internal_add(storage_domain, headers, query, wait)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 232, in
_internal_add\n return future.wait() if wait else future\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 55, in wait\n
return self._code(response)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 229, in
callback\n self._check_fault(response)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 132, in
_check_fault\n self._raise_error(response, body)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 118, in
_raise_error\n raise error\novirtsdk4.Error: Fault reason is \"Operation
Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it
is accessible from the host(s).]\". HTTP response code is 400.\n",
"invocation": {
"module_args": {
"backup": null,
"comment": null,
"critical_space_action_blocker": null,
"data_center": "Default",
"description": null,
"destroy": null,
"discard_after_delete": null,
"domain_function": "data",
"fcp": null,
"fetch_nested": false,
"format": null,
"glusterfs": null,
"host": "lnxvirt01-p55.classe.cornell.edu",
"id": null,
"iscsi": null,
"localfs": null,
"managed_block_storage": null,
"name": "hosted_storage",
"nested_attributes": [],
"nfs": null,
"poll_interval": 3,
"posixfs": null,
"state": "present",
"storage_format": null,
"timeout": 180,
"wait": true,
"warning_low_space": null,
"wipe_after_delete": null
}
},
"msg": "Fault reason is \"Operation Failed\". Fault
detail is \"[Storage domain cannot be reached. Please ensure it is accessible from
the host(s).]\". HTTP response code is 400."
},
"ansible_task": "Activate storage domain",
"ansible_type": "task",
"status": "FAILED",
"task_duration": 13
}
2024-09-11 08:16:25,551-0400 DEBUG ansible on_any args
<ansible.executor.task_result.TaskResult object at 0x7fac823de1f0> kwargs
ignore_errors:None 2024-09-11 08:16:25,552-0400 INFO ansible stats {
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
"ansible_playbook_duration": "00:59 Minutes",
"ansible_result": "type: <class 'dict'>\nstr:
{'localhost': {'ok': 20, 'failures': 1, 'unreachable': 0,
'changed': 1, 'skipped': 9, 'rescued': 0, 'ignored':
0}}",
"ansible_type": "finish",
"status": "FAILED"
}
2024-09-11 08:16:25,552-0400 INFO SUMMARY:
Duration Task Name
-------- --------
[ < 1 sec ] Execute just a specific set of steps
[ 00:03 ] Force facts gathering
[ 00:01 ] Check local VM dir stat
[ 00:01 ] Obtain SSO token using username/password credentials
[ 00:01 ] Fetch host facts
[ < 1 sec ] Fetch cluster ID
[ 00:01 ] Fetch cluster facts
[ 00:01 ] Fetch Datacenter facts
[ < 1 sec ] Fetch Datacenter ID
[ < 1 sec ] Fetch Datacenter name
[ < 1 sec ] Fetch cluster name
[ < 1 sec ] Fetch cluster version
[ < 1 sec ] Set storage_format
[ 00:15 ] Add iSCSI storage domain
[ 00:01 ] Get storage domain details
[ 00:01 ] Find the appliance OVF
[ 00:01 ] Get ovf data
[ < 1 sec ] Get disk size from ovf data
[ < 1 sec ] Get required size
[ FAILED ] Activate storage domain
2024-09-11 08:16:25,553-0400 DEBUG ansible on_any args
<ansible.executor.stats.AggregateStats object at 0x7fac83c8a700> kwargs
———
And here’s from the "ovirt-hosted-engine-setup-20240910222757-5ru4mo.log”:
———
2024-09-11 08:16:12,144-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Activate storage
domain]
2024-09-11 08:16:25,264-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 {'exception': 'Traceback (most recent call
last):\n File
"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py",
line 811, in main\n File
"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py",
line 660, in post_create_check\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py", line 3647, in add\n
return self._internal_add(storage_domain, headers, query, wait)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 232, in
_internal_add\n return future.wait() if wait else future\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 55, in wait\n
return self._code(response)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 229, in
callback\n self._check_fault(response)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 132, in
_check_fault\n self._raise_error(response, body)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 118, in
_raise_error\n raise error\novirtsdk4.Error: Fault reason is "Operation
Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is
accessible from the host(s).]". HTTP response code is 400.\n', 'msg':
'Fault reason is "Operation Failed". Fault detail is "[Storage domain
cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response
code is 400.', 'invocation': {'module_args': {'host':
'lnxvirt01-p55.classe.cornell.edu', 'data_center': 'Default',
'name': 'hosted_storage', 'wait': True, 'state':
'present', 'timeout': 180, 'poll_interval': 3,
'fetch_nested': False, 'nested_attributes': [], 'domain_function':
'data', 'id': None, 'description': None, 'comment': None,
'localfs': None, 'nfs': None, 'iscsi': None,
'managed_block_storage': None, 'posixfs': None, 'glusterfs': None,
'fcp': None, 'wipe_after_delete': None, 'backup': None,
'critical_space_action_blocker': None, 'warning_low_space': None,
'destroy': None, 'format': None, 'discard_after_delete': None,
'storage_format': None}}, '_ansible_no_log': False, 'changed':
False}
2024-09-11 08:16:25,364-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:113 ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is
accessible from the host(s).]". HTTP response code is 400.
2024-09-11 08:16:25,464-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed":
false, "msg": "Fault reason is \"Operation Failed\". Fault detail
is \"[Storage domain cannot be reached. Please ensure it is accessible from the
host(s).]\". HTTP response code is 400."}
2024-09-11 08:16:25,765-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 20 changed: 1 unreachable:
0 skipped: 9 failed: 1
2024-09-11 08:16:25,866-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils.run:226 ansible-playbook rc: 2
2024-09-11 08:16:25,866-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils.run:233 ansible-playbook stdout:
2024-09-11 08:16:25,867-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils.run:236 ansible-playbook stderr:
2024-09-11 08:16:25,867-0400 DEBUG otopi.plugins.otopi.dialog.human human.queryString:174
query OVEHOSTED_STORAGE_DOMAIN_TYPE
2024-09-11 08:16:25,868-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204
DIALOG:SEND Please specify the storage you would like to use (glusterfs,
iscsi, fc, nfs)[nfs]:
———