[ovirt-users] Moving hosted engine to iscsi storage domain

11 Sep 2024

      Hello,

We are attempting to move a hosted engine from an NFS to an iSCSI storage domain, following the normal backup / restore procedure.

With a little intervention in the new hosted engine VM, everything seems to be working with the new engine and the process gets to the point of trying to create the new iSCSI hosted_storage domain.   At this point, it fails with "Storage domain cannot be reached. Please ensure it is accessible from the host(s).”

The host itself does see the storage, and everything looks fine using iscsiadm and multipath commands.  However, if I look at the new hosted_storage domain in the new engine, it’s stuck “unattached” and the “hosted-engine --deploy --restore-from-file …” command loops at the "Please specify the storage you would like to use” step.

Please see below for an excerpt of the output and logs, and let me know what additional information I can provide.

Many thanks,
Devin

Here’s the output from the  “hosted-engine --deploy --restore-from-file …” command.
———
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : iSCSI login]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get iSCSI LUNs]
[ INFO  ] ok: [localhost]
          The following luns have been found on the requested target:
                  [1]     mpathm  1024.0GiB       IFT     DS 3000 Series
                          status: free, paths: 8 active
                    Please select the destination LUN (1) [1]:
[ INFO  ] iSCSI discard after delete is disabled
[ INFO  ] Creating Storage Domain
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage interface to be up]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir existence]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major version]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor version]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Add iSCSI storage domain]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Add Fibre Channel storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get storage domain details]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Find the appliance OVF]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get ovf data]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get disk size from ovf data]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Get required size]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Remove unsuitable storage domain]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Check storage domain free space]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]\". HTTP response code is 400.”}
———

Here’s an excerpt from the "ovirt-hosted-engine-setup-ansible-create_storage_domain-20240911081524-c7yrkf.log”:
———
2024-09-11 08:16:12,354-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml', 'ansible_task': 'ovirt.ovirt.hosted_engine_setup : Activate storage domain'}
2024-09-11 08:16:12,354-0400 DEBUG ansible on_any args TASK: ovirt.ovirt.hosted_engine_setup : Activate storage domain  kwargs is_conditional:False  2024-09-11 08:16:12,355-0400 DEBUG ansible on_any args localhost TASK: ovirt.ovirt.hosted_engine_setup : Activate storage domain  kwargs  2024-09-11 08:16:25,550-0400 DEBUG var changed: host "localhost" var "ansible_play_hosts" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var "ansible_play_batch" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var "play_hosts" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_no_log": false,
        "changed": false,
        "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\", line 811, in main\n  File \"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\", line 660, in post_create_check\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py\", line 3647, in add\n    return self._internal_add(storage_domain, headers, query, wait)\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n    return future.wait() if wait else future\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 55, in wait\n    return self._code(response)\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 229, in callback\n    self._check_fault(response)\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n    self._raise_error(response, body)\n  File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n    raise error\novirtsdk4.Error: Fault reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]\". HTTP response code is 400.\n",
        "invocation": {
            "module_args": {
                "backup": null,
                "comment": null,
                "critical_space_action_blocker": null,
                "data_center": "Default",
                "description": null,
                "destroy": null,
                "discard_after_delete": null,
                "domain_function": "data",
                "fcp": null,
                "fetch_nested": false,
                "format": null,
                "glusterfs": null,
                "host": "lnxvirt01-p55.classe.cornell.edu",
                "id": null,
                "iscsi": null,
                "localfs": null,
                "managed_block_storage": null,
                "name": "hosted_storage",
                "nested_attributes": [],
                "nfs": null,
                "poll_interval": 3,
                "posixfs": null,
                "state": "present",
                "storage_format": null,
                "timeout": 180,
                "wait": true,
                "warning_low_space": null,
                "wipe_after_delete": null
            }
        },
        "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]\". HTTP response code is 400."
    },
    "ansible_task": "Activate storage domain",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 13
}
2024-09-11 08:16:25,551-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7fac823de1f0>  kwargs ignore_errors:None  2024-09-11 08:16:25,552-0400 INFO ansible stats {
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
    "ansible_playbook_duration": "00:59 Minutes",
    "ansible_result": "type: <class 'dict'>\nstr: {'localhost': {'ok': 20, 'failures': 1, 'unreachable': 0, 'changed': 1, 'skipped': 9, 'rescued': 0, 'ignored': 0}}",
    "ansible_type": "finish",
    "status": "FAILED"
}
2024-09-11 08:16:25,552-0400 INFO SUMMARY:
Duration        Task Name
--------        --------
[ < 1 sec ]     Execute just a specific set of steps
[  00:03  ]     Force facts gathering
[  00:01  ]     Check local VM dir stat
[  00:01  ]     Obtain SSO token using username/password credentials
[  00:01  ]     Fetch host facts
[ < 1 sec ]     Fetch cluster ID
[  00:01  ]     Fetch cluster facts
[  00:01  ]     Fetch Datacenter facts
[ < 1 sec ]     Fetch Datacenter ID
[ < 1 sec ]     Fetch Datacenter name
[ < 1 sec ]     Fetch cluster name
[ < 1 sec ]     Fetch cluster version
[ < 1 sec ]     Set storage_format
[  00:15  ]     Add iSCSI storage domain
[  00:01  ]     Get storage domain details
[  00:01  ]     Find the appliance OVF
[  00:01  ]     Get ovf data
[ < 1 sec ]     Get disk size from ovf data
[ < 1 sec ]     Get required size
[ FAILED  ]     Activate storage domain
2024-09-11 08:16:25,553-0400 DEBUG ansible on_any args <ansible.executor.stats.AggregateStats object at 0x7fac83c8a700>  kwargs 
———

And here’s from the "ovirt-hosted-engine-setup-20240910222757-5ru4mo.log”:
———
2024-09-11 08:16:12,144-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
2024-09-11 08:16:25,264-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'exception': 'Traceback (most recent call last):\n  File "/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py", line 811, in main\n  File "/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py", line 660, in post_create_check\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py", line 3647, in add\n    return self._internal_add(storage_domain, headers, query, wait)\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n    return future.wait() if wait else future\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 55, in wait\n    return self._code(response)\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 229, in callback\n    self._check_fault(response)\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n    self._raise_error(response, body)\n  File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n    raise error\novirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400.\n', 'msg': 'Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400.', 'invocation': {'module_args': {'host': 'lnxvirt01-p55.classe.cornell.edu', 'data_center': 'Default', 'name': 'hosted_storage', 'wait': True, 'state': 'present', 'timeout': 180, 'poll_interval': 3, 'fetch_nested': False, 'nested_attributes': [], 'domain_function': 'data', 'id': None, 'description': None, 'comment': None, 'localfs': None, 'nfs': None, 'iscsi': None, 'managed_block_storage': None, 'posixfs': None, 'glusterfs': None, 'fcp': None, 'wipe_after_delete': None, 'backup': None, 'critical_space_action_blocker': None, 'warning_low_space': None, 'destroy': None, 'format': None, 'discard_after_delete': None, 'storage_format': None}}, '_ansible_no_log': False, 'changed': False}
2024-09-11 08:16:25,364-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400.
2024-09-11 08:16:25,464-0400 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]\". HTTP response code is 400."}
2024-09-11 08:16:25,765-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 20 changed: 1 unreachable: 0 skipped: 9 failed: 1
2024-09-11 08:16:25,866-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:226 ansible-playbook rc: 2
2024-09-11 08:16:25,866-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:233 ansible-playbook stdout:
2024-09-11 08:16:25,867-0400 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:236 ansible-playbook stderr:
2024-09-11 08:16:25,867-0400 DEBUG otopi.plugins.otopi.dialog.human human.queryString:174 query OVEHOSTED_STORAGE_DOMAIN_TYPE
2024-09-11 08:16:25,868-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]:
———

[ovirt-users] Moving hosted engine to iscsi storage domain

Devin A. Bougie