TASK [ovirt.hosted_engine_setup : Activate storage domain] ******************************************
...
Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.

usually means that the engine failed to activate that storage domain; unfortunately engine error messages are not always that clear (see https://bugzilla.redhat.com/1554922 ) but this is often due to fact the the NFS share or the iSCSI lun or whatever you used wasn't really clean.
Are you manually cleaning it between one attempt and the next one?   

On Tue, Apr 2, 2019 at 10:50 AM Callum Smith <callum@well.ox.ac.uk> wrote:
Dear Simone,

With no changes, we're now seeing this baffling error:

TASK [ovirt.hosted_engine_setup : Parse OVF] ********************************************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:120
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=
gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73
363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, '/root\n', '')
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=
gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73
363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320 `" && echo a
nsible-tmp-1553937522.31-129798476242320="` echo /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320 `" ) && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'ansible-tmp-1553937522.31-129798476242320=/root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320\n', '')
Using module file /opt/ansible/lib/ansible/modules/files/xml.py
<virthyp04.virt.in.bmrc.ox.ac.uk> PUT /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmp8wMU8o TO /root/.ansible/tmp/ansible-tmp-1553937522.31-12979847624
2320/AnsiballZ_xml.py
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentica
tions=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'sftp> put /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmp8wMU8o /root/.ansible/tmp/ansible-tmp-1553937522.31-129
798476242320/AnsiballZ_xml.py\n', '')
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=
gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73
363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/ /root/.ansible/tmp/ansible-tmp-1
553937522.31-129798476242320/AnsiballZ_xml.py && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=
gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73
363c -tt virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/AnsiballZ_xml.py && sle
ep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, '\r\n{"count": 1, "matches": [{"Disk": {"{http://schemas.dmtf.org/ovf/envelope/1/}wipe-after-delete": "false", "{http://
"5f2be758-82d7-4c07-a220-9060e782dc7a", "{http://schemas.dmtf.org/ovf/envelope/1/}parentRef": "", "{http://schemas.dmtf.org/ovf/envelope/1/}fileRef": "6f76686
b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5", "{http://schemas.dmtf.org/ovf/envelope/1/}actual_size": "51", "{http://schemas.dmtf.org/o
vf/envelope/1/}volume-format": "COW", "{http://schemas.dmtf.org/ovf/envelope/1/}boot": "true", "{http://schemas.dmtf.org/ovf/envelope/1/}size": "51", "{http:/
pe/1/}diskId": "72bc3948-5d8d-4877-bac8-7db4995045b5", "{http://schemas.dmtf.org/ovf/envelope/1/}disk-interface": "VirtIO"}}], "changed": false, "actions": {"
xpath": "/ovf:Envelope/Section/Disk", "state": "present", "namespaces": {"vssd": "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingDa
ta", "rasd": "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData", "xsi": "http://www.w3.org/2001/XMLSchema-instance", "ovf": "http://schemas.dmtf.org/ovf/envelope/1/"}}, "msg": 1, "invocation": {"module_args": {"xpath": "/ovf:Envelope/Section/Disk", "count": false, "set_children": null, "xmlstring": null, "strip_cdata_tags": false, "attribute": "ovf:size", "pretty_print": false, "add_children": null, "value": null, "content": "attribute", "state": "present", "namespaces": {"vssd": "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData", "rasd": "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData", "xsi": "http://www.w3.org/2001/XMLSchema-instance", "ovf": "http://schemas.dmtf.org/ovf/envelope/1/"}, "input_type": "yaml", "print_match": false, "path": "/var/tmp/localvmMNxnwL/master/vms/074a62d4-44f9-4ffe-a172-2702a9fe96df/074a62d4-44f9-4ffe-a172-2702a9fe96df.ovf", "backup": false}}}\r\n', 'Shared connection to virthyp04.virt.in.bmrc.ox.ac.uk closed.\r\n')
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1553937522.31-129798476242320/ > /dev/null 2>&1 && sleep 0'"'"''
    "actions": {
        "namespaces": {
            "ovf": "http://schemas.dmtf.org/ovf/envelope/1/",
        },
        "state": "present",
        "xpath": "/ovf:Envelope/Section/Disk"
    },
    "changed": false,
    "count": 1,
    "invocation": {
        "module_args": {
            "add_children": null,
            "attribute": "ovf:size",
            "backup": false,
            "content": "attribute",
            "count": false,
            "input_type": "yaml",
            "namespaces": {
                "ovf": "http://schemas.dmtf.org/ovf/envelope/1/",
                "xsi": "http://www.w3.org/2001/XMLSchema-instance"
            },
            "path": "/var/tmp/localvmMNxnwL/master/vms/074a62d4-44f9-4ffe-a172-2702a9fe96df/074a62d4-44f9-4ffe-a172-2702a9fe96df.ovf",
            "pretty_print": false,
            "print_match": false,
            "set_children": null,
            "state": "present",
            "strip_cdata_tags": false,
            "value": null,
            "xmlstring": null,
            "xpath": "/ovf:Envelope/Section/Disk"
        }
    },
    "matches": [
        {
            "Disk": {
                "{http://schemas.dmtf.org/ovf/envelope/1/}actual_size": "51",
                "{http://schemas.dmtf.org/ovf/envelope/1/}boot": "true",
                "{http://schemas.dmtf.org/ovf/envelope/1/}disk-interface": "VirtIO",
                "{http://schemas.dmtf.org/ovf/envelope/1/}disk-type": "System",
                "{http://schemas.dmtf.org/ovf/envelope/1/}diskId": "72bc3948-5d8d-4877-bac8-7db4995045b5",
                "{http://schemas.dmtf.org/ovf/envelope/1/}fileRef": "6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5",
                "{http://schemas.dmtf.org/ovf/envelope/1/}parentRef": "",
                "{http://schemas.dmtf.org/ovf/envelope/1/}size": "51",
                "{http://schemas.dmtf.org/ovf/envelope/1/}vm_snapshot_id": "5f2be758-82d7-4c07-a220-9060e782dc7a",
                "{http://schemas.dmtf.org/ovf/envelope/1/}volume-format": "COW",
                "{http://schemas.dmtf.org/ovf/envelope/1/}volume-type": "Sparse",
                "{http://schemas.dmtf.org/ovf/envelope/1/}wipe-after-delete": "false"
            }
        }
    ],
    "msg": 1
}

TASK [ovirt.hosted_engine_setup : Get required size] ************************************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:132
    "ansible_facts": {
        "required_size": "65498251264"
    },
    "changed": false
}

TASK [ovirt.hosted_engine_setup : debug] ************************************************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:139
    "required_size": "65498251264"
}

TASK [ovirt.hosted_engine_setup : Remove unsuitable storage domain] *********************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:140
    "changed": false,
    "skip_reason": "Conditional result was False"
}

TASK [ovirt.hosted_engine_setup : debug] ************************************************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:151
    "remove_storage_domain_details": {
        "changed": false,
        "skip_reason": "Conditional result was False",
        "skipped": true
    }
}

TASK [ovirt.hosted_engine_setup : Check storage domain free space] **********************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:152
    "changed": false,
    "skip_reason": "Conditional result was False"
}

TASK [ovirt.hosted_engine_setup : Activate storage domain] ******************************************
task path: /etc/ansible/playbook/ovirt-ansible-hosted-engine-setup/tasks/create_storage_domain.yml:161
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, '/root\n', '')
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848 `" && echo ansible-tmp-1553937525.89-247224387363848="` echo /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848 `" ) && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'ansible-tmp-1553937525.89-247224387363848=/root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848\n', '')
Using module file /opt/ansible/lib/ansible/modules/cloud/ovirt/ovirt_storage_domain.py
<virthyp04.virt.in.bmrc.ox.ac.uk> PUT /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmpQtVJtM TO /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c '[virthyp04.virt.in.bmrc.ox.ac.uk]'
<virthyp04.virt.in.bmrc.ox.ac.uk> (0, 'sftp> put /etc/ansible/.ansible/tmp/ansible-local-32213KmUe6/tmpQtVJtM /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py\n', '')
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/ /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c -tt virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/AnsiballZ_ovirt_storage_domain.py && sleep 0'"'"''
<virthyp04.virt.in.bmrc.ox.ac.uk> (1, '\r\n{"msg": "Fault reason is \\"Operation Failed\\". Fault detail is \\"[]\\". HTTP response code is 400.", "failed": true, "exception": "Traceback (most recent call last):\\n  File \\"/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py\\", line 664, in main\\n    storage_domains_module.post_create_check(sd_id)\\n  File \\"/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py\\", line 526, in post_create_check\\n    id=storage_domain.id,\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\\", line 3053, in add\\n    return self._internal_add(storage_domain, headers, query, wait)\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 232, in _internal_add\\n    return future.wait() if wait else future\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 55, in wait\\n    return self._code(response)\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 229, in callback\\n    self._check_fault(response)\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 132, in _check_fault\\n    self._raise_error(response, body)\\n  File \\"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\\", line 118, in _raise_error\\n    raise error\\nError: Fault reason is \\"Operation Failed\\". Fault detail is \\"[]\\". HTTP response code is 400.\\n", "invocation": {"module_args": {"comment": null, "warning_low_space": null, "fetch_nested": false, "localfs": null, "data_center": "Default", "id": null, "iscsi": null, "state": "present", "wipe_after_delete": null, "destroy": null, "fcp": null, "description": null, "format": null, "nested_attributes": [], "host": "virthyp04.virt.in.bmrc.ox.ac.uk", "discard_after_delete": null, "wait": true, "domain_function": "data", "name": "hosted_storage", "critical_space_action_blocker": null, "posixfs": null, "poll_interval": 3, "glusterfs": null, "nfs": null, "timeout": 180, "backup": null}}}\r\n', 'Shared connection to virthyp04.virt.in.bmrc.ox.ac.uk closed.\r\n')
<virthyp04.virt.in.bmrc.ox.ac.uk> Failed to connect to the host via ssh: Shared connection to virthyp04.virt.in.bmrc.ox.ac.uk closed.
<virthyp04.virt.in.bmrc.ox.ac.uk> ESTABLISH SSH CONNECTION FOR USER: root
<virthyp04.virt.in.bmrc.ox.ac.uk> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/etc/ansible/.ansible/cp/2c1e73363c virthyp04.virt.in.bmrc.ox.ac.uk '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1553937525.89-247224387363848/ > /dev/null 2>&1 && sleep 0'"'"''
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py", line 664, in main
    storage_domains_module.post_create_check(sd_id)
  File "/tmp/ansible_ovirt_storage_domain_payload_w8oO0Y/__main__.py", line 526, in post_create_check
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 3053, in add
    return self._internal_add(storage_domain, headers, query, wait)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add
    return future.wait() if wait else future
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback
    self._check_fault(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
    self._raise_error(response, body)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
    raise error
Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400.
fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "backup": null,
            "comment": null,
            "critical_space_action_blocker": null,
            "data_center": "Default",
            "description": null,
            "destroy": null,
            "discard_after_delete": null,
            "domain_function": "data",
            "fcp": null,
            "fetch_nested": false,
            "format": null,
            "glusterfs": null,
            "host": "virthyp04.virt.in.bmrc.ox.ac.uk",
            "id": null,
            "iscsi": null,
            "localfs": null,
            "name": "hosted_storage",
            "nested_attributes": [],
            "nfs": null,
            "poll_interval": 3,
            "posixfs": null,
            "state": "present",
            "timeout": 180,
            "wait": true,
            "warning_low_space": null,
            "wipe_after_delete": null
        }
    },
    "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."
}
        to retry, use: --limit @/etc/ansible/playbook/ovirt.retry

PLAY RECAP ******************************************************************************************
virthyp04.virt.in.bmrc.ox.ac.uk : ok=216  changed=69   unreachable=0    failed=1

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 17:48, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Fri, Mar 29, 2019 at 6:14 PM Callum Smith <callum@well.ox.ac.uk> wrote:
So close now:

fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": true, "cmd": ["dd", "bs=20480", "count=1", "oflag=direct", "if=/var/tmp/localvmDBMVgn/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6", "of=/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6"], "delta": "0:00:00.005134", "end": "2019-03-29 17:04:27.952367", "msg": "non-zero return code", "rc": 1, "start": "2019-03-29 17:04:27.947233", "stderr": "dd: failed to open ‘/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6’: Permission denied", "stderr_lines": ["dd: failed to open ‘/rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6’: Permission denied"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/etc/ansible/playbook/ovirt.retry

ls -laZ /rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6
-rw-rw----. vdsm kvm system_u:object_r:nfs_t:s0       /rhev/data-center/mnt/10.141.15.248:_export_virtman_hosted__storage/b4f93b28-1497-44b0-9eaf-5e5e2b71bce8/images/645c4286-71e4-4cce-9049-345903929e1b/e208e0f9-0f4d-4d0d-9104-10d8a26bfab6

Any ideas on this one? I can almost touch this deployment now... Looking at the command it should run as `vdsm` so should work fine, could this be SELinux?

Yes, it should be executed as vdsm user:

Did you tried executing it with -vvv ?
Can you please share the playbook you are using to trigger that role?

 

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 15:50, Callum Smith <callum@well.ox.ac.uk> wrote:

Guilty, will roll back and try again!

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 15:35, Simone Tiraboschi <stirabos@redhat.com> wrote:

The error comes from here:

TASK [ovirt.hosted_engine_setup : Parse OVF] ***************************************************************************************************************************************
fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": false, "msg": "missing parameter(s) required by 'attribute': value"}

but are you really using it with ansible 2.8 alpha 1?
I'd strongly suggest to switch back to a stable release of ansible which is currently 2.7.9.

In the next ansible build it will be just a warning as for: https://github.com/ansible/ansible/pull/54336

https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/pull/150/files already address this on ovirt-ansible-hosted-engine-setup to be compatible with future ansible releases.


On Fri, Mar 29, 2019 at 3:53 PM Callum Smith <callum@well.ox.ac.uk> wrote:
The OVF in question is here:

<ovf:Envelope ovf:version="0.9" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1/" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><References><File ovf:description="074a62d4-44f9-4ffe-a172-2702a9fe96df" ovf:href="6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5" ovf:id="72bc3948-5d8d-4877-bac8-7db4995045b5" ovf:size="54760833024" /></References><Section xsi:type="ovf:NetworkSection_Type"><Info>List of Networks</Info></Section><Section xsi:type="ovf:DiskSection_Type"><Disk ovf:actual_size="51" ovf:boot="true" ovf:disk-interface="VirtIO" ovf:disk-type="System" ovf:diskId="72bc3948-5d8d-4877-bac8-7db4995045b5" ovf:fileRef="6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5" ovf:format="http://www.vmware.com/specifications/vmdk.html#sparse" ovf:parentRef="" ovf:size="51" ovf:vm_snapshot_id="5f2be758-82d7-4c07-a220-9060e782dc7a" ovf:volume-format="COW" ovf:volume-type="Sparse" ovf:wipe-after-delete="false" /></Section><Content ovf:id="out" xsi:type="ovf:VirtualSystem_Type"><Name>074a62d4-44f9-4ffe-a172-2702a9fe96df</Name><TemplateId>074a62d4-44f9-4ffe-a172-2702a9fe96df</TemplateId><Description>Created by OVABuilder</Description><Domain /><CreationDate>2019/03/19 08:35:09</CreationDate><TimeZone /><IsAutoSuspend>false</IsAutoSuspend><VmType>1</VmType><default_display_type>0</default_display_type><default_boot_sequence>1</default_boot_sequence><Section ovf:id="074a62d4-44f9-4ffe-a172-2702a9fe96df" ovf:required="false" xsi:type="ovf:OperatingSystemSection_Type"><Info>Guest OS</Info><Description>OtherLinux</Description></Section><Section xsi:type="ovf:VirtualHardwareSection_Type"><Info>4 CPU, 16384 Memory</Info><System><vssd:VirtualSystemType>RHEVM 4.6.0.163</vssd:VirtualSystemType></System><Item><rasd:Caption>4 virtual CPU</rasd:Caption><rasd:Description>Number of virtual CPU</rasd:Description><rasd:InstanceId>1</rasd:InstanceId><rasd:ResourceType>3</rasd:ResourceType><rasd:num_of_sockets>1</rasd:num_of_sockets><rasd:cpu_per_socket>4</rasd:cpu_per_socket></Item><Item><rasd:Caption>16384 MB of memory</rasd:Caption><rasd:Description>Memory Size</rasd:Description><rasd:InstanceId>2</rasd:InstanceId><rasd:ResourceType>4</rasd:ResourceType><rasd:AllocationUnits>MegaBytes</rasd:AllocationUnits><rasd:VirtualQuantity>16384</rasd:VirtualQuantity></Item><Item><rasd:Caption>Drive 1</rasd:Caption><rasd:InstanceId>72bc3948-5d8d-4877-bac8-7db4995045b5</rasd:InstanceId><rasd:ResourceType>17</rasd:ResourceType><rasd:HostResource>6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5</rasd:HostResource><rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent><rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template><rasd:ApplicationList /><rasd:StorageId>00000000-0000-0000-0000-000000000000</rasd:StorageId><rasd:StoragePoolId>00000000-0000-0000-0000-000000000000</rasd:StoragePoolId><rasd:CreationDate>2019/03/19 08:35:09</rasd:CreationDate><rasd:LastModified>2019/03/19 08:35:09</rasd:LastModified></Item><Item><rasd:Caption>Ethernet 0 rhevm</rasd:Caption><rasd:InstanceId>3</rasd:InstanceId><rasd:ResourceType>10</rasd:ResourceType><rasd:ResourceSubType>3</rasd:ResourceSubType><rasd:Connection>rhevm</rasd:Connection><rasd:Name>eth0</rasd:Name><rasd:speed>1000</rasd:speed></Item><Item><rasd:Caption>Graphics</rasd:Caption><rasd:InstanceId>5</rasd:InstanceId><rasd:ResourceType>20</rasd:ResourceType><rasd:VirtualQuantity>1</rasd:VirtualQuantity></Item></Section></Content></ovf:Envelope>

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 14:42, Callum Smith <callum@well.ox.ac.uk> wrote:

Ok so we're getting very close now, weird OVF error:

Full ansible log attached

Only error in the engine.log looks normal/expected to me:
2019-03-29 14:32:44,370Z ERROR [org.ovirt.engine.core.bll.pm.FenceProxyLocator] (EE-ManagedThreadFactory-engineScheduled-Thread-71) [4405d6db] Can not run fence action on host 'vir
thyp04.virt.in.bmrc.ox.ac.uk', no suitable proxy host was found.

<ovirt.20190329133000.ansible.log>

Feeling damn close to success here, but have managed to replicate this issue twice re-running the installer.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 11:50, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Fri, Mar 29, 2019 at 12:36 PM Callum Smith <callum@well.ox.ac.uk> wrote:
ip link del ovirtmgmt has done the job....

Another issue, but this is likely due to randomised MAC addresses:
fatal: [virthyp04.virt.in.bmrc.ox.ac.uk]: FAILED! => {"changed": true, "cmd": ["virt-install", "-n", "HostedEngineLocal", "--os-variant", "rhel7", "--virt-type", "kvm", "--memory", "4096", "--vcpus", "64", "--network", "network=default,mac=fe:58:6c:da:1e:cc,model=virtio", "--disk", "/var/tmp/localvmOCYiyF/images/6f76686b-199c-4cb3-bbbe-86fc34365745/72bc3948-5d8d-4877-bac8-7db4995045b5", "--import", "--disk", "path=/var/tmp/localvmOCYiyF/seed.iso,device=cdrom", "--noautoconsole", "--rng", "/dev/random", "--graphics", "vnc", "--video", "vga", "--sound", "none", "--controller", "usb,model=none", "--memballoon", "none", "--boot", "hd,menu=off", "--clock", "kvmclock_present=yes"], "delta": "0:00:01.355834", "end": "2019-03-29 11:31:02.100143", "msg": "non-zero return code", "rc": 1, "start": "2019-03-29 11:31:00.744309", "stderr": "ERROR    unsupported configuration: Unable to use MAC address starting with reserved value 0xFE - 'fe:58:6c:da:1e:cc' - \nDomain installation does not appear to have been successful.\nIf it was, you can restart your domain by running:\n  virsh --connect qemu:///system start HostedEngineLocal\notherwise, please restart your installation.", "stderr_lines": ["ERROR    unsupported configuration: Unable to use MAC address starting with reserved value 0xFE - 'fe:58:6c:da:1e:cc' - ", "Domain installation does not appear to have been successful.", "If it was, you can restart your domain by running:", "  virsh --connect qemu:///system start HostedEngineLocal", "otherwise, please restart your installation."], "stdout": "\nStarting install...", "stdout_lines": ["", "Starting install..."]}

Seems it doesn't take into account reserved values when generating.

If not specified by the user,
a random unicast MAC address is randomly generated here:

for sure it's unicast but maybe we should make it more robust on reserved values.
Simply try again for now; I'll open an issue to track it, thanks!
 

I hope this feedback is valuable - I have a good feeling about the current deploy otherwise though.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 29 Mar 2019, at 11:01, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Fri, Mar 29, 2019 at 11:56 AM Callum Smith <callum@well.ox.ac.uk> wrote:
Dear Simone,

It doesn't seem to want to work:

# Settings
he_ansible_host_name: "virthyp04.virt.in.bmrc.ox.ac.uk"
he_admin_password:  <snip>
he_appliance_password:  <snip>

# Resources
he_mem_size_MB: "4096"

# Storage
he_domain_type: "nfs"
he_storage_domain_addr: <snip>
he_storage_domain_path: <snip>

# Network
he_vm_ip_addr: "10.141.31.240"
he_vm_ip_prefix: "20"
he_dns_addr: ["10.141.31.251","10.141.31.252","10.141.31.253"]
he_default_gateway_4: "10.141.31.254"
he_gateway: he_default_gateway_4
he_force_ip4: true
he_bridge_if: bond0.910
#he_just_collect_network_interfaces: true

# Email
he_smtp_port: 25
he_smtp_server: smtp.ox.ac.uk
he_source_email: ovirt@bmrc.ox.ac.uk

# Ansible Stuff
ansible_ssh_user: root
ansible_become: false

host_key_checking: false

I've attached the output of the ansible command as a log file, this is what happens when the IF bond0.910 is assigned the IP and `ovirtmgmt` is not defined on the host.


TASK [ovirt.hosted_engine_setup : debug] *******************************************************************************************************************************************
    "target_address_v4": {
        "changed": true, 
        "cmd": "ip addr show ovirtmgmt | grep 'inet ' | cut -d' ' -f6 | cut -d'/' -f1", 
        "delta": "0:00:00.008744", 
        "end": "2019-03-29 10:26:07.510481", 
        "failed": false, 
        "rc": 0, 
        "start": "2019-03-29 10:26:07.501737", 
        "stderr": "", 
        "stderr_lines": [], 
        "stdout": "", 
        "stdout_lines": []
    }
}

according to the logs ovirtmgmt is still there.

can you please share the output of 'ip a'?
 

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 28 Mar 2019, at 16:23, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Thu, Mar 28, 2019 at 1:44 PM Callum Smith <callum@well.ox.ac.uk> wrote:
Dear Simone,

This is my experience too, but I'm now hitting this error on the hosted-engine install at the part where it registers the hypervisor as the first host in the engine:

2019-03-28 12:40:50,025Z INFO  [org.ovirt.engine.core.bll.host.HostConnectivityChecker] (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Engine managed to communicate with VDSM
 agent on host 'virthyp04.virt.in.bmrc.ox.ac.uk' with address 'virthyp04.virt.in.bmrc.ox.ac.uk' ('db571f8a-fc85-40d3-b86f-c0038e3cd7e7')
2019-03-28 12:40:53,111Z WARN  [org.ovirt.engine.core.bll.network.NetworkConfigurator] (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Failed to find a valid interface for the
 management network of host virthyp04.virt.in.bmrc.ox.ac.uk. If the interface ovirtmgmt is a bridge, it should be torn-down manually.
2019-03-28 12:40:53,111Z ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [49f371c1] Exception: org.ovirt.engine.cor
e.bll.network.NetworkConfigurator$NetworkConfiguratorException: Interface ovirtmgmt is invalid for management network

The host's ovirtmgmt network connection is a statically assigned IP on a VLAN on a bond, how should I be configuring this if not manually?

If you need to deploy over vlan 123 over bond0 simply configure a device exactly called bond0.123 and statically configure your IP address there.
Choose it for hosted-engine deployment, nothing more: ovirtmgmt will be automatically created over that device and the vlan ID will be set at engine level for the whole management network.
 

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 27 Mar 2019, at 17:09, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Wed, Mar 27, 2019 at 4:27 PM Callum Smith <callum@well.ox.ac.uk> wrote:
It's ok, migrating to 4.3.2 on the oVirt node (from 4.3.0) did the job of fixing it.

It is a bug if you intend on using the ovirtmgmt network to deploy your ansible from

This is a bit tricky: when the engine brings up the host it also creates the management bridge and this could lead to a temporary network down on the selected interface for the bridge creation time (a couple of seconds?)
I tried it on a LAN and ansible ssh connection always survived but I 'm not sure it's always true.
 
, and you need it to have an IP address already on that range! But - it works as expected with the ovirtmgmt bridge setup so nothing to worry about.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 27 Mar 2019, at 14:57, Simone Tiraboschi <stirabos@redhat.com> wrote:



On Wed, Mar 27, 2019 at 3:24 PM Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,

We're trying to deploy our hosted engine remotely using the ansible hosted engine playbook, which has been a rocky road but we're now at the point where it's installing, and failing. We've got a pre-defined bond/VLAN setup for our interface which has the correct bond0 bond0.123 and ovirtmgmt bridge on top but we're hitting the classic error:

Failed to find a valid in
terface for the management network of host virthyp04.virt.in.bmrc.ox.ac.uk. If the interface ovirtmgmt is a bridge, it should be torn-down manually.

Does this bug still exist in the latest (4.3) version, and is installing using ansible with this network configuration impossible?

I don't think it's a bug; please avoid manually creating ovirtmgmt and simply set:
he_bridge_if: "bond0.123", in ansible variable file and the management bridge will be created for you at host-deploy time.
 

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SBOZ6FRBRQKE7FMTEHT6FBKHOW3PIPHK/



_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QOGFRHXDVOIM4WA5N4JT7RBSW7PJIYJQ/