On Mon, Apr 15, 2019 at 9:38 PM <dskryminskiy(a)devexperts.com> wrote:
I do a fresh install oVirt 4.3 on CentOS 7.6.1810, but in the end of playbook it falls
down:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"The system may not be provisioned according to the playbook results: please check
the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing
ansible-playbook
[ INFO ] Stage: Clean up
[ INFO ] Cleaning temporary resources
[ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs]
[ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the
local VM]
[ INFO ] ok: [localhost]
[ INFO ] Generating answer file
'/var/lib/ovirt-hosted-engine-setup/answers/answers-20190415123351.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix
accordingly or re-deploy from scratch.
Log file is located at
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20190415122114-rk9ve5.log
Log files can be found here:
https://ufile.io/kylza1g2
From 86fc34ovirt-hosted-engine-setup-20190415122114-rk9ve5.log:
2019-04-15 12:31:27,182-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Wait for the host to be up]
2019-04-15 12:32:48,145-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:32:48,548-0500 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 TASK [ovirt.hosted_engine_setup :
debug]
2019-04-15 12:32:48,850-0500 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 host_result_up_check: {'failed':
False, 'attempts': 16, u'changed': False, u'ansible_facts':
{u'ovirt_hosts': [{u'comment': u'', u'update_available':
False,
u'protocol': u'stomp', u'affinity_labels': [], u'hooks':
[],
u'cluster': {u'href':
u'/ovirt-engine/api/clusters/fe46b0b2-5fa3-11e9-b5d4-00163e4e516d',
u'id': u'fe46b0b2-5fa3-11e9-b5d4-00163e4e516d'}, u'href':
u'/ovirt-engine/api/hosts/49a7c18c-853d-4b00-8291-337474dd0aca',
u'devices': [], u'id': u'49a7c18c-853d-4b00-8291-337474dd0aca',
u'vgpu_placement': u'consolidated', u'external_status':
u'ok',
u'statistics': [], u'certificate': {u'organization':
u'ny2.thesmallexchange.com', u'subject':
u'O=ny2.thesmallexchange.com,CN=prod-srv005.ny2.thesmallexchange.com'},
u'nics': [], u'storage_connection_extensions': [], u'port':
54321,
u'hardware_information': {u'supported_rng_sources': []},
u'memory': 0,
u'ksm': {u'enabled': False}, u'se_linux': {}, u'type':
u'rhel',
u'status': u'non_operational', u'tags': [],
u'katello_errata': [],
u'external_network_provider_configurations': [], u'status_detail':
u'none', u'ssh': {u'port': 22, u'fingerprint':
u'SHA256:nbN07yOtyxAL3IAIUcm7LwaJzvUcJE20sJFsVErm3Ys'}, u'address':
u'prod-srv005.ny2.thesmallexchange.com', u'numa_nodes': [],
u'device_passthrough': {u'enabled': False}, u'unmanaged_networks':
[],
u'permissions': [], u'numa_supported': False,
u'power_management':
{u'kdump_detection': True, u'enabled': False, u'pm_proxies': [],
u'automatic_pm_enabled': True}, u'name':
u'prod-srv005.ny2.thesmallexchange.com', u'max_scheduling_memory': 0,
u'summary': {u'total': 0}, u'auto_numa_status':
u'unknown',
u'transparent_huge_pages': {u'enabled': False},
u'network_attachments': [], u'os': {u'custom_kernel_cmdline':
u''},
u'cpu': {u'speed': 0.0, u'topology': {}}, u'kdump_status':
u'unknown',
u'spm': {u'priority': 5, u'status': u'none'}}]}}
2019-04-15 12:32:49,151-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Check host status]
2019-04-15 12:32:49,554-0500 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 {u'msg': u'The host has been set in
non_operational status, please check engine logs, fix accordingly and
re-deploy.\n', u'changed': False, u'_ansible_no_log': False}
2019-04-15 12:32:49,654-0500 ERROR
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:107 fatal: [localhost]: FAILED! =>
{"changed": false, "msg": "The host has been set in
non_operational
status, please check engine logs, fix accordingly and re-deploy.\n"}
engine.log ends with:
2019-04-15 12:31:45,356-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [193cb7ba] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host
prod-srv005.ny2.thesmallexchange.com. Stage:
Termination.
2019-04-15 12:31:45,427-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [193cb7ba] EVENT_ID:
VDS_ANSIBLE_INSTALL_STARTED(560), Ansible host-deploy playbook
execution has started on host
prod-srv005.ny2.thesmallexchange.com.
2019-04-15 12:31:45,432-05 INFO
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engine-Thread-1) [193cb7ba] Executing Ansible
command: /usr/bin/ansible-playbook --ssh-common-args=-F
/var/lib/ovirt-engine/.ssh/config -v
--private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa
--inventory=/tmp/ansible-inventory4071758905872594629
--extra-vars=host_deploy_cluster_name="Default"
--extra-vars=host_deploy_vdsm_port="54321"
--extra-vars=host_deploy_cluster_switch_type="legacy"
--extra-vars=ansible_port="22"
--extra-vars=host_deploy_ovn_central="192.168.122.203"
--extra-vars=host_deploy_cluster_version="4.3"
--extra-vars=host_deploy_vnc_tls="false"
--extra-vars=host_deploy_post_tasks="/etc/ovirt-engine/ansible/ovirt-host-deploy-post-tasks.yml"
--extra-vars=host_deploy_firewall_type="FIREWALLD"
--extra-vars=host_deploy_gluster_enabled="false"
--extra-vars=host_deploy_ovn_tunneling_interface="10.128.0.5"
--extra-vars=host_deploy_virt_enabled="true"
--extra-vars=host_deploy_override_firewall="true"
/usr/share/ovirt-engine/playbooks/ovirt-host-deploy.yml [Logfile:
/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20190415123145-prod-srv005.ny2.thesmallexchange.com-193cb7ba.log]
2019-04-15 12:32:09,248-05 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [] Attempting to
update VMs/Templates Ovf.
2019-04-15 12:32:42,047-05 INFO
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor]
(EE-ManagedThreadFactory-engine-Thread-1) [193cb7ba] Ansible playbook
command has exited with value: 0
2019-04-15 12:32:42,052-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [193cb7ba] EVENT_ID:
VDS_ANSIBLE_INSTALL_FINISHED(561), Ansible host-deploy playbook
execution has successfully finished on host
prod-srv005.ny2.thesmallexchange.com.
which is before the failure.
The second attempt to fetch engine logs failed:
2019-04-15 12:33:30,822-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Fetch logs from the engine VM]
2019-04-15 12:33:31,224-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:33:31,626-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Set destination directory path]
2019-04-15 12:33:31,928-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:33:32,330-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Create destination directory]
2019-04-15 12:33:33,033-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 changed: [localhost]
2019-04-15 12:33:33,335-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
include_tasks]
2019-04-15 12:33:33,737-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:33:34,140-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Find the local appliance image]
2019-04-15 12:33:34,743-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:33:35,145-0500 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 TASK [ovirt.hosted_engine_setup :
debug]
2019-04-15 12:33:35,447-0500 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:103 app_img: {u'files': [], 'failed':
False, u'examined': 0, u'msg': u'/images was skipped as it does not
seem to be a valid directory or it cannot be accessed\n', u'changed':
False, u'matched': 0}
2019-04-15 12:33:35,849-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Set local_vm_disk_path]
2019-04-15 12:33:36,150-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 skipping: [localhost]
2019-04-15 12:33:36,653-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Give the vm time to flush dirty buffers]
2019-04-15 12:33:46,996-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
2019-04-15 12:33:47,298-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
Copy engine logs]
2019-04-15 12:33:48,001-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 TASK [ovirt.hosted_engine_setup :
include_tasks]
2019-04-15 12:33:48,403-0500 INFO
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 ok: [localhost]
So we cannot know for sure what happened.
Perhaps you can check other logs on the host (system, libvirt, etc.)
to try to find out?
Adding also Simone.
Best regards,
--
Didi