Local (Deployment) VM Can't Reach "centos-ceph-pacific" Repo
Hi All, OK, new issue: :-( If I'm reading things right the local (deployment) vm can't get to the centos-ceph-pacific repo. The repo is installed on the host machine (along with all of the relevant dependant repos). I thought that local 192.168.222.0/24 network was nated out the virbr0 virtual bridge - am I wrong in this (ie do we need to update/change our routing tables/whatever)? Here is the (relevant part) of the log: ~~~ 2022-10-11 16:22:14,749+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.engine_setup : Install oVirt Engine package] 2022-10-11 16:26:03,643+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'results': [], 'rc': 1, 'msg': "Failed to download metadata for repo 'centos-ceph-pacific': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'invocation': {'module_args': {'name': ['ovirt-engine'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'cacheonly': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'allowerasing': False, 'nobest': False, 'conf_file': None, 'disable_excludes': None, 'download_dir': None, 'list': None, 'releasever': None}}, '_ansible_no_log': False, 'changed': False, ' _ansible_delegated_vars': {'ansible_host': '192.168.222.77', 'ansible_port': None, 'ansible_user': 'root', 'ansible_connection': 'smart'}} 2022-10-11 16:26:03,744+1100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost -> 192.168.222.77]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'centos-ceph-pacific': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []} 2022-10-11 16:26:04,045+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine] 2022-10-11 16:26:04,947+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 changed: [localhost -> 192.168.222.77] 2022-10-11 16:26:05,449+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path] 2022-10-11 16:26:05,950+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost -> localhost] 2022-10-11 16:26:06,352+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory] 2022-10-11 16:26:06,953+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 changed: [localhost -> localhost] 2022-10-11 16:26:07,355+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] 2022-10-11 16:26:07,856+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost] 2022-10-11 16:26:08,357+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image] 2022-10-11 16:26:08,959+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost -> localhost] 2022-10-11 16:26:09,460+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path] 2022-10-11 16:26:09,862+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost -> localhost] 2022-10-11 16:26:10,363+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] 2022-10-11 16:26:20,986+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 ok: [localhost -> localhost] 2022-10-11 16:26:21,388+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs] 2022-10-11 16:26:27,901+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 changed: [localhost] 2022-10-11 16:26:28,403+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Change ownership of copied engine logs] 2022-10-11 16:26:29,005+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 changed: [localhost -> localhost] 2022-10-11 16:26:29,506+1100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure] 2022-10-11 16:26:29,908+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': 'There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n', '_ansible_no_log': False, 'changed': False} 2022-10-11 16:26:30,008+1100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} 2022-10-11 16:26:30,410+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 161 changed: 59 unreachable: 0 skipped: 80 failed: 1 2022-10-11 16:26:30,510+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:226 ansible-playbook rc: 2 2022-10-11 16:26:30,510+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:233 ansible-playbook stdout: 2022-10-11 16:26:30,510+1100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:236 ansible-playbook stderr: 2022-10-11 16:26:30,510+1100 DEBUG otopi.plugins.gr_he_ansiblesetup.core.misc misc._closeup:475 {'otopi_host_net': {'ansible_facts': {'otopi_host_net': ['bond0.40', 'ens1', 'bond0.20']}, '_ansible_no_log': False, 'changed': False}, 'otopi_localvm_dir': {'changed': True, 'path': '/var/tmp/localvmfq67djtv', 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0700', 'state': 'directory', 'secontext': 'unconfined_u:object_r:user_tmp_t:s0', 'size': 6, 'invocation': {'module_args': {'state': 'directory', 'path': '/var/tmp', 'prefix': 'localvm', 'suffix': ''}}, '_ansible_no_log': False}, 'otopi_appliance_disk_size': {'ansible_facts': {'virtual_size': '53689188352'}, '_ansible_no_log': False, 'changed': False}, 'ansible-playbook_rc': 2} 2022-10-11 16:26:30,511+1100 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 509, in _closeup raise RuntimeError(_('Failed executing ansible-playbook')) RuntimeError: Failed executing ansible-playbook 2022-10-11 16:26:30,511+1100 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook 2022-10-11 16:26:30,512+1100 DEBUG otopi.context context.dumpEnvironment:765 ENVIRONMENT DUMP - BEGIN 2022-10-11 16:26:30,512+1100 DEBUG otopi.context context.dumpEnvironment:775 ENV BASE/error=bool:'True' 2022-10-11 16:26:30,512+1100 DEBUG otopi.context context.dumpEnvironment:775 ENV BASE/exceptionInfo=list:'[(<class 'RuntimeError'>, RuntimeError('Failed executing ansible-playbook',), <traceback object at 0x7efe2a39fa08>)]' 2022-10-11 16:26:30,512+1100 DEBUG otopi.context context.dumpEnvironment:775 ENV OVEHOSTED_CORE/localVMDir=str:'/var/tmp/localvmfq67djtv' 2022-10-11 16:26:30,512+1100 DEBUG otopi.context context.dumpEnvironment:775 ENV OVEHOSTED_STORAGE/ovfSizeGB=int:'51' 2022-10-11 16:26:30,513+1100 DEBUG otopi.context context.dumpEnvironment:779 ENVIRONMENT DUMP - END ~~~ Anyone got any ideas - thanks: Dulux-Oz
Follow Up: So after doing more research, etc, I've discovered the following: - Because the install failed the V-Bridge (virbr0 - 192.168.222.1) and the Local VM (HostedEngineLocal - 192.168.222.77) still exist on the Host. - SSHing into the Local VM I discovered that I could ping out to an ip address, but not to a hostname (I used mirrorlist.centos.org, as the initial error in this run was complaining that the Local VM couldn't download the centos-ceph-pacific meta-data). - The Local VM's /etc/resolv.conf file has a single entry pointing to the V-Bridge. - I did a "dig" to mirrorlist.centos.org and that command returned the correct ip addresses, etc, and used the V-Bridge to do it. HOWEVER, with some further experimentation (eg doing 2 pings to the same host in a row, as well as other experiments) the underlying issue *seems* to be that the dns requests are timing out - once the Local VM has resolved the hostname<=>ip address mapping everything *seems* to run OK. So, I'm wondering if any of the following solutions are viable (of course, I could be completely off-track): - Is there some way of adding the DNS servers that the deployment script asks about to the Local VM's /etc/resolv.conf file (ie bypass the V-Bridge when doing a DNS lookup to make the response time faster)? - There is a `timeout` setting (default: 30 sec) in dnf config file. Is there some way of changing this to a larger value on the Local VM. - Use the `--ansible-extra-vars=he_pause_host=true` (or similar) flag to pause the install, go into the Local VM and make one or both of the above changes. If *anyone* has any ideas I'd love to hear them. Cheers Dulux-Oz
Further follow up: I managed to get this sorted. It turns out that the "hosted-engine --deploy --4" command was causing the issue. By removing the "--4" flag (even though we don't use IPv6), and letting the deployment script complain about duel-stack environments (to which I had to answer "yes" to continue the deployment) the script then went through its process OK - even doing the dnf updates. Hope this helps someone in the future. Cheers Dulux-Oz
participants (1)
-
Matthew J Black