[ovirt-users] Re: ovirt metrics ansible error

Shirly, I appreciate the help with this. Unfortunately I am still running in to the same problem. So far I've tried to install/enable/start NetworkManager on the existing "master0" server and re-ran the plans from the installer VM. I ran in to the same problem waiting for control plane pods and same errors in syslog. So I wiped everything out, killed the template along with the installer and master VMs. On oVirt engine (I am running 4.3.3.7-1 stable) I did have ovirt-engine-metrics-1.3.0x rpm installed, no yum updates available on an update check. So I installed http://resources.ovirt.org/pub/yum-repo/ovirt-release43-pre.rpm then proceeded to install the latest version of ovirt-engine-metrics which gave me: ovirt-engine-metrics-1.3.1-1.el7.noarch on hosted engine. After that package was installed I proceeded to follow steps from beginning outlined at: https://ovirt.org/documentation/metrics-install-guide/Installing_Metrics_... -- I ran in to the docker check issue again (same as my initial email) so I disabled that and again got as far as starting control plane pods before failure. Not sure where to go from here at this point. The only thing I can think of that I did differently vs the instructions outlined above is that I have not crated the wildcard DNS record, however I did set configs to create /etc/hosts entries and they /etc/hosts on the machines have the proper IPs assigned for all hostnames (automatically added by the ansible plays). Any ideas how I can get past the plane pods issue? Thanks! On Tue, May 28, 2019 at 4:23 AM Shirly Radco <sradco(a)redhat.com&gt; wrote: > Hi, > > The latest release of 4.3.z should already include a fix for this issue, > ovirt-engine-metrics-1.3.1 rpm. > > The issue is that it requires the NetworkManagar to be installed, running > and enabled for it to work. > > You can install it manually on the master0 vm , start and enable it or > you can also install the updated rpm from the nightly builds if your > environment is oVirt 4.2.z: > > https://resources.ovirt.org/pub/ovirt-4.2-snapshot/rpm/el7/noarch/ovirt-e... > > Relevant patches are: > https://gerrit.ovirt.org/#/c/99959/ > https://gerrit.ovirt.org/#/c/99718/ > > Best regards, > > -- > > Shirly Radco > > BI Senior Software Engineer > > Red Hat <https://www.redhat.com/> > > <https://www.redhat.com/> > > > On Mon, May 27, 2019 at 4:41 PM Jayme <jaymef(a)gmail.com&gt; wrote: > >> I managed to get past that but am running in to another problem later in >> the process on the control plane pods to appear task. I thought perhaps a >> glitch in the process from the failed docker step previously so after a few >> more runs I tried killing everything and restarting the metrics process >> again from the very beginning and end up hitting the same issue with >> control plane pods even though all other steps/tasks seem to be working. >> >> I'm just getting this: >> >> TASK [openshift_control_plane : Wait for control plane pods to appear] >> ********* >> Monday 27 May 2019 13:31:54 +0000 (0:00:00.180) 0:14:33.857 >> ************ >> FAILED - RETRYING: Wait for control plane pods to appear (60 retries >> left). >> FAILED - RETRYING: Wait for control plane pods to appear (59 retries >> left). >> FAILED - RETRYING: Wait for control plane pods to appear (58 retries >> left). >> FAILED - RETRYING: Wait for control plane pods to appear (57 retries >> left). >> FAILED - RETRYING: Wait for control plane pods to appear (56 retries >> left). >> >> It eventually counts all the way down to zero and fails. >> >> In syslog of the master0 server I'm seeing some errors related to cni >> config >> >> May 27 13:39:07 master0 ansible-oc_obj: Invoked with files=None kind=pod >> force=False all_namespaces=None field_selector=None namespace=kube-system >> delete_after=False kubeconfig=/etc/origin/master/admin.kubeconfig >> content=None state=list debug=False selector=None name= >> master-api-master0.xxxxxx.com >> May 27 13:39:09 master0 origin-node: W0527 13:39:09.064230 20150 >> cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d >> May 27 13:39:09 master0 origin-node: E0527 13:39:09.064670 20150 >> kubelet.go:2101] Container runtime network not ready: NetworkReady=false >> reason:NetworkPluginNotReady message:docker: network plugin is not ready: >> cni config uninitialized >> May 27 13:39:13 master0 ansible-oc_obj: Invoked with files=None kind=pod >> force=False all_namespaces=None field_selector=None namespace=kube-system >> delete_after=False kubeconfig=/etc/origin/master/admin.kubeconfig >> content=None state=list debug=False selector=None name= >> master-api-master0.xxxxxx.com >> May 27 13:39:14 master0 origin-node: W0527 13:39:14.066911 20150 >> cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d >> May 27 13:39:14 master0 origin-node: E0527 13:39:14.067321 20150 >> kubelet.go:2101] Container runtime network not ready: NetworkReady=false >> reason:NetworkPluginNotReady message:docker: network plugin is not ready: >> cni config uninitialized >> May 27 13:39:14 master0 origin-node: E0527 13:39:14.814705 20150 >> summary.go:102] Failed to get system container stats for >> "/system.slice/origin-node.service": failed to get cgroup stats for >> "/system.slice/origin-node.service": failed to get container info for >> "/system.slice/origin-node.service": unknown container >> "/system.slice/origin-node.service" >> May 27 13:39:19 master0 origin-node: W0527 13:39:19.069450 20150 >> cni.go:172] Unable to update cni config: No networks found in /etc/cni/net.d >> May 27 13:39:19 master0 origin-node: E0527 13:39:19.069850 20150 >> kubelet.go:2101] Container runtime network not ready: NetworkReady=false >> reason:NetworkPluginNotReady message:docker: network plugin is not ready: >> cni config uninitialized >> >> On Mon, May 27, 2019 at 9:35 AM Shirly Radco <sradco(a)redhat.com&gt; wrote: >> >>> Hi Jayme, >>> >>> Thank you for reaching out. >>> Please try rerunning the ansible playbook. >>> If this doesn't work, try adding to the integ.ini in the metrics vm >>> openshift_disable_check=docker_storage >>> and rerun the ansible playbook again. >>> >>> Please update how it goes. >>> >>> Best regards, >>> >>> -- >>> >>> Shirly Radco >>> >>> BI Senior Software Engineer >>> >>> Red Hat <https://www.redhat.com/> >>> >>> <https://www.redhat.com/> >>> >>> >>> On Sun, May 26, 2019 at 9:34 PM Jayme <jaymef(a)gmail.com&gt; wrote: >>> >>>> I'm running in to this ansible error during oVirt metrics installation >>>> (following procedures at: >>>> https://ovirt.org/documentation/metrics-install-guide/Installing_Metrics_... >>>> ) >>>> >>>> This is happening late in the process, after successfully deploying >>>> the installation VM and then running second step from the metrics VM. >>>> >>>> CHECK [memory_availability : master0.xxxxxx.com] >>>> ************************************************************************************************* >>>> fatal: [master0.xxxxxxx.com]: FAILED! => {"changed": true, "checks": >>>> {"disk_availability": {}, "docker_image_availability": {"changed": true}, >>>> "docker_storage": {"failed": true, "failures": [["OpenShiftCheckException", >>>> "Could not find imported module support code for docker_info. Looked for >>>> either AnsibleDockerClient.py or docker_common.py\nTraceback (most recent >>>> call last):\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py\", >>>> line 225, in run_check\n result = check.run()\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_storage.py\", >>>> line 53, in run\n docker_info = self.execute_module(\"docker_info\", >>>> {})\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/__init__.py\", >>>> line 211, in execute_module\n result = self._execute_module(module_name, >>>> module_args, self.tmp, self.task_vars)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py\", >>>> line 809, in _execute_module\n (module_style, shebang, module_data, >>>> module_path) = self._configure_module(module_name=module_name, >>>> module_args=module_args, task_vars=task_vars)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py\", >>>> line 203, in _configure_module\n environment=final_environment)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 1023, in modify_module\n environment=environment)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 859, in _find_module_utils\n recursive_finder(module_name, >>>> b_module_data, py_module_names, py_module_cache, zf)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 621, in recursive_finder\n raise AnsibleError(' >>>> '.join(msg))\nAnsibleError: Could not find imported module support code for >>>> docker_info. Looked for either AnsibleDockerClient.py or >>>> docker_common.py\n"]], "msg": "Could not find imported module support code >>>> for docker_info. Looked for either AnsibleDockerClient.py or >>>> docker_common.py\nTraceback (most recent call last):\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py\", >>>> line 225, in run_check\n result = check.run()\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_storage.py\", >>>> line 53, in run\n docker_info = self.execute_module(\"docker_info\", >>>> {})\n File >>>> \"/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/__init__.py\", >>>> line 211, in execute_module\n result = self._execute_module(module_name, >>>> module_args, self.tmp, self.task_vars)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py\", >>>> line 809, in _execute_module\n (module_style, shebang, module_data, >>>> module_path) = self._configure_module(module_name=module_name, >>>> module_args=module_args, task_vars=task_vars)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py\", >>>> line 203, in _configure_module\n environment=final_environment)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 1023, in modify_module\n environment=environment)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 859, in _find_module_utils\n recursive_finder(module_name, >>>> b_module_data, py_module_names, py_module_cache, zf)\n File >>>> \"/usr/lib/python2.7/site-packages/ansible/executor/module_common.py\", >>>> line 621, in recursive_finder\n raise AnsibleError(' >>>> '.join(msg))\nAnsibleError: Could not find imported module support code for >>>> docker_info. Looked for either AnsibleDockerClient.py or >>>> docker_common.py\n"}, "memory_availability": {}, "package_availability": >>>> {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", >>>> "bash-completion", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", >>>> "flannel", "glusterfs-fuse", "httpd-tools", "iptables", >>>> "iptables-services", "iscsi-initiator-utils", "libselinux-python", >>>> "nfs-utils", "ntp", "openssl", "origin", "origin-clients", >>>> "origin-hyperkube", "origin-node", "pyparted", "python-httplib2", >>>> "yum-utils"]}}}, "package_version": {"changed": false, "invocation": >>>> {"module_args": {"package_list": [{"check_multi": false, "name": "origin", >>>> "version": ""}, {"check_multi": false, "name": "origin-master", "version": >>>> ""}, {"check_multi": false, "name": "origin-node", "version": ""}], >>>> "package_mgr": "yum"}}}}, "msg": "One or more checks failed", >>>> "playbook_context": "install"} >>>> >>>> NO MORE HOSTS LEFT >>>> ******************************************************************************************************************************************* >>>> >>>> PLAY RECAP >>>> *************************************************************************************************************************************************** >>>> localhost : ok=35 changed=1 unreachable=0 >>>> failed=0 skipped=16 rescued=0 ignored=0 >>>> master0.xxxxxxx.com : ok=96 changed=6 unreachable=0 failed=1 >>>> skipped=165 rescued=0 ignored=0 >>>> >>>> >>>> INSTALLER STATUS >>>> ********************************************************************************************************************************************* >>>> Initialization : Complete (0:00:16) >>>> Health Check : In Progress (0:00:36) >>>> This phase can be restarted by running: >>>> playbooks/openshift-checks/pre-install.yml >>>> Sunday 26 May 2019 16:36:25 +0000 (0:00:36.151) 0:01:56.339 >>>> ************ >>>> >>>> =============================================================================== >>>> Run health checks (install) - EL >>>> --------------------------------------------------------------------------------------------------------------------- >>>> 36.15s >>>> os_firewall : Ensure iptables services are not enabled >>>> ------------------------------------------------------------------------------------------------ >>>> 2.74s >>>> openshift_repos : Ensure libselinux-python is installed >>>> ----------------------------------------------------------------------------------------------- >>>> 1.77s >>>> openshift_repos : refresh cache >>>> ----------------------------------------------------------------------------------------------------------------------- >>>> 1.60s >>>> Gather Cluster facts >>>> ---------------------------------------------------------------------------------------------------------------------------------- >>>> 1.51s >>>> container_runtime : Fixup SELinux permissions for docker >>>> ---------------------------------------------------------------------------------------------- >>>> 1.33s >>>> container_runtime : Place additional/blocked/insecure registries in >>>> /etc/containers/registries.conf >>>> --------------------------------------------------- 1.30s >>>> Ensure openshift-ansible installer package deps are installed >>>> ----------------------------------------------------------------------------------------- >>>> 1.29s >>>> container_runtime : Install Docker >>>> -------------------------------------------------------------------------------------------------------------------- >>>> 1.17s >>>> Initialize openshift.node.sdn_mtu >>>> --------------------------------------------------------------------------------------------------------------------- >>>> 1.13s >>>> os_firewall : Install firewalld packages >>>> -------------------------------------------------------------------------------------------------------------- >>>> 1.13s >>>> container_runtime : Set various Docker options >>>> -------------------------------------------------------------------------------------------------------- >>>> 1.11s >>>> install NetworkManager >>>> -------------------------------------------------------------------------------------------------------------------------------- >>>> 1.10s >>>> openshift_repos : Configure correct origin release repository >>>> ----------------------------------------------------------------------------------------- >>>> 1.05s >>>> container_runtime : Get current installed Docker version >>>> ---------------------------------------------------------------------------------------------- >>>> 1.04s >>>> openshift_repos : Configure origin gpg keys >>>> ----------------------------------------------------------------------------------------------------------- >>>> 1.04s >>>> openshift_repos : Remove openshift_additional.repo file >>>> ----------------------------------------------------------------------------------------------- >>>> 0.99s >>>> container_runtime : Setup the docker-storage for overlay >>>> ---------------------------------------------------------------------------------------------- >>>> 0.96s >>>> Detecting Operating System from ostree_booted >>>> --------------------------------------------------------------------------------------------------------- >>>> 0.95s >>>> Gather Cluster facts >>>> ---------------------------------------------------------------------------------------------------------------------------------- >>>> 0.92s >>>> >>>> >>>> Failure summary: >>>> >>>> >>>> 1. Hosts: master0.xxxxxxx.com >>>> Play: OpenShift Health Checks >>>> Task: Run health checks (install) - EL >>>> Message: One or more checks failed >>>> Details: check "docker_storage": >>>> Could not find imported module support code for >>>> docker_info. Looked for either AnsibleDockerClient.py or docker_common.py >>>> Traceback (most recent call last): >>>> File >>>> "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", >>>> line 225, in run_check >>>> result = check.run() >>>> File >>>> "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_storage.py", >>>> line 53, in run >>>> docker_info = self.execute_module("docker_info", {}) >>>> File >>>> "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/__init__.py", >>>> line 211, in execute_module >>>> result = self._execute_module(module_name, >>>> module_args, self.tmp, self.task_vars) >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line >>>> 809, in _execute_module >>>> (module_style, shebang, module_data, module_path) = >>>> self._configure_module(module_name=module_name, module_args=module_args, >>>> task_vars=task_vars) >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line >>>> 203, in _configure_module >>>> environment=final_environment) >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line >>>> 1023, in modify_module >>>> environment=environment) >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line >>>> 859, in _find_module_utils >>>> recursive_finder(module_name, b_module_data, >>>> py_module_names, py_module_cache, zf) >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line >>>> 621, in recursive_finder >>>> raise AnsibleError(' '.join(msg)) >>>> AnsibleError: Could not find imported module support >>>> code for docker_info. Looked for either AnsibleDockerClient.py or >>>> docker_common.py >>>> >>>> >>>> The execution of "install_okd.yaml" includes checks designed to fail >>>> early if the requirements of the playbook are not met. One or more of these >>>> checks failed. To disregard these results,explicitly disable checks by >>>> setting an Ansible variable: >>>> openshift_disable_check=docker_storage >>>> Failing check names are shown in the failure details above. Some >>>> checks may be configurable by variables if your requirements are different >>>> from the defaults; consult check documentation. >>>> Variables can be set in the inventory or passed on the command line >>>> using the -e flag to ansible-playbook. >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/SEFPOF36T7G... >>>> >>>