[ovirt-users] Re: Problems provisioning 4.4.6 hosted engine

On Thu, May 13, 2021 at 3:34 PM Sketch <ovirt(a)rednsx.org&gt; wrote: > This is a new system is CentOS 8.3, with the oVirt-4.4 repo and all > updates applied. When I try to install the hosted engine with my engine > backup from 4.3.10, the installation fails with a too many open files > error. My 8.3 hosts already had 1M system max files, which is more than > any of my CentOS 7/oVirt 4.3 hosts have. I tried increasing it to 2M with > no luck, so my suspicion is that the error is on the engine itself? > > I tried provisioning a new engine just to test, and I get SSH key errors > instead of this one. > > Any suggestions? > > 2021-05-12 23:09:44,731-0700 ERROR ansible failed { > "ansible_host": "localhost", > "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", > "ansible_result": { > "_ansible_no_log": false, > "exception": "Traceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py\", line 665, in _execute\n result = self._handler.run(task_vars=variables)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/wait_for_connection.py\", line 122, in run\n self._remove_tmp_path(self._connection._shell.tmpdir)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/__init__.py\", line 417, in _remove_tmp_path\n tmp_rm_res = self._low_level_execute_command(cmd, sudoable=False)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/__init__.py\", line 1085, in _low_level_execute_command\n rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/connection/ssh.py\", line 1191, in exec_command\n cmd = self._build_command(*args)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/connection/s > sh.py\", line 562, in _build_command\n self.sshpass_pipe = os.pipe()\nOSError: [Errno 24] Too many open files\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py\", line 147, in run\n res = self._execute()\n File \"/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py\", line 673, in _execute\n self._handler.cleanup()\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/__init__.py\", line 128, in cleanup\n self._remove_tmp_path(self._connection._shell.tmpdir)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/__init__.py\", line 417, in _remove_tmp_path\n tmp_rm_res = self._low_level_execute_command(cmd, sudoable=False)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/action/__init__.py\", line 1085, in _low_level_execute_command\n rc, stdout, stderr = self._connection.exec_command > (cmd, in_data=in_data, sudoable=sudoable)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/connection/ssh.py\", line 1191, in exec_command\n cmd = self._build_command(*args)\n File \"/usr/lib/python3.6/site-packages/ansible/plugins/connection/ssh.py\", line 562, in _build_command\n self.sshpass_pipe = os.pipe()\nOSError: [Errno 24] Too many open files\n", > "msg": "Unexpected failure during module execution.", > "stdout": "" > }, > "ansible_task": "Wait for the local VM", > "ansible_type": "task", > "status": "FAILED", > "task_duration": 3605 So I suppose it failed after 3605 seconds, or 721 attempts (of 5 seconds each). Do you see the VM in 'virsh list'? Can you see the VM running (e.g. 'ps auxww | grep qemu')? Can you try to ssh to it from the host (search the logs for local_vm_ip for its local/private temporary address)? Perhaps open its console (Perhaps 'virsh console HostedEngineLocal')? That said, I'd personally also consider it a bug in ansible, unless you made some relevant custom changes - the bug is that it seems to leak open files. Thanks and best regards,