hosted setup failed with error on 'Initialize lockspace volume'

Hi, I am trying to perform a self hosted ovirt cluster. All goes well and towards the end the setup fails at the task 'Initialize lockspace volume' with the following error message: { "attempts":5, "changed":true, "cmd":[ "hosted-engine", "--reinitialize-lockspace", "--force" ], "delta":"0:00:00.108243", "end":"2023-05-03 22:50:53.930482", "msg":"non-zero return code", "rc":1, "start":"2023-05-03 22:50:53.822239", "stderr":"Traceback (most recent call last):\n File \"/usr/lib64/python3.9/runpy.py\", line 197, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1122, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1464, in __request\n response = self.__transport.request(\n File \"/usr/lib64/py thon3.9/xmlrpc/client.py\", line 1166, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1178, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1291, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1321, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect\n self.sock.connect(base64.b16decode(self.host ))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines":[ "Traceback (most recent call last):", " File \"/usr/lib64/python3.9/runpy.py\", line 197, in _run_module_as_main", " return _run_code(code, main_globals, None,", " File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1122, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1464, in __request", " response = self.__transport.request(", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1166, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1178, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1291, in send_request", " self.send_content(connection, request_body)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1321, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.9/http/client.py\", line 980, in send", " self.connect()", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory" ], "stdout":"", "stdout_lines":[ ] } The underlying storage is NFS on top of Ceph (which is not listed in the ovirt documentation). Does this ring any bell to anybody? Please let me know if you need more information in this regard or contents from any of the log files. Thanks

Hi, Did you follow https://www.ovirt.org/develop/troubleshooting-nfs-storage-issues.html ? In general those are several aspects that would prevent ovengine from writing to the directory and in addition to checking it manually you can use nfs-check.py described over there. I think the fact that your NFS is over Ceph should be something that is 'abstracted away' or you might ask yourself - how NFS share implemented by Ceph is different from any other NFS share? BR, Konstantin Am 04.05.23, 00:40 schrieb "destfinal--- via Users" <users@ovirt.org <mailto:users@ovirt.org>>: Hi, I am trying to perform a self hosted ovirt cluster. All goes well and towards the end the setup fails at the task 'Initialize lockspace volume' with the following error message: { "attempts":5, "changed":true, "cmd":[ "hosted-engine", "--reinitialize-lockspace", "--force" ], "delta":"0:00:00.108243", "end":"2023-05-03 22:50:53.930482", "msg":"non-zero return code", "rc":1, "start":"2023-05-03 22:50:53.822239", "stderr":"Traceback (most recent call last):\n File \"/usr/lib64/python3.9/runpy.py\", line 197, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1122, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1464, in __request\n response = self.__transport.request(\n File \"/usr/lib64/py thon3.9/xmlrpc/client.py\", line 1166, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1178, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1291, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1321, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.9/http/client.py\", line 980, in send\n self.connect()\n File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect\n self.sock.connect(base64.b16decode(self.host ))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines":[ "Traceback (most recent call last):", " File \"/usr/lib64/python3.9/runpy.py\", line 197, in _run_module_as_main", " return _run_code(code, main_globals, None,", " File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1122, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1464, in __request", " response = self.__transport.request(", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1166, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1178, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1291, in send_request", " self.send_content(connection, request_body)", " File \"/usr/lib64/python3.9/xmlrpc/client.py\", line 1321, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python3.9/http/client.py\", line 1280, in endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.9/http/client.py\", line 1040, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.9/http/client.py\", line 980, in send", " self.connect()", " File \"/usr/lib/python3.9/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory" ], "stdout":"", "stdout_lines":[ ] } The underlying storage is NFS on top of Ceph (which is not listed in the ovirt documentation). Does this ring any bell to anybody? Please let me know if you need more information in this regard or contents from any of the log files. Thanks _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org <mailto:users@ovirt.org>/message/DXB2JDUJENTDD2CV7SRCYIWUFJQC43XA/

Thanks very much, Konstantin. "you can use nfs-check.py described over there" - thats a useful lead. Will try (temporarily I reverted to glusterfs, to get this moving; but the ultimate goal is to have the ceph as the storage). "I think the fact that your NFS is over Ceph should be something that is 'abstracted away' " - Yes fully agree. In an iteration earlier, IIRC, vanilla NFS worked with no issues. This is definitely something to do with Ceph's NFS. Having said that, the installation process successfully writes (copies) the engine VM in to the storage but only fails at the 'reinitialize-lockspace' task. Do you (or anybody else) know what the 'reinitialize-lockspace' does do? Thanks
participants (2)
-
destfinal@googlemail.com
-
Volenbovskyi, Konstantin