oVirt 4.4.0 HE deployment on GlusterFS fails during health check
Hi, I have the following two components: 1.) A freshly installed VM host (oVirt Node 4.4.0 release ISO) 2.) 3 storage hosts, also freshly installed from oVirt Node 4.4.0 release ISO The storage hosts have been successfully installed with Gluster (through Cockpit). They have two volumes, both of which I can mount and read/write from a client. On the VM host, I ran "hosted-engine --deploy" (no backups imported). When prompted for storage, I answered "glusterfs" and specified "storagehost1:/engine" as storage for the HE deployment. For mount options, I specified "backup-volfile-servers=storagehost2:/engine:storagehost3:/engine" (Not the real hostnames, but all of them are resolvable via internal DNS) Everything seems to works fine, I also see the "engine" volume become populated with data. At some point I could ping and SSH login to the HE. When the setup proceed to health check, it failed and the whole process was aborted :-( "hosted-engine --vm-status" reported "failed liveliness check" when it was reachable via SSH. At some point the engine went down and, to my surprise, shows a grub prompt after the restart when doing a "hosted-engine --console". [ INFO ] TASK [ovirt.hosted_engine_setup : Check engine VM health] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 180, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.160595", "end": "2020-06-12 17:50:05.675774", "rc": 0, "start": "2020-06-12 17:50:05.515179", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"host-id\": 1, \"host-ts\": 11528, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Powering down\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"vmhost\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2c447835\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 11528, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528 (Fri Jun 12 17:49:57 2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12 17:49:57 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu Jan 1 04:12:48 1970\\n\", \"live-data\": true}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"host-id\": 1, \"host-ts\": 11528, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Powering down\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"vmhost\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2c447835\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 11528, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528 (Fri Jun 12 17:49:57 2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12 17:49:57 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu Jan 1 04:12:48 1970\\n\", \"live-data\": true}, \"global_maintenance\": false}"]} A second attempt failed at exactly the same stage. I can see the following in the setup log: ovirt-hosted-engine-setup-20200612151212-j9zwd2.log: 2020-06-12 17:33:18,314+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': 'non-zero return code', 'cmd': ['hosted-engine', '--reinitialize-lockspace', '--force'], 'stdout': '', 'stderr': 'Traceback (most recent call last):\n File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main\n "__main__", mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_ lockspace.py", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File "/usr/lib/python3.6/site-packages/ov irt_hosted_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__\n return self.__send(self.__name, args)\n File "/usr/lib64/python3.6/xmlrpc/client .py", line 1452, in __request\n verbose=self.__verbose\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request\n return self.single_request(host, handler, request_body, verbose)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_requ est\n http_conn = self.send_request(host, handler, request_body, verbose)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request\n self.send_content(connection, request_body)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_con tent\n connection.endheaders(request_body)\n File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output\n self.sen d(msg)\n File "/usr/lib64/python3.6/http/client.py", line 974, in send\n self.connect()\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 74, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory', 'rc': 1, 'start': '2020-06-12 17:33:17.982995', 'end': '2020-06-12 17:33:18.164197', 'delta': '0:00:00.181202', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'hosted-engine --reinitialize-lockspace --force', 'warn': True, '_u ses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Traceback (most recent call last):', ' File "/usr/lib64/python 3.6/runpy.py", line 193, in _run_module_as_main', ' "__main__", mod_spec)', ' File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code', ' exec(code, run_globals)', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py", li ne 30, in <module>', ' ha_cli.reset_lockspace(force)', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 286, in reset_lockspace', ' stats = broker.get_stats_from_storage()', ' File "/usr/lib/python3.6/site-packages/ovirt_hoste d_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage', ' result = self._proxy.get_stats()', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__', ' return self.__send(self.__name, args)', ' File "/usr/lib64/python3.6/xmlrpc/client. py", line 1452, in __request', ' verbose=self.__verbose', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request', ' return self.single_request(host, handler, request_body, verbose)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in sing le_request', ' http_conn = self.send_request(host, handler, request_body, verbose)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request', ' self.send_content(connection, request_body)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 13 09, in send_content', ' connection.endheaders(request_body)', ' File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders', ' self._send_output(message_body, encode_chunked=encode_chunked)', ' File "/usr/lib64/python3.6/http/client.py", line 1036, in _se nd_output', ' self.send(msg)', ' File "/usr/lib64/python3.6/http/client.py", line 974, in send', ' self.connect()', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 74, in connect', ' self.sock.connect(base64.b16decode(self.h ost))', 'FileNotFoundError: [Errno 2] No such file or directory'], '_ansible_no_log': False, 'attempts': 5} Bonus question: I can see the following in the gluster volume log on the storage host: [2020-06-12 17:30:38.368773] E [MSGID: 113072] [posix-inode-fd-ops.c:1976:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2020-06-12 17:30:38.368832] E [MSGID: 115067] [server-rpc-fops_v2.c:1372:server4_writev_cbk] 0-engine-server: 190586: WRITEV 4 (baf5fbaa-0dc0-4ee4-bfd9-fa98f8853827), client: CTX_ID:be1aca7a-5ba7-453d-94fd-d24f8c33511e-GRAPH_ID:0-PID:121217-HOST:vmhost-PC_NAME:engine-client-0-RECON_NO:-0, error-xlator: engine-posix [Invalid argument] There are some entries every few minutes like this (not like 100s per minute). Since gluster is completely new for me, I'm still trying to find out whether this might be related. Thanks for any input! Best regards Oliver
Your gluster mount option is not correct. You need 'backup-volfile-servers=storagehost2:storagehost3' (without the volume name as they all have thaylt volume) . Best Regards, Strahil Nikolov На 13 юни 2020 г. 10:47:28 GMT+03:00, Oliver Leinfelder <oliver.leinfelder@gmail.com> написа:
Hi,
I have the following two components:
1.) A freshly installed VM host (oVirt Node 4.4.0 release ISO) 2.) 3 storage hosts, also freshly installed from oVirt Node 4.4.0 release ISO
The storage hosts have been successfully installed with Gluster (through Cockpit). They have two volumes, both of which I can mount and read/write from a client.
On the VM host, I ran "hosted-engine --deploy" (no backups imported).
When prompted for storage, I answered "glusterfs" and specified "storagehost1:/engine" as storage for the HE deployment. For mount options, I specified "backup-volfile-servers=storagehost2:/engine:storagehost3:/engine"
(Not the real hostnames, but all of them are resolvable via internal DNS)
Everything seems to works fine, I also see the "engine" volume become populated with data. At some point I could ping and SSH login to the HE.
When the setup proceed to health check, it failed and the whole process
was aborted :-(
"hosted-engine --vm-status" reported "failed liveliness check" when it was reachable via SSH. At some point the engine went down and, to my surprise, shows a grub prompt after the restart when doing a "hosted-engine --console".
[ INFO ] TASK [ovirt.hosted_engine_setup : Check engine VM health] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 180, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.160595", "end": "2020-06-12 17:50:05.675774", "rc": 0, "start": "2020-06-12 17:50:05.515179", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"host-id\": 1, \"host-ts\": 11528, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Powering down\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"vmhost\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2c447835\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 11528, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528
(Fri Jun 12 17:49:57 2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12
17:49:57 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu
Jan 1 04:12:48 1970\\n\", \"live-data\": true}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"host-id\": 1, \"host-ts\": 11528,
\"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Powering down\", \"reason\": \"failed liveliness
check\"}, \"hostname\": \"vmhost\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2c447835\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 11528, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528
(Fri Jun 12 17:49:57 2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12
17:49:57 2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu
Jan 1 04:12:48 1970\\n\", \"live-data\": true}, \"global_maintenance\": false}"]}
A second attempt failed at exactly the same stage.
I can see the following in the setup log:
ovirt-hosted-engine-setup-20200612151212-j9zwd2.log: 2020-06-12 17:33:18,314+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': 'non-zero return code', 'cmd': ['hosted-engine', '--reinitialize-lockspace', '--force'], 'stdout': '',
'stderr': 'Traceback (most recent call last):\n File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main\n "__main__", mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_ lockspace.py", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File "/usr/lib/python3.6/site-packages/ov irt_hosted_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__\n return self.__send(self.__name, args)\n File "/usr/lib64/python3.6/xmlrpc/client .py", line 1452, in __request\n verbose=self.__verbose\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request\n return
self.single_request(host, handler, request_body, verbose)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_requ est\n http_conn = self.send_request(host, handler, request_body, verbose)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in
send_request\n self.send_content(connection, request_body)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_con tent\n connection.endheaders(request_body)\n File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output\n self.sen d(msg)\n File "/usr/lib64/python3.6/http/client.py", line 974, in send\n self.connect()\n File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
line 74, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory', 'rc': 1, 'start': '2020-06-12 17:33:17.982995', 'end': '2020-06-12 17:33:18.164197', 'delta': '0:00:00.181202', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'hosted-engine --reinitialize-lockspace --force', 'warn': True, '_u ses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True,
'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Traceback (most recent call last):', ' File "/usr/lib64/python 3.6/runpy.py", line 193, in _run_module_as_main', ' "__main__", mod_spec)', ' File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code', ' exec(code, run_globals)', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py",
li ne 30, in <module>', ' ha_cli.reset_lockspace(force)', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 286, in reset_lockspace', ' stats = broker.get_stats_from_storage()', ' File "/usr/lib/python3.6/site-packages/ovirt_hoste d_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage', ' result = self._proxy.get_stats()', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__', ' return self.__send(self.__name, args)', ' File "/usr/lib64/python3.6/xmlrpc/client. py", line 1452, in __request', ' verbose=self.__verbose', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request', ' return self.single_request(host, handler, request_body, verbose)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in sing le_request', ' http_conn = self.send_request(host, handler, request_body, verbose)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request', ' self.send_content(connection, request_body)', ' File "/usr/lib64/python3.6/xmlrpc/client.py", line 13 09, in send_content', ' connection.endheaders(request_body)', ' File
"/usr/lib64/python3.6/http/client.py", line 1249, in endheaders', ' self._send_output(message_body, encode_chunked=encode_chunked)', ' File "/usr/lib64/python3.6/http/client.py", line 1036, in _se nd_output', ' self.send(msg)', ' File "/usr/lib64/python3.6/http/client.py", line 974, in send', ' self.connect()', ' File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
line 74, in connect', ' self.sock.connect(base64.b16decode(self.h ost))', 'FileNotFoundError: [Errno 2] No such file or directory'], '_ansible_no_log': False, 'attempts': 5}
Bonus question: I can see the following in the gluster volume log on the storage host: [2020-06-12 17:30:38.368773] E [MSGID: 113072] [posix-inode-fd-ops.c:1976:posix_writev] 0-engine-posix: write failed: offset 0, [Invalid argument] [2020-06-12 17:30:38.368832] E [MSGID: 115067] [server-rpc-fops_v2.c:1372:server4_writev_cbk] 0-engine-server: 190586:
WRITEV 4 (baf5fbaa-0dc0-4ee4-bfd9-fa98f8853827), client: CTX_ID:be1aca7a-5ba7-453d-94fd-d24f8c33511e-GRAPH_ID:0-PID:121217-HOST:vmhost-PC_NAME:engine-client-0-RECON_NO:-0,
error-xlator: engine-posix [Invalid argument]
There are some entries every few minutes like this (not like 100s per minute). Since gluster is completely new for me, I'm still trying to find out whether this might be related.
Thanks for any input!
Best regards Oliver _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3BQMCIGCEOLIOV...
Hi, Your gluster mount option is not correct.
You need 'backup-volfile-servers=storagehost2:storagehost3' (without the volume name as they all have thaylt volume) .
yes, of course. I'm sorry but the appended volume name was a mistake I made for the email and not during deployment where only specified the FQDNs without the volname. As mentioned, the mount generally seema to work as data ist written during deployment. It fails later during health check :-( Best regards Oliver
participants (2)
-
Oliver Leinfelder -
Strahil Nikolov