1:40 p.m.
Your gluster mount option is not correct.
You need 'backup-volfile-servers=storagehost2:storagehost3' (without the volume
name as they all have thaylt volume) .
Best Regards,
Strahil Nikolov
На 13 юни 2020 г. 10:47:28 GMT+03:00, Oliver Leinfelder
<oliver.leinfelder(a)gmail.com> написа:
>Hi,
>
>I have the following two components:
>
>1.) A freshly installed VM host (oVirt Node 4.4.0 release ISO)
>2.) 3 storage hosts, also freshly installed from oVirt Node 4.4.0
>release ISO
>
>The storage hosts have been successfully installed with Gluster
>(through
>Cockpit). They have two volumes, both of which I can mount and
>read/write from a client.
>
>On the VM host, I ran "hosted-engine --deploy" (no backups imported).
>
>When prompted for storage, I answered "glusterfs" and specified
>"storagehost1:/engine" as storage for the HE deployment. For mount
>options, I specified
>"backup-volfile-servers=storagehost2:/engine:storagehost3:/engine"
>
>(Not the real hostnames, but all of them are resolvable via internal
>DNS)
>
>Everything seems to works fine, I also see the "engine" volume become
>populated with data. At some point I could ping and SSH login to the
>HE.
>
>When the setup proceed to health check, it failed and the whole process
>
>was aborted :-(
>
>"hosted-engine --vm-status" reported "failed liveliness check"
when it
>was reachable via SSH. At some point the engine went down and, to my
>surprise, shows a grub prompt after the restart when doing a
>"hosted-engine --console".
>
>[ INFO ] TASK [ovirt.hosted_engine_setup : Check engine VM health]
>[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 180,
"changed":
>true, "cmd": ["hosted-engine", "--vm-status",
"--json"], "delta":
>"0:00:00.160595", "end": "2020-06-12 17:50:05.675774",
"rc": 0,
>"start":
>"2020-06-12 17:50:05.515179", "stderr": "",
"stderr_lines": [],
>"stdout": "{\"1\": {\"host-id\": 1,
\"host-ts\": 11528, \"score\":
>3400,
>\"engine-status\": {\"vm\": \"up\",
\"health\": \"bad\", \"detail\":
>\"Powering down\", \"reason\": \"failed liveliness
check\"},
>\"hostname\": \"vmhost\", \"maintenance\": false,
\"stopped\": false,
>\"crc32\": \"2c447835\", \"conf_on_shared_storage\":
true,
>\"local_conf_timestamp\": 11528, \"extra\":
>\"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528
>
>(Fri Jun 12 17:49:57
>2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12
>
>17:49:57
>2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu
>
>Jan 1 04:12:48 1970\\n\", \"live-data\": true},
>\"global_maintenance\":
>false}", "stdout_lines": ["{\"1\":
{\"host-id\": 1, \"host-ts\": 11528,
>
>\"score\": 3400, \"engine-status\": {\"vm\":
\"up\", \"health\":
>\"bad\", \"detail\": \"Powering down\",
\"reason\": \"failed liveliness
>
>check\"}, \"hostname\": \"vmhost\",
\"maintenance\": false,
>\"stopped\":
>false, \"crc32\": \"2c447835\",
\"conf_on_shared_storage\": true,
>\"local_conf_timestamp\": 11528, \"extra\":
>\"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=11528
>
>(Fri Jun 12 17:49:57
>2020)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=11528 (Fri Jun 12
>
>17:49:57
>2020)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStop\\nstopped=False\\ntimeout=Thu
>
>Jan 1 04:12:48 1970\\n\", \"live-data\": true},
>\"global_maintenance\":
>false}"]}
>
>A second attempt failed at exactly the same stage.
>
>I can see the following in the setup log:
>
>ovirt-hosted-engine-setup-20200612151212-j9zwd2.log:
>2020-06-12 17:33:18,314+0200 DEBUG
>otopi.ovirt_hosted_engine_setup.ansible_utils
>ansible_utils._process_output:103 {'msg': 'non-zero return code',
>'cmd':
>['hosted-engine', '--reinitialize-lockspace', '--force'],
'stdout': '',
>
>'stderr': 'Traceback (most recent call last):\n
> File "/usr/lib64/python3.6/runpy.py", line 193, in
>_run_module_as_main\n "__main__", mod_spec)\n File
>"/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code,
>run_globals)\n File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_
>lockspace.py", line 30, in <module>\n ha_cli.reset_lockspace(force)\n
>File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>
>line 286, in reset_lockspace\n stats =
>broker.get_stats_from_storage()\n File
>"/usr/lib/python3.6/site-packages/ov
>irt_hosted_engine_ha/lib/brokerlink.py", line 148, in
>get_stats_from_storage\n result = self._proxy.get_stats()\n File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__\n
>return
>self.__send(self.__name, args)\n File
>"/usr/lib64/python3.6/xmlrpc/client
>.py", line 1452, in __request\n verbose=self.__verbose\n File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request\n return
>
>self.single_request(host, handler, request_body, verbose)\n File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_requ
>est\n http_conn = self.send_request(host, handler, request_body,
>verbose)\n File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in
>
>send_request\n self.send_content(connection, request_body)\n File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_con
>tent\n connection.endheaders(request_body)\n File
>"/usr/lib64/python3.6/http/client.py", line 1249, in endheaders\n
>self._send_output(message_body, encode_chunked=encode_chunked)\n File
>"/usr/lib64/python3.6/http/client.py", line 1036, in _send_output\n
>self.sen
>d(msg)\n File "/usr/lib64/python3.6/http/client.py", line 974, in
>send\n self.connect()\n File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
>
>line 74, in connect\n
>self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError:
>[Errno
>2] No such file or directory', 'rc': 1, 'start': '2020-06-12
>17:33:17.982995', 'end': '2020-06-12 17:33:18.164197',
'delta':
>'0:00:00.181202', 'changed': True, 'invocation':
{'module_args':
>{'_raw_params': 'hosted-engine --reinitialize-lockspace --force',
>'warn': True, '_u
>ses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends':
True,
>
>'argv': None, 'chdir': None, 'executable': None,
'creates': None,
>'removes': None, 'stdin': None}}, 'stdout_lines': [],
'stderr_lines':
>['Traceback (most recent call last):', ' File "/usr/lib64/python
>3.6/runpy.py", line 193, in _run_module_as_main', '
"__main__",
>mod_spec)', ' File "/usr/lib64/python3.6/runpy.py", line 85, in
>_run_code', ' exec(code, run_globals)', ' File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py",
>
>li
>ne 30, in <module>', ' ha_cli.reset_lockspace(force)', ' File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>
>line 286, in reset_lockspace', ' stats =
>broker.get_stats_from_storage()', ' File
>"/usr/lib/python3.6/site-packages/ovirt_hoste
>d_engine_ha/lib/brokerlink.py", line 148, in get_stats_from_storage',
>' result = self._proxy.get_stats()', ' File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__',
'
>return self.__send(self.__name, args)', ' File
>"/usr/lib64/python3.6/xmlrpc/client.
>py", line 1452, in __request', ' verbose=self.__verbose', '
File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request',
'
>return self.single_request(host, handler, request_body, verbose)', '
>File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in sing
>le_request', ' http_conn = self.send_request(host, handler,
>request_body, verbose)', ' File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request',
>' self.send_content(connection, request_body)', ' File
>"/usr/lib64/python3.6/xmlrpc/client.py", line 13
>09, in send_content', ' connection.endheaders(request_body)', '
File
>
>"/usr/lib64/python3.6/http/client.py", line 1249, in endheaders',
'
>self._send_output(message_body, encode_chunked=encode_chunked)', '
>File
>"/usr/lib64/python3.6/http/client.py", line 1036, in _se
>nd_output', ' self.send(msg)', ' File
>"/usr/lib64/python3.6/http/client.py", line 974, in send', '
>self.connect()', ' File
>"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
>
>line 74, in connect', ' self.sock.connect(base64.b16decode(self.h
>ost))', 'FileNotFoundError: [Errno 2] No such file or directory'],
>'_ansible_no_log': False, 'attempts': 5}
>
>
>Bonus question: I can see the following in the gluster volume log on
>the
>storage host:
>[2020-06-12 17:30:38.368773] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1976:posix_writev] 0-engine-posix: write failed:
>offset 0, [Invalid argument]
>[2020-06-12 17:30:38.368832] E [MSGID: 115067]
>[server-rpc-fops_v2.c:1372:server4_writev_cbk] 0-engine-server: 190586:
>
>WRITEV 4 (baf5fbaa-0dc0-4ee4-bfd9-fa98f8853827), client:
>CTX_ID:be1aca7a-5ba7-453d-94fd-d24f8c33511e-GRAPH_ID:0-PID:121217-HOST:vmhost-PC_NAME:engine-client-0-RECON_NO:-0,
>
>error-xlator: engine-posix [Invalid argument]
>
>There are some entries every few minutes like this (not like 100s per
>minute). Since gluster is completely new for me, I'm still trying to
>find out whether this might be related.
>
>Thanks for any input!
>
>Best regards
>Oliver
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/3BQMCIGCEOLIOV3LSW47GVPKSMOOK7IL/