<div dir="auto"><div>Hi,<div dir="auto"><br></div><div dir="auto">How many hosts you have? Check hosted-engine.conf on all hosts including the one you have problem with and look if all host_id values are unique. It might happen that you have several hosts with host_id=1</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Artem</div><br><div class="gmail_quote"><div dir="ltr">ср, 28 мар. 2018 г., 20:49 Jamie Lawrence <<a href="mailto:jlawrence@squaretrade.com">jlawrence@squaretrade.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I still can't resolve this issue.<br>
<br>
I have a host that is stuck in a cycle; it will be marked non responsive, then come back up, ending with an "finished activation" message in the GUI. Then it repeats.<br>
<br>
The root cause seems to be sanlock. I'm just unclear on why it started or how to resolve it. The only "approved" knob I'm aware of is --reinitialize-lockspace and the manual equivalent, neither of which fix anything.<br>
<br>
Anyone have a guess?<br>
<br>
-j<br>
<br>
- - - vdsm.log - - - -<br>
<br>
2018-03-28 10:38:22,207-0700 INFO (monitor/b41eb20) [storage.SANLock] Acquiring host id for domain b41eb20a-eafb-481b-9a50-a135cf42b15e (id=1, async=True) (clusterlock:284)<br>
2018-03-28 10:38:22,208-0700 ERROR (monitor/b41eb20) [storage.Monitor] Error acquiring host id 1 for domain b41eb20a-eafb-481b-9a50-a135cf42b15e (monitor:568)<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 565, in _acquireHostId<br>
self.domain.acquireHostId(self.hostId, async=True)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 828, in acquireHostId<br>
self._manifest.acquireHostId(hostId, async)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 453, in acquireHostId<br>
self._domainLock.acquireHostId(hostId, async)<br>
File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", line 315, in acquireHostId<br>
raise se.AcquireHostIdFailure(self._sdUUID, e)<br>
AcquireHostIdFailure: Cannot acquire host id: (u'b41eb20a-eafb-481b-9a50-a135cf42b15e', SanlockException(22, 'Sanlock lockspace add failure', 'Invalid argument'))<br>
2018-03-28 10:38:23,078-0700 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:23,085-0700 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=[u'b41eb20a-eafb-481b-9a50-a135cf42b15e']) from=::1,54450, task_id=186d7e8b-7b4e-485d-a9e0-c0cb46eed621 (api:46)<br>
2018-03-28 10:38:23,085-0700 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={u'b41eb20a-eafb-481b-9a50-a135cf42b15e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000812547', 'lastCheck': '0.4', 'valid': True}} from=::1,54450, task_id=186d7e8b-7b4e-485d-a9e0-c0cb46eed621 (api:52)<br>
2018-03-28 10:38:23,086-0700 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:23,092-0700 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=periodic/6 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x1d44110> at 0x1d44150> timeout=15, duration=150 at 0x7f076c05fb90> task#=83985 at 0x7f082c08e510>, traceback:<br>
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap<br>
self.__bootstrap_inner()<br>
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner<br>
self.run()<br>
File: "/usr/lib64/python2.7/threading.py", line 765, in run<br>
self.__target(*self.__args, **self.__kwargs)<br>
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run<br>
ret = func(*args, **kwargs)<br>
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run<br>
self._execute_task()<br>
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task<br>
task()<br>
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__<br>
self._callable()<br>
File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 213, in __call__<br>
self._func()<br>
File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 578, in __call__<br>
stats = hostapi.get_stats(self._cif, self._samples.stats())<br>
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats<br>
ret['haStats'] = _getHaInfo()<br>
File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo<br>
stats = instance.get_all_stats()<br>
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 93, in get_all_stats<br>
stats = broker.get_stats_from_storage()<br>
File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage<br>
result = self._proxy.get_stats()<br>
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__<br>
return self.__send(self.__name, args)<br>
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request<br>
verbose=self.__verbose<br>
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request<br>
return self.single_request(host, handler, request_body, verbose)<br>
File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request<br>
response = h.getresponse(buffering=True)<br>
File: "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse<br>
response.begin()<br>
File: "/usr/lib64/python2.7/httplib.py", line 444, in begin<br>
version, status, reason = self._read_status()<br>
File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status<br>
line = self.fp.readline(_MAXLINE + 1)<br>
File: "/usr/lib64/python2.7/socket.py", line 476, in readline<br>
data = self._sock.recv(self._rbufsize) (executor:363)<br>
2018-03-28 10:38:23,274-0700 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:24,297-0700 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)<br>
<br>
2018-03-28 10:38:24,306-0700 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=[u'b41eb20a-eafb-481b-9a50-a135cf42b15e']) from=::1,54450, task_id=6a60e316-e4d7-415d-970a-a998710a5899 (api:46)<br>
2018-03-28 10:38:24,306-0700 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={u'b41eb20a-eafb-481b-9a50-a135cf42b15e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000812547', 'lastCheck': '1.6', 'valid': True}} from=::1,54450, task_id=6a60e316-e4d7-415d-970a-a998710a5899 (api:52)<br>
2018-03-28 10:38:24,307-0700 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:24,374-0700 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::ffff:10.181.26.150,46064 (api:46)<br>
2018-03-28 10:38:24,377-0700 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:10.181.26.150,46064 (api:52)<br>
2018-03-28 10:38:24,379-0700 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:24,529-0700 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,54454 (api:46)<br>
2018-03-28 10:38:24,532-0700 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,54454 (api:52)<br>
2018-03-28 10:38:24,533-0700 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)<br>
2018-03-28 10:38:24,545-0700 INFO (jsonrpc/6) [api.host] START getAllVmIoTunePolicies() from=::1,54454 (api:46)<br>
2018-03-28 10:38:24,546-0700 INFO (jsonrpc/6) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'message': 'Done', 'code': 0}, 'io_tune_policies_dict': {'588a1394-4f28-4fb8-bcad-5b08d78ecd00': {'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': u'/var/run/vdsm/storage/b41eb20a-eafb-481b-9a50-a135cf42b15e/a9d01d59-f146-47e5-b514-d10f8867678e/8f0c9f7a-ae6a-476e-b6f3-a830dcb79e87', 'name': 'vda'}]}}} from=::1,54454 (api:52)<br>
2018-03-28 10:38:24,547-0700 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:29,319-0700 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:29,327-0700 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=[u'b41eb20a-eafb-481b-9a50-a135cf42b15e']) from=::1,54450, task_id=c27c5e13-3b31-4182-9c14-11463c9b590a (api:46)<br>
2018-03-28 10:38:29,327-0700 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={u'b41eb20a-eafb-481b-9a50-a135cf42b15e': {'code': 0, 'actual': True, 'version': 4, 'acquired': False, 'delay': '0.000812547', 'lastCheck': '6.6', 'valid': True}} from=::1,54450, task_id=c27c5e13-3b31-4182-9c14-11463c9b590a (api:52)<br>
2018-03-28 10:38:29,328-0700 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:30,471-0700 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)<br>
2018-03-28 10:38:30,475-0700 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::1,54450 (api:46)<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank" rel="noreferrer">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
</blockquote></div></div></div>