thanks very much ,I got it.<br><div id="origbody"><div style="background: #f2f2f2;">----- Original Message -----<br>From: Martin Sivak <msivak@redhat.com><br>To: dhy336 <dhy336@sina.com><br>Cc: users <users@ovirt.org><br>Subject: Re: Re: Re: [ovirt-users] �ظ���Re: Hosted-engine can not_switch<br>Date: 2018-04-20 18:28<br></div><br><br>Hi,<br>No, this is not an error. You killed the host without moving it to<br>maintenance first. The engine has no way to distinguish this from<br>temporary network failure for example. Give it some time and the host<br>will move its status to one of the error states and handle the highly<br>available VMs on it (if fencing is properly configured).<br>Best regards<br>Martin Sivak<br>On Fri, Apr 20, 2018 at 12:13 PM, <dhy336@sina.com> wrote:<br>> this process is not error ?<br>> ----- Original Message -----<br>> From: Martin Sivak <msivak@redhat.com><br>> To: dhy336 <dhy336@sina.com><br>> Cc: users <users@ovirt.org><br>> Subject: Re: Re: [ovirt-users] �ظ���Re: Hosted-engine can not_switch<br>> Date: 2018-04-20 18:05<br>><br>><br>> Hi,<br>> the engine does not know you killed the host. It will notice<br>> eventually and handle the situation. Just give it time (5 minutes or<br>> so).<br>> Best regards<br>> --<br>> Martin Sivak<br>> SLA / oVirt<br>> On Fri, Apr 20, 2018 at 12:00 PM, <dhy336@sina.com> wrote:<br>>> Hi�� thanks for your feedback. I hava another qeustions<br>>><br>>> I deploy two node for hosted engine, first hosted engine VM run in<br>>> 192.168.122.65, I power off this host, hosted-engine VM switch<br>>> another host��but ovirt engine still connect 192.168.122.65. if restart<br>>> ovirt-engine server, it is work.<br>>><br>>><br>>> 2018-04-20 17:13:04,692+08 ERROR<br>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]<br>>> (EE-ManagedThreadFactory-en gineScheduled-Thread-98) [] Command<br>>> 'GetAllVmStatsVDSCommand(HostName = hosted-engine2,<br>>> VdsIdVDSCommandParametersBase:{hos<br>>> tId='a5428ef7-9df6-4a86-91de-7e36fda340fa'})' execution failed:<br>>> java.net.NoRouteToHostException: No route to host<br>>> 6568 2018-04-20 17:13:04,693+08 INFO<br>>> [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]<br>>> (EE-ManagedThreadFactory-engi neScheduled-Thread-98) [] Failed to fetch<br>>> vms info for host 'hosted-engin2' - skipping VMs monitoring.<br>>> 6569 2018-04-20 17:13:19,710+08 INFO<br>>> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)<br>>> [] Connecting to hosted-engine2/192.168.122.656570 2018-04-20<br>>> 17:13:22,730+08 ERROR<br>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]<br>>> (EE-ManagedThreadFactory-en gineScheduled-Thread-45) [] Command<br>>> 'GetAllVmStatsVDSCommand(HostName = hosted-engine-tchyp2,<br>>> VdsIdVDSCommandParametersBase:{hos<br>>> tId='a5428ef7-9df6-4a86-91de-7e36fda340fa'})' execution failed:<br>>> java.net.NoRouteToHostException: No route to host<br>>> 6571 2018-04-20 17:13:22,732+08 INFO<br>>> [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]<br>>> (EE-ManagedThreadFactory-engi neScheduled-Thread-45) [] Failed to fetch<br>>> vms info for host 'hosted-engine2' - skipping VMs monitoring.<br>>><br>>> ----- Original Message -----<br>>> From: Martin Sivak <msivak@redhat.com><br>>> To: dhy336 <dhy336@sina.com><br>>> Cc: users <users@ovirt.org><br>>> Subject: Re: [ovirt-users] �ظ���Re: Hosted-engine can not_switch<br>>> Date: 2018-04-20 16:40<br>>><br>>><br>>> Hi,<br>>> your ovirt-hosted-engine-ha package is too old. You need at least<br>>> 2.1.9 to properly support 4.2 engine. The same applies to vdsm. Please<br>>> upgrade the node.<br>>> Best regards<br>>> Martin Sivak<br>>> On Fri, Apr 20, 2018 at 3:58 AM, <dhy336@sina.com> wrote:<br>>>> Hi I find some error logs in /var/log/ovirt-hosted-engine-ha/broker.<br>>>><br>>>> [root@hosted-engine2 ~]# ll /rhev/data-center/mnt<br>>>> total 0<br>>>> drwxr-xr-x. 3 vdsm kvm 76 Apr 18 22:28 192.168.122.218:_exports_data<br>>>> drwxr-xr-x. 3 vdsm kvm 76 Apr 18 22:12<br>>>> 192.168.122.218:_exports_hosted-engine-test1<br>>>> [root@hosted-engine2 ~]# ll<br>>>> /rhev/data-center/mnt/192.168.122.218\:_exports_hosted-engine-test1/<br>>>> total 0<br>>>> drwxr-xr-x. 5 vdsm kvm 50 Apr 18 22:14<br>>>> 8a734205-65b7-4801-b7f0-d380eb45dbae<br>>>> -rwxr-xr-x. 1 vdsm kvm 0 Apr 20 09:54 __DIRECT_IO_TEST__<br>>>><br>>>> uuid 8a734205-65b7-4801-b7f0-d380eb45dbae is in<br>>>> /rhev/data-center/mnt/192.168.122.218\:_exports_hosted-engine-test1/<br>>>> but broker find it in /rhev/data-center/mnt, is it my version is error?<br>>>> my<br>>>> ovirt-hosted-engine-ha version is 2.1.5, vdsm is 4.20.5��<br>>>> ovirt-engine is 4.2<br>>>><br>>>> MainThread::INFO::2018-04-19<br>>>><br>>>><br>>>> 19:26:31,479::listener::41::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__)<br>>>> Initializing SocketServer<br>>>> MainThread::INFO::2018-04-19<br>>>><br>>>><br>>>> 19:26:31,480::listener::56::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__)<br>>>> SocketServer ready<br>>>> Thread-1::INFO::2018-04-19<br>>>><br>>>><br>>>> 19:26:31,558::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)<br>>>> Connection established<br>>>> Thread-1::ERROR::2018-04-19<br>>>><br>>>><br>>>> 19:26:31,559::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)<br>>>> Error handling request, data: 'set-storage-domain FilesystemBackend<br>>>> dom_type=nfs3 sd_uuid=8a734205-65b7-4801-b7f0-d380eb45dbae'<br>>>> Traceback (most recent call last):<br>>>> File<br>>>><br>>>><br>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",<br>>>> line 166, in handle<br>>>> data)<br>>>> File<br>>>><br>>>><br>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",<br>>>> line 299, in _dispatch<br>>>> .set_storage_domain(client, sd_type, **options)<br>>>> File<br>>>><br>>>><br>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",<br>>>> line 66, in set_storage_domain<br>>>> self._backends[client].connect()<br>>>> File<br>>>><br>>>><br>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",<br>>>> line 462, in connect<br>>>> self._dom_type)<br>>>> File<br>>>><br>>>><br>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",<br>>>> line 107, in get_domain_path<br>>>> " in {1}".format(sd_uuid, parent))<br>>>> BackendFailureException: path to storage domain<br>>>> 8a734205-65b7-4801-b7f0-d380eb45dbae not found in /rhev/data-center/mnt<br>>>> Thread-1::INFO::2018-04-19<br>>>><br>>>><br>>>> 19:26:31,563::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)<br>>>> Connection closed<br>>>> Thread-2::INFO::2018-04-19<br>>>><br>>>><br>>>> 19:26:44,601::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)<br>>>> Connection established<br>>>><br>>>> ----- ԭʼ�ʼ� -----<br>>>> �����ˣ�<dhy336@sina.com><br>>>> �ռ��ˣ�"Martin Sivak" <msivak@redhat.com><br>>>> �����ˣ�users <users@ovirt.org><br>>>> ���⣺[ovirt-users] �ظ���Re: Hosted-engine can not_switch<br>>>> ���ڣ�2018��04��20�� 09��30��<br>>>><br>>>> libvirt has not error logs . I only find some error for vdsm.<br>>>> vdsm log is:<br>>>> 2018-04-20 09:24:52,610+0800 INFO (jsonrpc/1) [vdsm.api] FINISH<br>>>> getVolumeInfo return={'info': {'status': 'OK', 'domain':<br>>>> '8a734205-65b7-4801-b7f0-d380eb45dbae', 'voltype': 'LEAF', 'description':<br>>>> 'hosted-engine.lockspace', 'parent':<br>>>> '00000000-0000-0000-0000-000000000000',<br>>>> 'format': 'RAW', 'generation': 0, 'image':<br>>>> '611272bd-c2cc-42bc-94e2-9aa52e754c35', 'ctime': '1524032037',<br>>>> 'disktype':<br>>>> '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1048576',<br>>>> 'children': [], 'pool': '', 'capacity': '1048576', 'uuid':<br>>>> u'7037aac6-7c8e-4efd-82f7-ca618c953fe6', 'truesize': '1048576', 'type':<br>>>> 'PREALLOCATED', 'lease': {'owners': [], 'version': None}}}<br>>>> from=::1,48306,<br>>>> task_id=03a7938e-8afb-4b16-b8dd-126c2b1f5d52 (api:52)<br>>>> 2018-04-20 09:24:52,611+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC<br>>>> call Volume.getInfo succeeded in 0.03 seconds (__init__:630)<br>>>> 2018-04-20 09:24:54,113+0800 ERROR (periodic/3) [virt.periodic.Operation]<br>>>> <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x1e92f90> operation<br>>>> failed<br>>>> (periodic:215)<br>>>> Traceback (most recent call last):<br>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 213,<br>>>> in __call__<br>>>> self._func()<br>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 522,<br>>>> in __call__<br>>>> self._send_metrics()<br>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 538,<br>>>> in _send_metrics<br>>>> vm_sample.interval)<br>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 45, in<br>>>> produce<br>>>> networks(vm, stats, first_sample, last_sample, interval)<br>>>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 322,<br>>>> in<br>>>> networks<br>>>> if nic.name.startswith('hostdev'):<br>>>> AttributeError: name<br>>>> 2018-04-20 09:24:54,800+0800 INFO (Reactor thread)<br>>>> [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:48308<br>>>> (protocoldetector:61)<br>>>> 2018-04-20 09:24:54,810+0800 INFO (Reactor thread)<br>>>> [ProtocolDetector.Detector] Detected protocol stomp from ::1:48308<br>>>> (protocoldetector:125)<br>>>> 2018-04-20 09:24:54,810+0800 INFO (Reactor thread) [Broker.StompAdapter]<br>>>> Processing CONNECT request (stompreactor:103)<br>>>> 2018-04-20 09:24:54,818+0800 INFO (JsonRpc (StompReactor))<br>>>> [Broker.StompAdapter] Subscribe command received (stompreactor:132)<br>>>> 2018-04-20 09:24:55,119+0800 INFO (jsonrpc/6) [api.host] START<br>>>> getHardwareInfo() from=::1,48308 (api:46)<br>>>><br>>>> ----- ԭʼ�ʼ� -----<br>>>> �����ˣ�Martin Sivak <msivak@redhat.com><br>>>> �ռ��ˣ�dhy336 <dhy336@sina.com><br>>>> �����ˣ�users <users@ovirt.org><br>>>> ���⣺Re: [ovirt-users] Hosted-engine can not switch<br>>>> ���ڣ�2018��04��19�� 20��16��<br>>>><br>>>><br>>>> We need more than just this small log snippet. Please check the vdsm<br>>>> and libvirt logs as well.<br>>>> Best regards<br>>>> Martin Sivak<br>>>> On Thu, Apr 19, 2018 at 2:05 PM, <dhy336@sina.com> wrote:<br>>>>> Hi��<br>>>>> I deploy three node with hosted engine, I force shut down a node which<br>>>>> Host-engine VM is run, But hosted engine VM in other nodes can not run.<br>>>>><br>>>>> I find some error in /var/log/ovirt-hosted-engine-ha/agent.log<br>>>>><br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:35,787::hosted_engine::1192::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state)<br>>>>> Cleaning state for non-running VM<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:42,587::hosted_engine::1176::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state)<br>>>>> Vdsm state for VM clean<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:42,589::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)<br>>>>> Starting vm using `/usr/sbin/hosted-engine --vm-start`<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,599::hosted_engine::1131::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)<br>>>>> stdout:<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,600::hosted_engine::1132::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)<br>>>>> stderr: Virtual machine does not exist: {'vmId':<br>>>>> u'08bbd680-a8a7-4267-82e7-89f36e87e930'}<br>>>>><br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,600::hosted_engine::1144::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)<br>>>>> Engine VM started on localhost<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,609::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>>>>> Trying: notify time=1524139007.61 type=state_transition<br>>>>> detail=EngineStart-EngineStarting hostname='hosted-engine2'<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,670::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>>>>> Success, was notification of state_transition<br>>>>> (EngineStart-EngineStarting)<br>>>>> sent? sent<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:47,670::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)<br>>>>> Initializing VDSM<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:50,095::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)<br>>>>> Connecting the storage<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:50,096::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server)<br>>>>> Validating storage server<br>>>>> MainThread::INFO::2018-04-19<br>>>>><br>>>>><br>>>>><br>>>>> 19:56:52,449::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)<br>>>>> Storage domain reported as valid and reconnect is not forced.<br>>>>><br>>>>><br>>>>> _______________________________________________<br>>>>> Users mailing list<br>>>>> Users@ovirt.org<br>>>>> http://lists.ovirt.org/mailman/listinfo/users<br>>>>><br>>>> _______________________________________________<br>>>> Users mailing list<br>>>> Users@ovirt.org<br>>>> http://lists.ovirt.org/mailman/listinfo/users<br></div>