[ovirt-users] 回复：Re: Hosted-engine can not_switch

Fri Apr 20 10:05:15 UTC 2018

Hi,

the engine does not know you killed the host. It will notice
eventually and handle the situation. Just give it time (5 minutes or
so).

Best regards

--
Martin Sivak
SLA / oVirt

On Fri, Apr 20, 2018 at 12:00 PM,  <dhy336 at sina.com> wrote:
> Hi， thanks for your feedback. I hava another qeustions
>
> I deploy two node for hosted engine, first hosted engine VM run in
> 192.168.122.65, I power off this host, hosted-engine VM switch
> another host，but ovirt engine still connect 192.168.122.65.  if restart
> ovirt-engine server, it is work.
>
>
>  2018-04-20 17:13:04,692+08 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (EE-ManagedThreadFactory-en     gineScheduled-Thread-98) [] Command
> 'GetAllVmStatsVDSCommand(HostName = hosted-engine2,
> VdsIdVDSCommandParametersBase:{hos
> tId='a5428ef7-9df6-4a86-91de-7e36fda340fa'})' execution failed:
> java.net.NoRouteToHostException: No route to host
> 6568 2018-04-20 17:13:04,693+08 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]
> (EE-ManagedThreadFactory-engi     neScheduled-Thread-98) [] Failed to fetch
> vms info for host 'hosted-engin2' - skipping VMs monitoring.
> 6569 2018-04-20 17:13:19,710+08 INFO
> [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
> [] Connecting to hosted-engine2/192.168.122.656570 2018-04-20
> 17:13:22,730+08 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (EE-ManagedThreadFactory-en     gineScheduled-Thread-45) [] Command
> 'GetAllVmStatsVDSCommand(HostName = hosted-engine-tchyp2,
> VdsIdVDSCommandParametersBase:{hos
> tId='a5428ef7-9df6-4a86-91de-7e36fda340fa'})' execution failed:
> java.net.NoRouteToHostException: No route to host
> 6571 2018-04-20 17:13:22,732+08 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]
> (EE-ManagedThreadFactory-engi     neScheduled-Thread-45) [] Failed to fetch
> vms info for host 'hosted-engine2' - skipping VMs monitoring.
>
> ----- Original Message -----
> From: Martin Sivak <msivak at redhat.com>
> To: dhy336 <dhy336 at sina.com>
> Cc: users <users at ovirt.org>
> Subject: Re: [ovirt-users] 回复：Re: Hosted-engine can not_switch
> Date: 2018-04-20 16:40
>
>
> Hi,
> your ovirt-hosted-engine-ha package is too old. You need at least
> 2.1.9 to properly support 4.2 engine. The same applies to vdsm. Please
> upgrade the node.
> Best regards
> Martin Sivak
> On Fri, Apr 20, 2018 at 3:58 AM, <dhy336 at sina.com> wrote:
>> Hi I find some error logs in /var/log/ovirt-hosted-engine-ha/broker.
>>
>> [root at hosted-engine2 ~]# ll /rhev/data-center/mnt
>> total 0
>> drwxr-xr-x. 3 vdsm kvm 76 Apr 18 22:28 192.168.122.218:_exports_data
>> drwxr-xr-x. 3 vdsm kvm 76 Apr 18 22:12
>> 192.168.122.218:_exports_hosted-engine-test1
>> [root at hosted-engine2 ~]# ll
>> /rhev/data-center/mnt/192.168.122.218\:_exports_hosted-engine-test1/
>> total 0
>> drwxr-xr-x. 5 vdsm kvm 50 Apr 18 22:14
>> 8a734205-65b7-4801-b7f0-d380eb45dbae
>> -rwxr-xr-x. 1 vdsm kvm 0 Apr 20 09:54 __DIRECT_IO_TEST__
>>
>> uuid 8a734205-65b7-4801-b7f0-d380eb45dbae is in
>> /rhev/data-center/mnt/192.168.122.218\:_exports_hosted-engine-test1/
>> but broker find it in /rhev/data-center/mnt, is it my version is error? my
>> ovirt-hosted-engine-ha version is 2.1.5, vdsm is 4.20.5，
>> ovirt-engine is 4.2
>>
>> MainThread::INFO::2018-04-19
>>
>> 19:26:31,479::listener::41::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__)
>> Initializing SocketServer
>> MainThread::INFO::2018-04-19
>>
>> 19:26:31,480::listener::56::ovirt_hosted_engine_ha.broker.listener.Listener::(__init__)
>> SocketServer ready
>> Thread-1::INFO::2018-04-19
>>
>> 19:26:31,558::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>> Connection established
>> Thread-1::ERROR::2018-04-19
>>
>> 19:26:31,559::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>> Error handling request, data: 'set-storage-domain FilesystemBackend
>> dom_type=nfs3 sd_uuid=8a734205-65b7-4801-b7f0-d380eb45dbae'
>> Traceback (most recent call last):
>> File
>>
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 166, in handle
>> data)
>> File
>>
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 299, in _dispatch
>> .set_storage_domain(client, sd_type, **options)
>> File
>>
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> line 66, in set_storage_domain
>> self._backends[client].connect()
>> File
>>
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 462, in connect
>> self._dom_type)
>> File
>>
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 107, in get_domain_path
>> " in {1}".format(sd_uuid, parent))
>> BackendFailureException: path to storage domain
>> 8a734205-65b7-4801-b7f0-d380eb45dbae not found in /rhev/data-center/mnt
>> Thread-1::INFO::2018-04-19
>>
>> 19:26:31,563::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>> Connection closed
>> Thread-2::INFO::2018-04-19
>>
>> 19:26:44,601::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>> Connection established
>>
>> ----- 原始邮件 -----
>> 发件人：<dhy336 at sina.com>
>> 收件人："Martin Sivak" <msivak at redhat.com>
>> 抄送人：users <users at ovirt.org>
>> 主题：[ovirt-users] 回复：Re: Hosted-engine can not_switch
>> 日期：2018年04月20日 09点30分
>>
>> libvirt has not error logs . I only find some error for vdsm.
>> vdsm log is:
>> 2018-04-20 09:24:52,610+0800 INFO (jsonrpc/1) [vdsm.api] FINISH
>> getVolumeInfo return={'info': {'status': 'OK', 'domain':
>> '8a734205-65b7-4801-b7f0-d380eb45dbae', 'voltype': 'LEAF', 'description':
>> 'hosted-engine.lockspace', 'parent':
>> '00000000-0000-0000-0000-000000000000',
>> 'format': 'RAW', 'generation': 0, 'image':
>> '611272bd-c2cc-42bc-94e2-9aa52e754c35', 'ctime': '1524032037', 'disktype':
>> '2', 'legality': 'LEGAL', 'mtime': '0', 'apparentsize': '1048576',
>> 'children': [], 'pool': '', 'capacity': '1048576', 'uuid':
>> u'7037aac6-7c8e-4efd-82f7-ca618c953fe6', 'truesize': '1048576', 'type':
>> 'PREALLOCATED', 'lease': {'owners': [], 'version': None}}} from=::1,48306,
>> task_id=03a7938e-8afb-4b16-b8dd-126c2b1f5d52 (api:52)
>> 2018-04-20 09:24:52,611+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC
>> call Volume.getInfo succeeded in 0.03 seconds (__init__:630)
>> 2018-04-20 09:24:54,113+0800 ERROR (periodic/3) [virt.periodic.Operation]
>> <vdsm.virt.sampling.VMBulkstatsMonitor object at 0x1e92f90> operation
>> failed
>> (periodic:215)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 213,
>> in __call__
>> self._func()
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 522,
>> in __call__
>> self._send_metrics()
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 538,
>> in _send_metrics
>> vm_sample.interval)
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 45, in
>> produce
>> networks(vm, stats, first_sample, last_sample, interval)
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 322, in
>> networks
>> if nic.name.startswith('hostdev'):
>> AttributeError: name
>> 2018-04-20 09:24:54,800+0800 INFO (Reactor thread)
>> [ProtocolDetector.AcceptorImpl] Accepted connection from ::1:48308
>> (protocoldetector:61)
>> 2018-04-20 09:24:54,810+0800 INFO (Reactor thread)
>> [ProtocolDetector.Detector] Detected protocol stomp from ::1:48308
>> (protocoldetector:125)
>> 2018-04-20 09:24:54,810+0800 INFO (Reactor thread) [Broker.StompAdapter]
>> Processing CONNECT request (stompreactor:103)
>> 2018-04-20 09:24:54,818+0800 INFO (JsonRpc (StompReactor))
>> [Broker.StompAdapter] Subscribe command received (stompreactor:132)
>> 2018-04-20 09:24:55,119+0800 INFO (jsonrpc/6) [api.host] START
>> getHardwareInfo() from=::1,48308 (api:46)
>>
>> ----- 原始邮件 -----
>> 发件人：Martin Sivak <msivak at redhat.com>
>> 收件人：dhy336 <dhy336 at sina.com>
>> 抄送人：users <users at ovirt.org>
>> 主题：Re: [ovirt-users] Hosted-engine can not switch
>> 日期：2018年04月19日 20点16分
>>
>>
>> We need more than just this small log snippet. Please check the vdsm
>> and libvirt logs as well.
>> Best regards
>> Martin Sivak
>> On Thu, Apr 19, 2018 at 2:05 PM, <dhy336 at sina.com> wrote:
>>> Hi，
>>> I deploy three node with hosted engine, I force shut down a node which
>>> Host-engine VM is run, But hosted engine VM in other nodes can not run.
>>>
>>> I find some error in /var/log/ovirt-hosted-engine-ha/agent.log
>>>
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:35,787::hosted_engine::1192::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state)
>>> Cleaning state for non-running VM
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:42,587::hosted_engine::1176::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state)
>>> Vdsm state for VM clean
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:42,589::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
>>> Starting vm using `/usr/sbin/hosted-engine --vm-start`
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,599::hosted_engine::1131::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
>>> stdout:
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,600::hosted_engine::1132::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
>>> stderr: Virtual machine does not exist: {'vmId':
>>> u'08bbd680-a8a7-4267-82e7-89f36e87e930'}
>>>
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,600::hosted_engine::1144::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
>>> Engine VM started on localhost
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,609::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
>>> Trying: notify time=1524139007.61 type=state_transition
>>> detail=EngineStart-EngineStarting hostname='hosted-engine2'
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,670::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
>>> Success, was notification of state_transition
>>> (EngineStart-EngineStarting)
>>> sent? sent
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:47,670::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
>>> Initializing VDSM
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:50,095::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> Connecting the storage
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:50,096::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server)
>>> Validating storage server
>>> MainThread::INFO::2018-04-19
>>>
>>>
>>> 19:56:52,449::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> Storage domain reported as valid and reconnect is not forced.
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users