[ovirt-users] hosted-engine unknow stale-data
Artem Tambovskiy
artem.tambovskiy at gmail.com
Thu Jan 18 18:37:37 UTC 2018
Hi,
Ok, i decided to remove second host from the cluster.
I reinstalled from webUI it with hosted-engine action UNDEPLOY, and removed
it from the cluster aftewards.
All VM's are fine hosted engine running ok,
But hosted-engine --vm-status still showing 2 hosts.
How I can clean the traces of second host in a correct way?
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : ovirt1.telia.ru
Host ID : 1
Engine status : {"health": "good", "vm": "up",
"detail": "up"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 1b1b6f6d
local_conf_timestamp : 545385
Host timestamp : 545385
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=545385 (Thu Jan 18 21:34:25 2018)
host-id=1
score=3400
vm_conf_refresh_time=545385 (Thu Jan 18 21:34:25 2018)
conf_on_shared_storage=True
maintenance=False
state=GlobalMaintenance
stopped=False
--== Host 2 status ==--
conf_on_shared_storage : True
Status up-to-date : False
Hostname : ovirt1.telia.ru
Host ID : 2
Engine status : unknown stale-data
Score : 0
stopped : True
Local maintenance : False
crc32 : c7037c03
local_conf_timestamp : 7530
Host timestamp : 7530
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=7530 (Fri Jan 12 16:10:12 2018)
host-id=2
score=0
vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
conf_on_shared_storage=True
maintenance=False
state=AgentStopped
stopped=True
!! Cluster is in GLOBAL MAINTENANCE mode !!
Thank you in advance!
Regards,
Artem
On Wed, Jan 17, 2018 at 6:47 PM, Artem Tambovskiy <
artem.tambovskiy at gmail.com> wrote:
> Hello,
>
> Any further suggestions on how to fix the issue and make HA setup working?
> Can the complete removal of second host (with complete removal ovirt
> configuration files and packages) from cluster and adding it again solve
> the issue? Or it might completly ruin the cluster?
>
> Regards,
> Artem
>
> 16 янв. 2018 г. 17:00 пользователь "Artem Tambovskiy" <
> artem.tambovskiy at gmail.com> написал:
>
> Hi Martin,
>>
>> Thanks for feedback.
>>
>> All hosts and hosted-engine running 4.1.8 release.
>> The strange thing : I can see that host ID is set to 1 on both hosts at
>> /etc/ovirt-hosted-engine/hosted-engine.conf file.
>> I have no idea how this happen, the only thing I have changed recently is
>> that I have changed mnt_options in order to add backup-volfile-servers
>> by using hosted-engine --set-shared-config command
>>
>> Both agent and broker are running on second host
>>
>> [root at ovirt2 ovirt-hosted-engine-ha]# ps -ef | grep ovirt-ha-
>> vdsm 42331 1 26 14:40 ? 00:31:35 /usr/bin/python
>> /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker --no-daemon
>> vdsm 42332 1 0 14:40 ? 00:00:16 /usr/bin/python
>> /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon
>>
>> but I saw some tracebacks during the broker start
>>
>> [root at ovirt2 ovirt-hosted-engine-ha]# systemctl status ovirt-ha-broker -l
>> ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability
>> Communications Broker
>> Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service;
>> enabled; vendor preset: disabled)
>> Active: active (running) since Tue 2018-01-16 14:40:15 MSK; 1h 58min
>> ago
>> Main PID: 42331 (ovirt-ha-broker)
>> CGroup: /system.slice/ovirt-ha-broker.service
>> └─42331 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker
>> --no-daemon
>>
>> Jan 16 14:40:15 ovirt2.telia.ru systemd[1]: Started oVirt Hosted Engine
>> High Availability Communications Broker.
>> Jan 16 14:40:15 ovirt2.telia.ru systemd[1]: Starting oVirt Hosted Engine
>> High Availability Communications Broker...
>> Jan 16 14:40:16 ovirt2.telia.ru ovirt-ha-broker[42331]: ovirt-ha-broker
>> ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error
>> handling request, data: 'set-storage-domain FilesystemBackend
>> dom_type=glusterfs sd_uuid=4a7f8717-9bb0-4d80-8016-498fa4b88162'
>> Traceback (most
>> recent call last):
>> File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 166, in handle
>> data)
>> File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>> line 299, in _dispatch
>>
>> .set_storage_domain(client, sd_type, **options)
>> File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>> line 66, in set_storage_domain
>>
>> self._backends[client].connect()
>> File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 462, in connect
>>
>> self._dom_type)
>> File
>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
>> line 107, in get_domain_path
>> " in
>> {1}".format(sd_uuid, parent))
>>
>> BackendFailureException: path to storage domain
>> 4a7f8717-9bb0-4d80-8016-498fa4b88162 not found in
>> /rhev/data-center/mnt/glusterSD
>>
>>
>>
>> I have tried to issue hosted-engine --connect-storage on second host
>> followed by agent & broker restart
>> But there is no any visible improvements.
>>
>> Regards,
>> Artem
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jan 16, 2018 at 4:18 PM, Martin Sivak <msivak at redhat.com> wrote:
>>
>>> Hi everybody,
>>>
>>> there are couple of things to check here.
>>>
>>> - what version of hosted engine agent is this? The logs look like
>>> coming from 4.1
>>> - what version of engine is used?
>>> - check the host ID in /etc/ovirt-hosted-engine/hosted-engine.conf on
>>> both hosts, the numbers must be different
>>> - it looks like the agent or broker on host 2 is not active (or there
>>> would be a report)
>>> - the second host does not see data from the first host (unknown
>>> stale-data), wait for a minute and check again, then check the storage
>>> connection
>>>
>>> And then the general troubleshooting:
>>>
>>> - put hosted engine in global maintenance mode (and check that it is
>>> visible from the other host using he --vm-status)
>>> - mount storage domain (hosted-engine --connect-storage)
>>> - check sanlock client status to see if proper lockspaces are present
>>>
>>> Best regards
>>>
>>> Martin Sivak
>>>
>>> On Tue, Jan 16, 2018 at 1:16 PM, Derek Atkins <derek at ihtfp.com> wrote:
>>> > Why are both hosts reporting as ovirt 1?
>>> > Look at the hostname fields to see what mean.
>>> >
>>> > -derek
>>> > Sent using my mobile device. Please excuse any typos.
>>> >
>>> > On January 16, 2018 7:11:09 AM Artem Tambovskiy <
>>> artem.tambovskiy at gmail.com>
>>> > wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> Yes, I followed exactly the same procedure while reinstalling the
>>> hosts
>>> >> (the only difference that I have SSH key configured instead of the
>>> >> password).
>>> >>
>>> >> Just reinstalled the second host one more time, after 20 min the host
>>> >> still haven't reached active score of 3400 (Hosted Engine HA:Not
>>> Active) and
>>> >> I still don't see crown icon for this host.
>>> >>
>>> >> hosted-engine --vm-status from ovirt1 host
>>> >>
>>> >> [root at ovirt1 ~]# hosted-engine --vm-status
>>> >>
>>> >>
>>> >> --== Host 1 status ==--
>>> >>
>>> >> conf_on_shared_storage : True
>>> >> Status up-to-date : True
>>> >> Hostname : ovirt1.telia.ru
>>> >> Host ID : 1
>>> >> Engine status : {"health": "good", "vm": "up",
>>> >> "detail": "up"}
>>> >> Score : 3400
>>> >> stopped : False
>>> >> Local maintenance : False
>>> >> crc32 : 3f94156a
>>> >> local_conf_timestamp : 349144
>>> >> Host timestamp : 349144
>>> >> Extra metadata (valid at timestamp):
>>> >> metadata_parse_version=1
>>> >> metadata_feature_version=1
>>> >> timestamp=349144 (Tue Jan 16 15:03:45 2018)
>>> >> host-id=1
>>> >> score=3400
>>> >> vm_conf_refresh_time=349144 (Tue Jan 16 15:03:45 2018)
>>> >> conf_on_shared_storage=True
>>> >> maintenance=False
>>> >> state=EngineUp
>>> >> stopped=False
>>> >>
>>> >>
>>> >> --== Host 2 status ==--
>>> >>
>>> >> conf_on_shared_storage : True
>>> >> Status up-to-date : False
>>> >> Hostname : ovirt1.telia.ru
>>> >> Host ID : 2
>>> >> Engine status : unknown stale-data
>>> >> Score : 0
>>> >> stopped : True
>>> >> Local maintenance : False
>>> >> crc32 : c7037c03
>>> >> local_conf_timestamp : 7530
>>> >> Host timestamp : 7530
>>> >> Extra metadata (valid at timestamp):
>>> >> metadata_parse_version=1
>>> >> metadata_feature_version=1
>>> >> timestamp=7530 (Fri Jan 12 16:10:12 2018)
>>> >> host-id=2
>>> >> score=0
>>> >> vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
>>> >> conf_on_shared_storage=True
>>> >> maintenance=False
>>> >> state=AgentStopped
>>> >> stopped=True
>>> >>
>>> >>
>>> >> hosted-engine --vm-status output from ovirt2 host
>>> >>
>>> >> [root at ovirt2 ovirt-hosted-engine-ha]# hosted-engine --vm-status
>>> >>
>>> >>
>>> >> --== Host 1 status ==--
>>> >>
>>> >> conf_on_shared_storage : True
>>> >> Status up-to-date : False
>>> >> Hostname : ovirt1.telia.ru
>>> >> Host ID : 1
>>> >> Engine status : unknown stale-data
>>> >> Score : 3400
>>> >> stopped : False
>>> >> Local maintenance : False
>>> >> crc32 : 6d3606f1
>>> >> local_conf_timestamp : 349264
>>> >> Host timestamp : 349264
>>> >> Extra metadata (valid at timestamp):
>>> >> metadata_parse_version=1
>>> >> metadata_feature_version=1
>>> >> timestamp=349264 (Tue Jan 16 15:05:45 2018)
>>> >> host-id=1
>>> >> score=3400
>>> >> vm_conf_refresh_time=349264 (Tue Jan 16 15:05:45 2018)
>>> >> conf_on_shared_storage=True
>>> >> maintenance=False
>>> >> state=EngineUp
>>> >> stopped=False
>>> >>
>>> >>
>>> >> --== Host 2 status ==--
>>> >>
>>> >> conf_on_shared_storage : True
>>> >> Status up-to-date : False
>>> >> Hostname : ovirt1.telia.ru
>>> >> Host ID : 2
>>> >> Engine status : unknown stale-data
>>> >> Score : 0
>>> >> stopped : True
>>> >> Local maintenance : False
>>> >> crc32 : c7037c03
>>> >> local_conf_timestamp : 7530
>>> >> Host timestamp : 7530
>>> >> Extra metadata (valid at timestamp):
>>> >> metadata_parse_version=1
>>> >> metadata_feature_version=1
>>> >> timestamp=7530 (Fri Jan 12 16:10:12 2018)
>>> >> host-id=2
>>> >> score=0
>>> >> vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
>>> >> conf_on_shared_storage=True
>>> >> maintenance=False
>>> >> state=AgentStopped
>>> >> stopped=True
>>> >>
>>> >>
>>> >> Also I saw some log messages in webGUI about time drift like
>>> >>
>>> >> "Host ovirt2.telia.ru has time-drift of 5305 seconds while maximum
>>> >> configured value is 300 seconds." that is a bit weird as haven't
>>> touched any
>>> >> time settings since I installed the cluster.
>>> >> both host have the same time and timezone (MSK) but hosted engine
>>> lives in
>>> >> UTC timezone. Is it mandatory to have everything in sync and in the
>>> same
>>> >> timezone?
>>> >>
>>> >> Regards,
>>> >> Artem
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Jan 16, 2018 at 2:20 PM, Kasturi Narra <knarra at redhat.com>
>>> wrote:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> I now see that your hosted engine is up and running. Can you
>>> let me
>>> >>> know how did you try reinstalling the host? Below is the procedure
>>> which is
>>> >>> used and hope you did not miss any step while reinstalling. If no,
>>> can you
>>> >>> try reinstalling again and see if that works ?
>>> >>>
>>> >>> 1) Move the host to maintenance
>>> >>> 2) click on reinstall
>>> >>> 3) provide the password
>>> >>> 4) uncheck 'automatically configure host firewall'
>>> >>> 5) click on 'Deploy' tab
>>> >>> 6) click Hosted Engine deployment as 'Deploy'
>>> >>>
>>> >>> And once the host installation is done, wait till the active score
>>> of the
>>> >>> host shows 3400 in the general tab then check hosted-engine
>>> --vm-status.
>>> >>>
>>> >>> Thanks
>>> >>> kasturi
>>> >>>
>>> >>> On Mon, Jan 15, 2018 at 4:57 PM, Artem Tambovskiy
>>> >>> <artem.tambovskiy at gmail.com> wrote:
>>> >>>>
>>> >>>> Hello,
>>> >>>>
>>> >>>> I have uploaded 2 archives with all relevant logs to shared hosting
>>> >>>> files from host 1 (which is currently running all VM's including
>>> >>>> hosted_engine) - https://yadi.sk/d/PttRoYV63RTvhK
>>> >>>> files from second host - https://yadi.sk/d/UBducEsV3RTvhc
>>> >>>>
>>> >>>> I have tried to restart both ovirt-ha-agent and ovirt-ha-broker but
>>> it
>>> >>>> gives no effect. I have also tried to shutdown hosted_engine VM,
>>> stop
>>> >>>> ovirt-ha-agent and ovirt-ha-broker services disconnect storage and
>>> connect
>>> >>>> it again - no effect as well.
>>> >>>> Also I tried to reinstall second host from WebGUI - this lead to the
>>> >>>> interesting situation - now hosted-engine --vm-status shows that
>>> both
>>> >>>> hosts have the same address.
>>> >>>>
>>> >>>> [root at ovirt1 ~]# hosted-engine --vm-status
>>> >>>>
>>> >>>> --== Host 1 status ==--
>>> >>>>
>>> >>>> conf_on_shared_storage : True
>>> >>>> Status up-to-date : True
>>> >>>> Hostname : ovirt1.telia.ru
>>> >>>> Host ID : 1
>>> >>>> Engine status : {"health": "good", "vm": "up",
>>> >>>> "detail": "up"}
>>> >>>> Score : 3400
>>> >>>> stopped : False
>>> >>>> Local maintenance : False
>>> >>>> crc32 : a7758085
>>> >>>> local_conf_timestamp : 259327
>>> >>>> Host timestamp : 259327
>>> >>>> Extra metadata (valid at timestamp):
>>> >>>> metadata_parse_version=1
>>> >>>> metadata_feature_version=1
>>> >>>> timestamp=259327 (Mon Jan 15 14:06:48 2018)
>>> >>>> host-id=1
>>> >>>> score=3400
>>> >>>> vm_conf_refresh_time=259327 (Mon Jan 15 14:06:48 2018)
>>> >>>> conf_on_shared_storage=True
>>> >>>> maintenance=False
>>> >>>> state=EngineUp
>>> >>>> stopped=False
>>> >>>>
>>> >>>>
>>> >>>> --== Host 2 status ==--
>>> >>>>
>>> >>>> conf_on_shared_storage : True
>>> >>>> Status up-to-date : False
>>> >>>> Hostname : ovirt1.telia.ru
>>> >>>> Host ID : 2
>>> >>>> Engine status : unknown stale-data
>>> >>>> Score : 0
>>> >>>> stopped : True
>>> >>>> Local maintenance : False
>>> >>>> crc32 : c7037c03
>>> >>>> local_conf_timestamp : 7530
>>> >>>> Host timestamp : 7530
>>> >>>> Extra metadata (valid at timestamp):
>>> >>>> metadata_parse_version=1
>>> >>>> metadata_feature_version=1
>>> >>>> timestamp=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>> host-id=2
>>> >>>> score=0
>>> >>>> vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>> conf_on_shared_storage=True
>>> >>>> maintenance=False
>>> >>>> state=AgentStopped
>>> >>>> stopped=True
>>> >>>>
>>> >>>> Gluster seems working fine. all gluster nodes showing connected
>>> state.
>>> >>>>
>>> >>>> Any advises on how to resolve this situation are highly appreciated!
>>> >>>>
>>> >>>> Regards,
>>> >>>> Artem
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jan 15, 2018 at 11:45 AM, Kasturi Narra <knarra at redhat.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hello Artem,
>>> >>>>>
>>> >>>>> Can you check if glusterd service is running on host1 and
>>> all
>>> >>>>> the peers are in connected state ? If yes, can you restart
>>> ovirt-ha-agent
>>> >>>>> and broker services and check if things are working fine ?
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>> kasturi
>>> >>>>>
>>> >>>>> On Sat, Jan 13, 2018 at 12:33 AM, Artem Tambovskiy
>>> >>>>> <artem.tambovskiy at gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> Explored logs on both hosts.
>>> >>>>>> broker.log shows no errors.
>>> >>>>>>
>>> >>>>>> agent.log looking not good:
>>> >>>>>>
>>> >>>>>> on host1 (which running hosted engine) :
>>> >>>>>>
>>> >>>>>> MainThread::ERROR::2018-01-12
>>> >>>>>> 21:51:03,883::agent::205::ovirt_hosted_engine_ha.agent.agent
>>> .Agent::(_run_agent)
>>> >>>>>> Traceback (most recent call last):
>>> >>>>>> File
>>> >>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/age
>>> nt/agent.py",
>>> >>>>>> line 191, in _run_agent
>>> >>>>>> return action(he)
>>> >>>>>> File
>>> >>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/age
>>> nt/agent.py",
>>> >>>>>> line 64, in action_proper
>>> >>>>>> return he.start_monitoring()
>>> >>>>>> File
>>> >>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/age
>>> nt/hosted_engine.py",
>>> >>>>>> line 411, in start_monitoring
>>> >>>>>> self._initialize_sanlock()
>>> >>>>>> File
>>> >>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/age
>>> nt/hosted_engine.py",
>>> >>>>>> line 749, in _initialize_sanlock
>>> >>>>>> "Failed to initialize sanlock, the number of errors has"
>>> >>>>>> SanlockInitializationError: Failed to initialize sanlock, the
>>> number
>>> >>>>>> of errors has exceeded the limit
>>> >>>>>>
>>> >>>>>> MainThread::ERROR::2018-01-12
>>> >>>>>> 21:51:03,884::agent::206::ovirt_hosted_engine_ha.agent.agent
>>> .Agent::(_run_agent)
>>> >>>>>> Trying to restart agent
>>> >>>>>> MainThread::WARNING::2018-01-12
>>> >>>>>> 21:51:08,889::agent::209::ovirt_hosted_engine_ha.agent.agent
>>> .Agent::(_run_agent)
>>> >>>>>> Restarting agent, attempt '1'
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:08,919::hosted_engine::242::ovirt_hosted_engine_ha.age
>>> nthosted_engine.HostedEngine::(_get_hostname)
>>> >>>>>> Found certificate common name: ovirt1.telia.ru
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:08,921::hosted_engine::604::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_vdsm)
>>> >>>>>> Initializing VDSM
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:11,398::hosted_engine::630::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Connecting the storage
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:11,399::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(validate_storage_server)
>>> >>>>>> Validating storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:13,725::storage_server::239::ovirt_hosted_engine_ha.li
>>> bstorage_server.StorageServer::(connect_storage_server)
>>> >>>>>> Connecting storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:18,390::storage_server::246::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> >>>>>> Connecting storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:18,423::storage_server::253::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> >>>>>> Refreshing the storage domain
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:18,689::hosted_engine::663::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Preparing images
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:18,690::image::126::ovirt_hosted_engine_ha.lib.image.I
>>> mage::(prepare_images)
>>> >>>>>> Preparing images
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,895::hosted_engine::666::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Refreshing vm.conf
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,895::config::493::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_vm_conf)
>>> >>>>>> Reloading vm.conf from the shared storage domain
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,896::config::416::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Trying to get a fresher copy of vm configuration from the
>>> OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,896::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> Extracting Engine VM OVF from the OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,897::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> OVF_STORE volume path:
>>> >>>>>> /var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/5
>>> cabd8e1-5f4b-469e-becc-227469e03f5c/8048cbd7-77e2-4805-9af4-d109fa36dfcf
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,915::config::435::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Found an OVF for HE VM, trying to convert
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,918::config::440::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Got vm.conf from OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,919::hosted_engine::509::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_broker)
>>> >>>>>> Initializing ha-broker connection
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,919::brokerlink::130::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Starting monitor ping, options {'addr': '80.239.162.97'}
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,922::brokerlink::141::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Success, id 140547104457680
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,922::brokerlink::130::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Starting monitor mgmt-bridge, options {'use_ssl': 'true',
>>> 'bridge_name':
>>> >>>>>> 'ovirtmgmt', 'address': '0'}
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,936::brokerlink::141::ovirt_hosted_engine_ha.lib.br
>>> okerlinkBrokerLink::(start_monitor)
>>> >>>>>> Success, id 140547104458064
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,936::brokerlink::130::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Starting monitor mem-free, options {'use_ssl': 'true', 'address':
>>> '0'}
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,938::brokerlink::141::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Success, id 140547104458448
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,939::brokerlink::130::ovirt_hosted_engine_ha.lib.br
>>> okerlinkBrokerLink::(start_monitor)
>>> >>>>>> Starting monitor cpu-load-no-engine, options {'use_ssl': 'true',
>>> 'vm_uuid':
>>> >>>>>> 'b366e466-b0ea-4a09-866b-d0248d7523a6', 'address': '0'}
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,940::brokerlink::141::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Success, id 140547104457552
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,941::brokerlink::130::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Starting monitor engine-health, options {'use_ssl': 'true',
>>> 'vm_uuid':
>>> >>>>>> 'b366e466-b0ea-4a09-866b-d0248d7523a6', 'address': '0'}
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:21,942::brokerlink::141::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(start_monitor)
>>> >>>>>> Success, id 140547104459792
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:26,951::brokerlink::179::ovirt_hosted_engine_ha.lib.br
>>> okerlink.BrokerLink::(set_storage_domain)
>>> >>>>>> Success, id 140546772847056
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:26,952::hosted_engine::601::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_broker)
>>> >>>>>> Broker initialized, all submonitors started
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:51:27,049::hosted_engine::704::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_sanlock)
>>> >>>>>> Ensuring lease for lockspace hosted-engine, host id 1 is acquired
>>> (file:
>>> >>>>>> /var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/0
>>> 93faa75-5e33-4559-84fa-1f1f8d48153b/911c7637-b49d-463e-b186-
>>> 23b404e50769)
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:53:48,067::hosted_engine::745::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_sanlock)
>>> >>>>>> Failed to acquire the lock. Waiting '5's before the next attempt
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:56:14,088::hosted_engine::745::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_sanlock)
>>> >>>>>> Failed to acquire the lock. Waiting '5's before the next attempt
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 21:58:40,111::hosted_engine::745::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_sanlock)
>>> >>>>>> Failed to acquire the lock. Waiting '5's before the next attempt
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:06,133::hosted_engine::745::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_sanlock)
>>> >>>>>> Failed to acquire the lock. Waiting '5's before the next attempt
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> agent.log from second host
>>> >>>>>>
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:37,241::hosted_engine::630::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Connecting the storage
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:37,242::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(validate_storage_server)
>>> >>>>>> Validating storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:39,540::hosted_engine::639::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Storage domain reported as valid and reconnect is not forced.
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:41,939::hosted_engine::453::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> >>>>>> Current state EngineUnexpectedlyDown (score: 0)
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,150::config::493::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_vm_conf)
>>> >>>>>> Reloading vm.conf from the shared storage domain
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,150::config::416::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Trying to get a fresher copy of vm configuration from the
>>> OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,151::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> Extracting Engine VM OVF from the OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,153::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> OVF_STORE volume path:
>>> >>>>>> /var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/5
>>> cabd8e1-5f4b-469e-becc-227469e03f5c/8048cbd7-77e2-4805-9af4-d109fa36dfcf
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,174::config::435::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Found an OVF for HE VM, trying to convert
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,179::config::440::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Got vm.conf from OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:52,189::hosted_engine::604::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_vdsm)
>>> >>>>>> Initializing VDSM
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:54,586::hosted_engine::630::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Connecting the storage
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:54,587::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(validate_storage_server)
>>> >>>>>> Validating storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:56,903::hosted_engine::639::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Storage domain reported as valid and reconnect is not forced.
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:59,299::states::682::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine::(score)
>>> >>>>>> Score is 0 due to unexpected vm shutdown at Fri Jan 12 21:57:48
>>> 2018
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:01:59,299::hosted_engine::453::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> >>>>>> Current state EngineUnexpectedlyDown (score: 0)
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,659::config::493::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_vm_conf)
>>> >>>>>> Reloading vm.conf from the shared storage domain
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,659::config::416::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Trying to get a fresher copy of vm configuration from the
>>> OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,660::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> Extracting Engine VM OVF from the OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,663::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> OVF_STORE volume path:
>>> >>>>>> /var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/5
>>> cabd8e1-5f4b-469e-becc-227469e03f5c/8048cbd7-77e2-4805-9af4-d109fa36dfcf
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,683::config::435::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Found an OVF for HE VM, trying to convert
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,688::config::440::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Got vm.conf from OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:09,698::hosted_engine::604::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_vdsm)
>>> >>>>>> Initializing VDSM
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:12,112::hosted_engine::630::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Connecting the storage
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:12,113::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(validate_storage_server)
>>> >>>>>> Validating storage server
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:14,444::hosted_engine::639::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Storage domain reported as valid and reconnect is not forced.
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:16,859::states::682::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine::(score)
>>> >>>>>> Score is 0 due to unexpected vm shutdown at Fri Jan 12 21:57:47
>>> 2018
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:16,859::hosted_engine::453::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> >>>>>> Current state EngineUnexpectedlyDown (score: 0)
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,100::config::493::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_vm_conf)
>>> >>>>>> Reloading vm.conf from the shared storage domain
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,100::config::416::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Trying to get a fresher copy of vm configuration from the
>>> OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,101::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> Extracting Engine VM OVF from the OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,103::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> >>>>>> OVF_STORE volume path:
>>> >>>>>> /var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/5
>>> cabd8e1-5f4b-469e-becc-227469e03f5c/8048cbd7-77e2-4805-9af4-d109fa36dfcf
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,125::config::435::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Found an OVF for HE VM, trying to convert
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,129::config::440::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>> >>>>>> Got vm.conf from OVF_STORE
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,130::states::667::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine::(consume)
>>> >>>>>> Engine down, local host does not have best score
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:27,139::hosted_engine::604::ovirt_hosted_engine_ha.age
>>> nt.hosted_engineHostedEngine::(_initialize_vdsm)
>>> >>>>>> Initializing VDSM
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:29,584::hosted_engine::630::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> >>>>>> Connecting the storage
>>> >>>>>> MainThread::INFO::2018-01-12
>>> >>>>>> 22:02:29,586::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(validate_storage_server)
>>> >>>>>> Validating storage server
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Any suggestions how to resolve this .
>>> >>>>>>
>>> >>>>>> regards,
>>> >>>>>> Artem
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Fri, Jan 12, 2018 at 7:08 PM, Artem Tambovskiy
>>> >>>>>> <artem.tambovskiy at gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Trying to fix one thing I broke another :(
>>> >>>>>>>
>>> >>>>>>> I fixed mnt_options for hosted engine storage domain and
>>> installed
>>> >>>>>>> latest security patches to my hosts and hosted engine. All VM's
>>> up and
>>> >>>>>>> running, but hosted_engine --vm-status reports about issues:
>>> >>>>>>>
>>> >>>>>>> [root at ovirt1 ~]# hosted-engine --vm-status
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --== Host 1 status ==--
>>> >>>>>>>
>>> >>>>>>> conf_on_shared_storage : True
>>> >>>>>>> Status up-to-date : False
>>> >>>>>>> Hostname : ovirt2
>>> >>>>>>> Host ID : 1
>>> >>>>>>> Engine status : unknown stale-data
>>> >>>>>>> Score : 0
>>> >>>>>>> stopped : False
>>> >>>>>>> Local maintenance : False
>>> >>>>>>> crc32 : 193164b8
>>> >>>>>>> local_conf_timestamp : 8350
>>> >>>>>>> Host timestamp : 8350
>>> >>>>>>> Extra metadata (valid at timestamp):
>>> >>>>>>> metadata_parse_version=1
>>> >>>>>>> metadata_feature_version=1
>>> >>>>>>> timestamp=8350 (Fri Jan 12 19:03:54 2018)
>>> >>>>>>> host-id=1
>>> >>>>>>> score=0
>>> >>>>>>> vm_conf_refresh_time=8350 (Fri Jan 12 19:03:54 2018)
>>> >>>>>>> conf_on_shared_storage=True
>>> >>>>>>> maintenance=False
>>> >>>>>>> state=EngineUnexpectedlyDown
>>> >>>>>>> stopped=False
>>> >>>>>>> timeout=Thu Jan 1 05:24:43 1970
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --== Host 2 status ==--
>>> >>>>>>>
>>> >>>>>>> conf_on_shared_storage : True
>>> >>>>>>> Status up-to-date : False
>>> >>>>>>> Hostname : ovirt1.telia.ru
>>> >>>>>>> Host ID : 2
>>> >>>>>>> Engine status : unknown stale-data
>>> >>>>>>> Score : 0
>>> >>>>>>> stopped : True
>>> >>>>>>> Local maintenance : False
>>> >>>>>>> crc32 : c7037c03
>>> >>>>>>> local_conf_timestamp : 7530
>>> >>>>>>> Host timestamp : 7530
>>> >>>>>>> Extra metadata (valid at timestamp):
>>> >>>>>>> metadata_parse_version=1
>>> >>>>>>> metadata_feature_version=1
>>> >>>>>>> timestamp=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>>>>> host-id=2
>>> >>>>>>> score=0
>>> >>>>>>> vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>>>>> conf_on_shared_storage=True
>>> >>>>>>> maintenance=False
>>> >>>>>>> state=AgentStopped
>>> >>>>>>> stopped=True
>>> >>>>>>> [root at ovirt1 ~]#
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> from second host situation looks a bit different:
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> [root at ovirt2 ~]# hosted-engine --vm-status
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --== Host 1 status ==--
>>> >>>>>>>
>>> >>>>>>> conf_on_shared_storage : True
>>> >>>>>>> Status up-to-date : True
>>> >>>>>>> Hostname : ovirt2
>>> >>>>>>> Host ID : 1
>>> >>>>>>> Engine status : {"reason": "vm not running
>>> on
>>> >>>>>>> this host", "health": "bad", "vm": "down", "detail": "unknown"}
>>> >>>>>>> Score : 0
>>> >>>>>>> stopped : False
>>> >>>>>>> Local maintenance : False
>>> >>>>>>> crc32 : 78eabdb6
>>> >>>>>>> local_conf_timestamp : 8403
>>> >>>>>>> Host timestamp : 8402
>>> >>>>>>> Extra metadata (valid at timestamp):
>>> >>>>>>> metadata_parse_version=1
>>> >>>>>>> metadata_feature_version=1
>>> >>>>>>> timestamp=8402 (Fri Jan 12 19:04:47 2018)
>>> >>>>>>> host-id=1
>>> >>>>>>> score=0
>>> >>>>>>> vm_conf_refresh_time=8403 (Fri Jan 12 19:04:47 2018)
>>> >>>>>>> conf_on_shared_storage=True
>>> >>>>>>> maintenance=False
>>> >>>>>>> state=EngineUnexpectedlyDown
>>> >>>>>>> stopped=False
>>> >>>>>>> timeout=Thu Jan 1 05:24:43 1970
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --== Host 2 status ==--
>>> >>>>>>>
>>> >>>>>>> conf_on_shared_storage : True
>>> >>>>>>> Status up-to-date : False
>>> >>>>>>> Hostname : ovirt1.telia.ru
>>> >>>>>>> Host ID : 2
>>> >>>>>>> Engine status : unknown stale-data
>>> >>>>>>> Score : 0
>>> >>>>>>> stopped : True
>>> >>>>>>> Local maintenance : False
>>> >>>>>>> crc32 : c7037c03
>>> >>>>>>> local_conf_timestamp : 7530
>>> >>>>>>> Host timestamp : 7530
>>> >>>>>>> Extra metadata (valid at timestamp):
>>> >>>>>>> metadata_parse_version=1
>>> >>>>>>> metadata_feature_version=1
>>> >>>>>>> timestamp=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>>>>> host-id=2
>>> >>>>>>> score=0
>>> >>>>>>> vm_conf_refresh_time=7530 (Fri Jan 12 16:10:12 2018)
>>> >>>>>>> conf_on_shared_storage=True
>>> >>>>>>> maintenance=False
>>> >>>>>>> state=AgentStopped
>>> >>>>>>> stopped=True
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> WebGUI shows that engine running on host ovirt1.
>>> >>>>>>> Gluster looks fine
>>> >>>>>>> [root at ovirt1 ~]# gluster volume status engine
>>> >>>>>>> Status of volume: engine
>>> >>>>>>> Gluster process TCP Port RDMA Port
>>> >>>>>>> Online Pid
>>> >>>>>>>
>>> >>>>>>> ------------------------------------------------------------
>>> ------------------
>>> >>>>>>> Brick ovirt1.teliaru:/oVirt/engine 49169 0 Y
>>> >>>>>>> 3244
>>> >>>>>>> Brick ovirt2.telia.ru:/oVirt/engine 49179 0
>>> Y
>>> >>>>>>> 20372
>>> >>>>>>> Brick ovirt3.telia.ru:/oVirt/engine 49206 0
>>> Y
>>> >>>>>>> 16609
>>> >>>>>>> Self-heal Daemon on localhost N/A N/A
>>> Y
>>> >>>>>>> 117868
>>> >>>>>>> Self-heal Daemon on ovirt2.telia.ru N/A N/A
>>> Y
>>> >>>>>>> 20521
>>> >>>>>>> Self-heal Daemon on ovirt3 N/A N/A
>>> Y
>>> >>>>>>> 25093
>>> >>>>>>>
>>> >>>>>>> Task Status of Volume engine
>>> >>>>>>>
>>> >>>>>>> ------------------------------------------------------------
>>> ------------------
>>> >>>>>>> There are no active volume tasks
>>> >>>>>>>
>>> >>>>>>> How to resolve this issue?
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> _______________________________________________
>>> >>>>>>> Users mailing list
>>> >>>>>>> Users at ovirt.org
>>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> _______________________________________________
>>> >>>>>> Users mailing list
>>> >>>>>> Users at ovirt.org
>>> >>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> >> _______________________________________________
>>> >> Users mailing list
>>> >> Users at ovirt.org
>>> >> http://lists.ovirt.org/mailman/listinfo/users
>>> >>
>>> >
>>> > _______________________________________________
>>> > Users mailing list
>>> > Users at ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180118/bebc7038/attachment.html>
More information about the Users
mailing list