Hi,
> Host id is not set
This is an internal error report that should normally not happen. It
means the ovirt-ha-agent asked for a storage operation before it
registered itself with the broker. If this happens seldom then it looks
like a race condition.
I'm seeing this events also on my setup randomly and seldom.
versions:
ovirt-release42-4.2.7.1-1.el7.noarch
ovirt-hosted-engine-ha-2.2.18-1.el7.noarch
ovirt-host-4.2.3-1.el7.x86_64
...
I have emails from 15 nov, 21 nov, 01 dec, 05 dec...
Last event at 09 dec 16:35:
-----------------
MainThread::INFO::2018-12-09
16:34:05,223::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
Current state EngineUp (score: 3400)
MainThread::INFO::2018-12-09
16:34:15,360::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
Current state EngineUp (score: 3400)
....... ALL OK HERE UNTIL......
MainThread::ERROR::2018-12-09
16:34:25,373::hosted_engine::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Unhandled monitoring loop exception
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 428, in start_monitoring
self._monitoring_loop()
.....
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 135, in get_stats_from_storage
result = self._proxy.get_stats()
.....
error: [Errno 2] No such file or directory
.....
-------------------------
MainThread::ERROR::2018-12-09
16:34:25,380::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Trying to restart agent
.....
MainThread::INFO::2018-12-09
16:34:46,353::brokerlink::77::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor storage-domain, options {'sd_uuid':
'0571ac7b-a28e-4e20-9cd8-4803e40ec602'}
MainThread::INFO::2018-12-09
16:34:46,354::brokerlink::85::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id storage-domain
.....
MainThread::INFO::2018-12-09
16:35:25,062::brokerlink::68::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition (EngineStarting-EngineUp)
sent? sent
.....
MainThread::INFO::2018-12-09
16:35:31,209::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
Current state EngineUp (score: 3400)
-----------------------------
Are anybody open this bug report ? I do not have account in RH bugzilla..
(Accidentally this (9 dec) log file is rotated while i wrote this email...)
On Fri, Dec 14, 2018 at 12:35 PM fsoyer <fsoyer(a)systea.fr
<mailto:fsoyer@systea.fr>> wrote:
In borker.log I found this, just before 05:59am:
Thread-3::INFO::2018-12-13
05:58:45,634::mem_free::51::mem_free.MemFree::(action) memFree:
82101
Thread-1::INFO::2018-12-13
05:58:46,322::ping::60::ping.Ping::(action) Successfully pinged
10.0.1.254
Thread-5::INFO::2018-12-13
05:58:46,611::engine_health::241::engine_health.EngineHealth::(_result_from_stats)
VM is up on this host with healthy engine
Thread-2::INFO::2018-12-13
05:58:49,144::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
StatusStorageThread::ERROR::2018-12-13
05:58:54,935::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed to update state.
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 82, in run
if (self._status_broker._inquire_whiteboard_lock() or
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 190, in _inquire_whiteboard_lock
self.host_id, self._lease_file)
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 128, in host_id
raise ex.HostIdNotLockedError("Host id is not set")
HostIdNotLockedError: Host id is not set
StatusStorageThread::ERROR::2018-12-13
05:58:54,937::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_restart)
Trying to restart the broker
"Host is not set" ???
--
Regards,
*Frank*
Le Vendredi, Décembre 14, 2018 12:27 CET, Martin Sivak
<msivak(a)redhat.com <mailto:msivak@redhat.com>> a écrit:
> Hi,
> check the broker.log as well. The connect is used to talk to
> ovirt-ha-broker service socket.
> Best regards
> Martin Sivak
> On Fri, Dec 14, 2018 at 12:20 PM fsoyer <fsoyer(a)systea.fr
> <mailto:fsoyer@systea.fr>> wrote:
>
> I think I have it in agent.log. What can be this "file not
> found" ?
>
> MainThread::ERROR::2018-12-13
>
05:59:03,909::hosted_engine::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Unhandled monitoring loop exception
> Traceback (most recent call last):
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 428, in start_monitoring
> self._monitoring_loop()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 447, in _monitoring_loop
> for old_state, state, delay in self.fsm:
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
> line 127, in next
> new_data = self.refresh(self._state.data)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py",
> line 81, in refresh
> stats.update(self.hosted_engine.collect_stats())
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 736, in collect_stats
> all_stats = self._broker.get_stats_from_storage()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 135, in get_stats_from_storage
> result = self._proxy.get_stats()
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in
> __call__
> return self.__send(self.__name, args)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in
> __request
> verbose=self.__verbose
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in
> request
> return self.single_request(host, handler,
> request_body, verbose)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in
> single_request
> self.send_content(h, request_body)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
> send_content
> connection.endheaders(request_body)
> File "/usr/lib64/python2.7/httplib.py", line 1037, in
> endheaders
> self._send_output(message_body)
> File "/usr/lib64/python2.7/httplib.py", line 881, in
> _send_output
> self.send(msg)
> File "/usr/lib64/python2.7/httplib.py", line 843, in send
> self.connect()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
> line 52, in connect
> self.sock.connect(base64.b16decode(self.host))
> File "/usr/lib64/python2.7/socket.py", line 224, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 2] No such file or directory
> MainThread::ERROR::2018-12-13
>
05:59:04,043::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Traceback (most recent call last):
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent
> return action(he)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper
> return he.start_monitoring()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 435, in start_monitoring
> self.publish(stopped)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 337, in publish
> self._push_to_storage(blocks)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 707, in _push_to_storage
> self._broker.put_stats_on_storage(self.host_id, blocks)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 105, in put_stats_on_storage
> self._proxy.put_stats(host_id, xmlrpclib.Binary(data))
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in
> __call__
> return self.__send(self.__name, args)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in
> __request
> verbose=self.__verbose
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in
> request
> return self.single_request(host, handler,
> request_body, verbose)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in
> single_request
> self.send_content(h, request_body)
> File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
> send_content
> connection.endheaders(request_body)
> File "/usr/lib64/python2.7/httplib.py", line 1037, in
> endheaders
> self._send_output(message_body)
> File "/usr/lib64/python2.7/httplib.py", line 881, in
> _send_output
> self.send(msg)
> File "/usr/lib64/python2.7/httplib.py", line 843, in send
> self.connect()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
> line 52, in connect
> self.sock.connect(base64.b16decode(self.host))
> File "/usr/lib64/python2.7/socket.py", line 224, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 2] No such file or directory
>
> MainThread::ERROR::2018-12-13
>
05:59:04,043::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Trying to restart agent
> MainThread::INFO::2018-12-13
> 05:59:04,044::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
> Agent shutting down
> MainThread::INFO::2018-12-13
> 05:59:14,923::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
> ovirt-hosted-engine-ha agent 2.2.16 started
>
>
>
> --
>
> Cordialement,
>
> *Frank Soyer *
> Mob. 06 72 28 38 53 - Fix. 05 49 50 52 34
>
> <
http://www.systea.net>
>
> *Systea IG*
> Administration systèmes, réseaux et bases de données
>
www.systea.net <
http://www.systea.net>
> Membre du réseau Les Professionnels du Numérique
> <
http://www.spn.asso.fr>
>
> <
http://www.kogite.fr>
>
> *KoGite*
> Hébergement de proximité
>
www.kogite.fr <
http://www.kogite.fr>
>
>
>
> Le Vendredi, Décembre 14, 2018 12:11 CET, Martin Sivak
> <msivak(a)redhat.com <mailto:msivak@redhat.com>> a écrit:
>> Hi,
>>
>> no StartState is not common, it is only ever entered when the
>> agent
>> boots up. So something restarted or killed the agent process.
>> Check
>> the agent log in /var/log/ovirt-hosted-engine-ha for errors.
>>
>> Best regards
>>
>> Martin Sivak
>>
>> On Fri, Dec 14, 2018 at 12:05 PM fsoyer <fsoyer(a)systea.fr
>> <mailto:fsoyer@systea.fr>> wrote:
>> >
>> > Hi Martin,
>> > my problem is that nobody restarted the agent. Do you mean
>> that this is not a normal behavior ? Is it possible that it
>> restarts itself ?
>> >
>> > Thanks
>> > --
>> >
>> > Regards,
>> >
>> > Frank
>> >
>> >
>> >
>> > Le Jeudi, Décembre 13, 2018 15:25 CET, Martin Sivak
>> <msivak(a)redhat.com <mailto:msivak@redhat.com>> a écrit:
>> >
>> >
>> > Hi,
>> >
>> > those are state change notifications from the hosted engine
>> agent. It
>> > basically means somebody restarted the ha-agent process and
>> it found
>> > out the VM is still running fine and returned to the proper
>> state.
>> >
>> > Configuring it is possible using the broker.conf file in
>> > /etc/ovirt-hosted-engine-ha (look for the notification
>> section) or the
>> > hosted-engine tool (search --help for set config) depending
>> on the
>> > version of hosted engine you are using.
>> >
>> > Best regards
>> >
>> > --
>> > Martin Sivak
>> >
>> >
>> > On Thu, Dec 13, 2018 at 3:10 PM fsoyer <fsoyer(a)systea.fr
>> <mailto:fsoyer@systea.fr>> wrote:
>> > >
>> > > Hi,
>> > > I don't find revelant answer about this. Sorry il this
>> was already asked.
>> > > I receive randomly (one or two tims a week, differents
>> hours) 3 mails with this subjects :
>> > > first : ovirt-hosted-engine state transition
>> StartState-ReinitializeFSM
>> > > second : ovirt-hosted-engine state transition
>> ReinitializeFSM-EngineStarting
>> > > third : ovirt-hosted-engine state transition
>> EngineStarting-EngineUp
>> > > all at exactly the same time. The "events" in GUI
doesn't
>> indicate anything about this. No impact on engine or VMs.
>> > > So I wonder what this messages means ? And, if case of
>> just "info" messages, is there a way to disable them ?
>> > >
>> > > Thanks.
>> > > --
>> > >
>> > > Reagrds,
>> > >
>> > > Frank
>> > >
>> > > _______________________________________________
>> > > Users mailing list -- users(a)ovirt.org
>> <mailto:users@ovirt.org>
>> > > To unsubscribe send an email to users-leave(a)ovirt.org
>> <mailto:users-leave@ovirt.org>
>> > > Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
>> > > oVirt Code of Conduct:
>>
https://www.ovirt.org/community/about/community-guidelines/
>> > > List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CVEHTWILWDE...
>> >
>> >
>> >
>> >
>
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement:
https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5VJVLDA5J2C...