5:23 p.m.
Hello,
I currently restarted ovirt-ha-agent and I don't see any
"startMonitoringDomain" in vdsm.log (see attachement).
I attach vdsm.log and agent.log from restart of agent to timeout. (Agent
sleeps and continues and exits in about 30 Minutes)
In agent-log is states every 7 seconds:
MainThread::INFO::2017-02-03
15:10:21,915::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
MainThread::INFO::2017-02-03
15:10:29,058::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
MainThread::INFO::2017-02-03
15:10:36,206::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
MainThread::INFO::2017-02-03
15:10:43,346::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
VDSM domain monitor status: PENDING
until
uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96, host_id=3): timeout during
domain acquisition
MainThread::WARNING::2017-02-03
15:11:19,111::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Error while monitoring engine: Failed to start monitoring domain
(sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96, host_id=3): timeout
during domain acquisition
MainThread::WARNING::2017-02-03
15:11:19,111::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Unexpected error
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 443, in start_monitoring
self._initialize_domain_monitor()
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 816, in _initialize_domain_monitor
raise Exception(msg)
Exception: Failed to start monitoring domain
(sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96, host_id=3): timeout
during domain acquisition
MainThread::INFO::2017-02-03
15:11:19,112::hosted_engine::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Sleeping 60 seconds
Am 03.02.2017 um 13:39 schrieb Simone Tiraboschi:
I see there an ERROR on stopMonitoringDomain but I cannot see the
correspondent startMonitoringDomain; could you please look for it?
On Fri, Feb 3, 2017 at 1:16 PM, Ralf Schenk <rs(a)databay.de
<mailto:rs@databay.de>> wrote:
Hello,
attached is my vdsm.log from the host with hosted-engine-ha around
the time-frame of agent timeout that is not working anymore for
engine (it works in Ovirt and is active). It simply isn't working
for engine-ha anymore after Update.
At 2017-02-02 19:25:34,248 you'll find an error corresponoding to
agent timeout error.
Bye
Am 03.02.2017 um 11:28 schrieb Simone Tiraboschi:
>
> 3. Three of my hosts have the hosted engine deployed for
> ha. First all three where marked by a crown (running was
> gold and others where silver). After upgrading the 3 Host
> deployed hosted engine ha is not active anymore.
>
> I can't get this host back with working
> ovirt-ha-agent/broker. I already rebooted, manually
> restarted the services but It isn't able to get cluster
> state according to
> "hosted-engine --vm-status". The other hosts state the
> host status as "unknown stale-data"
>
> I already shut down all agents on all hosts and issued a
> "hosted-engine --reinitialize-lockspace" but that didn't
> help.
>
> Agents stops working after a timeout-error according to log:
>
> MainThread::INFO::2017-02-02
>
19:24:52,040::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:24:59,185::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:25:06,333::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:25:13,554::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:25:20,710::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:25:27,865::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::ERROR::2017-02-02
>
19:25:27,866::hosted_engine::815::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor)
> Failed to start monitoring domain
> (sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96,
> host_id=3): timeout during domain acquisition
> MainThread::WARNING::2017-02-02
>
19:25:27,866::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Error while monitoring engine: Failed to start monitoring
> domain (sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96,
> host_id=3): timeout during domain acquisition
> MainThread::WARNING::2017-02-02
>
19:25:27,866::hosted_engine::472::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Unexpected error
> Traceback (most recent call last):
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 443, in start_monitoring
> self._initialize_domain_monitor()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 816, in _initialize_domain_monitor
> raise Exception(msg)
> Exception: Failed to start monitoring domain
> (sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96,
> host_id=3): timeout during domain acquisition
> MainThread::ERROR::2017-02-02
>
19:25:27,866::hosted_engine::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Shutting down the agent because of 3 failures in a row!
> MainThread::INFO::2017-02-02
>
19:25:32,087::hosted_engine::841::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status)
> VDSM domain monitor status: PENDING
> MainThread::INFO::2017-02-02
>
19:25:34,250::hosted_engine::769::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor)
> Failed to stop monitoring domain
> (sd_uuid=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96): Storage
> domain is member of pool:
> u'domain=7c8deaa8-be02-4aaf-b9b4-ddc8da99ad96'
> MainThread::INFO::2017-02-02
>
19:25:34,254::agent::143::ovirt_hosted_engine_ha.agent.agent.Agent::(run)
> Agent shutting down
>
> Simone, Martin, can you please follow up on this?
>
>
> Ralph, could you please attach vdsm logs from on of your hosts
> for the relevant time frame?
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70 <tel:+49%202405%20408370>
fax +49 (0) 24 05 / 40 83 759 <tel:+49%202405%204083759>
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari,
Dipl.-Kfm. Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
