2:20 a.m.
Good day for all.
I have some issues with Ovirt 4.2.6. But now the main this of it:
I have two Centos 7 Nodes with same config and last Ovirt 4.2.6 with Hostedengine with
disk on NFS storage.
Also some of virtual machines working good.
But, when HostedEngine running on one node (srv02.local) everything is fine.
After migrating to another node (srv00.local), i see that agent cannot to check
livelinness of HostedEngine. After few minutes HostedEngine going to reboot and after some
time i see some situation. After migration to another node (srv00.local) all looks OK.
hosted-engine --vm-status commang when HosterEngine on srv00 node:
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : srv02.local
Host ID : 1
Engine status : {"reason": "vm not running on this
host", "health": "bad", "vm":
"down_unexpected", "detail": "unknown"}
Score : 0
stopped : False
Local maintenance : False
crc32 : ecc7ad2d
local_conf_timestamp : 78328
Host timestamp : 78328
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=78328 (Tue Sep 18 12:44:18 2018)
host-id=1
score=0
vm_conf_refresh_time=78328 (Tue Sep 18 12:44:18 2018)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Fri Jan 2 03:49:58 1970
--== Host 2 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : srv00.local
Host ID : 2
Engine status : {"reason": "failed liveliness
check", "health": "bad", "vm": "up",
"detail": "Up"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 1d62b106
local_conf_timestamp : 326288
Host timestamp : 326288
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=326288 (Tue Sep 18 12:44:21 2018)
host-id=2
score=3400
vm_conf_refresh_time=326288 (Tue Sep 18 12:44:21 2018)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
Log agent.log from srv00.local:
MainThread::INFO::2018-09-18
12:40:51,749::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:40:52,052::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:01,066::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:01,374::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Global metadata: {'maintenance': False}
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Host srv02.local.pioner.kz (id 1):
{'conf_on_shared_storage': True, 'extra': 'meta
data_parse_version=1\nmetadata_feature_version=1\ntimestamp=78128 (Tue Sep 18 12:40:58
2018)\nhost-id=1\ns
core=0\nvm_conf_refresh_time=78128 (Tue Sep 18 12:40:58
2018)\nconf_on_shared_storage=True\nmaintenance=Fa
lse\nstate=EngineUnexpectedlyDown\nstopped=False\ntimeout=Fri Jan 2 03:49:58 1970\n',
'hostname': 'srv02.
local.pioner.kz', 'alive': True, 'host-id': 1,
'engine-status': {'reason': 'vm not running on this host',
'health': 'bad', 'vm': 'down_unexpected',
'detail': 'unknown'}, 'score': 0, 'stopped': False,
'maintenance
': False, 'crc32': 'e18e3f22', 'local_conf_timestamp': 78128,
'host-ts': 78128}
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Local (id 2): {'engine-health': {'reason':
'failed liveliness check', 'health': 'b
ad', 'vm': 'up', 'detail': 'Up'}, 'bridge':
True, 'mem-free': 12763.0, 'maintenance': False, 'cpu-load': 0
.0364, 'gateway': 1.0, 'storage-domain': True}
MainThread::INFO::2018-09-18
12:41:11,393::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:11,703::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:21,716::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:22,020::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:31,033::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:31,344::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
As we can see, agent thinking that HostedEngine just in powering up mode. I cannot to do
anythink with it. I allready reinstalled many times srv00 node without success.
One time i even has to uninstall ovirt* and vdsm* software. Also here one interesting
point, after installing just "yum install
http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm" on this node i try to
install this node from engine web interface with "Deploy" action. But,
installation was unsuccesfull, before i didnt install ovirt-hosted-engine-ha on this node.
I dont see in documentation that its need bofore installation of new hosts. But this is
for information and checking. After installing ovirt-hosted-engine-ha node was installed
with HostedEngine support. But the main issue not changed.
Thanks in advance for help.
BR,
Alexandr
7:50 a.m.
On Tue, Sep 18, 2018 at 9:21 AM <asm(a)pioner.kz> wrote:
Good day for all.
I have some issues with Ovirt 4.2.6. But now the main this of it:
I have two Centos 7 Nodes with same config and last Ovirt 4.2.6 with
Hostedengine with disk on NFS storage.
Also some of virtual machines working good.
But, when HostedEngine running on one node (srv02.local) everything is
fine.
After migrating to another node (srv00.local), i see that agent cannot to
check livelinness of HostedEngine. After few minutes HostedEngine going to
reboot and after some time i see some situation. After migration to another
node (srv00.local) all looks OK.
hosted-engine --vm-status commang when HosterEngine on srv00 node:
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : srv02.local
Host ID : 1
Engine status : {"reason": "vm not running on this
host", "health": "bad", "vm":
"down_unexpected", "detail": "unknown"}
Score : 0
stopped : False
Local maintenance : False
crc32 : ecc7ad2d
local_conf_timestamp : 78328
Host timestamp : 78328
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=78328 (Tue Sep 18 12:44:18 2018)
host-id=1
score=0
vm_conf_refresh_time=78328 (Tue Sep 18 12:44:18 2018)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Fri Jan 2 03:49:58 1970
--== Host 2 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : srv00.local
Host ID : 2
Engine status : {"reason": "failed liveliness
check",
"health": "bad", "vm": "up", "detail":
"Up"}
vm: up refers to vm status at virt level polling a local vdsm, health: bad
refers instead to a live check on the engine portal over http.
Bad name resolution or network routing issues can cause this. I'd suggest
to check if everything is fine on network side.
Score : 3400
stopped : False
Local maintenance : False
crc32 : 1d62b106
local_conf_timestamp : 326288
Host timestamp : 326288
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=326288 (Tue Sep 18 12:44:21 2018)
host-id=2
score=3400
vm_conf_refresh_time=326288 (Tue Sep 18 12:44:21 2018)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
Log agent.log from srv00.local:
MainThread::INFO::2018-09-18
12:40:51,749::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:40:52,052::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:01,066::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:01,374::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Global metadata: {'maintenance': False}
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Host srv02.local.pioner.kz (id 1):
{'conf_on_shared_storage': True, 'extra': 'meta
data_parse_version=1\nmetadata_feature_version=1\ntimestamp=78128 (Tue Sep
18 12:40:58 2018)\nhost-id=1\ns
core=0\nvm_conf_refresh_time=78128 (Tue Sep 18 12:40:58
2018)\nconf_on_shared_storage=True\nmaintenance=Fa
lse\nstate=EngineUnexpectedlyDown\nstopped=False\ntimeout=Fri Jan 2
03:49:58 1970\n', 'hostname': 'srv02.
local.pioner.kz', 'alive': True, 'host-id': 1,
'engine-status':
{'reason': 'vm not running on this host',
'health': 'bad', 'vm': 'down_unexpected',
'detail': 'unknown'}, 'score':
0, 'stopped': False, 'maintenance
': False, 'crc32': 'e18e3f22', 'local_conf_timestamp': 78128,
'host-ts':
78128}
MainThread::INFO::2018-09-18
12:41:11,393::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(refresh) Local (id 2): {'engine-health': {'reason':
'failed
liveliness check', 'health': 'b
ad', 'vm': 'up', 'detail': 'Up'}, 'bridge':
True, 'mem-free': 12763.0,
'maintenance': False, 'cpu-load': 0
.0364, 'gateway': 1.0, 'storage-domain': True}
MainThread::INFO::2018-09-18
12:41:11,393::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:11,703::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:21,716::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:22,020::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
MainThread::INFO::2018-09-18
12:41:31,033::states::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE
ngine::(consume) VM is powering up..
MainThread::INFO::2018-09-18
12:41:31,344::hosted_engine::491::ovirt_hosted_engine_ha.agent.hosted_engine.
HostedEngine::(_monitoring_loop) Current state EngineStarting (score: 3400)
As we can see, agent thinking that HostedEngine just in powering up mode.
I cannot to do anythink with it. I allready reinstalled many times srv00
node without success.
One time i even has to uninstall ovirt* and vdsm* software. Also here one
interesting point, after installing just "yum install
http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm" on this node
i try to install this node from engine web interface with "Deploy" action.
But, installation was unsuccesfull, before i didnt install
ovirt-hosted-engine-ha on this node. I dont see in documentation that its
need bofore installation of new hosts. But this is for information and
checking. After installing ovirt-hosted-engine-ha node was installed with
HostedEngine support. But the main issue not changed.
Thanks in advance for help.
BR,
Alexandr
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7KGDIM3X3G4...
11:27 p.m.
New subject: Ovirt-engine-ha cannot to see live status of Hosted Engine
Hi! How i can to check the network? All the same on two nodes except IP address. Pinges
working fine, and others.
Here also broker log srom srv00. You cal see moment when HostedEngine migrated to another
host:
Thread-2::INFO::2018-09-18 12:48:07,531::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-3::INFO::2018-09-18 12:48:07,767::mem_free::51::mem_free.MemFree::(action) memFree:
12774
Thread-1::INFO::2018-09-18 12:48:07,901::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-2::INFO::2018-09-18 12:48:17,555::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-3::INFO::2018-09-18 12:48:17,812::mem_free::51::mem_free.MemFree::(action) memFree:
12766
Thread-3::INFO::2018-09-18 12:48:26,852::mem_free::51::mem_free.MemFree::(action) memFree:
12757
Thread-1::INFO::2018-09-18 12:48:27,453::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-2::INFO::2018-09-18 12:48:27,587::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-5::WARNING::2018-09-18
12:48:30,495::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-3::INFO::2018-09-18 12:48:36,894::mem_free::51::mem_free.MemFree::(action) memFree:
12759
Thread-2::INFO::2018-09-18 12:48:37,619::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-1::INFO::2018-09-18 12:48:37,727::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-3::INFO::2018-09-18 12:48:46,944::mem_free::51::mem_free.MemFree::(action) memFree:
12762
Thread-2::INFO::2018-09-18 12:48:47,651::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-1::INFO::2018-09-18 12:48:48,006::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-5::WARNING::2018-09-18
12:48:50,603::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-3::INFO::2018-09-18 12:48:57,021::mem_free::51::mem_free.MemFree::(action) memFree:
12736
Thread-1::INFO::2018-09-18 12:48:57,285::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-2::INFO::2018-09-18 12:48:57,679::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-4::INFO::2018-09-18
12:49:04,920::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load)
System load total=0.0401, engine=0.0030, non-e
ngine=0.0371
Thread-3::INFO::2018-09-18 12:49:07,064::mem_free::51::mem_free.MemFree::(action) memFree:
12740
Thread-1::INFO::2018-09-18 12:49:07,561::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-2::INFO::2018-09-18 12:49:07,760::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-5::WARNING::2018-09-18
12:49:10,715::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-5::WARNING::2018-09-18
12:49:10,823::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-7::WARNING::2018-09-18
12:49:12,961::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-3::INFO::2018-09-18 12:49:17,114::mem_free::51::mem_free.MemFree::(action) memFree:
12739
Thread-1::INFO::2018-09-18 12:49:17,817::ping::60::ping.Ping::(action) Successfully pinged
192.168.2.248
Thread-2::INFO::2018-09-18 12:49:17,888::mgmt_bridge::62::mgmt_bridge.MgmtBridge::(action)
Found bridge ovirtmgmt with ports
Thread-5::WARNING::2018-09-18
12:49:19,945::engine_health::233::engine_health.EngineHealth::(_result_from_stats) bad
health status: Hosted Engine is not up!
Thread-7::INFO::2018-09-18
12:49:25,650::engine_health::191::engine_health.EngineHealth::(_result_from_stats) VM
successfully migrated away from this host.
4:07 a.m.
New subject: Ovirt-engine-ha cannot to see live status of Hosted Engine
Hi! You was right.
The problem was due to an error in the hosts file. The FQDN of engine has another IP in
this file on this host from previous instalation.
Thank you very muth.
Please help me with my another question. I know that you can help me.
8:05 a.m.
New subject: Ovirt-engine-ha cannot to see live status of Hosted Engine
Hello,
we are experiencing probably the same situation, and it is serious failure of the ovirt!
After last upgrade of centos 7 hosts and hosted engine (see versions below), the hosted
engine does not come up, making the whole ovirt cluster USELESS. I can't connect to
engine console any way. I even tried clean install of one centos 7 host with hosted engine
on nfs storage with the same result, engine comes up, but failing liveliness check.
Here are the versions of clean install, which is also failing:
ovirt-engine-appliance.noarch 4.2-20180903.1.el7
@ovirt-4.2
ovirt-engine-sdk-python.noarch 3.6.9.1-1.el7
@ovirt-4.2
ovirt-host.x86_64 4.2.3-1.el7
@ovirt-4.2
ovirt-host-dependencies.x86_64 4.2.3-1.el7
@ovirt-4.2
ovirt-host-deploy.noarch 1.7.4-1.el7
@ovirt-4.2
ovirt-hosted-engine-ha.noarch 2.2.16-1.el7
@ovirt-4.2
ovirt-hosted-engine-setup.noarch 2.2.26-1.el7
@ovirt-4.2
ovirt-imageio-common.x86_64 1.4.4-0.el7
@ovirt-4.2
ovirt-imageio-daemon.noarch 1.4.4-0.el7
@ovirt-4.2
ovirt-provider-ovn-driver.noarch 1.2.14-1.el7
@ovirt-4.2
ovirt-release42.noarch 4.2.6.1-1.el7
installed
ovirt-setup-lib.noarch 1.1.5-1.el7
@ovirt-4.2
ovirt-vmconsole.noarch 1.0.5-4.el7.centos
@ovirt-4.2
ovirt-vmconsole-host.noarch 1.0.5-4.el7.centos
@ovirt-4.2
python-ovirt-engine-sdk4.x86_64 4.2.8-2.el7
@ovirt-4.2
This is a total fail of "intended for production use" system. Even more, when I
try to restore regular engine backup to new engine before engine-setup on new clean centos
7 host, it is failing also.
What logs would you need to get it analyzed? I can supply it right back.
Regards,
Pavel
2:37 a.m.
New subject: Ovirt-engine-ha cannot to see live status of Hosted Engine
On Wed, Sep 19, 2018 at 3:06 PM Pavel Stržínek <pavel.strzinek(a)gmail.com>
wrote:
Hello,
we are experiencing probably the same situation, and it is serious failure
of the ovirt! After last upgrade of centos 7 hosts and hosted engine (see
versions below), the hosted engine does not come up, making the whole ovirt
cluster USELESS. I can't connect to engine console any way. I even tried
clean install of one centos 7 host with hosted engine on nfs storage with
the same result, engine comes up, but failing liveliness check.
Here are the versions of clean install, which is also failing:
ovirt-engine-appliance.noarch
4.2-20180903.1.el7 @ovirt-4.2
ovirt-engine-sdk-python.noarch
3.6.9.1-1.el7 @ovirt-4.2
ovirt-host.x86_64
4.2.3-1.el7 @ovirt-4.2
ovirt-host-dependencies.x86_64
4.2.3-1.el7 @ovirt-4.2
ovirt-host-deploy.noarch
1.7.4-1.el7 @ovirt-4.2
ovirt-hosted-engine-ha.noarch
2.2.16-1.el7 @ovirt-4.2
ovirt-hosted-engine-setup.noarch
2.2.26-1.el7 @ovirt-4.2
ovirt-imageio-common.x86_64
1.4.4-0.el7 @ovirt-4.2
ovirt-imageio-daemon.noarch
1.4.4-0.el7 @ovirt-4.2
ovirt-provider-ovn-driver.noarch
1.2.14-1.el7 @ovirt-4.2
ovirt-release42.noarch
4.2.6.1-1.el7 installed
ovirt-setup-lib.noarch
1.1.5-1.el7 @ovirt-4.2
ovirt-vmconsole.noarch
1.0.5-4.el7.centos @ovirt-4.2
ovirt-vmconsole-host.noarch
1.0.5-4.el7.centos @ovirt-4.2
python-ovirt-engine-sdk4.x86_64
4.2.8-2.el7 @ovirt-4.2
This is a total fail of "intended for production use" system. Even more,
when I try to restore regular engine backup to new engine before
engine-setup on new clean centos 7 host, it is failing also.
What logs would you need to get it analyzed? I can supply it right back.
vdsm.log from the host trying to start the engine VM.
Did you tried to connect to the engine VM over VNC?
Regards,
Pavel
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/53IE4M52M4T...
1786
days inactive
2287
days old
5 comments
3 participants
participants (3)
-
asm＠pioner.kz -
Pavel Stržínek -
Simone Tiraboschi