8:07 a.m.
Hello,
We've been trying to setup oVirt environment for few days but we have issue with
hosted-engine-setup (ansible script).
We managed to fix few small things and have them merged upstream but unfortunately right
now the installation process fails on getting host facts.
It looks like it cannot proceed because it fails when connecting to ovirt-engine API of
the bootstrap VM.
The oVirt API / webpanel is working, I tested it via a browser and I can login without
issues using the admin password chosen earlier in the process.
2018-05-18 15:26:47,800+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:100 TASK [Wait for the host to be up]
2018-05-18 15:39:14,025+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:94 {u'_ansible_parsed': True,
u'_ansible_no_log': False, u'changed': False, u'attempts': 120,
u'invocation': {u'module_args': {
u'pattern': u'name=host01.redacted', u'fetch_nested': False,
u'nested_attributes': []}}, u'ansible_facts': {u'ovirt_hosts':
[]}}
2018-05-18 15:39:14,127+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:98 fatal: [localhost]: FAILED! =>
{"ansible_facts": {"ovirt_hosts": []}, "attempts": 120,
"changed": false}
May 18 13:34:34 host01 python: ansible-ovirt_hosts_facts Invoked with
pattern=name=host01.redacted fetch_nested=False nested_attributes=[]
auth={'timeout': 0, 'url':
'https://ovirt-dev.redacted/ovirt-engine/api', 'insecure': True,
'kerberos': False, 'compress': True, 'headers': None,
'token': 'R--token-redacted', 'ca_file': None}
Do you have idea what/where is issue and how to fix it?
8:11 a.m.
On Wed, May 23, 2018 at 8:35 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
Hello,
We've been trying to setup oVirt environment for few days but we have
issue with hosted-engine-setup (ansible script).
We managed to fix few small things and have them merged upstream but
unfortunately right now the installation process fails on getting host
facts.
It looks like it cannot proceed because it fails when connecting to
ovirt-engine API of the bootstrap VM.
The oVirt API / webpanel is working, I tested it via a browser and I can
login without issues using the admin password chosen earlier in the process.
2018-05-18 15:26:47,800+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:100 TASK [Wait for the host to be up]
2018-05-18 15:39:14,025+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:94 {u'_ansible_parsed': True,
u'_ansible_no_log': False, u'changed': False, u'attempts': 120,
u'invocation': {u'module_args': {
u'pattern': u'name=host01.redacted', u'fetch_nested': False,
u'nested_attributes': []}}, u'ansible_facts': {u'ovirt_hosts':
[]}}
2018-05-18 15:39:14,127+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:98 fatal: [localhost]: FAILED! =>
{"ansible_facts": {"ovirt_hosts": []}, "attempts": 120,
"changed": false}
May 18 13:34:34 host01 python: ansible-ovirt_hosts_facts Invoked with
pattern=name=host01.redacted fetch_nested=False nested_attributes=[]
auth={'timeout': 0, 'url':
'https://ovirt-dev.redacted/ovirt-engine/api',
'insecure': True, 'kerberos': False, 'compress': True,
'headers': None,
'token': 'R--token-redacted', 'ca_file': None}
Do you have idea what/where is issue and how to fix it?
Hi,
it correctly gets host facts from engine APIs, the issue now is simply that
no host appears in the reported facts and this means that engine, for same
reason, failed to configure the host where you are running
hosted-engine-setup.
To better understand what it's happening you have to check host-deploy
logs; they are available under /var/log/ovirt-engine/host-deploy/ on your
engine VM.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
8:55 a.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy logs; they are
available under /var/log/ovirt-engine/host-deploy/ on your engine VM.
Unfortunately there is no logs under that directory. It's empty.
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
5:21 a.m.
On Fri, May 25, 2018 at 9:20 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy
logs; they are available under /var/log/ovirt-engine/host-deploy/ on your
engine VM.
Unfortunately there is no logs under that directory. It's empty.
So it probably failed to reach the host due to a name resolution issue or
something like that.
Can you please double check it in /var/log/ovirt-engine/engine.log on the
engine VM ?
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
4:15 p.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Fri, 2018-05-25 at 11:21 +0200, Simone Tiraboschi wrote:
On Fri, May 25, 2018 at 9:20 AM, Mariusz Kozakowski
<mariusz.kozakowski@dsg.dk<mailto:mariusz.kozakowski@dsg.dk>> wrote:
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy logs; they are
available under /var/log/ovirt-engine/host-deploy/ on your engine VM.
Unfortunately there is no logs under that directory. It's empty.
So it probably failed to reach the host due to a name resolution issue or something like
that.
Can you please double check it in /var/log/ovirt-engine/engine.log on the engine VM ?
Thanks - it helped a bit. At least now we have logs for host-deploy, but still no
success.
Few parts I found in engine log:
2018-05-28 11:07:39,473+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException:
VDSNetworkException: Message timeout which can be caused by communication issues (Failed
with error VDS_NETWORK_ERROR and code 5022)
2018-05-28 11:07:39,485+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Host installation failed for host
'098c3c99-921d-46f0-bdba-86370a2dc895', 'host01.redacted': Failed to
configure management network on the host
2018-05-28 11:20:04,705+02 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,711+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='098c3c99-921d-46f0-bdba-86370a2dc895',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
11ebbdeb
2018-05-28 11:20:04,715+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] FINISH,
SetVdsStatusVDSCommand, log id: 11ebbdeb
2018-05-28 11:20:04,769+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Host
'host01.redacted' is set to Non-Operational, it is missing the following networks:
'ovirtmgmt'
2018-05-28 11:20:04,786+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host host01.redacted does not comply with the cluster
Default networks, the following networks are missing on host: 'ovirtmgmt'
2018-05-28 11:20:04,807+02 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,814+02 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] EVENT_ID: VDS_DETECTED(13),
Status of host host01.redacted was set to NonOperational.
2018-05-28 11:20:04,833+02 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Running command:
HandleVdsVersionCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,837+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Host
'host01.redacted'(098c3c99-921d-46f0-bdba-86370a2dc895) is already in
NonOperational status for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds
command is skipped.
Full log as attachment.
6:57 a.m.
On Mon, May 28, 2018 at 11:44 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
On Fri, 2018-05-25 at 11:21 +0200, Simone Tiraboschi wrote:
On Fri, May 25, 2018 at 9:20 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy
logs; they are available under /var/log/ovirt-engine/host-deploy/ on your
engine VM.
Unfortunately there is no logs under that directory. It's empty.
So it probably failed to reach the host due to a name resolution issue or
something like that.
Can you please double check it in /var/log/ovirt-engine/engine.log on the
engine VM ?
Thanks - it helped a bit. At least now we have logs for host-deploy, but
still no success.
Few parts I found in engine log:
2018-05-28 11:07:39,473+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)
2018-05-28 11:07:39,485+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Host installation
failed for host '098c3c99-921d-46f0-bdba-86370a2dc895',
'host01.redacted': Failed to configure management network on the host
The issue is on network configuration:
you have to check /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log to
understand why it failed.
2018-05-28 11:20:04,705+02 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Running
command: SetNonOperationalVdsCommand internal: true. Entities affected
: ID: 098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,711+02 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='098c3c99-921d-46f0-bdba-86370a2dc895',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
11ebbdeb
2018-05-28 11:20:04,715+02 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] FINISH,
SetVdsStatusVDSCommand, log id: 11ebbdeb
2018-05-28 11:20:04,769+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Host
'host01.redacted' is set to Non-Operational, it is missing the following
networks: 'ovirtmgmt'
2018-05-28 11:20:04,786+02 WARN [org.ovirt.engine.core.
dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37)
[5ba0ae45] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host
host01.redacted does not comply with the cluster Default networks, the
following networks are missing on host: 'ovirtmgmt'
2018-05-28 11:20:04,807+02 INFO [org.ovirt.engine.core.bll.
HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37)
[7937fb47] Running command: HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities affected : ID: 098c3c99-921d-46f0-bdba-86370a2dc895
Type: VDS
2018-05-28 11:20:04,814+02 INFO [org.ovirt.engine.core.
dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37)
[7937fb47] EVENT_ID: VDS_DETECTED(13), Status of host host01.redacted was
set to NonOperational.
2018-05-28 11:20:04,833+02 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Running
command: HandleVdsVersionCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,837+02 INFO [org.ovirt.engine.core.
vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-37)
[4c10675c] Host 'host01.redacted'(098c3c99-921d-46f0-bdba-86370a2dc895)
is already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
SetNonOperationalVds command is skipped.
Full log as attachment.
12:06 p.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Mon, 2018-05-28 at 12:57 +0200, Simone Tiraboschi wrote:
The issue is on network configuration:
you have to check /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log to understand why
it failed.
From same time frame, vdsm.log. Can this be related?
2018-05-28 11:07:34,481+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats()
from=::1,45816 (api:46)
2018-05-28 11:07:34,482+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats
return={'status': {'message': 'Done', 'code': 0},
'statsList': (suppressed)} from=::1,45816 (api:52)
2018-05-28 11:07:34,483+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call
Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
2018-05-28 11:07:34,489+0200 INFO (jsonrpc/2) [api.host] START getAllVmIoTunePolicies()
from=::1,45816 (api:46)
2018-05-28 11:07:34,489+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmIoTunePolicies
return={'status': {'message': 'Done', 'code': 0},
'io_tune_policies_dict': {'405f8ec0-03f9-43cb-a7e1-343a4c30453f':
{'policy': [], 'current_values': []}}} from=::1,45816 (api:52)
2018-05-28 11:07:34,490+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:573)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vdsm.api] START
getConnectedStoragePoolsList(options=None) from=internal,
task_id=6f517c47-a9f3-4913-bf9d-661355262c38 (api:46)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vdsm.api] FINISH
getConnectedStoragePoolsList return={'poollist': []} from=internal,
task_id=6f517c47-a9f3-4913-bf9d-661355262c38 (api:52)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool
to go up (clientIF:707)
2018-05-28 11:07:38,982+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker
name=jsonrpc/7 running <Task <JsonRpcTask {'params': {}, 'jsonrpc':
'2.0', 'method': u'Host.getCapabilities', 'id':
u'b5990e16-65c9-4137-aeed-271c415f9df5'} at 0x7eff50088910> timeout=60,
duration=180 at 0x2fc3650> task#=1 at 0x3590710>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in
run
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task
task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__
self._callable()
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 523, in
__call__
self._handler(self._ctx, self._req)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 566, in
_serveRequest
response = self._handle_request(req, ctx)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in
_handle_request
res = method(**params)
File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
_dynamicMethod
result = fn(*methodArgs)
File: "<string>", line 2, in getCapabilities
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
method
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in
getCapabilities
c = caps.get()
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 201, in get
liveSnapSupported = _getLiveSnapshotSupport(cpuarch.effective())
File: "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 41, in
__call__
value = self.func(*args)
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 92, in
_getLiveSnapshotSupport
capabilities = _getCapsXMLStr()
File: "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 41, in
__call__
value = self.func(*args)
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 60, in
_getCapsXMLStr
return _getFreshCapsXMLStr()
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 55, in
_getFreshCapsXMLStr
return libvirtconnection.get().getCapabilities()
File: "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line
130, in wrapper
ret = f(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in
wrapper
return func(inst, *args, **kwargs)
File: "/usr/lib64/python2.7/site-packages/libvirt.py", line 3669, in
getCapabilities
ret = libvirtmod.virConnectGetCapabilities(self._o) (executor:363)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vdsm.api] START
getConnectedStoragePoolsList(options=None) from=internal,
task_id=2ef7c0a3-4cf9-436e-ad6b-ee04e2a0cf3a (api:46)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vdsm.api] FINISH
getConnectedStoragePoolsList return={'poollist': []} from=internal,
task_id=2ef7c0a3-4cf9-436e-ad6b-ee04e2a0cf3a (api:52)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool
to go up (clientIF:707)
supervdsm hase nothing at 11:07:39.
5:45 a.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Mon, 2018-05-28 at 14:30 +0200, Mariusz Kozakowski wrote:
On Mon, 2018-05-28 at 12:57 +0200, Simone Tiraboschi wrote:
The issue is on network configuration:
you have to check /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log to understand why
it failed.
From same time frame, vdsm.log. Can this be related?
2018-05-28 11:07:34,481+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats()
from=::1,45816 (api:46)
2018-05-28 11:07:34,482+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats
return={'status': {'message': 'Done', 'code': 0},
'statsList': (suppressed)} from=::1,45816 (api:52)
2018-05-28 11:07:34,483+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call
Host.getAllVmStats succeeded in 0.01 seconds (__init__:573)
2018-05-28 11:07:34,489+0200 INFO (jsonrpc/2) [api.host] START getAllVmIoTunePolicies()
from=::1,45816 (api:46)
2018-05-28 11:07:34,489+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmIoTunePolicies
return={'status': {'message': 'Done', 'code': 0},
'io_tune_policies_dict': {'405f8ec0-03f9-43cb-a7e1-343a4c30453f':
{'policy': [], 'current_values': []}}} from=::1,45816 (api:52)
2018-05-28 11:07:34,490+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getAllVmIoTunePolicies succeeded in 0.00 seconds (__init__:573)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vdsm.api] START
getConnectedStoragePoolsList(options=None) from=internal,
task_id=6f517c47-a9f3-4913-bf9d-661355262c38 (api:46)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vdsm.api] FINISH
getConnectedStoragePoolsList return={'poollist': []} from=internal,
task_id=6f517c47-a9f3-4913-bf9d-661355262c38 (api:52)
2018-05-28 11:07:35,555+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool
to go up (clientIF:707)
2018-05-28 11:07:38,982+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker
name=jsonrpc/7 running <Task <JsonRpcTask {'params': {}, 'jsonrpc':
'2.0', 'method': u'Host.getCapabilities', 'id':
u'b5990e16-65c9-4137-aeed-271c415f9df5'} at 0x7eff50088910> timeout=60,
duration=180 at 0x2fc3650> task#=1 at 0x3590710>, traceback:
File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File: "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in
run
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task
task()
File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__
self._callable()
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 523, in
__call__
self._handler(self._ctx, self._req)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 566, in
_serveRequest
response = self._handle_request(req, ctx)
File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in
_handle_request
res = method(**params)
File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in
_dynamicMethod
result = fn(*methodArgs)
File: "<string>", line 2, in getCapabilities
File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
method
ret = func(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in
getCapabilities
c = caps.get()
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 201, in get
liveSnapSupported = _getLiveSnapshotSupport(cpuarch.effective())
File: "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 41, in
__call__
value = self.func(*args)
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 92, in
_getLiveSnapshotSupport
capabilities = _getCapsXMLStr()
File: "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 41, in
__call__
value = self.func(*args)
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 60, in
_getCapsXMLStr
return _getFreshCapsXMLStr()
File: "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 55, in
_getFreshCapsXMLStr
return libvirtconnection.get().getCapabilities()
File: "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line
130, in wrapper
ret = f(*args, **kwargs)
File: "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in
wrapper
return func(inst, *args, **kwargs)
File: "/usr/lib64/python2.7/site-packages/libvirt.py", line 3669, in
getCapabilities
ret = libvirtmod.virConnectGetCapabilities(self._o) (executor:363)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vdsm.api] START
getConnectedStoragePoolsList(options=None) from=internal,
task_id=2ef7c0a3-4cf9-436e-ad6b-ee04e2a0cf3a (api:46)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vdsm.api] FINISH
getConnectedStoragePoolsList return={'poollist': []} from=internal,
task_id=2ef7c0a3-4cf9-436e-ad6b-ee04e2a0cf3a (api:52)
2018-05-28 11:07:40,561+0200 INFO (vmrecovery) [vds] recovery: waiting for storage pool
to go up (clientIF:707)
supervdsm hase nothing at 11:07:39.
Found another one, maybe related:
2018-05-28 11:02:44,371+0200 ERROR (vm/405f8ec0) [virt.vm]
(vmId='405f8ec0-03f9-43cb-a7e1-343a4c30453f') Failed to connect to guest agent
channel (vm:2500)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2498, in
_vmDependentInit
self.guestAgent.start()
File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 249, in
start
self._prepare_socket()
File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 291, in
_prepare_socket
supervdsm.getProxy().prepareVmChannel(self._socketName)
File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in
__call__
return callMethod()
File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in
<lambda>
**kwargs)
File "<string>", line 2, in prepareVmChannel
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
OSError: [Errno 2] No such file or directory:
'/var/lib/libvirt/qemu/channels/405f8ec0-03f9-43cb-a7e1-343a4c30453f.com.redhat.rhevm.vdsm'
Interesting part: there is no file with ending: com.redhat.rhevm.vdsm, but there is
`/var/lib/libvirt/qemu/channels/405f8ec0-03f9-43cb-a7e1-343a4c30453f.org.qemu.guest_agent.0`
6:41 a.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
Hi,
we managed to get a bit forward, but we still face issues.
2018-06-05 09:38:42,556+02 INFO [org.ovirt.engine.core.bll.host.HostConnectivityChecker]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Engine managed to communicate with
VDSM agent on host 'host01.redacted' with address 'host01.redacted'
('8af21ab3-ce7a-49a5-a526-94b65aa3da29')
2018-06-05 09:38:47,488+02 WARN [org.ovirt.engine.core.bll.network.NetworkConfigurator]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Failed to find a valid interface for
the management network of host host01.redacted. If the interface ovirtmgmt is a bridge, it
should be torn-down manually.
2018-06-05 09:38:47,488+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Exception:
org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException:
Interface ovirtmgmt is invalid for management network
Our network configuration, bond0.1111 is bridged into ovirtmgmt:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[…]
11: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet 1.2.3.42/24 brd 1.2.3.255 scope global noprefixroute ovirtmgmt
valid_lft forever preferred_lft forever
inet6 fe80::e8dd:fff:fe33:4bba/64 scope link
valid_lft forever preferred_lft forever
12: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5ef3:fcff:feda:b618/64 scope link
valid_lft forever preferred_lft forever
13: bond0.3019@bond0<mailto:bond0.3019@bond0>:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.3019 state UP
group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
14: bond0.1111@bond0<mailto:bond0.1111@bond0>:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovirtmgmt state UP
group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
15: br0.3019: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet 19.2.3.22/16 brd 192.168.255.255 scope global noprefixroute br0.3019
valid_lft forever preferred_lft forever
inet6 fe80::5ef3:fcff:feda:b618/64 scope link
valid_lft forever preferred_lft forever
31: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
qlen 1000
link/ether da:aa:73:7e:d7:93 brd ff:ff:ff:ff:ff:ff
32: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group
default qlen 1000
link/ether 52:54:00:a6:75:67 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
33: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state
DOWN group default qlen 1000
link/ether 52:54:00:a6:75:67 brd ff:ff:ff:ff:ff:ff
40: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0
state UNKNOWN group default qlen 1000
link/ether fe:16:3e:2d:0d:55 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe2d:d55/64 scope link
valid_lft forever preferred_lft forever
On Mon, 2018-05-28 at 12:57 +0200, Simone Tiraboschi wrote:
On Mon, May 28, 2018 at 11:44 AM, Mariusz Kozakowski
<mariusz.kozakowski@dsg.dk<mailto:mariusz.kozakowski@dsg.dk>> wrote:
On Fri, 2018-05-25 at 11:21 +0200, Simone Tiraboschi wrote:
On Fri, May 25, 2018 at 9:20 AM, Mariusz Kozakowski
<mariusz.kozakowski@dsg.dk<mailto:mariusz.kozakowski@dsg.dk>> wrote:
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy logs; they are
available under /var/log/ovirt-engine/host-deploy/ on your engine VM.
Unfortunately there is no logs under that directory. It's empty.
So it probably failed to reach the host due to a name resolution issue or something like
that.
Can you please double check it in /var/log/ovirt-engine/engine.log on the engine VM ?
Thanks - it helped a bit. At least now we have logs for host-deploy, but still no
success.
Few parts I found in engine log:
2018-05-28 11:07:39,473+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException:
VDSNetworkException: Message timeout which can be caused by communication issues (Failed
with error VDS_NETWORK_ERROR and code 5022)
2018-05-28 11:07:39,485+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [1a4cf85e] Host installation failed for host
'098c3c99-921d-46f0-bdba-86370a2dc895', 'host01.redacted': Failed to
configure management network on the host
The issue is on network configuration:
you have to check /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log to understand why
it failed.
2018-05-28 11:20:04,705+02 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,711+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='098c3c99-921d-46f0-bdba-86370a2dc895',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
11ebbdeb
2018-05-28 11:20:04,715+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] FINISH,
SetVdsStatusVDSCommand, log id: 11ebbdeb
2018-05-28 11:20:04,769+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Host
'host01.redacted' is set to Non-Operational, it is missing the following networks:
'ovirtmgmt'
2018-05-28 11:20:04,786+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host host01.redacted does not comply with the cluster
Default networks, the following networks are missing on host: 'ovirtmgmt'
2018-05-28 11:20:04,807+02 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,814+02 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] EVENT_ID: VDS_DETECTED(13),
Status of host host01.redacted was set to NonOperational.
2018-05-28 11:20:04,833+02 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Running command:
HandleVdsVersionCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,837+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Host
'host01.redacted'(098c3c99-921d-46f0-bdba-86370a2dc895) is already in
NonOperational status for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds
command is skipped.
Full log as attachment.
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
4:09 a.m.
On Tue, Jun 5, 2018 at 10:05 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)sallinggroup.com> wrote:
Hi,
we managed to get a bit forward, but we still face issues.
2018-06-05 09:38:42,556+02 INFO
[org.ovirt.engine.core.bll.host.HostConnectivityChecker]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Engine managed to
communicate with VDSM agent on host 'host01.redacted' with address
'host01.redacted' ('8af21ab3-ce7a-49a5-a526-94b65aa3da29')
2018-06-05 09:38:47,488+02 WARN [org.ovirt.engine.core.bll.network.NetworkConfigurator]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Failed to find a
valid interface for the management network of host host01.redacted. If the
interface ovirtmgmt is a bridge, it should be torn-down manually.
2018-06-05 09:38:47,488+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2617aebd] Exception:
org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException:
Interface ovirtmgmt is invalid for management network
Our network configuration, bond0.1111 is bridged into ovirtmgmt:
But did you manually created the bridge or did the engine created it for
you triggered by hosted-engine-setup?
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[…]
11: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet 1.2.3.42/24 brd 1.2.3.255 scope global noprefixroute ovirtmgmt
valid_lft forever preferred_lft forever
inet6 fe80::e8dd:fff:fe33:4bba/64 scope link
valid_lft forever preferred_lft forever
12: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5ef3:fcff:feda:b618/64 scope link
valid_lft forever preferred_lft forever
13: bond0.3019@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master br0.3019 state UP group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
14: bond0.1111@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue master ovirtmgmt state UP group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
15: br0.3019: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 5c:f3:fc:da:b6:18 brd ff:ff:ff:ff:ff:ff
inet 19.2.3.22/16 brd 192.168.255.255 scope global noprefixroute
br0.3019
valid_lft forever preferred_lft forever
inet6 fe80::5ef3:fcff:feda:b618/64 scope link
valid_lft forever preferred_lft forever
31: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
group default qlen 1000
link/ether da:aa:73:7e:d7:93 brd ff:ff:ff:ff:ff:ff
32: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 52:54:00:a6:75:67 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
33: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master
virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:a6:75:67 brd ff:ff:ff:ff:ff:ff
40: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:2d:0d:55 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe2d:d55/64 scope link
valid_lft forever preferred_lft forever
On Mon, 2018-05-28 at 12:57 +0200, Simone Tiraboschi wrote:
On Mon, May 28, 2018 at 11:44 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
On Fri, 2018-05-25 at 11:21 +0200, Simone Tiraboschi wrote:
On Fri, May 25, 2018 at 9:20 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
On Thu, 2018-05-24 at 14:11 +0200, Simone Tiraboschi wrote:
To better understand what it's happening you have to check host-deploy
logs; they are available under /var/log/ovirt-engine/host-deploy/ on your
engine VM.
Unfortunately there is no logs under that directory. It's empty.
So it probably failed to reach the host due to a name resolution issue or
something like that.
Can you please double check it in /var/log/ovirt-engine/engine.log on the
engine VM ?
Thanks - it helped a bit. At least now we have logs for host-deploy, but
still no success.
Few parts I found in engine log:
2018-05-28 11:07:39,473+02 ERROR [org.ovirt.engine.core.bll.hos
tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1)
[1a4cf85e] Exception: org.ovirt.engine.core.common.errors.EngineException:
EngineException: org.ovirt.engine.core.vdsbroke
r.vdsbroker.VDSNetworkException: VDSGenericException:
VDSNetworkException: Message timeout which can be caused by communication
issues (Failed with error VDS_NETWORK_ERROR and code 5022)
2018-05-28 11:07:39,485+02 ERROR [org.ovirt.engine.core.bll.hos
tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1)
[1a4cf85e] Host installation failed for host
'098c3c99-921d-46f0-bdba-86370a2dc895',
'host01.redacted': Failed to configure management network on the host
The issue is on network configuration:
you have to check /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log
to understand why it failed.
2018-05-28 11:20:04,705+02 INFO [org.ovirt.engine.core.b
ll.SetNonOperationalVdsCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37)
[5ba0ae45] Running command: SetNonOperationalVdsCommand internal: true.
Entities affected : ID: 098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,711+02 INFO [org.ovirt.engine.core.v
dsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37)
[5ba0ae45] START, SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='098c3c99-921d-46f0-bdba-86370a2dc895',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
11ebbdeb
2018-05-28 11:20:04,715+02 INFO [org.ovirt.engine.core.v
dsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-37)
[5ba0ae45] FINISH, SetVdsStatusVDSCommand, log id: 11ebbdeb
2018-05-28 11:20:04,769+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] Host
'host01.redacted' is set to Non-Operational, it is missing the following
networks: 'ovirtmgmt'
2018-05-28 11:20:04,786+02 WARN [org.ovirt.engine.core.d
al.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [5ba0ae45] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host host01.redacted does not comply
with the cluster Default networks, the following networks are missing on
host: 'ovirtmgmt'
2018-05-28 11:20:04,807+02 INFO [org.ovirt.engine.core.b
ll.HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] Running
command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true.
Entities affected : ID: 098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,814+02 INFO [org.ovirt.engine.core.d
al.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [7937fb47] EVENT_ID:
VDS_DETECTED(13), Status of host host01.redacted was set to NonOperational.
2018-05-28 11:20:04,833+02 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-37) [4c10675c] Running
command: HandleVdsVersionCommand internal: true. Entities affected : ID:
098c3c99-921d-46f0-bdba-86370a2dc895 Type: VDS
2018-05-28 11:20:04,837+02 INFO [org.ovirt.engine.core.v
dsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-37)
[4c10675c] Host 'host01.redacted'(098c3c99-921d-46f0-bdba-86370a2dc895)
is already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
SetNonOperationalVds command is skipped.
Full log as attachment.
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
12:49 p.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Tue, 2018-06-05 at 10:09 +0200, Simone Tiraboschi wrote:
But did you manually created the bridge or did the engine created it for you triggered by
hosted-engine-setup?
Manually. Before we had br0.1111. Should we go back with network configuration to
br0.1111, and no ovritmgmt network created?
Also, what we should use as anwers here?
OVEHOSTED_NETWORK/bridgeIf=str:bond0.1111
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
5:03 a.m.
On Tue, Jun 5, 2018 at 10:16 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)sallinggroup.com> wrote:
On Tue, 2018-06-05 at 10:09 +0200, Simone Tiraboschi wrote:
But did you manually created the bridge or did the engine created it for
you triggered by hosted-engine-setup?
Manually. Before we had br0.1111. Should we go back with network
configuration to br0.1111, and no ovritmgmt network created?
Yes, I'm pretty sure that the issue is there, see also
https://bugzilla.redhat.com/show_bug.cgi?id=1317125
It will work for sure if the management bridge has been created in the past
by the engine but we had a lot of failure reports trying to consume
management bridge manually created with wrong options.
Letting the engine creating it with the right configuration is by far the
safest option.
Also, what we should use as anwers here?
OVEHOSTED_NETWORK/bridgeIf=str:bond0.1111
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
Yes, this should be fine.
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
4:17 p.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
Ok, removed the manual created ovirtmgmt network. Now it's complaining about missing
ovirtmgmt network when installing host:
2018-06-05 13:21:32,091+02 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(SSL Stomp Reactor) [2ed0af08] Connecting to host01.recacted/1.2.3.42
2018-06-05 13:21:32,250+02 INFO [org.ovirt.engine.core.bll.host.HostConnectivityChecker]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Engine managed to communicate with
VDSM agent on host 'host01.recacted' with address 'host01.recacted'
('9956cebf-59ab-426b-ace6-25342705445e')
2018-06-05 13:22:09,007+02 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78) [2b3212b5] Lock Acquired to object
'EngineLock:{exclusiveLocks='[5eb1152b-e9fb-4e2a-abea-f2d9823b724a=PROVIDER]',
sharedLocks=''}'
2018-06-05 13:22:09,024+02 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78) [2b3212b5] Running command:
SyncNetworkProviderCommand internal: true.
2018-06-05 13:22:09,254+02 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils]
(default task-64) [] User admin@internal<mailto:admin@internal> successfully logged
in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search
ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
ovirt-ext=token:password-access
2018-06-05 13:22:09,526+02 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78) [2b3212b5] Lock freed to object
'EngineLock:{exclusiveLocks='[5eb1152b-e9fb-4e2a-abea-f2d9823b724a=PROVIDER]',
sharedLocks=''}'
2018-06-05 13:24:32,603+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host01.recacted command
CollectVdsNetworkDataAfterInstallationVDS failed: Message timeout which can be caused by
communication issues
2018-06-05 13:24:32,603+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand'
return value
'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7b51b9cb<mailto:'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7b51b9cb>'
2018-06-05 13:24:32,603+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] HostName = host01.recacted
2018-06-05 13:24:32,616+02 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Command
'CollectVdsNetworkDataAfterInstallationVDSCommand(HostName = host01.recacted,
CollectHostNetworkDataVdsCommandParameters:{hostId='9956cebf-59ab-426b-ace6-25342705445e',
vds='Host[host01.recacted,9956cebf-59ab-426b-ace6-25342705445e]'})' execution
failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by
communication issues
2018-06-05 13:24:32,616+02 WARN [org.ovirt.engine.core.vdsbroker.VdsManager]
(EE-ManagedThreadFactory-engine-Thread-3) [33333924] Host 'host01.recacted' is not
responding.
2018-06-05 13:24:32,622+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-3) [33333924] EVENT_ID:
VDS_HOST_NOT_RESPONDING(9,027), Host host01.recacted is not responding. Host cannot be
fenced automatically because power management for the host is disabled.
2018-06-05 13:24:32,616+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException:
VDSNetworkException: Message timeout which can be caused by communication issues (Failed
with error VDS_NETWORK_ERROR and code 5022)
2018-06-05 13:24:32,628+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Host installation failed for host
'9956cebf-59ab-426b-ace6-25342705445e', 'host01.redacted': Failed to
configure management network on the host
2018-06-05 13:24:32,633+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='9956cebf-59ab-426b-ace6-25342705445e',
status='NonOperational', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
2fa6c4ab
2018-06-05 13:26:19,101+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='9956cebf-59ab-426b-ace6-25342705445e',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
460b0642
2018-06-05 13:26:19,106+02 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] FINISH,
SetVdsStatusVDSCommand, log id: 460b0642
2018-06-05 13:26:19,161+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] Host
'host01.redacted' is set to Non-Operational, it is missing the following networks:
'ovirtmgmt'
2018-06-05 13:26:19,196+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host host01.redacted does not comply with the cluster
Default networks, the following networks are missing on host: 'ovirtmgmt'
2018-06-05 13:26:19,235+02 INFO
[org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [647a0b6c] Running command:
HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID:
9956cebf-59ab-426b-ace6-25342705445e Type: VDS
Answers used for ansible deploy:
OVEHOSTED_NETWORK/bridgeIf=str:br0.1111
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
Do you have idea why the ovirtmgmt wasn't created during deploy? Or should we use
other settings for network for deploy script?
On Tue, 2018-06-05 at 11:03 +0200, Simone Tiraboschi wrote:
On Tue, Jun 5, 2018 at 10:16 AM, Mariusz Kozakowski
<mariusz.kozakowski@sallinggroup.com<mailto:mariusz.kozakowski@sallinggroup.com>>
wrote:
On Tue, 2018-06-05 at 10:09 +0200, Simone Tiraboschi wrote:
But did you manually created the bridge or did the engine created it for you triggered by
hosted-engine-setup?
Manually. Before we had br0.1111. Should we go back with network configuration to
br0.1111, and no ovritmgmt network created?
Yes, I'm pretty sure that the issue is there, see also
https://bugzilla.redhat.com/show_bug.cgi?id=1317125
It will work for sure if the management bridge has been created in the past by the engine
but we had a lot of failure reports trying to consume management bridge manually created
with wrong options.
Letting the engine creating it with the right configuration is by far the safest option.
Also, what we should use as anwers here?
OVEHOSTED_NETWORK/bridgeIf=str:bond0.1111
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
Yes, this should be fine.
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com<http://dansksupermarked.com>
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
7:43 a.m.
On Tue, Jun 5, 2018 at 1:40 PM, Mariusz Kozakowski <
mariusz.kozakowski(a)sallinggroup.com> wrote:
Ok, removed the manual created ovirtmgmt network. Now it's
complaining
about missing ovirtmgmt network when installing host:
2018-06-05 13:21:32,091+02 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(SSL Stomp Reactor) [2ed0af08] Connecting to host01.recacted/1.2.3.42
2018-06-05 13:21:32,250+02 INFO
[org.ovirt.engine.core.bll.host.HostConnectivityChecker]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Engine managed to
communicate with VDSM agent on host 'host01.recacted' with address
'host01.recacted' ('9956cebf-59ab-426b-ace6-25342705445e')
2018-06-05 13:22:09,007+02 INFO [org.ovirt.engine.core.
bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78)
[2b3212b5] Lock Acquired to object 'EngineLock:{exclusiveLocks='[
5eb1152b-e9fb-4e2a-abea-f2d9823b724a=PROVIDER]', sharedLocks=''}'
2018-06-05 13:22:09,024+02 INFO [org.ovirt.engine.core.
bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78)
[2b3212b5] Running command: SyncNetworkProviderCommand internal: true.
2018-06-05 13:22:09,254+02 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils]
(default task-64) [] User admin@internal successfully logged in with
scopes: ovirt-app-api ovirt-ext=token-info:authz-search
ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
ovirt-ext=token:password-access
2018-06-05 13:22:09,526+02 INFO [org.ovirt.engine.core.
bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-78)
[2b3212b5] Lock freed to object 'EngineLock:{exclusiveLocks='[
5eb1152b-e9fb-4e2a-abea-f2d9823b724a=PROVIDER]', sharedLocks=''}'
2018-06-05 13:24:32,603+02 ERROR [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1)
[33333924] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM
host01.recacted command CollectVdsNetworkDataAfterInstallationVDS failed:
Message timeout which can be caused by communication issues
2018-06-05 13:24:32,603+02 INFO [org.ovirt.engine.core.
vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand'
return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.
VDSInfoReturn@7b51b9cb'
2018-06-05 13:24:32,603+02 INFO [org.ovirt.engine.core.
vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] HostName =
host01.recacted
2018-06-05 13:24:32,616+02 ERROR [org.ovirt.engine.core.
vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Command '
CollectVdsNetworkDataAfterInstallationVDSCommand(HostName =
host01.recacted, CollectHostNetworkDataVdsCommandParameters:{hostId='
9956cebf-59ab-426b-ace6-25342705445e', vds='Host[host01.recacted,
9956cebf-59ab-426b-ace6-25342705445e]'})' execution failed:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues
2018-06-05 13:24:32,616+02 WARN [org.ovirt.engine.core.vdsbroker.VdsManager]
(EE-ManagedThreadFactory-engine-Thread-3) [33333924] Host
'host01.recacted' is not responding.
2018-06-05 13:24:32,622+02 WARN [org.ovirt.engine.core.
dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-3)
[33333924] EVENT_ID: VDS_HOST_NOT_RESPONDING(9,027), Host host01.recacted
is not responding. Host cannot be fenced automatically because power
management for the host is disabled.
2018-06-05 13:24:32,616+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Exception:
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: Message timeout which can be
caused by communication issues (Failed with error VDS_NETWORK_ERROR and
code 5022)
2018-06-05 13:24:32,628+02 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] Host installation
failed for host '9956cebf-59ab-426b-ace6-25342705445e',
'host01.redacted': Failed to configure management network on the host
2018-06-05 13:24:32,633+02 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [33333924] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='9956cebf-59ab-426b-ace6-25342705445e',
status='NonOperational', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
2fa6c4ab
2018-06-05 13:26:19,101+02 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] START,
SetVdsStatusVDSCommand(HostName = host01.redacted,
SetVdsStatusVDSCommandParameters:{hostId='9956cebf-59ab-426b-ace6-25342705445e',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
460b0642
2018-06-05 13:26:19,106+02 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] FINISH,
SetVdsStatusVDSCommand, log id: 460b0642
2018-06-05 13:26:19,161+02 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45) [374e0c] Host
'host01.redacted' is set to Non-Operational, it is missing the following
networks: 'ovirtmgmt'
2018-06-05 13:26:19,196+02 WARN [org.ovirt.engine.core.
dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-45)
[374e0c] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host
host01.redacted does not comply with the cluster Default networks, the
following networks are missing on host: 'ovirtmgmt'
2018-06-05 13:26:19,235+02 INFO [org.ovirt.engine.core.bll.
HandleVdsCpuFlagsOrClusterChangedCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-45)
[647a0b6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand
internal: true. Entities affected : ID: 9956cebf-59ab-426b-ace6-25342705445e
Type: VDS
Answers used for ansible deploy:
OVEHOSTED_NETWORK/bridgeIf=str:br0.1111
> OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
>
Do you have idea why the ovirtmgmt wasn't created during deploy? Or should
we use other settings for network for deploy script?
Can you please share your /var/log/vdsm/vdsm.log and
/var/log/vdsm/supervdsm.log
?
On Tue, 2018-06-05 at 11:03 +0200, Simone Tiraboschi wrote:
On Tue, Jun 5, 2018 at 10:16 AM, Mariusz Kozakowski <mariusz.kozakowski@
sallinggroup.com> wrote:
On Tue, 2018-06-05 at 10:09 +0200, Simone Tiraboschi wrote:
But did you manually created the bridge or did the engine created it for
you triggered by hosted-engine-setup?
Manually. Before we had br0.1111. Should we go back with network
configuration to br0.1111, and no ovritmgmt network created?
Yes, I'm pretty sure that the issue is there, see also
https://bugzilla.redhat.com/show_bug.cgi?id=1317125
It will work for sure if the management bridge has been created in the
past by the engine but we had a lot of failure reports trying to consume
management bridge manually created with wrong options.
Letting the engine creating it with the right configuration is by far the
safest option.
Also, what we should use as anwers here?
OVEHOSTED_NETWORK/bridgeIf=str:bond0.1111
OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
Yes, this should be fine.
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
8:56 a.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Tue, 2018-06-05 at 13:43 +0200, Simone Tiraboschi wrote:
Can you please share your /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log ?
Yes, please find the attached.
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
10:37 a.m.
On Tue, Jun 5, 2018 at 2:22 PM, Mariusz Kozakowski <
mariusz.kozakowski(a)sallinggroup.com> wrote:
On Tue, 2018-06-05 at 13:43 +0200, Simone Tiraboschi wrote:
Can you please share your /var/log/vdsm/vdsm.log and
/var/log/vdsm/supervdsm.log ?
Yes, please find the attached.
On a seccesful setup in the logs you should see something like
2018-06-05
10:51:53,451+0200 INFO (jsonrpc/7) [api.network] START
setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True,
u'nic':
u'eth0', u'mtu': 1500, u'switch': u'legacy',
u'dhcpv6': False, u'STP':
u'no', u'bridged': u'true', u'defaultRoute': True,
u'bootproto': u'dhcp'}},
bondings={}, options={u'connectivityCheck': u'true',
u'connectivityTimeout': 120}) from=::ffff:192.168.122.37,49716,
flow_id=41ca4fc1 (api:46)
but it's completely missing on your logs.
Can you please attach also the whole engine.log and host-deploy logs?
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
5:26 p.m.
On Tue, Jun 5, 2018 at 4:37 PM, Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
On Tue, Jun 5, 2018 at 2:22 PM, Mariusz Kozakowski <mariusz.kozakowski@
sallinggroup.com> wrote:
> On Tue, 2018-06-05 at 13:43 +0200, Simone Tiraboschi wrote:
>
> Can you please share your /var/log/vdsm/vdsm.log and
> /var/log/vdsm/supervdsm.log ?
>
>
> Yes, please find the attached.
>
> On a seccesful setup in the logs you should see something like
2018-06-05 10:51:53,451+0200 INFO (jsonrpc/7) [api.network] START
setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True,
u'nic':
u'eth0', u'mtu': 1500, u'switch': u'legacy',
u'dhcpv6': False, u'STP':
u'no', u'bridged': u'true', u'defaultRoute': True,
u'bootproto': u'dhcp'}},
bondings={}, options={u'connectivityCheck': u'true',
u'connectivityTimeout': 120}) from=::ffff:192.168.122.37,49716,
flow_id=41ca4fc1 (api:46)
but it's completely missing on your logs.
Can you please attach also the whole engine.log and host-deploy logs?
I tried to deploy hosted-engine over vlan over a bond and everything worked
as expected but I also found a case where it fails: SetupNetworks is going
to fail if bond0.123 is correctly configured with an IPv4 address while the
untagged bond0 lacks IPv4 configuration.
Simply configuring a static IPv4 address from an unused subnet on the
untagged bond is a valid workaround.
I just opened https://bugzilla.redhat.com/show_bug.cgi?id=1586280
>
> --
>
> Best regards/Pozdrawiam/MfG
>
> *Mariusz Kozakowski*
>
> Site Reliability Engineer
>
> Dansk Supermarked Group
> Baltic Business Park
> ul. 1 Maja 38-39
> 71-627 Szczecin
> dansksupermarked.com
>
10:25 p.m.
New subject: oVirt hosted-engine-setup issues with getting host facts
On Tue, 2018-06-05 at 23:26 +0200, Simone Tiraboschi wrote:
I tried to deploy hosted-engine over vlan over a bond and everything worked as expected
but I also found a case where it fails: SetupNetworks is going to fail if bond0.123 is
correctly configured with an IPv4 address while the untagged bond0 lacks IPv4
configuration.
Simply configuring a static IPv4 address from an unused subnet on the untagged bond is a
valid workaround.
I just opened https://bugzilla.redhat.com/show_bug.cgi?id=1586280
Ok, I removed all bridges, so IPs are configured on bond0.id, added dummy IP for bond0,
flushed iptables. And it works. So thank you for your help!
One last question - does the bond0 IP is still needed or it's just issue during
install and now we can delete it?
Cheers
--
Best regards/Pozdrawiam/MfG
Mariusz Kozakowski
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
9:12 a.m.
On Wed, Jun 6, 2018 at 2:49 PM, Mariusz Kozakowski <
mariusz.kozakowski(a)sallinggroup.com> wrote:
On Tue, 2018-06-05 at 23:26 +0200, Simone Tiraboschi wrote:
I tried to deploy hosted-engine over vlan over a bond and everything
worked as expected but I also found a case where it fails: SetupNetworks is
going to fail if bond0.123 is correctly configured with an IPv4 address
while the untagged bond0 lacks IPv4 configuration.
Simply configuring a static IPv4 address from an unused subnet on the
untagged bond is a valid workaround.
I just opened https://bugzilla.redhat.com/show_bug.cgi?id=1586280
Ok, I removed all bridges, so IPs are configured on bond0.id, added dummy IP for bond0,
flushed iptables. And it works. So thank you for your help!
Thanks for the report.
One last question - does the bond0 IP is still needed or it's just issue during
install and now we can delete it?
Please keep it since vdsm could potentially try to rollback your network
configuration to its initial status if something is reported to be down for
a relevant amount of time.
Cheers
--
Best regards/Pozdrawiam/MfG
*Mariusz Kozakowski*
Site Reliability Engineer
Dansk Supermarked Group
Baltic Business Park
ul. 1 Maja 38-39
71-627 Szczecin
dansksupermarked.com
8:19 a.m.
Hi Mariusz,
I'm sorry to hear you're having issues. Could you please provide a link to
a pastebin or something similar for the full log file?
Thanks!
-Phillip Bailey
On Wed, May 23, 2018 at 2:35 AM, Mariusz Kozakowski <
mariusz.kozakowski(a)dsg.dk> wrote:
Hello,
We've been trying to setup oVirt environment for few days but we have
issue with hosted-engine-setup (ansible script).
We managed to fix few small things and have them merged upstream but
unfortunately right now the installation process fails on getting host
facts.
It looks like it cannot proceed because it fails when connecting to
ovirt-engine API of the bootstrap VM.
The oVirt API / webpanel is working, I tested it via a browser and I can
login without issues using the admin password chosen earlier in the process.
2018-05-18 15:26:47,800+0200 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:100 TASK [Wait for the host to be up]
2018-05-18 15:39:14,025+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:94 {u'_ansible_parsed': True,
u'_ansible_no_log': False, u'changed': False, u'attempts': 120,
u'invocation': {u'module_args': {
u'pattern': u'name=host01.redacted', u'fetch_nested': False,
u'nested_attributes': []}}, u'ansible_facts': {u'ovirt_hosts':
[]}}
2018-05-18 15:39:14,127+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:98 fatal: [localhost]: FAILED! =>
{"ansible_facts": {"ovirt_hosts": []}, "attempts": 120,
"changed": false}
May 18 13:34:34 host01 python: ansible-ovirt_hosts_facts Invoked with
pattern=name=host01.redacted fetch_nested=False nested_attributes=[]
auth={'timeout': 0, 'url':
'https://ovirt-dev.redacted/ovirt-engine/api',
'insecure': True, 'kerberos': False, 'compress': True,
'headers': None,
'token': 'R--token-redacted', 'ca_file': None}
Do you have idea what/where is issue and how to fix it?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
2391
days inactive
2406
days old
19 comments
4 participants
participants (4)
-
Mariusz Kozakowski -
Mariusz Kozakowski -
Phillip Bailey -
Simone Tiraboschi